Offline data storage and export to USB flash drive

Prerequisites

Configuration of the SiteController

SiteController.cfg

The following code block example shows settings in the SiteController configuration file, which is located in /opt/azeti/SiteController/config/SiteController.cfg  There are also other entries in the configuration file, but they are not related to the topic which is described in this document.

The comment lines, starting by #, explain the settings.


Site template

The control of the LED for the visualization of the USB export process is completely realized by configuration of the SiteController. For this purpose, the site template of the SiteController should (also) inherit a component template like the following one:

Preparation of a USB flash drive

The USB flash drive, on which to store the collected data, must have a single formatted partition containing a file named "Password.txt" in the root folder. The file "Password.txt" needs to contain a clear text password, which must match the one preconfigured in the SiteController.cfg.

The USB flash drive may already contain other files, even older export files, because they get a unique name. Of course, the USB flash drive should have enough free space.

Export procedure

There are 4 states regarding the USB data export.

USB export stateshort descriptionLED indicator
normal_operation

To indicate that the SiteController is running in regular operation and there's no USB export in progress.

LED is permanently glowing
in_progressThe USB export is in progress. Do not remove the USB drive in that state!LED flashes every second
errorThere was an error while exporting the Database to the USB drive.LED is permanently off
successThe export of the database has finished successfully. You may remove the USB drive now.LED flashes every half a second


The following diagram shows two possible flows of operation:


The LED to observe is the 'GPO1', the most left one in the LED bar

Steps to do the data export

  1. Insert the prepared USB flash drive, the LED should permanently glow (normal_operation)
  2. It takes one or two seconds, to see the next state of the USB data export, either 'error' (LED permanently off) or 'in_progress' (LED flashes every second)
  3. In case of 'error', remove the USB flash drive
  4. In case of 'in progress', wait until the shows the 'success' state (LED flashes every half a second)
  5. In case of 'success', remove the USB flash drive

In case of the error state

If it is possible to open a shell on the device, it would be possible to crosscheck the file /opt/azeti/SiteController/log/usb_exporter.log to find the cause why the export failed. If there is no possibility to open a shell, you could crosscheck the USB flash drive regarding free space, partitions, password.

Usage of the exported data


The database file could be copied into the /opt/azeti/SiteController/persistent folder of another regular (stopped) SiteController installation of same version as the data was gathered with. Make sure there are no other files in that persistent folder and the SiteController.cfg is properly configured to have access to the cloud before starting the SiteController.

Of course, as the data_store.db file on the USB drive is just a sqlite3 database file, it could easily reviewed or processed directly with an appropriate sqlite client, too.

Procedure to restore the formerly exported data to cloud

The following steps describe how to get the exported data to the cloud, but only these data (not mixed with other data), separated in a new site.

Preparation of the cloud

  1. Verify in the Control Panel at / Management / Templates / Site Templates if there is a site template checked as default.
  2. If so, open the drop-down menu at the right side and press the Delete Default


Preparation of a SiteController

The SiteController might be an azeti-machina or any other SiteController installation.

SSH access is required to open a shell and to copy the database file. In the following documentation all actions with the shell assume the execution with root permission by carrying out the sudo su after opening the shell, which could be done once and at this point.

The SiteController should have a tested connection to cloud with appropriate [data_store] settings in the /opt/azeti/SiteController/config/SiteController.cfg.

Follow these steps after the cloud connection is established:

  1. Stop the SiteController execution 

  2. Empty the running site template 

  3. Stop the mosquitto MQTT broker (To avoid any remaining data which does not belong to the data of the exported database)

  4. Remove the mosquitto database, as well as the SiteController data store database

  5. For the same benefit change the Serial (serial=<my serial>) of the SiteController to a new one in the SiteController.cfg.
  6. After carrying out a /opt/azeti/SiteController/run_SiteController.py start the new site should appear in cloud with a blank Admin and Life table:
  7. After this verification stop the SiteController execution again by 

Database restoration steps


With this prepared environment (SiteController suite is still not running)  This can be done by several means, like using the Windows WinSCP. In case of trouble regarding permissions you could copy it first to your home directory at the SiteController device and then with su permissions to /opt/azeti/SiteController/persistent .

  1. Copy (SFTP transfer, e.g. WinSCP or Filezilla) the database, retrieved with the USB flash drive, e.g. data_store-NIFE103_test-20181127T174239+0000.db, to /opt/azeti/SiteController/persistent
  2. Rename this database to data_store.db 

  3. Copy the site template that was originally used within the SiteController that you restore, e.g. sensor_config-NIFE103_test-20181127T174239+0000.xml, into /opt/azeti/SiteController/config and rename it to sensor_config.xml 

  4. Start the upload of the data by starting the SiteController with /opt/azeti/SiteController/run_SiteController.py start

  5. The upload of the historical data can take quite some time, depending on the used network connection, but at least 15 minutes for most scenarios. You'll now be able to see the data in your azeti Cloud organization under the SiteController that was used for restore.