Scheduling, Timing and Polling Intervals
Introduction
This page will go into the technical depths about the timing of data acquisition in general.Some abbreviations will be used for the sake of simplicity.
DA = Data Acquisition
Publish strategy
A data acquisition device module is publishing raw results to the internal broker, which are processed then to calibrated results. Each sensor gateway of a DA device publishes its raw result, which contains one or more keys. From one or more values behind those keys a calibrated result can be derived. Each DA device module is configured according to the XML schema definition (XSD). Part of this definition is the publish_strategy for the sensor gateway.
| publish_strategy | description |
|---|---|
| always | All raw result entries of the sensor gateway will be always processed to calibrated results (if a sensor for that is configured). |
| on_change | A raw result entry will be processed to a calibrated result only, if there is a change to the previous result. |
The default publish strategy (if not explicitly defined in the configuration) is on_change.
Passive modules without polling and timeout
These modules are not polling and are also not waiting up to a timeout time. They just listen and forward a result, when present.
| DA module | description |
|---|---|
| trapd | SNMP trap daemon |
| gps_tcp_client | listen to GPS NMEA packet received from a TCP connection |
| gps_serial_client | listen to GPS NMEA packet received from a serial port |
Modules with interval and timeout
http_server
The http_server works passive regarding incoming notifications (no polling, no timeout) and is driven by actions to request images. These requests have a configurable timeout. If this timeout is exceeded the action ends up with an error.
| unit | min | max | default |
|---|---|---|---|
| seconds | 1s | undefined | 60s |
GPIO DIO
The GPIO DIO waits for input changes with a timeout parameter. The timeout for that is configurable with the attribute polling_interval, with a default of 10000ms (10s). If the corresponding sensor_gateway is configured to publish_strategy="always" then a raw_result of the GPIO input/output states will be generated all "polling_interval" milliseconds.
polling_interval
| unit | min | max | default |
|---|---|---|---|
milliseconds | 1 ms | undefined | 10 000 ms |
da_external_checker
The da_external_checker is a data acquisition module that periodically invokes an external script.
interval
Mandatory parameter with the period in milliseconds to wait until try to launch the script again. This value is expected to be bigger or equal to 1000 (1 second)
| unit | min | max | default |
|---|---|---|---|
milliseconds | 1000 ms | undefined | undefined |
timeout
Mandatory parameter with the amount of time in milliseconds that a script could run.
| unit | min | max | default |
|---|---|---|---|
milliseconds | 1 ms | undefined | undefined |
ble_scanner
The ble_scanner is a data acquisition module which listen to advertisements sent by BLE (Bluetooth Low Energy) devices.
alive_timeout
Mandatory parameter with the amount of time in seconds that a BLE device should be kept in memory and marked as alive (or in the neighborhood).
| unit | min | max | default |
|---|---|---|---|
| seconds | 1s | undefined | undefined |
AP (wifi) scanner
The ap_scanner is a data acquisition module which scans Wireless Access Points on the neighborhood.
interval
Mandatory parameter which defines the interval in milliseconds between each scan.
| unit | min | max | default |
|---|---|---|---|
milliseconds | 1000 ms | undefined | undefined |
timeout
Mandatory parameter which defines the maximum amount of time that the scan is allowed to run, in milliseconds.
| unit | min | max | default |
|---|---|---|---|
milliseconds | 1000 ms | undefined | undefined |
PLC engine interface
poll_interval
If not configured, the default is taken.
| unit | min | max | default |
|---|---|---|---|
milliseconds | 50 ms | undefined | 1000 ms |
In difference to the other DA modules the poll_interval is configured for the complete module but not separated in the sensor gateways.
Modules with scheduled polling
The modules, listed in this section use a configuration schema to control the scheduling, which is defined in the XML schema definition (XSD).
Schema
This complex schema definition is currently only partially used in the DA modules, which are listed in this section. Different DA modules use this definition in a different way.
The timing resolution of these modules is limited to a second. Even though the units are milliseconds the values get rounded internally to seconds.
retry_algorithm
is a value of following enumeration
| retry algorithm | description |
|---|---|
| linear | linear timing steps |
| binary_exp | binary exponential timing steps, like the following set: 1,2,4,8,16,32,64,... |
Only the linear retry algorithm (which is the default) is used for now. As linear is the default, there is no need to put it to the configuration.
da_rest_client, da_icmp, snmpgetd
These three DA modules use the schema definition above in the same way.The used parameters are:
polling_interval
time between each query
| unit | min | max | default | remark |
|---|---|---|---|---|
milliseconds | 1000 ms 1 | undefined | undefined | rounded to seconds |
retry_interval_before_alert
time for which a query could be retried, after that an error is reported and it need to wait for the next query cycle.
| unit | min | max | default | remark |
|---|---|---|---|---|
milliseconds | 1000 ms 1 | undefined | undefined | rounded to seconds |
fixed_timeout
timeout for each query try
| unit | min | max | default | remark |
|---|---|---|---|---|
milliseconds | 1000 ms 1 | undefined | undefined | rounded to seconds |
1 It is possible to configure a minimum value >= 0 milliseconds. This misconfiguration is intercepted by the DA modules.
These three DA modules essentially calculate
and perform the query up to trials times until they either get the result or specify the error of the query. This is done each polling_interval cycle.
Example
The following configuration example for the scheduling configuration
...
<scheduling>
<polling_interval>15000</polling_interval>
<error_handling>
<retry retry_interval_before_alert="9000"/>
<timeout_handling>
<fixed_timeout>3000</fixed_timeout>
</timeout_handling>
</error_handling>
</scheduling>
...
means: All 15 seconds the module tries 3 times to do the query. Each trial is limited to 3 seconds.
modbusd
The DA module modbusd uses the parameters
- polling_interval
- retry_interval_before_alert
fixed_timeout
with the same resolution (rounded to seconds) and limitations as in the tables above in this section. Usually the modbusd is configured with a polling_interval only. The difference to the other three DA modules of this section is the error handling, if configured. In case of a failed primary query in the polling_interval query cycle the modbusd continues to poll with the retry_polling_interval until the retry_interval_before_alert is reached.
Configuration of the data export to the cloud
Sensor events are always sent to cloud. There is no way to restrict this.
There is the possibility to configure specific for each sensor if or when its data (its calibrated results) are sent to the cloud.
| upload strategy | description |
|---|---|
| never | means never. |
| on_change | means first and on change only. - If there is no change in the sensor value for lifetime then the value is sent only once. |
| interval | means first 1, on change and on interval (if any new in between), like a heartbeat. The interval is the upload interval to cloud. If there is no new incoming sensor result in the interval ("dead" sensor), then no value will be sent to cloud (no heartbeat). |
| always | Each calibrated result of a sensor is uploaded, regardless its previous values. |
1 First means the first value of a sensor available on start of data_store.
In case of a value change for sensors with 'on_change' flag or with 'interval' flag the latest old value (with latest old timestamp) and the new value (with the new timestamp) is sent to the cloud.
The default upload strategy can be set in the SiteController.cfg, which is by default 'interval'.
The interval of 'interval' is equal to the upload_interval, which can be also set in the SiteController.cfg. The default is 60 seconds.
The upload_interval has a high influence to the overall performance of the data upload to the cloud. The default of 60 seconds is a good value. It is recommended to set the upload_interval not less then 15 seconds. All sensor data do have their individual time stamp about their acquisition. The upload of the sensor data (hd - historical data) does have a delay on principle.
[data_store] ... # default upload strategy (never, on_change, interval, always) upload_strategy = interval # upload interval to server in seconds upload_interval = 60 ...
The default upload strategy, defined in the SiteController.cfg is valid for all sensors, which have no specific upload strategy configured. The upload interval cannot be changed specifically for a sensor in the sensor configuration.
A change of the SiteController.cfg is active after the restart of the SiteController.
The upload strategy can be configured specific for a sensor:
<sensor sensor_id="Battery_Voltage"> <sensor_class>voltage</sensor_class> <upload>never</upload> </sensor>
If a sensor is configured to be never uploaded but events are configured for that sensor, which will be uploaded, then you find a warning in log whilst processing the sensor configuration:
2016-07-01 09:44:24,786:1010:[data_store.py:745]:WARNING:Upload to server switched off for a sensor with events! sensor: Battery_Voltage
Example
The SiteController.cfg is on default regarding the upload_interval (60s) and the upload_strategy (interval).
In the example configuration above are 2 sensors configured, each with an own sensor gateway at the data acquisition device. Usually more then one sensors are using the same sensor gateway, which would be possible also here.Both sensor gateways are configured to have the publish_strategy always (default is on_change).
In the example configuration the sensor rest_pressures_sensor/tire is configured to have the upload_strategy always. The other sensor rest_temperatures_sensor/cooler remains with the default upload_strategy interval.
Both gateways of the device 'rest_server' emit their raw results all 15 seconds, regardless if there was a change in the values, because both gateways are configured with the publish_strategy="always"
The sensor rest_pressures_sensor/tire is configured with the upload strategy <upload>always</upload>. This way in the cloud the sensor data are present with a difference in their timestamps of 15 seconds.
The sensor rest_temperatures_sensor/cooler has no specific upload strategy in its configuration, so it uses the default upload strategy interval which is set to 60 seconds (see the Example snippet of the SiteController.cfg above). Herewith the sensor data are present in the cloud with a difference in their timestamps of 60 seconds.
In both cases additional values are uploaded to cloud, if there is a change in the value.