Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Current »

Introduction

This page will go into the technical depths about the timing of data acquisition in general.Some abbreviations will be used for the sake of simplicity.

DA = Data Acquisition

Publish strategy

A data acquisition device module is publishing raw results to the internal broker, which are processed then to calibrated results. Each sensor gateway of a DA device publishes its raw result, which contains one or more keys. From one or more values behind those keys a calibrated result can be derived. Each DA device module is configured according to the XML schema definition (XSD). Part of this definition is the publish_strategy for the sensor gateway.

publish_strategydescription
alwaysAll raw result entries of the sensor gateway will be always processed to calibrated results (if a sensor for that is configured).
on_changeA raw result entry will be processed to a calibrated result only, if there is a change to the previous result.

The default publish strategy (if not explicitly defined in the configuration) is on_change.

Passive modules without polling and timeout

These modules are not polling and are also not waiting up to a timeout time. They just listen and forward a result, when present.

DA moduledescription
trapdSNMP trap daemon
gps_tcp_clientlisten to GPS NMEA packet received from a TCP connection
gps_serial_clientlisten to GPS NMEA packet received from a serial port
On this page:

Modules with interval and timeout

http_server

The http_server works passive regarding incoming notifications (no polling, no timeout) and is driven by actions to request images. These requests have a configurable timeout. If this timeout is exceeded the action ends up with an error.

unitminmaxdefault
seconds1sundefined60s

GPIO DIO

The GPIO DIO waits for input changes with a timeout parameter.  The timeout for that is configurable with the attribute polling_interval, with a default of 10000ms (10s). If the corresponding sensor_gateway is configured to  publish_strategy="always" then a raw_result of the GPIO input/output states will be generated all "polling_interval" milliseconds.

polling_interval

unitminmaxdefault

milliseconds

1 msundefined10 000 ms

da_external_checker

The da_external_checker is a data acquisition module that periodically invokes an external script.

interval

Mandatory parameter with the period in milliseconds to wait until try to launch the script again. This value is expected to be bigger or equal to 1000 (1 second) 

unitminmaxdefault

milliseconds

1000 msundefinedundefined

timeout

Mandatory parameter with the amount of time in milliseconds that a script could run.

unitminmaxdefault

milliseconds

1 msundefinedundefined

ble_scanner

The ble_scanner is a data acquisition module which listen to advertisements sent by BLE (Bluetooth Low Energy) devices.

alive_timeout

Mandatory parameter with the amount of time in seconds that a BLE device should be kept in memory and marked as alive (or in the neighborhood).

unitminmaxdefault
seconds1sundefinedundefined

AP (wifi) scanner

The ap_scanner is a data acquisition module which scans Wireless Access Points on the neighborhood.

interval

Mandatory parameter which defines the interval in milliseconds between each scan.

unitminmaxdefault

milliseconds

1000 msundefined

undefined

timeout

Mandatory parameter which defines the maximum amount of time that the scan is allowed to run, in milliseconds.

unitminmaxdefault

milliseconds

1000 msundefinedundefined

PLC engine interface

poll_interval

If not configured, the default is taken.

unitminmaxdefault

milliseconds

50 msundefined1000 ms

In difference to the other DA modules the poll_interval is configured for the complete module but not separated in the sensor gateways.

Modules with scheduled polling

The modules, listed in this section use a configuration schema to control the scheduling, which is defined in the XML schema definition (XSD).

Schema

This complex schema definition is currently only partially used in the DA modules, which are listed in this section. Different DA modules use this definition in a different way. 

The timing resolution of these modules is limited to a second. Even though the units are milliseconds the values get rounded internally to seconds.

retry_algorithm

is a value of following enumeration

retry algorithmdescription
linearlinear timing steps
binary_expbinary exponential timing steps, like the following set: 1,2,4,8,16,32,64,...

Only the linear retry algorithm (which is the default) is used for now. As linear is the default, there is no need to put it to the configuration.

da_rest_client, da_icmp, snmpgetd

These three DA modules use the schema definition above in the same way.The used parameters are:

polling_interval

time between each query

unitminmaxdefaultremark

milliseconds

1000 ms 1
undefined

undefined

rounded to seconds

retry_interval_before_alert

time for which a query could be retried, after that an error is reported and it need to wait for the next query cycle.

unitminmaxdefaultremark

milliseconds

1000 ms 1
undefined

undefined

rounded to seconds

fixed_timeout

timeout for each query try

unitminmaxdefault

remark

milliseconds

1000 ms 1
undefined

undefined

rounded to seconds

1 It is possible to configure a minimum value >= 0 milliseconds. This misconfiguration is intercepted by the DA modules.

 These three DA modules essentially calculate and perform the query up to trials times until they either get the result or specify the error of the query. This is done each polling_interval cycle.

Example

The following configuration example for the scheduling configuration

...
            <scheduling>
              <polling_interval>15000</polling_interval>
				<error_handling>
					<retry retry_interval_before_alert="9000"/>
					<timeout_handling>
						<fixed_timeout>3000</fixed_timeout>
					</timeout_handling>
				</error_handling>
            </scheduling>
...

means: All 15 seconds the module tries 3 times to do the query. Each trial is limited to 3 seconds.

modbusd

The DA module modbusd uses the parameters

  • polling_interval
  • retry_interval_before_alert
  • fixed_timeout

with the same resolution (rounded to seconds) and limitations as in the tables above in this section. Usually the modbusd is configured with a polling_interval only. The difference to the other three DA modules of this section is the error handling, if configured. In case of a failed primary query in the polling_interval query cycle the modbusd continues to poll with the retry_polling_interval until the retry_interval_before_alert is reached.

Configuration of the data export to the cloud

Sensor events are always sent to cloud. There is no way to restrict this.

There is the possibility to configure specific for each sensor if or when its data (its calibrated results) are sent to the cloud.

upload strategydescription
nevermeans never.
on_changemeans first and on change only. - If there is no change in the sensor value for lifetime then the value is sent only once.
intervalmeans first 1, on change and on interval (if any new in between), like a heartbeat. The interval is the upload interval to cloud. If there is no new incoming sensor result in the interval ("dead" sensor), then no value will be sent to cloud (no heartbeat).
alwaysEach calibrated result of a sensor is uploaded, regardless its previous values.

1 First means the first value of a sensor available on start of data_store.

In case of a value change for sensors with 'on_change' flag or with 'interval' flag the latest old value (with latest old timestamp) and the new value (with the new timestamp) is sent to the cloud. 

The default upload strategy can be set in the SiteController.cfg, which is by default 'interval'.

The interval of 'interval' is equal to the upload_interval, which can be also set in the SiteController.cfg. The default is 60 seconds.

The upload_interval has a high influence to the overall performance of the data upload to the cloud. The default of 60 seconds is a good value. It is recommended to set the upload_interval not less then 15 seconds. All sensor data do have their individual time stamp about their acquisition. The upload of the sensor data (hd - historical data) does have a delay on principle.

Example snippet of the SiteController.cfg
[data_store]
...
# default upload strategy (never, on_change, interval, always)
upload_strategy = interval
# upload interval to server in seconds
upload_interval = 60
...

The default upload strategy, defined in the SiteController.cfg is valid for all sensors, which have no specific upload strategy configured. The upload interval cannot be changed specifically for a sensor in the sensor configuration.

A change of the SiteController.cfg is active after the restart of the SiteController.

The upload strategy can be configured specific for a sensor:

example sensor configuration snippet
<sensor sensor_id="Battery_Voltage">
	<sensor_class>voltage</sensor_class>
	<upload>never</upload>
</sensor>

If a sensor is configured to be never uploaded but events are configured for that sensor, which will be uploaded, then you find a warning in log whilst processing the sensor configuration:

2016-07-01 09:44:24,786:1010:[data_store.py:745]:WARNING:Upload to server switched off for a sensor with events! sensor: Battery_Voltage

Example

Example snippet of the SiteController.cfg
[data_store]
...
# default upload strategy (never, on_change, interval, always)
upload_strategy = interval
# upload interval to server in seconds
upload_interval = 60
...

The SiteController.cfg is on default regarding the upload_interval (60s) and the upload_strategy (interval).

Example component template for polling and upload configuration
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<component_template config_version="1.0" schema_version="1.5.16-Aspirator">
  <description>DA rest client</description>
  <class>multi-sensor</class>
  <vendor>azeti</vendor>
  <version>1</version>
  <sensors>
    <!-- REST sensors -->
    <sensor sensor_id="rest_temperatures_sensor/cooler">
      <sensor_class>unknown</sensor_class>
      <sensor_gateway sensor_gateway_id="rest_temperatures_gw">
        <demux>
          <keys>
            <key>cooler</key>
          </keys>
        </demux>
      </sensor_gateway>
    </sensor>
    <sensor sensor_id="rest_pressures_sensor/tire">
      <sensor_class>unknown</sensor_class>
      <upload>always</upload>
      <sensor_gateway sensor_gateway_id="rest_pressures_gw">
        <demux>
          <keys>
            <key>tire</key>
          </keys>
        </demux>
      </sensor_gateway>
    </sensor>
  </sensors>
  <devices>
    <device device_id="rest_server">
      <rest_client_device>
        <host>http://192.168.118.26:7088</host>
        <form_auth>
          <uri>/rest/login.php</uri>
          <form_data>
            <form_key_value>
              <pair_key>username</pair_key>
              <pair_value>operator</pair_value>
            </form_key_value>
            <form_key_value>
              <pair_key>password</pair_key>
              <pair_value>LetMeIn</pair_value>
            </form_key_value>
            <form_key_value>
              <pair_key>t</pair_key>
              <pair_value>OK</pair_value>
            </form_key_value>
          </form_data>
          <method>post</method>
        </form_auth>
      </rest_client_device>
      <sensor_gateways>
        <sensor_gateway publish_strategy="always" sensor_gateway_id="rest_temperatures_gw">
          <rest_client>
            <uri>/rest/temperature.php</uri>
            <scheduling>
              <polling_interval>15000</polling_interval>
              <error_handling>
                <retry retry_interval_before_alert="9000"/>
                <timeout_handling>
                  <fixed_timeout>3000</fixed_timeout>
                </timeout_handling>
              </error_handling>
            </scheduling>
          </rest_client>
        </sensor_gateway>
        <sensor_gateway publish_strategy="always" sensor_gateway_id="rest_pressures_gw">
          <rest_client>
            <uri>/rest/pressure.php</uri>
            <scheduling>
              <polling_interval>15000</polling_interval>
              <error_handling>
                <retry retry_interval_before_alert="9000"/>
                <timeout_handling>
                  <fixed_timeout>3000</fixed_timeout>
                </timeout_handling>
              </error_handling>
            </scheduling>
          </rest_client>
        </sensor_gateway>
      </sensor_gateways>
    </device>
  </devices>
</component_template>

In the example configuration above are 2 sensors configured, each with an own sensor gateway at the data acquisition device. Usually more then one sensors are using the same sensor gateway, which would be possible also here.Both sensor gateways are configured to have the publish_strategy always (default is on_change).

In the example configuration the sensor rest_pressures_sensor/tire is configured to have the upload_strategy always. The other sensor rest_temperatures_sensor/cooler remains with the default upload_strategy interval.

Both gateways of the device 'rest_server' emit their raw results all 15 seconds, regardless if there was a change in the values, because both gateways are configured with the publish_strategy="always"

The sensor rest_pressures_sensor/tire is configured with the upload strategy <upload>always</upload>. This way in the cloud the sensor data are present with a difference in their timestamps of 15 seconds.

The sensor rest_temperatures_sensor/cooler has no specific upload strategy in its configuration, so it uses the default upload strategy interval which is set to 60 seconds (see the Example snippet of the SiteController.cfg above). Herewith the sensor data are present in the cloud with a difference in their timestamps of  60 seconds.

  • No labels