Modbus Monitoring - Best Practices and Performance Insights
Introduction
This document presents several performance considerations related to azeti's Modbus daemon and the Modbus interface in general. These considerations are based on the results of performance measurements and tests we did in our laboratories.
Article Scope
The article explains technical backgrounds about the timing and scheudling of MODBUS based checks and gives advice on how to plan your monitoring configuration and optimize the response times of particular checks depending on their importance (priority).
There are several factors that can influence the response time of a Modbus check in the SONARPLEX system, therefore a deep understanding of the several factors that influence response time is necessary when designing installations that include Modbus checks.
The data showed is based in dry contact checks, as they are the more time-critical checks on the system.
Factors influencing response time
The modbus response time is influenced by the following factors: Modbus device, service reaper frequency configuration, Modbus delay configuration, load of the system.
| Factors that can influence delays in the Modbus checks response | |
|---|---|
| Modbus device | a Modbus-capable device. Devices add some delay, as they take some time to change their internal status after an event occurs. This is influenced by the mark, model and firmware revision of the device, as well as by the baud rate configured in the device. |
| Service reaper frequency | our system use a reaper to process information that comes from Modbus in the messages queue. The frequency of the reaper means how often this is called. As an example, a reaper set to 30 seconds means that our message can take up to 30 seconds to be processed, while a reaper that runs every second will process it quickly. However, lowering the reaper too much may affect system performance, specially in single-core systems, increasing the load of the system and delaying the process of the events. |
| Modbus delay | there is the possibility to include a delay between Modbus calls. This will make the Modbus interface poll with less frequency the devices connected to the bus. This will add a delay in the reading of a status change in the device, and at the same time will create less load on the system. |
| System load | the system load is the number of processes that are waiting at a certain moment. A high load may affect negatively the performance of the system, and therefore delay the Modbus checks response. A backup process, for example, can lower the number of checks per minute as is an intensive process that will compete for the resources. |
| Plugins installed | Some plugins may use the Modbus interface intensively and therefore may affect performance. An example is the check_modbus_access_m2m-wtsc (for example, it may bring down the number of checks per minute from 700 to 500 with 50 ms. latency). |
| Frequency of changes | If the inputs of the dry contacts have many values changing at a high frequency, it might create a high load on the system due to the processing of all the status changes, therefore causing delays. |
| Faulty device | This is a special case. One faulty device (not powered, not cabled) that has services attached, may lower the performance of the queue to less than one third, due to the timeouts assigned to the checks. As an example, a queue with 700 checks per minute might go as low as 230 due to a T38I disconnection. It is important to take care of the faulty devices as soon as properly to keep the system running in optimal conditions. |
| Use of SONARMANAGER | The use of SONARMANAGER will cause a fall on the number of checks. The reason is the status file that has to be collected and sent to the SONARMANAGER regularly (and every time the user refreshes), therefore uses CPU time and lower the Modbus checks per minute (for example, they might drop from 700 to 540 with 50 ms latency) |
| Configuration changes | Every configuration change will need a restart of the daemon, so please be patient, it will need a few minutes to reach high performance after you change the configuration. |
On this page:
Recommendations for a SONARPLEX 600M
Based on information derived from our lab-tests, we recommend the following configuration for a SONARPLEX 600M appliance:
| Sonarplex 600M parameters recommendation | |
|---|---|
| Modbus device | install the latest firmware available |
| Service reaper frequency | 3 seconds (default is 10 s.) |
| Modbus delay | 50 ms. (is the default option). You might raise this value if experiencing too much load. |
| Logs activation | the deactivation of the logs can lead to up to a 20% of performance increase. However, it will lead to having less available information in a forensic issue investigation. |
Modbus Support Tool
The Modbus support tool allows you to read the queue status and statistics, configure priorities and restart the daemon, among other capabilites. In order to fine-tune your checks configuration, you need to know how many checks are performed in the bus per service in a minute. Please use the Modbus Support tool integrated in the Modbus Daemon. The following sections show you the most important options you need to know to be able of fine-tuning your Modbus configuration.
The Modbus support tool can be accessed through the Administration Web Interface, e.g. http://example.com:81/cgi-bin/modbusd_query.cgi
Command SHOW_QUEUE
This command allows you to see the queue selecting the port you are interested. In the 600M system, use ttyS2 for COM1 and ttyS3 for COM2. The result of showing the queue will let you see, along with the result of the bus read and other interesting information how many checks have been done per minute in a service. See the following example, where you have a service with priority 1 and two with priority 10.
As you see in the table, the check with priority 1 has been done 10 times more (173 checks/minute) than the checks with priority 10 (17 checks/minute).
Influence of check priorities
The check priority drives the allocation of check execution slots over time in the Modbus interface. The higher the priority, the more often a check will be executed within the overall number of checks.
Command RECONFIGURE_READ_PRIORITY
The scheduled checks are usually used to gather analog information, like temperature, humidity, liquid level, brightness, pressure, radioactivity, oxygen level, and so on. You may realize that the analog checks are performed several times per minute. This is for most use-cases not a problem, but in some situations you do not need this information to be retrieved so often, and may want to lower the frequency of this checks in order that other kind of checks (i.e. dry contacts) have more checks on the bus. This is done in the Modbus setup too, with the command RECONFIGURE_READ_PRIORITY. The default (0) will be 11 in most cases.At the moment it is adjusting the priority for all the analog checks at the same time.
Command SHOW_STATS
This command will show you statistics about the Modbus port (Modbus serial device). You will be able to see the total checks executed in the last minute and the minute before, the duration of the checks, the maximum time that a check is in cache before its information is refreshed with a new read, etc.
Command RECONFIGURE_DELAY
This command allows the user to change the delay between reads in the bus in milliseconds. The higher is this value, the lower will be the frequency of checks in the bus. Increasing this value will lower the load on the system, as there will be less checks performed per minute.
Performance Tests
Following is an overview of performance profiling tests we did in our laboratories. Please be aware that your system configuration may vary and influence response time, and that this are the Modbus statistics, to which you have to add the delay time due to internal processing in SONARPLEX.
Use the Modbus Support Tool to review detail metrics.
Test environment
| Devices, plugins and tools | Models and versions |
|---|---|
| azeti model | azeti 600M |
| SONARPLEX Version | v5.1.1b |
| SONARPLEX Extensions | Modbus daemon version 1.7.3 check_modbus_fast_dry3 (v. 1.7.777) for dry contact services. check_modbus_fast_read3 (v.1.4.735) for scheduled checks. |
| SONARMANAGER Version | v2.7.2.24 |
| Modbus devices | Temco Controls T3 Fast Switches (T3-8I13O) Temco Controls Temperature and Humidity sensors (HUM2) |
Please contact support in case you need further information about how to prepare a check environment and which parameters you need to configure.
Configured delay vs. Checks per minute
We show the average number of Modbus checks per minute related to the configured delay in the Modbus daemon. Checks are counted separately per bus (i.e. COM1 and COM2 may have different check count).
| Delay (ms) | Modbus checks (min) |
|---|---|
| 50 | 710 |
| 100 | 420 |
| 150 | 320 |
| 200 | 240 |
| 250 | 210 |
See the graph below.
This test has been done with Temco T3 devices with a baudrate of 19200 bps, that have an average response time (time from request to answer) of 2.50 ms. We have upgraded the T32I for performance improvements, but the latest firmware does not show any perceptible improvement (from hw revision 49 to 65). You may experience lower or higher response times if you have different devices.
Influence of the delay on the bus
The more delay we introduce in the bus, the less checks will be performed per minute. Increasing the delay may be useful to lower the load on your system.
Number of Checks vs. Checks per minute
The following table shows how many checks will be done in the bus for a particular service, depending on how many Modbus services are configured in the SONARPLEX for this particular port.
| Number of checks | Checks per minute |
|---|---|
| 70 | 9-10 |
| 60 | 10-12 |
| 55 | 10-12 |
| 50 | 13-16 |
| 40 | 21-23 |
| 30 | 21-23 |
| 25 | 24-29 |
| 20 | 33-37 |
| 10 | 55-60 |
Environment: this test has been done with Temco T3 devices and a delay of 50 ms.
Influence of the number of checks
The number of checks configured service checks influence the checks that you can do in a the bus per minute. The more configured checks, the less checks can be done per minute for each individual check, and vice versa.
Modbus Check Priority
As the Modbus bandwidth is a finite resource, our Modbus services can be assigned a priority that makes them come more often in the checks queue. As an example, there is probably no need to do 10 checks a minute in case you are going to gather the temperature every minute, and all this extra checks can be assigned to more critical services, like a door opening or a firing alarm. The priority number means how much a service will be delayed in reference to other services. Thus, the maximum priority is 1, and if a service has a priority of 10, the service with priority 10 will perform 1/10 of checks per minute. In this section, we will explain further the default priorities and how can they be adjusted to fulfill your installation needs.
Default priorities
The following table show the default priority on the bus depending on the type of service. Door access is assigned the maximum priority with a high margin, to avoid that a high number of service checks delay the keypad readings. Following are dry contacts, which are intended to be checked as much as possible, and after them come the scheduled checks, used for analog readings like humidity and temperature.
| Type of service | Default priority |
|---|---|
| Door access | 1 |
| Dry contacts | 10 |
| Scheduled checks | 11 |
Impact of the Modbus priority for scheduled checks (simple read)
In the following tables, we show the effect of changing the priority of scheduled checks in the rest of the checks in the bus. Observe in the examples how changing the priority of scheduled checks influence the number of its checks and even the dry contact checks. In an environment with too many dry-contacts you might not notice a big difference.
Table legend
| Color | Title | Explanation |
|---|---|---|
| Yellow | Configured Service Checks | This is the number of configured checks for each kind (Immediate dry contact/Scheduled check) |
| Grey(vertical) | Check Type | Immediate Dry Contact: a change in status for this services will be delivered as soon as it is detected. Scheduled check: a change in status for this services will just be read by SONARPLEX at scheduled times. Usually it is an analog read (temperature, humidity, pressure and similar). |
| White | Priority - Test # | Priority assigned to the each individual configured service check. |
| Blue | Individual Check Executions/min | Number of checks per minute that are performed on the bus for every individual configured check. |
Test Overview
| Check Type | Configured Service Checks | Priority - Test #1 | Individual Check Executions/min | Priority - Test #2 | Individual Check Executions/min |
|---|---|---|---|---|---|
| Immediate Dry Contact | 60 | 10 | 9-10 | 10 | 10-11 |
| Scheduled Check | 10 | 11 | 9-11 | 50 | 2 |
In the previous table, we have a system with 60 dry contacts and 10 scheduled checks. After changing the priority of the scheduled checks from 11 to 50 (test 1 and 2, respectively), we see that they reduce their number of checks per minute from 9-11 to 2, and that the checks per minute for the dry-contacts increase slightly. The increase of the number of dry contact checks per minute and decrease of scheduled checks after changing the priorities can be better shown in the following graph:
Test #1
Influence of check priorities
60 Service checks are configured overall. 10 services have priority 1 while 50 have priority 10. In one minute there are around 700-720 execution slots available for services. Increasing the priority of the scheduled checks in a factor of 5 (up to 50), will have as a consequence that the execution rate is divided by 5 (from 9-11 till 2).
| Configured Service Checks | Priority - Test #1 | Individual Check Executions/min | Priority - Test #2 | Individual Check Executions/min | Priority - Test #3 | Individual Check Executions/min |
|---|---|---|---|---|---|---|
| 45 | 10 | 12-13 | 10 | 12-13 | 10 | 14-15 |
| 10 | 11 | 12 | 50 | 2-3 | 100 | 1-2 |
In the previous table, we have a system with 45 dry contacts and 10 scheduled checks. After changing the priority of the scheduled checks from 11 to 50 and later to 100 (test 1, 2 and 3 respectively), we see that they reduce their number of checks per minute from 12 to 2-3 and later to 1-2, and that the checks per minute for the dry-contacts increase slightly from 12-13 to 14-15 in the latest test. The increase of the number of dry contact checks per minute and decrease of scheduled checks after changing the priorities can be better shown in the following graph:
Influence of check priorities
Test #2:
Number of checks configured: 55. With a lower number of total configured checks, the influence of lowering the priority for scheduled checks is more apparent (changing from 12-13 until 14-15)
| Configured Service Checks | Priority - Test #1 | Individual Check Executions/min | Priority - Test #2 | Individual Check Executions/min | Priority - Test #3 | Individual Check Executions/min |
|---|---|---|---|---|---|---|
| 30 | 10 | 14-16 | 10 | 17-18 | 10 | 19-21 |
| 10 | 11 | 14 | 50 | 3-4 | 100 | 2 |
In the previous table, we have a system with 30 dry contacts and 10 scheduled checks. After changing the priority of the scheduled checks from 11 to 50 and later to 100 (test 1, 2 and 3 respectively), we see that they reduce their number of checks per minute from 14 to 3-4 and later to 2, and that the checks per minute for the dry-contacts increase slightly from 14-16 to 17-18 and 19-21 in the latest test. The increase of the number of dry contact checks per minute and decrease of scheduled checks after changing the priorities can be better shown in the following graph:
Influence of check priorities
Test #3:
Number of checks configured: 40. With a lower number of total configured checks, the influence of lowering the priority for scheduled checks is more apparent (changing from 14-16 until 19-21).
Impact of the Modbus priority for immediate checks (dry contacts)
The default priorities have been adjusted in order that they fulfill most use-cases. However, you may find situations in which you want some specific check to be executed faster than the others, and therefore want to adjust the priorities. In the following table, you can see the effect of adjusting services (dry contacts) priorities, and the effect it will have in the checks/minute. You can change the priority for dry contact services directly in the service configuration, using the WebGUI or SONARMANAGER.
| Check Priority | Configured Service Checks - Test #1 | Individual Check Executions/min | Configured Service Checks - Test #2 | Individual Check Executions/min | Configured Service Checks - Test #3 | Individual Check Executions/min |
|---|---|---|---|---|---|---|
| 1 | 0 | 0 | 1 | 140 | 3 | 100 |
| 10 | 30 | 14-16 | 30 | 14-15 | 30 | 10-11 |
| 11 | 10 | 14 | 10 | 13 | 10 | 9 |
In the test 1, we have system with 30 services with priority 10 and 10 with priority 11. The results are similar checks for both priorities. For test 2, we create a service with priority 1, and as a result this service having a checks per minute up to 140, assuring a quick response from the system. In the test 3, we give this priority to three services, which causes a slight reduction in the number of checks for priority 1, but still they will be having a higher number of checks per minute than the services with priority 10 or 11, as shown in the table. The influence of the 3 tests in relation to the number of checks per minute on every kind of priority can be seen in the following graphic:
Changing priority for higher read frequency in important services
40 Service checks are configured overall.In one minute there are around 700-720 execution slots available for services (ideal conditions). 30 services have priority 10, taking around 14-16 slots each, while 10 have priority 11, taking around 14 slots each. Creating a check with priority 1 will have as result that the frequency of the checks is 10 times faster (takes 140 execution slots) and the other checks frequency descent a bit. Assigning 3 checks this high priority lower the average slot usage for this priority level (each takes 100 slots, up to a total of 300), as they have to share the allocated slots, as well as the average for the lower priority checks, as they have to share the remaining slots.
Adjusting important parameters
Changing the service reaper frequency
- Open the Administration Web Interface > Configuration > System > Load Configuration
- Adjust Service reaper frequency in seconds
Changing the modbus delay
- Go to the Modbus Tool (IP address http://your-device-ip:81/cgi-bin/modbusd_query.cgi).
- Select the option RECONFIGURE_DELAY in the Command drop-down box .
Update device firmware
- Please update to the latest stable SONARPLEX software version and use the latest stable extension versions.
Check the system load
- Open the Administration Web Interface > Configuration > Summary
- You will find it under Performance Status.
- style