[Changelog] flespi analytics

shal

shal We finally installed pending update. Now analytics precision is the same as everywhere in the flespi - in microseconds.

shal

We implemented the new interval selector type="inactive". It can be used to detect connection/reporting problems with the device in realtime and historic mode.

In contrast to all other selector types, in the "inactive" selector interval is created even if there is no data from the device. When device does not send the data for the specified period of time, a new "active" interval is created and its "end" time is automatically incremented every 60 seconds if there is still no data. Once a device is back online and delivers messages to the platform, the interval will be marked as non-active with updated and correct "begin" and "end" times of the inactivity.

shal

New feature is now available in calculators configuration - special boolean property update_onchange.

This property controls the behavior of analytics system when calculator configuration is changed.

With TRUE value (default) the system will synchronize all devices intervals according to a new calculator configuration within update_period (10 days by default). So if you change selectors, counters, or validation fields - you will immediately see the effect with intervals created, updated, deleted events. This is standard behavior of flespi analytics from the very beginning.

While FALSE value will prevent any automatic synchronization to currently existent intervals and will only use the new configuration for new device messages processing. This is convenient to use in production level calculators with thousands of devices assigned to prevent flooding of your handlers with intervals update events.

In order to provide our users with precise control we also added new method that will force analytics system to apply current configuration of calculator to a specific range of device messages: POST /gw/calcs/XXXX/devices/XXXX/recalculate. This can be convenient to force intervals synchronization for historical device messages beyond standard update_period.

shal

We tuned analytics system to use update_delay calculator's property to prevent the creation of false-positive short-living intervals.

The system will delay intervals generation which begin time is less then current time minus update_delay.

This can be effectively used in systems where data delivery can be delayed due to network issues or internal device's logic - for example to correctly delay processing of crash events from Teltonika (which are delivered out of ordinary data stream) or prevent false-positive notifications with only partial data sent by the device.

shal

Two new features now available in analytics.

Filter in the interval with counter type=parameter and method=distinct only distinct values of encountered parameter.
New function available in expressions: previous("parameter-name"). It returns previous value of parameter when possible and operates in similar way as via #parameter-name notation however if the value is not yet known it will return null instead of error. You can check for availability of previous parameter for sequential calculation using not(isnull(previous("parameter-name"))). This function can be used in device plugins as well.

And as a reminder, for those who works with plugins and calculators - we have nice expression testing tool that simplifies a lot expressions construction and evaluation.

shal

We changed geofences hit test implementation with new R-Tree index to improve the performance of such intervals selectors. Such change may introduce slightly different hit-test results between past and current algorithms which may trigger unexpected interval updates in some cases.

This is just an informational message, but in case you will notice that your geofences selection in intervals generates something unexpected please contact us in the flespi built-in chat with all details.

shal

We have added new type of interval counter - message. It is used to store in the interval full JSON representation of message(-s) validated by some condition, with first, last or each method choices. For the first or last method the value of interval counter will be JSON object. For each method the value will be an array of JSON objects.

shal

To improve expressions that are using json keys new notation has been developed.
Instead of function json('field_name', 'json_path') you may now use field_name['json-path'].

For example you have BLE beacons information in the message:

{
"ble.beacons": [ 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:1000", 
			"rssi": -70 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:2000", 
			"rssi": -65 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:3000", 
			"rssi": -90 
		} 
	]
	}

In order to access first beacon you may use construction: ble.beacons['/0/id'] in the expression. If unable to extract by this path it will return null.

shal

We added 2 additional features for message counter - the possibility to filter fields in the message(-s) added to the interval and possibility to find and add to the interval with some minimum or maximum extremum.

Here is the configuration of counter you may use to add to the interval time and position information when maximum speed was reached within the interval:

For each method counter will generate JSON array with all messages with the minimum/maximum value after expression calculation.

shal

We added validate_message option for selector of types expression, datetime, geofences. It allows to pass into selector for intervals markup operation only those that pass this filter while for interval counters all messages will be available.

Previously similar functionality could be achieved by utilizing merge_unknown option which continues active interval if calculation was not possible. However with this special validate_message filter it is possible to perform much more advanced interval detection with even simpler and straightforward configuration.

Now inside calculator configuration we have three places for validate_message filter:

inside calculator configuration - will filter all messages before passing to cauclator selector and counters;
inside selector configuration - will filter messages for this particular selector before passing them to intervals markup engine;
inside counter configuration - will filter messages for this particular counter during interval field calculation;

shal

We enhanced intervals generation system with reason_code field which defines the reason why this or that interval event was generated. Full actual list of all reasons is available in REST API:

reason_code is attached as a user property (available for MQTT 5.0 protocol clients only) to all events and intervals generated by flespi analytics:

At the same time it is appended as a JSON field into calculator log and visible in LOGS & MESSAGES tab of calculator to detect easily the reason of particular analytics event:

shal

We defined new type of interval selector - calc. It defines intervals based on the intervals created by referenced calculator. Special invert option can be enabled to create opposite intervals - for example if referenced calculator detects trips, opposite calculator will detect all parkings with the reversed logic. validate_interval option can be used to select very specific intervals in referenced calculator.

Think of this interval selector as a building block to create complex logic with various calculators. It applications are vary from setting up specific and different counters for selected intervals generated by another (base, referenced) calculator up to creating matching inverted calculator to have consistent ON/OFF events with same begin/end timings - when parking is always starts on time when trip stopped and next trip always starts from time when parking stopped.

When used in reverse mode it is desirable to match min_active value to max_inactive for both calculators to have maximum consistency.

shal

We implemented new function json_array_count(X) that can be used in expressions which will return number of elements in array X. For example the code: json_array_count(ble.beacons) will return quantity of BLE beacons in the device vicinity.

shal

Interval counter of type calc which can insert into one calculator intervals generated by another calculators (for example insert into trip associated stops) was enhanced with 2 new options: method and validate_interval:

method defines how to insert referenced intervals into the counter field. Either each to generate the array (default method) or insert first or last referenced interval.

validate_interval defines optional validation expression that can filter referenced intervals.

shal

We implemented a new counter type - variable. Value of this counter is dynamically updated from the left to the right with each message pass and it is possible to access it in expressions of all other counters and validate_interval, validate_message fields as well.

This opens a totally new possibility to have some state during counters calculations.

For example it is possible to ignore X count or Y seconds initial messages from the moment interval has started. Or ignore all other iButton/RFID codes except the very first.

Variable value is accessed with variable("name") call. Check this artificial sample of 2 counters which will generate a timestamp of the 5th harsh acceleration event withing interval under name harsh.driving.started:

Please note that variables are not available during intervals markup with selectors, but once interval is marked up and its calculation began variables can be used everywhere.

shal

In flespi analytics we updated the behavior of interval calculations for messages with timestamp.key parameters. timestamp.key is a special parameter that is usually constructed from the original timestamp sent by device with event.seqnum (record/event sequence number) added in fraction part. This method is used to store and process multiple messages with the same timestamp but different record sequence numbers.

Previously the value inside timestamp.key was taken as interval begin and end which could result in intervals with fraction ranges and duration.

Now we will always take just the original timestamp value as was reported by the device for the interval begin or end.

This behavior change may result in a slightly higher level of events from the flespi analytics system towards your integration as some intervals will have timing changed.

shal

New function is now available in analytics expressions: weekday(timestamp). It will return day of week for its argument starting from Monday (1) to Sunday(7).

shal

We are currently upgrading the flespi analytics system with a new communication and storage architecture, moving its operational data out of RAM into MQTT and local cache. This process started on Monday March 18 and will take around a week.

The general architecture of the analytics system and its algorithms for device messages processing remained the same, we just changed the communication system between services and instead of a RAM cache we will use a distributed drive-backed mapped cache.

We do not expect any impact on the intervals calculation system - everything is covered with a very large set of cruel unit and runtime tests and we are keeping an eye on any possible problem that may popup. However if you notice something strange in the analytics system behavior please let us know in flespi chat.

Currently we have already upgraded 15% of nodes where analytics calculations are running and will slowly upgrade remaining nodes.

After the upgrade we will have a lot of new functional possibilities available inside analytics. And the very first in our features list is the possibility to use device metadata in calculators to configure flexible per-device algorithms with a single calculator. We will implement it shortly after the upgrade.

shal

We adjusted the contents of interval JSON generated by flespi calculators to always reflect in its JSON fields all defined counters.

Counters that were not able to calculate a value within interval messages will now store null values to the interval JSON. Previously such counters did not add any value to the interval which we consider inconsistent.

Please adopt your integration to that change if required.

shal

We've modified the logic inside inactivity detection selector in flespi analytics. The change in logic affected only the indicated end time of detected inactive interval. It was extended up to the moment of connectivity with the device was restored.

Previously the end time of such interval was a last message with latency beyond configured. This is indicated on a screenshot below from a inactivity detection calculator with 60 seconds configured allowed data latency. At the top of screenshot you can see interval and at the bottom last included messages (highlighted):

As you can see at 07:26:05 connectivity was restored, but the end of interval was indicated as 07:25:04 - it was the timestamp of last message with latency above configured 60 seconds.

The new logic will finish such interval on the last reported message before the connectivity was restored. Here is the same data calculated with a new logic:

In this logic message with timestamp 07:26:04 was included into inactivity period, because connectivity was restored at 07:26:05.

« Previous Page Next Page »