Here we will post information about flespi analytics new features and bugfixes.

Fixed bug with last interval sometimes reset to null in flespi/state MQTT. Bug may occurred when last interval was inactive.

Again modified trigger event for 'last' interval change. Previous fix wasn't complete. Modified detection of recalculation time ranges when receiving new message. Previous implementation used too large ranges. The detection of time ranges for recalculation upon receiving of new messages is primary for analytics. In worse case this can lead to problems in intervals detections or splitting them into multiple parts.

There was a bug in the latest update that affected event triggering system. Thus interval 'created' event triggered for each message. Bug was fixed and update was installed.

adsa By the way, we recommend to disable/enable all devices assigned to calculators in order to correctly recalculate intervals that may be wrong due to bug installed this morning. You can do it with simple REST API calls : curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":false}' 'https://flespi.io/gw/calcs/all/devices/all' curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":true}' 'https://flespi.io/gw/calcs/all/devices/all' if("undefined"!==typeof hljs)hljs._ha();else if("undefined"===typeof hljsLoading){hljsLoading=1;var a=document.getElementsByTagName("head")[0],e=document.createElement("link");e.type="text/css";e.rel="stylesheet";e.href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.7.0/styles/default.min.css";a.appendChild(e);e=document.createElement("script");e.type="text/javascript";e.onload=function(){var d={},f=0;hljs._hb=function(b){b.removeAttribute("data-hljs");var c=b.innerHTML;c in d?b.innerHTML=d[c]:(7

After a long pause we have more updates to the flespi analytics engine. First of all it is possible to use date/time functions in expressions - month(X), day(X), hour(X), minute(X). This can be used for intervals validation e.g. hour(begin)>8 or just to filter some messages with hour(X)>12 if you need to detect intervals for some shifts only. Any kind of usage of course. And additionally we have added active MQTT topic for device assigned to calculator . It will contain the same value as last whenever last detected interval is active at the moment. Once it is no more active null will be published into MQTT. active topic can be used as a foundation for analytics-based own notifications system.

Additional new feature: now it is possible to test actual textual parameters values in expressions . For example you can test for non-empty driver identifier with expression ibutton.code!="" or generate intervals when it is changed with expression like ibutton.code == $ibutton.code .

For those who used merge_unknown=true option in calculators and in general in flespi analytics in selectors configuration - we detected a bug that in some very special cases may lead to generation of false intervals contain messages with unspecified by selector values. In general the logic of merge unknown option has been rewritten and this may lead to different results comparing to previous logic for messages which ON or OFF state according to interval selectors configuration is unknown. So the bug has been fixed and we recommend you to recalculate currently generated intervals. This is easy to achieve by issuing via REST API any change to calculator configuration or by disabling and re-enabling assigned device .

We have added possibility to access JSON objects and arrays in expressions . Imagine that you have next message: {"key.s":{"key2":2, "key3":3}, "key.p":"key2", "key.a": [10, 9]} In order to access value under "key2" it is possible to use any of next formats (by index, by name or using name from other message parameter): json('key.s', 0) == json('key.s', 'key2') == json('key.s', key.p) == 2 To access values inside arrays use zero-based index specification: json('key.a', 0) == 10

Calculators received new parameter - update_period . It specifies the validity period for the generated intervals and covers any update to the intervals - due to new messages or manual parameter changes in the calculator. Any interval which end-time lies outside of specified update_period (in seconds from the current moment of time) will be ignored. The default value for update_period is 365 days and we recommend to keep in balance next formula: update_period < messages_ttl (in devices) < intervals_ttl . For example, you may set update_period to 30 days, messages_ttl to 90 days (to keep original messages from devices up to 90 days) and intervals_ttl to 10 years - to keep lightweight report results for up to 10 years. One important notice is that if you want to change calculator settings and want the system to resync all current intervals according to new settings you should set update_interval >= intervals_ttl to cover the whole range. You can read more details in calculators knowledge base .

We have implemented different strategy for intervals synchronization during constant recalculation in flespi analytics. This may affect some intervals that begin and end at the same time - both zero-second duration interval and siblings. You may expect a little bit different behavior (we suppose it is more correct now) and probably some set of extra update events when synchronization is running for the first time. Whenever you have any questions or the intervals behavior is different then expected please contact us in the chat.

[Changelog] flespi analytics

shal

We installed major update to the analytics engine.

There are two new features available for you now.

First of all - analytics, calculators, intervals and all events are now much easier to debug with toolbox. You can find in the logs & messages section of the corresponding calculator next log entries about corresponding events:

synced/unsynced - log event fires once the device assigned to the calculator changes its synced state. At the moment you assigned the device to the calculator one of the calculation jobs should pick it up. This can be monitored with the unsynced event. Once all initial calculations are made, synced event is generated. This event reflects the state of the calculation system - if it operates with device or is suspended for some reason.
created/updated/deleted - log event fires together with the same interval event. This can be used mostly to assist you in debugging your listeners for MQTT events.
activated/deactivated - this will reflect active the state change for the assigned device, see below regarding second new feature.

The second new feature is about two new events that should make application of flespi analytics in notification handling systems much easier. Initially flespi analytics was designed as a reports engine that provides updates in realtime. That's why we have information about created, updated, deleted intervals and can access the last or currently active interval. But our users often asked us about a simple system that will allow them to catch baisc events like enter/exit geofence, input activation/deactivation, speed control, and so on that are happening to the device right now. For such use cases analytics was a little cumbersome and didn't provide convenient tools. Until now.

So now there are two super-simple MQTT topics that you may use to receive update for the current state of the device:

flespi/interval/gw/calcs/+/devices/+/activated - to handle event when something has just happened (ON).
flespi/interval/gw/calcs/+/devices/+/deactivated - to handle event when something that was active and happening no more (OFF).

These topics will reflect the state of the device currently known to flespi. If you receive activated event, it means the interval configured in calculator's selector is active right now. If it was activated somewhere in the past (by uploading blackbox for example) and currently last interval is not active, you will not receive this event.

Some more information you can read here: https://flespi.com/kb/working-with-devices-assigned-to-calculator

If you want to catch these events in your solution you can wrap them as a flespi message in the MQTT channel and forward to your system with standard flespi streams. Here is a good guide for this: https://flespi.com/kb/how-to-stream-intervals

If you have questions, please contact our team via HelpBox - we will be glad to answer them.

shal

shal We finally installed pending update. Now analytics precision is the same as everywhere in the flespi - in microseconds.

shal

We implemented the new interval selector type="inactive". It can be used to detect connection/reporting problems with the device in realtime and historic mode.

In contrast to all other selector types, in the "inactive" selector interval is created even if there is no data from the device. When device does not send the data for the specified period of time, a new "active" interval is created and its "end" time is automatically incremented every 60 seconds if there is still no data. Once a device is back online and delivers messages to the platform, the interval will be marked as non-active with updated and correct "begin" and "end" times of the inactivity.

shal

New feature is now available in calculators configuration - special boolean property update_onchange.

This property controls the behavior of analytics system when calculator configuration is changed.

With TRUE value (default) the system will synchronize all devices intervals according to a new calculator configuration within update_period (10 days by default). So if you change selectors, counters, or validation fields - you will immediately see the effect with intervals created, updated, deleted events. This is standard behavior of flespi analytics from the very beginning.

While FALSE value will prevent any automatic synchronization to currently existent intervals and will only use the new configuration for new device messages processing. This is convenient to use in production level calculators with thousands of devices assigned to prevent flooding of your handlers with intervals update events.

In order to provide our users with precise control we also added new method that will force analytics system to apply current configuration of calculator to a specific range of device messages: POST /gw/calcs/XXXX/devices/XXXX/recalculate. This can be convenient to force intervals synchronization for historical device messages beyond standard update_period.

shal

We tuned analytics system to use update_delay calculator's property to prevent the creation of false-positive short-living intervals.

The system will delay intervals generation which begin time is less then current time minus update_delay.

This can be effectively used in systems where data delivery can be delayed due to network issues or internal device's logic - for example to correctly delay processing of crash events from Teltonika (which are delivered out of ordinary data stream) or prevent false-positive notifications with only partial data sent by the device.

shal

Two new features now available in analytics.

Filter in the interval with counter type=parameter and method=distinct only distinct values of encountered parameter.
New function available in expressions: previous("parameter-name"). It returns previous value of parameter when possible and operates in similar way as via #parameter-name notation however if the value is not yet known it will return null instead of error. You can check for availability of previous parameter for sequential calculation using not(isnull(previous("parameter-name"))). This function can be used in device plugins as well.

And as a reminder, for those who works with plugins and calculators - we have nice expression testing tool that simplifies a lot expressions construction and evaluation.

shal

We changed geofences hit test implementation with new R-Tree index to improve the performance of such intervals selectors. Such change may introduce slightly different hit-test results between past and current algorithms which may trigger unexpected interval updates in some cases.

This is just an informational message, but in case you will notice that your geofences selection in intervals generates something unexpected please contact us in the flespi built-in chat with all details.

shal

We have added new type of interval counter - message. It is used to store in the interval full JSON representation of message(-s) validated by some condition, with first, last or each method choices. For the first or last method the value of interval counter will be JSON object. For each method the value will be an array of JSON objects.

shal

To improve expressions that are using json keys new notation has been developed.
Instead of function json('field_name', 'json_path') you may now use field_name['json-path'].

For example you have BLE beacons information in the message:

{
"ble.beacons": [ 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:1000", 
			"rssi": -70 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:2000", 
			"rssi": -65 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:3000", 
			"rssi": -90 
		} 
	]
	}

In order to access first beacon you may use construction: ble.beacons['/0/id'] in the expression. If unable to extract by this path it will return null.

shal

We added 2 additional features for message counter - the possibility to filter fields in the message(-s) added to the interval and possibility to find and add to the interval with some minimum or maximum extremum.

Here is the configuration of counter you may use to add to the interval time and position information when maximum speed was reached within the interval:

For each method counter will generate JSON array with all messages with the minimum/maximum value after expression calculation.

shal

We added validate_message option for selector of types expression, datetime, geofences. It allows to pass into selector for intervals markup operation only those that pass this filter while for interval counters all messages will be available.

Previously similar functionality could be achieved by utilizing merge_unknown option which continues active interval if calculation was not possible. However with this special validate_message filter it is possible to perform much more advanced interval detection with even simpler and straightforward configuration.

Now inside calculator configuration we have three places for validate_message filter:

inside calculator configuration - will filter all messages before passing to cauclator selector and counters;
inside selector configuration - will filter messages for this particular selector before passing them to intervals markup engine;
inside counter configuration - will filter messages for this particular counter during interval field calculation;

shal

We enhanced intervals generation system with reason_code field which defines the reason why this or that interval event was generated. Full actual list of all reasons is available in REST API:

reason_code is attached as a user property (available for MQTT 5.0 protocol clients only) to all events and intervals generated by flespi analytics:

At the same time it is appended as a JSON field into calculator log and visible in LOGS & MESSAGES tab of calculator to detect easily the reason of particular analytics event:

shal

We defined new type of interval selector - calc. It defines intervals based on the intervals created by referenced calculator. Special invert option can be enabled to create opposite intervals - for example if referenced calculator detects trips, opposite calculator will detect all parkings with the reversed logic. validate_interval option can be used to select very specific intervals in referenced calculator.

Think of this interval selector as a building block to create complex logic with various calculators. It applications are vary from setting up specific and different counters for selected intervals generated by another (base, referenced) calculator up to creating matching inverted calculator to have consistent ON/OFF events with same begin/end timings - when parking is always starts on time when trip stopped and next trip always starts from time when parking stopped.

When used in reverse mode it is desirable to match min_active value to max_inactive for both calculators to have maximum consistency.

shal

We implemented new function json_array_count(X) that can be used in expressions which will return number of elements in array X. For example the code: json_array_count(ble.beacons) will return quantity of BLE beacons in the device vicinity.

shal

Interval counter of type calc which can insert into one calculator intervals generated by another calculators (for example insert into trip associated stops) was enhanced with 2 new options: method and validate_interval:

method defines how to insert referenced intervals into the counter field. Either each to generate the array (default method) or insert first or last referenced interval.

validate_interval defines optional validation expression that can filter referenced intervals.

shal

We implemented a new counter type - variable. Value of this counter is dynamically updated from the left to the right with each message pass and it is possible to access it in expressions of all other counters and validate_interval, validate_message fields as well.

This opens a totally new possibility to have some state during counters calculations.

For example it is possible to ignore X count or Y seconds initial messages from the moment interval has started. Or ignore all other iButton/RFID codes except the very first.

Variable value is accessed with variable("name") call. Check this artificial sample of 2 counters which will generate a timestamp of the 5th harsh acceleration event withing interval under name harsh.driving.started:

Please note that variables are not available during intervals markup with selectors, but once interval is marked up and its calculation began variables can be used everywhere.

shal

In flespi analytics we updated the behavior of interval calculations for messages with timestamp.key parameters. timestamp.key is a special parameter that is usually constructed from the original timestamp sent by device with event.seqnum (record/event sequence number) added in fraction part. This method is used to store and process multiple messages with the same timestamp but different record sequence numbers.

Previously the value inside timestamp.key was taken as interval begin and end which could result in intervals with fraction ranges and duration.

Now we will always take just the original timestamp value as was reported by the device for the interval begin or end.

This behavior change may result in a slightly higher level of events from the flespi analytics system towards your integration as some intervals will have timing changed.

shal

New function is now available in analytics expressions: weekday(timestamp). It will return day of week for its argument starting from Monday (1) to Sunday(7).

shal

We are currently upgrading the flespi analytics system with a new communication and storage architecture, moving its operational data out of RAM into MQTT and local cache. This process started on Monday March 18 and will take around a week.

The general architecture of the analytics system and its algorithms for device messages processing remained the same, we just changed the communication system between services and instead of a RAM cache we will use a distributed drive-backed mapped cache.

We do not expect any impact on the intervals calculation system - everything is covered with a very large set of cruel unit and runtime tests and we are keeping an eye on any possible problem that may popup. However if you notice something strange in the analytics system behavior please let us know in flespi chat.

Currently we have already upgraded 15% of nodes where analytics calculations are running and will slowly upgrade remaining nodes.

After the upgrade we will have a lot of new functional possibilities available inside analytics. And the very first in our features list is the possibility to use device metadata in calculators to configure flexible per-device algorithms with a single calculator. We will implement it shortly after the upgrade.

shal

We adjusted the contents of interval JSON generated by flespi calculators to always reflect in its JSON fields all defined counters.

Counters that were not able to calculate a value within interval messages will now store null values to the interval JSON. Previously such counters did not add any value to the interval which we consider inconsistent.

Please adopt your integration to that change if required.