Here we will post information about flespi analytics new features and bugfixes.

Fixed bug with last interval sometimes reset to null in flespi/state MQTT. Bug may occurred when last interval was inactive.

Again modified trigger event for 'last' interval change. Previous fix wasn't complete. Modified detection of recalculation time ranges when receiving new message. Previous implementation used too large ranges. The detection of time ranges for recalculation upon receiving of new messages is primary for analytics. In worse case this can lead to problems in intervals detections or splitting them into multiple parts.

There was a bug in the latest update that affected event triggering system. Thus interval 'created' event triggered for each message. Bug was fixed and update was installed.

adsa By the way, we recommend to disable/enable all devices assigned to calculators in order to correctly recalculate intervals that may be wrong due to bug installed this morning. You can do it with simple REST API calls : curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":false}' 'https://flespi.io/gw/calcs/all/devices/all' curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":true}' 'https://flespi.io/gw/calcs/all/devices/all' if("undefined"!==typeof hljs)hljs._ha();else if("undefined"===typeof hljsLoading){hljsLoading=1;var a=document.getElementsByTagName("head")[0],e=document.createElement("link");e.type="text/css";e.rel="stylesheet";e.href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.7.0/styles/default.min.css";a.appendChild(e);e=document.createElement("script");e.type="text/javascript";e.onload=function(){var d={},f=0;hljs._hb=function(b){b.removeAttribute("data-hljs");var c=b.innerHTML;c in d?b.innerHTML=d[c]:(7

After a long pause we have more updates to the flespi analytics engine. First of all it is possible to use date/time functions in expressions - month(X), day(X), hour(X), minute(X). This can be used for intervals validation e.g. hour(begin)>8 or just to filter some messages with hour(X)>12 if you need to detect intervals for some shifts only. Any kind of usage of course. And additionally we have added active MQTT topic for device assigned to calculator . It will contain the same value as last whenever last detected interval is active at the moment. Once it is no more active null will be published into MQTT. active topic can be used as a foundation for analytics-based own notifications system.

Additional new feature: now it is possible to test actual textual parameters values in expressions . For example you can test for non-empty driver identifier with expression ibutton.code!="" or generate intervals when it is changed with expression like ibutton.code == $ibutton.code .

For those who used merge_unknown=true option in calculators and in general in flespi analytics in selectors configuration - we detected a bug that in some very special cases may lead to generation of false intervals contain messages with unspecified by selector values. In general the logic of merge unknown option has been rewritten and this may lead to different results comparing to previous logic for messages which ON or OFF state according to interval selectors configuration is unknown. So the bug has been fixed and we recommend you to recalculate currently generated intervals. This is easy to achieve by issuing via REST API any change to calculator configuration or by disabling and re-enabling assigned device .

We have added possibility to access JSON objects and arrays in expressions . Imagine that you have next message: {"key.s":{"key2":2, "key3":3}, "key.p":"key2", "key.a": [10, 9]} In order to access value under "key2" it is possible to use any of next formats (by index, by name or using name from other message parameter): json('key.s', 0) == json('key.s', 'key2') == json('key.s', key.p) == 2 To access values inside arrays use zero-based index specification: json('key.a', 0) == 10

Calculators received new parameter - update_period . It specifies the validity period for the generated intervals and covers any update to the intervals - due to new messages or manual parameter changes in the calculator. Any interval which end-time lies outside of specified update_period (in seconds from the current moment of time) will be ignored. The default value for update_period is 365 days and we recommend to keep in balance next formula: update_period < messages_ttl (in devices) < intervals_ttl . For example, you may set update_period to 30 days, messages_ttl to 90 days (to keep original messages from devices up to 90 days) and intervals_ttl to 10 years - to keep lightweight report results for up to 10 years. One important notice is that if you want to change calculator settings and want the system to resync all current intervals according to new settings you should set update_interval >= intervals_ttl to cover the whole range. You can read more details in calculators knowledge base .

We have implemented different strategy for intervals synchronization during constant recalculation in flespi analytics. This may affect some intervals that begin and end at the same time - both zero-second duration interval and siblings. You may expect a little bit different behavior (we suppose it is more correct now) and probably some set of extra update events when synchronization is running for the first time. Whenever you have any questions or the intervals behavior is different then expected please contact us in the chat.

[Changelog] flespi analytics

shal

We adjusted the contents of interval JSON generated by flespi calculators to always reflect in its JSON fields all defined counters.

Counters that were not able to calculate a value within interval messages will now store null values to the interval JSON. Previously such counters did not add any value to the interval which we consider inconsistent.

Please adopt your integration to that change if required.

shal

We've modified the logic inside inactivity detection selector in flespi analytics. The change in logic affected only the indicated end time of detected inactive interval. It was extended up to the moment of connectivity with the device was restored.

Previously the end time of such interval was a last message with latency beyond configured. This is indicated on a screenshot below from a inactivity detection calculator with 60 seconds configured allowed data latency. At the top of screenshot you can see interval and at the bottom last included messages (highlighted):

As you can see at 07:26:05 connectivity was restored, but the end of interval was indicated as 07:25:04 - it was the timestamp of last message with latency above configured 60 seconds.

The new logic will finish such interval on the last reported message before the connectivity was restored. Here is the same data calculated with a new logic:

In this logic message with timestamp 07:26:04 was included into inactivity period, because connectivity was restored at 07:26:05.

shal

We added access to device metadata directly from expressions in calculators.

You may configure in device metadata any specific per-device configuration, for example maximum speed limit or controlling geofence and use this data in calculator selectors or counters.

To access metadata you use new function metadata(“field-path”). For example, let's imagine that you store in the device's metadata two fields: maximum speed limit and coordinates of locked point. You can do this by executing the following REST API call after which device metadata will be the following:

"metadata": {
  "speed_limit": 100,
  "geofence": {
   "lat": 50,
   "lon": 20
  }
 }

Now you can create a calculator that will handle speed and generate overspeeding intervals according to the currently configured speed limit. To do this use the following configuration of calculator:

{
  "selectors": [
	{
  	"expression": "position.speed > metadata(\"speed_limit\")",
  	"type": "expression"
	}
  ],
  "name": "Overspeeding",
  "validate_message": "not(isnull(metadata(\"speed_limit\")))",
  "update_onchange": true
}

Validate message parameter used to disable calculator when no speed_limit parameter is defined in device’s metadata.

Same you can achieve with geofence. In order to generate interval when device moves 100 meters outside of specified center point you use the following expression in selector:

distance(metadata("/geofence/lat"), metadata("/geofence/lon"), position.latitude, position.longitude) > 100

Important note. Update_onchange option in calculator properties will determine the calculator action when you changed the metadata - should it recalculate everything (update_onchange=true) or just ignore change in metadata and initiate minor recalculation only when the device sends a new message.

P.S. Access to metadata is provided only for devices assigned to calculators and not available using calculation on-demand API call.

shal

We enhanced parameter interval counter with method=distinct. Now it is able to add to the interval (e.g. trip) a list of all unique geofences visited, BLE beacons detected or DTC codes reported.
Of course It will also work with other parameter that you may have in device message.

Check the details in this KB article covering these use cases.

shal

Now it is possible to add to the interval in one calculator aggregated data from intervals of another calculator. You can do this by utilizing two new methods for calculator counters of type=”calc”: aggregate and aggregate_by_field:

“aggregate” will aggregate all intervals into a single resulting object.
“aggregate_by_field” will take the first defined field as a key and aggregate all intervals with the same key into a single resulting object. The output of that counter will be an array of stats per each encountered key.

Also for counter type=”calc” we enabled the usage of flespi expressions in fields configuration. This is available for all methods, not only for aggregation.

You can read more information about the usage of this new functionality in a related KB article describing the method how to calculate aggregated time spent daily in each geofence.

shal

flespi analytics system received new intervals processing engine. This may affect how calculator's intervals are detected in real-time - created, updated and deleted.

We implemented a huge set of tests to ensure seamless upgrade of this engine however in such complex system as telematics analytics anything can go wrong.

We expect that it should now process data from your devices better and detect more complex edge cases but if you notice something strange with intervals detection please contact us in HelpBox with all details to look into your case.

shal

We enabled recently added geofences in analytics.

In order to activate them you need to assign geofences to the calculator either directly or via groups.

You have 2 options to use geofences in calculators.

Option 1

Use them to select intervals instead of explicitly defined geofences in calculator configuration:

Option 2

You may access the status of the device relative to geofence in any expression within the calculator. We implemented 2 new functions that you may use:

geofence() - will return the name of the geofence with the highest priority device is in now or null otherwise.
geofences() - will return the array of names device is currently in now.

You may enhance that usage by specifying other fields (or even a few fields) to return from the active geofence by using optional argument to these functions: geofence("metadata.max_speed") will return the maximum speed defined by geofence’s metadata.

For example to detect over-speeding intervals based on the maximum speed defined in geofence’s metadata you may use the following interval selector configuration:

Or if you want to count how much time vehicle moved above specified speed limit for the geofence you can use this inside interval counter as well:

Keep in mind that this functionality is not available in non-automated on-demand analytics mode - e.g. when you calculate intervals using REST API.

shal

We enabled geofences assigned directly to devices in flespi analytics.

So right now there are 3 variants how you can control geofences with calculators:

assign geofences to calculator either directly or indirectly via groups;
assign geofences to device;
specify geofences geometry directly in interval selector (only to detect entrance/exit by the selector);

When you hit test current message with geofence() or geofences() function in calculators' expressions both geofences linked to the device and calculator are analyzed together, based on their actual priority and geometry.

To simplify the setup we recommend to use in your flespi-based project one of these methods only. However if both device and calculator have linked geofences you will have a combined hit-test result in expressions. Later we may add to calculators the configurable selection of geofences source (device, calculator or both) if there will be appropriate demand.

shal

We adjusted the behavior of interval activated and deactivated events in flespi analytics.

Now activated event will be additionally generated when active interval ID has changed. For previous interval if it had active status beforehand additional deactivated event will be generated correspondingly.

For example if you configured calculator to control multiple adjacent geofences with the same name and device is immediately transitioning from zone1 to zone2. Earlier activated event would be skipped because device state for calculator was already active (it was in zone1). Now it will generate deactivated event for interval related to zone1 and activated event for new interval related to zone2.

This makes notification systems on top of flespi analytics easier to setup with more straightforward binding to flespi/interval/gw/calcs/{calc-id}/devices/{device-id}/activated,deactivated topic for all most important events.

shal

We implemented 2 new functions to work with JSON arrays inside expressions:

json_array_contains(X, Y): will return true if array X contains value Y or false otherwise. For string values wildcards can be used as well. For example next expression will test if DTC codes reported by device contain any DTC code that starts from "P": json_array_contains(dtc.codes, "P*").
json_array_find(X, Y): will apply expression Y to each item in JSON array X and return first matched item or null otherwise. For example to fetch BLE beacon with specific ID defined in device's metadata it is possible to use next construction: json_array_find(ble.beacons, "id == '" + tostring(metadata("sensor_id")) + "'"). If device's metadata contains sensor_id with value "ABCD" the function call will be equivalent of: json_array_find(ble.beacons, "id == 'ABCD'"). It's important to format expression in argument Y correctly, en-quoting all strings.

When applying these functions we suggest to use special tool to test your expression before using it in calculator or plugin configuration to avoid syntax mistake.

These new functions can be used in any flespi subsystem: calculators, plugins, webhooks, stream validators, REST API calls. The most common areas of application are DTC codes, BLE beacons and any other array structure reported by devices or constructed with plugins.

shal

We enhanced the flespi analytics system with the possibility to access values located in the previous interval when calculating or validating the next one.

This functionality is available via counter type=”interval” and validate_interval expression.

There you may operate with values calculated by current interval counters and apply some logic mixing values calculated/stored in previous interval. You can access such values using the previous(“interval-counter-or-property”) call.

For example, to know the difference between the end of previous interval and beginning of current interval you may use the counter of type=”interval” and expression: if(previous('end'), begin - previous('end'), 0).

Or if you would like to prevent intervals creation within 1 hour from the end of the previous interval you may put into validate_interval the following expression: begin - previous('end') >= 3600.

As a bonus and for really complex cases previous function will allow you to retrieve the data not only calculated by the counter by also attached to the interval via PUT /gw/calcs/{calc_id}/devices/{device_id}/intervals/{previous-interval-id} call. For example you may attach to some interval user data and subsequently copy its value to all further intervals until the next user-data. For example, for counter name=”datacache”, type=”interval” you may use the next expression: if(previous('userdata'), previous('userdata'), previous('datacache')) and attach to the interval custom value under 'userdata' key.