Here we will post information about flespi analytics new features and bugfixes.

Fixed bug with last interval sometimes reset to null in flespi/state MQTT. Bug may occurred when last interval was inactive.

Again modified trigger event for 'last' interval change. Previous fix wasn't complete. Modified detection of recalculation time ranges when receiving new message. Previous implementation used too large ranges. The detection of time ranges for recalculation upon receiving of new messages is primary for analytics. In worse case this can lead to problems in intervals detections or splitting them into multiple parts.

There was a bug in the latest update that affected event triggering system. Thus interval 'created' event triggered for each message. Bug was fixed and update was installed.

adsa By the way, we recommend to disable/enable all devices assigned to calculators in order to correctly recalculate intervals that may be wrong due to bug installed this morning. You can do it with simple REST API calls : curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":false}' 'https://flespi.io/gw/calcs/all/devices/all' curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":true}' 'https://flespi.io/gw/calcs/all/devices/all' if("undefined"!==typeof hljs)hljs._ha();else if("undefined"===typeof hljsLoading){hljsLoading=1;var a=document.getElementsByTagName("head")[0],e=document.createElement("link");e.type="text/css";e.rel="stylesheet";e.href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.7.0/styles/default.min.css";a.appendChild(e);e=document.createElement("script");e.type="text/javascript";e.onload=function(){var d={},f=0;hljs._hb=function(b){b.removeAttribute("data-hljs");var c=b.innerHTML;c in d?b.innerHTML=d[c]:(7

After a long pause we have more updates to the flespi analytics engine. First of all it is possible to use date/time functions in expressions - month(X), day(X), hour(X), minute(X). This can be used for intervals validation e.g. hour(begin)>8 or just to filter some messages with hour(X)>12 if you need to detect intervals for some shifts only. Any kind of usage of course. And additionally we have added active MQTT topic for device assigned to calculator . It will contain the same value as last whenever last detected interval is active at the moment. Once it is no more active null will be published into MQTT. active topic can be used as a foundation for analytics-based own notifications system.

Additional new feature: now it is possible to test actual textual parameters values in expressions . For example you can test for non-empty driver identifier with expression ibutton.code!="" or generate intervals when it is changed with expression like ibutton.code == $ibutton.code .

For those who used merge_unknown=true option in calculators and in general in flespi analytics in selectors configuration - we detected a bug that in some very special cases may lead to generation of false intervals contain messages with unspecified by selector values. In general the logic of merge unknown option has been rewritten and this may lead to different results comparing to previous logic for messages which ON or OFF state according to interval selectors configuration is unknown. So the bug has been fixed and we recommend you to recalculate currently generated intervals. This is easy to achieve by issuing via REST API any change to calculator configuration or by disabling and re-enabling assigned device .

We have added possibility to access JSON objects and arrays in expressions . Imagine that you have next message: {"key.s":{"key2":2, "key3":3}, "key.p":"key2", "key.a": [10, 9]} In order to access value under "key2" it is possible to use any of next formats (by index, by name or using name from other message parameter): json('key.s', 0) == json('key.s', 'key2') == json('key.s', key.p) == 2 To access values inside arrays use zero-based index specification: json('key.a', 0) == 10

Calculators received new parameter - update_period . It specifies the validity period for the generated intervals and covers any update to the intervals - due to new messages or manual parameter changes in the calculator. Any interval which end-time lies outside of specified update_period (in seconds from the current moment of time) will be ignored. The default value for update_period is 365 days and we recommend to keep in balance next formula: update_period < messages_ttl (in devices) < intervals_ttl . For example, you may set update_period to 30 days, messages_ttl to 90 days (to keep original messages from devices up to 90 days) and intervals_ttl to 10 years - to keep lightweight report results for up to 10 years. One important notice is that if you want to change calculator settings and want the system to resync all current intervals according to new settings you should set update_interval >= intervals_ttl to cover the whole range. You can read more details in calculators knowledge base .

We have implemented different strategy for intervals synchronization during constant recalculation in flespi analytics. This may affect some intervals that begin and end at the same time - both zero-second duration interval and siblings. You may expect a little bit different behavior (we suppose it is more correct now) and probably some set of extra update events when synchronization is running for the first time. Whenever you have any questions or the intervals behavior is different then expected please contact us in the chat.

[Changelog] flespi analytics

shal

Interval selector of type geofence now is able to control corridors - path of connected points of certain width.

With flespi analytics now it is possible to control next types of geofences:

circle - single point of certain radius
polygon - for precise zonal control
corridor - for precise route control with certain deviation possibility

shal

New function can be used in expressions from now: exists('param-name') will check if parameter named param-name is exists in the message/interval and return 1 or 0.

Typical usage can be the following. For example to detect movement state of the vehicle you can use engine ignition sensor if available or just check if it is moving over the time:
if(exists('engine.ignition.status'), $engine.ignition.status, mileage()>0.005)

shal

Currently analytics expressions support only double precision numbers as values. Even when you manipulate strings, we take their 32-bit hash as a value of string parameter.

But we are moving closer and closer to introduce various value types in analytics expressions. Already new expressions library is used in device plugins and sooner or later we will introduce it and in analytics expressions as well. Once introduced all values during expression evalution will have type attribute.
We will initially support next types: number (double precision floating point), boolean (true or false), string and null. It is not allowed to perform operations on the values of different type. For example you can not add true to 5 or 10 to "abc".

To simplify the transition period already in existing expression evaluations engine we introduced (dummy now) functions that will help you later. First of all this is type checks: isnumber(X), isboolean(X), isstring(X) and isnull(X) that will check if value is of given type. Also we introduced value cast functions: tonumber(X), toboolean(X), tostring(X) to convert value to different type.

And of course it is now also possible to use special keywords: null, true and false in expressions that in a new engine will have correct meaning and type. They are dummy now and true evaluates to one, while false and null evaluates to zero.

shal

We have changed implementation of mileage() function:

Previously if there were no position.latitude or position.longitude parameter in the current message function returned "impossible to calculate" status and now it will always return zero calculated mileage for all cases when not enough input information.
Previously it used 3D calculation (with vertical dimension) if altitude data was available at all and now it will also validate altitude values in the range from -4 km up to 15 km and switch to 2D calculation (only horizontal dimension) if altitude is out of this range.

shal

Expressions engine in flespi analytics is ready to be upgraded to the next version. Current version of expressions processing engine is called legacy and to be replaced with next version on the next Friday, April 2nd.

The key difference is that we introduce different types of values except for simple number: nulls, strings, and booleans.

This will make different values for some expression evaluations between legacy and next versions. Take for example counter with type=expression and expression="engine.ignition.state". In legacy version it will store 0 or 1 in the interval message and in next version it will store false or true.

Another key difference is that reference to always-valid message parameter ($some.parameter.name) for parameters that are not present in the message will return 0 in legacy version and null in next.

To simplify the upgrade we did our best to synchronize type casts as much as possible and in the current version and for all comparison operators we do automatic type casts to numbers. null and false are cast to zero, true castes to one.

A special set of functions can be used to wrap value and convert it to another type: tonumber, toboolean, tostring.

So what you should do in order to continue to use analytics seamlessly starting next Friday, April 2nd?

We are now evaluating expressions by both versions of libraries and report any differences in internal logs. The only change for the last 48 hours of such operation that we detected is related to different types of counter values like on example above with engine.ignition.status or with expressions that contain tests like "position.speed>0" will evaluate to 1/0 in legacy version and to true/false in next.

We recommend to wrap all such counters with tonumber(X) function: "tonumber(engine.ignition.status)" and "tonumber(position.speed>0)" so that both legacy and next library versions will evaluate expression to the number. And once the expressions engine is upgraded (will be reported in its changelog) you may update your implementation to work with real booleans, strings, or nulls if required.

More information about the next version of the expressions library you may read in the Knowledge Base.

shal

shal We performed upgrade process for the analytics services and new expressions processing engine is in effect from now.

shal

With a new expressions engine a few new functions are now possible to use in expressions:

>> and << operators that perform bit shift for number values (remember that the maximum size of integer that can be used is 53 bits only).
hex(X, [Y, Z]) function to convert a hexadecimal string to a number optionally extracting given amount of bits only.
~ and == operators can be used for matching string values to wildcards, case sensitive or case insensitive.
sqrt(X) function to extract square root from X number.
strftime(X, Y) function to format date into string similar to strftime.

More details about expressions, functions, and operators available in them can be read here.

shal

We enhanced json(X, Y) function. Now it is possible to use json path as a second argument to access values inside complex structures.
Imagine you have the following JSON as a message:
{"f1":[{"x":"z", "z":"p"}, 102, {"e":[1, 2, 3]}]}
You may access some fields with expressions:

json("f1", "/1") and json("f1", 1) will both return 102 (2nd element in array).
json("f1", "/0/x") will return "z" (access 1st element in array and after that field with name "x").
json("f1", "/2/1") will return 2 (access 1st element in array and after that 2nd element in next array).

shal

Analytics was enhanced with the new error() function. It will generate an error for expression evaluation when encountered.

And more significant change is the way we now handle if function: if(CONDITION, WHEN-TRUE, WHEN-FALSE)
Before this update both WHEN-TRUE and WHEN-FALSE were evaluated which was the result of such expressions like listed below to fail.

Now depending on the CONDITION only WHEN-TRUE or WHEN-FALSE will be evaluated further.

And expressions samples that are affected by the current update:

if (exists('position.valid'), position.valid, true) - when position.valid is unknown whole expression failed. As a workaround we advised to use $position.valid. Now you may use pure logical expression.
if (parameter != 0, something/parameter, 0) - this will always fail when the parameter is zero because something divided by zero is not possible at all. Now evaluation of something/parameter will be skipped if the parameter is zero.

adsa

REST API requests for calculator management are served with a new internal framework. This shouldn't cause any problems, but if you encounter unexpected behavior, please let us know.
This update allowed any device fields to be used as a selector when assigned to a calculator, i.e.
POST gw/calcs/all/devices/connected=true - assign all connected devices to all available calculators.

The following fields are available as a selector of already assigned devices: name, connected, configuration.
GET gw/calcs/all/devices/name=test* - get all assigned devices whose name matches test * mask.

shal

Be aware that tomorrow we will start upgrade process and move devices messages store from seconds to microseconds granularity. At the same time flespi analytics system will continue to operate with devices on a seconds granularity and thus may not react to some state changes that may be visible in the source messages.

Please read more information here.

We plan to move analytics it to a microseconds granularity a little bit later (October). Please prepare your receivers for MQTT events and REST intervals from analytics system to receive double precision timestamps in begin, end and duration properties for each interval. Also it may be so that within one second there may be generated multiple intervals.
Sample JSON for the old interval:

{
	"begin": 1631516960,
	"end": 1631516963,
	"duration": 3,
	"timestamp": 1631516970.987203,
	...
}

and same JSON for the new interval may look like this:

{
	"begin": 1631516960.700201,
	"end": 1631516963.874323,
	"duration": 3.174122,
	"timestamp": 1631516970.987203,
	...
}

shal

shal We finally installed pending update. Now analytics precision is the same as everywhere in the flespi - in microseconds.

shal

We introduced the change in analytics behavior at the moment when you are assigning a device to the calculator.
Previously only intervals within update_period were calculated. And from now all intervals from time_begin (by default - unlimited) are being generated. But then, when the calculator is changed, the assigned device is disabled/enabled or any new message from the device arrives - only intervals within the update_period are being modified.

Thus with new behavior, once you assign a device to the calculator you end up with the whole device history analyzed and split into intervals. If this is not a behavior you want please use the time_begin attribute during assigning a device to the calculator specifying the exact time from which to calculate intervals.

shal

New function is now available to use in expressions (selectors, counters, plugins, etc) - distance(lat1, lon1, lat2, lon2).
It will calculate distance between two points with specified latitude and longitude coordinates or return zero if unable to calculate or if one of the input parameters is zero or out of bounds.

And a new REST API method GET /gw/calcs/XXX/devices/YYY/intervals/last is available to quickly access the last interval for the device YYY assigned to calculator XXX.

shal

We implemented and installed a set of optimizations to the analytic services. If you notice any strange behavior in the interval events generation, please let us know via HelpBox.

shal

We installed major update to the analytics engine.

There are two new features available for you now.

First of all - analytics, calculators, intervals and all events are now much easier to debug with toolbox. You can find in the logs & messages section of the corresponding calculator next log entries about corresponding events:

synced/unsynced - log event fires once the device assigned to the calculator changes its synced state. At the moment you assigned the device to the calculator one of the calculation jobs should pick it up. This can be monitored with the unsynced event. Once all initial calculations are made, synced event is generated. This event reflects the state of the calculation system - if it operates with device or is suspended for some reason.
created/updated/deleted - log event fires together with the same interval event. This can be used mostly to assist you in debugging your listeners for MQTT events.
activated/deactivated - this will reflect active the state change for the assigned device, see below regarding second new feature.

The second new feature is about two new events that should make application of flespi analytics in notification handling systems much easier. Initially flespi analytics was designed as a reports engine that provides updates in realtime. That's why we have information about created, updated, deleted intervals and can access the last or currently active interval. But our users often asked us about a simple system that will allow them to catch baisc events like enter/exit geofence, input activation/deactivation, speed control, and so on that are happening to the device right now. For such use cases analytics was a little cumbersome and didn't provide convenient tools. Until now.

So now there are two super-simple MQTT topics that you may use to receive update for the current state of the device:

flespi/interval/gw/calcs/+/devices/+/activated - to handle event when something has just happened (ON).
flespi/interval/gw/calcs/+/devices/+/deactivated - to handle event when something that was active and happening no more (OFF).

These topics will reflect the state of the device currently known to flespi. If you receive activated event, it means the interval configured in calculator's selector is active right now. If it was activated somewhere in the past (by uploading blackbox for example) and currently last interval is not active, you will not receive this event.

Some more information you can read here: https://flespi.com/kb/working-with-devices-assigned-to-calculator

If you want to catch these events in your solution you can wrap them as a flespi message in the MQTT channel and forward to your system with standard flespi streams. Here is a good guide for this: https://flespi.com/kb/how-to-stream-intervals

If you have questions, please contact our team via HelpBox - we will be glad to answer them.

shal

We implemented the new interval selector type="inactive". It can be used to detect connection/reporting problems with the device in realtime and historic mode.

In contrast to all other selector types, in the "inactive" selector interval is created even if there is no data from the device. When device does not send the data for the specified period of time, a new "active" interval is created and its "end" time is automatically incremented every 60 seconds if there is still no data. Once a device is back online and delivers messages to the platform, the interval will be marked as non-active with updated and correct "begin" and "end" times of the inactivity.

shal

New feature is now available in calculators configuration - special boolean property update_onchange.

This property controls the behavior of analytics system when calculator configuration is changed.

With TRUE value (default) the system will synchronize all devices intervals according to a new calculator configuration within update_period (10 days by default). So if you change selectors, counters, or validation fields - you will immediately see the effect with intervals created, updated, deleted events. This is standard behavior of flespi analytics from the very beginning.

While FALSE value will prevent any automatic synchronization to currently existent intervals and will only use the new configuration for new device messages processing. This is convenient to use in production level calculators with thousands of devices assigned to prevent flooding of your handlers with intervals update events.

In order to provide our users with precise control we also added new method that will force analytics system to apply current configuration of calculator to a specific range of device messages: POST /gw/calcs/XXXX/devices/XXXX/recalculate. This can be convenient to force intervals synchronization for historical device messages beyond standard update_period.

shal

We tuned analytics system to use update_delay calculator's property to prevent the creation of false-positive short-living intervals.

The system will delay intervals generation which begin time is less then current time minus update_delay.

This can be effectively used in systems where data delivery can be delayed due to network issues or internal device's logic - for example to correctly delay processing of crash events from Teltonika (which are delivered out of ordinary data stream) or prevent false-positive notifications with only partial data sent by the device.

shal

Two new features now available in analytics.

Filter in the interval with counter type=parameter and method=distinct only distinct values of encountered parameter.
New function available in expressions: previous("parameter-name"). It returns previous value of parameter when possible and operates in similar way as via #parameter-name notation however if the value is not yet known it will return null instead of error. You can check for availability of previous parameter for sequential calculation using not(isnull(previous("parameter-name"))). This function can be used in device plugins as well.

And as a reminder, for those who works with plugins and calculators - we have nice expression testing tool that simplifies a lot expressions construction and evaluation.