[Changelog] flespi analytics

shal

Here we will post information about flespi analytics new features and bugfixes.

shal

Fixed bug with last interval sometimes reset to null in flespi/state MQTT. Bug may occurred when last interval was inactive.

shal

Again modified trigger event for 'last' interval change. Previous fix wasn't complete.

Modified detection of recalculation time ranges when receiving new message. Previous implementation used too large ranges. The detection of time ranges for recalculation upon receiving of new messages is primary for analytics. In worse case this can lead to problems in intervals detections or splitting them into multiple parts.

adsa

There was a bug in the latest update that affected event triggering system. Thus interval 'created' event triggered for each message. Bug was fixed and update was installed.

shal

adsa By the way, we recommend to disable/enable all devices assigned to calculators in order to correctly recalculate intervals that may be wrong due to bug installed this morning. You can do it with simple REST API calls:

curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":false}' 'https://flespi.io/gw/calcs/all/devices/all'

curl -X PUT --header 'Authorization: FlespiToken XXX' -d '{"enabled":true}' 'https://flespi.io/gw/calcs/all/devices/all'

shal

After a long pause we have more updates to the flespi analytics engine.

First of all it is possible to use date/time functions in expressions - month(X), day(X), hour(X), minute(X). This can be used for intervals validation e.g. hour(begin)>8 or just to filter some messages with hour(X)>12 if you need to detect intervals for some shifts only. Any kind of usage of course.

And additionally we have added active MQTT topic for device assigned to calculator. It will contain the same value as last whenever last detected interval is active at the moment. Once it is no more active null will be published into MQTT. active topic can be used as a foundation for analytics-based own notifications system.

shal

Additional new feature: now it is possible to test actual textual parameters values in expressions. For example you can test for non-empty driver identifier with expression ibutton.code!="" or generate intervals when it is changed with expression like ibutton.code == $ibutton.code.

shal

For those who used merge_unknown=true option in calculators and in general in flespi analytics in selectors configuration - we detected a bug that in some very special cases may lead to generation of false intervals contain messages with unspecified by selector values.

In general the logic of merge unknown option has been rewritten and this may lead to different results comparing to previous logic for messages which ON or OFF state according to interval selectors configuration is unknown.

So the bug has been fixed and we recommend you to recalculate currently generated intervals. This is easy to achieve by issuing via REST API any change to calculator configuration or by disabling and re-enabling assigned device.

shal

We have added possibility to access JSON objects and arrays in expressions.

Imagine that you have next message:
{"key.s":{"key2":2, "key3":3}, "key.p":"key2", "key.a": [10, 9]}

In order to access value under "key2" it is possible to use any of next formats (by index, by name or using name from other message parameter):
json('key.s', 0) == json('key.s', 'key2') == json('key.s', key.p) == 2

To access values inside arrays use zero-based index specification:
json('key.a', 0) == 10

shal

Calculators received new parameter - update_period.

It specifies the validity period for the generated intervals and covers any update to the intervals - due to new messages or manual parameter changes in the calculator. Any interval which end-time lies outside of specified update_period (in seconds from the current moment of time) will be ignored.

The default value for update_period is 365 days and we recommend to keep in balance next formula:
update_period < messages_ttl (in devices) < intervals_ttl.

For example, you may set update_period to 30 days, messages_ttl to 90 days (to keep original messages from devices up to 90 days) and intervals_ttl to 10 years - to keep lightweight report results for up to 10 years.

One important notice is that if you want to change calculator settings and want the system to resync all current intervals according to new settings you should set update_interval >= intervals_ttl to cover the whole range.

You can read more details in calculators knowledge base.

shal

We have implemented different strategy for intervals synchronization during constant recalculation in flespi analytics. This may affect some intervals that begin and end at the same time - both zero-second duration interval and siblings. You may expect a little bit different behavior (we suppose it is more correct now) and probably some set of extra update events when synchronization is running for the first time.

Whenever you have any questions or the intervals behavior is different then expected please contact us in the chat.

shal

We have installed new update to flespi analytics engine that modifies the behavior of active interval state - both active counter in generated interval and active MQTT topic/property for assigned device.

Now non-zero max_inactive property of interval selector can automatically trigger active interval recalculation due to timeout if no messages received within specified time and interval active state may reset to false.

In simple words it means that with active fields for assigned device it is now possible to control that device is in valid state within specified timeout - e.g. online (loss of GPRS connection) or have position information (loss of GPS visibility).

The control in realtime is done by subscribing via MQTT to the topic "flespi/state/gw/calcs/+/devices/+/active" and processing can be triggered once null message is received, meaning that interval for this device is no more active (e.g. timeouted).

shal

flespi analytics engine today received a great new feature - the possibility to intersect intervals generated by multiple calculators or, in other words, inject intervals detected by one calculator into another calculator.

This is implemented via counter of type="calc" which contain the configuration of where to look for intervals for injection like this stops counter from trips calculator:

In order for this counter to work correctly, the same device should be assigned to a referenced calculator and once the referenced calculator (stops on the sample above) intervals change, they are automatically added to our calculator interval (trips):

It is possible to limit fields to include from another calculator, like begin and duration fields in the sample on the screenshot - we do not need other fields generated by stops calculator by default like end, id, timestamp.

allow_start_before and allow_finish_after configuration parameters for the counter define the selection of sourced intervals. By default, they are set to false meaning only intervals located exactly inside bounds of our interval are retrieved.

shal

And one more great feature for flespi analytics. We implemented new counter of type=accumulator. This counter can accumulate value between intervals and store accumulated resulting value in each interval.

For example you may have trips that contain counter to calculate mileage in each trip. Your next task is to create accumulating total_mileage and total_duration counters and also you want to calculate how much time you were driving since beginning of each day.

You can achieve this by adding to you counters 3 more of type=accumulator:

total_mileage: configured with name=total_mileage, counter=mileage.
total_duration: configured with name=total_duration, counter=duration.
daily_duration: configured with name=daily_duration, counter=duration, reset_interval=day.

With configuration like on the screenshot:

And you end up (in device toolbox)with pretty intervals, where each trip contain information about total mileage, total duration and total daily mileage as a parameter:

shal

The experimental mark has been cleared from flespi analytics.
Now you may use it in production services and all blocking changes to the REST API are now informed in advance according to the flespi SLA.

shal

There is a certain limit on the size of each field generated by the counter into interval. Previously it was 32MB and today we installed the update that restricts maximum field size in interval to 8MB.
There is also a limit to 32MB of total size of interval in binary representation. Once interval larger then 32MB is calculated flespi analytics system will disable further calculation for the specified device assigned to given calculator and update its state to "synced=false".

shal

New option is now available during configuration of expression/geofence interval selectors - max_active. It can restrict maximum active duration for the interval and create new interval if it's condition is still actual.

shal

Interval selector of type geofence now is able to control corridors - path of connected points of certain width.

With flespi analytics now it is possible to control next types of geofences:

circle - single point of certain radius
polygon - for precise zonal control
corridor - for precise route control with certain deviation possibility

shal

New function can be used in expressions from now: exists('param-name') will check if parameter named param-name is exists in the message/interval and return 1 or 0.

Typical usage can be the following. For example to detect movement state of the vehicle you can use engine ignition sensor if available or just check if it is moving over the time:
if(exists('engine.ignition.status'), $engine.ignition.status, mileage()>0.005)

shal

Currently analytics expressions support only double precision numbers as values. Even when you manipulate strings, we take their 32-bit hash as a value of string parameter.

But we are moving closer and closer to introduce various value types in analytics expressions. Already new expressions library is used in device plugins and sooner or later we will introduce it and in analytics expressions as well. Once introduced all values during expression evalution will have type attribute.
We will initially support next types: number (double precision floating point), boolean (true or false), string and null. It is not allowed to perform operations on the values of different type. For example you can not add true to 5 or 10 to "abc".

To simplify the transition period already in existing expression evaluations engine we introduced (dummy now) functions that will help you later. First of all this is type checks: isnumber(X), isboolean(X), isstring(X) and isnull(X) that will check if value is of given type. Also we introduced value cast functions: tonumber(X), toboolean(X), tostring(X) to convert value to different type.

And of course it is now also possible to use special keywords: null, true and false in expressions that in a new engine will have correct meaning and type. They are dummy now and true evaluates to one, while false and null evaluates to zero.