Additional new feature: now it is possible to test actual textual parameters values in expressions. For example you can test for non-empty driver identifier with expression ibutton.code!="" or generate intervals when it is changed with expression like ibutton.code == $ibutton.code.
[Changelog] flespi analytics
a month later
- Edited
For those who used merge_unknown=true option in calculators and in general in flespi analytics in selectors configuration - we detected a bug that in some very special cases may lead to generation of false intervals contain messages with unspecified by selector values.
In general the logic of merge unknown option has been rewritten and this may lead to different results comparing to previous logic for messages which ON or OFF state according to interval selectors configuration is unknown.
So the bug has been fixed and we recommend you to recalculate currently generated intervals. This is easy to achieve by issuing via REST API any change to calculator configuration or by disabling and re-enabling assigned device.
14 days later
- Edited
We have added possibility to access JSON objects and arrays in expressions.
Imagine that you have next message:
{"key.s":{"key2":2, "key3":3}, "key.p":"key2", "key.a": [10, 9]}
In order to access value under "key2" it is possible to use any of next formats (by index, by name or using name from other message parameter):
json('key.s', 0) == json('key.s', 'key2') == json('key.s', key.p) == 2
To access values inside arrays use zero-based index specification:
json('key.a', 0) == 10
5 days later
- Edited
Calculators received new parameter - update_period.
It specifies the validity period for the generated intervals and covers any update to the intervals - due to new messages or manual parameter changes in the calculator. Any interval which end-time lies outside of specified update_period (in seconds from the current moment of time) will be ignored.
The default value for update_period is 365 days and we recommend to keep in balance next formula:
update_period < messages_ttl (in devices) < intervals_ttl.
For example, you may set update_period to 30 days, messages_ttl to 90 days (to keep original messages from devices up to 90 days) and intervals_ttl to 10 years - to keep lightweight report results for up to 10 years.
One important notice is that if you want to change calculator settings and want the system to resync all current intervals according to new settings you should set update_interval >= intervals_ttl to cover the whole range.
You can read more details in calculators knowledge base.
2 months later
We have implemented different strategy for intervals synchronization during constant recalculation in flespi analytics. This may affect some intervals that begin and end at the same time - both zero-second duration interval and siblings. You may expect a little bit different behavior (we suppose it is more correct now) and probably some set of extra update events when synchronization is running for the first time.
Whenever you have any questions or the intervals behavior is different then expected please contact us in the chat.
5 days later
We have installed new update to flespi analytics engine that modifies the behavior of active interval state - both active counter in generated interval and active MQTT topic/property for assigned device.
Now non-zero max_inactive property of interval selector can automatically trigger active interval recalculation due to timeout if no messages received within specified time and interval active state may reset to false.
In simple words it means that with active fields for assigned device it is now possible to control that device is in valid state within specified timeout - e.g. online (loss of GPRS connection) or have position information (loss of GPS visibility).
The control in realtime is done by subscribing via MQTT to the topic "flespi/state/gw/calcs/+/devices/+/active" and processing can be triggered once null message is received, meaning that interval for this device is no more active (e.g. timeouted).
- Edited
flespi analytics engine today received a great new feature - the possibility to intersect intervals generated by multiple calculators or, in other words, inject intervals detected by one calculator into another calculator.
This is implemented via counter of type="calc" which contain the configuration of where to look for intervals for injection like this stops counter from trips calculator:
In order for this counter to work correctly, the same device should be assigned to a referenced calculator and once the referenced calculator (stops on the sample above) intervals change, they are automatically added to our calculator interval (trips):
It is possible to limit fields to include from another calculator, like begin and duration fields in the sample on the screenshot - we do not need other fields generated by stops calculator by default like end, id, timestamp.
allow_start_before and allow_finish_after configuration parameters for the counter define the selection of sourced intervals. By default, they are set to false meaning only intervals located exactly inside bounds of our interval are retrieved.
8 days later
And one more great feature for flespi analytics. We implemented new counter of type=accumulator. This counter can accumulate value between intervals and store accumulated resulting value in each interval.
For example you may have trips that contain counter to calculate mileage in each trip. Your next task is to create accumulating total_mileage and total_duration counters and also you want to calculate how much time you were driving since beginning of each day.
You can achieve this by adding to you counters 3 more of type=accumulator:
- total_mileage: configured with name=total_mileage, counter=mileage.
- total_duration: configured with name=total_duration, counter=duration.
- daily_duration: configured with name=daily_duration, counter=duration, reset_interval=day.
With configuration like on the screenshot:
And you end up (in device toolbox)with pretty intervals, where each trip contain information about total mileage, total duration and total daily mileage as a parameter:
a month later
The experimental mark has been cleared from flespi analytics.
Now you may use it in production services and all blocking changes to the REST API are now informed in advance according to the flespi SLA.
There is a certain limit on the size of each field generated by the counter into interval. Previously it was 32MB and today we installed the update that restricts maximum field size in interval to 8MB.
There is also a limit to 32MB of total size of interval in binary representation. Once interval larger then 32MB is calculated flespi analytics system will disable further calculation for the specified device assigned to given calculator and update its state to "synced=false".
3 months later
New option is now available during configuration of expression/geofence interval selectors - max_active. It can restrict maximum active duration for the interval and create new interval if it's condition is still actual.
11 days later
Interval selector of type geofence now is able to control corridors - path of connected points of certain width.
With flespi analytics now it is possible to control next types of geofences:
- circle - single point of certain radius
- polygon - for precise zonal control
- corridor - for precise route control with certain deviation possibility
6 months later
- Edited
New function can be used in expressions from now: exists('param-name') will check if parameter named param-name is exists in the message/interval and return 1 or 0.
Typical usage can be the following. For example to detect movement state of the vehicle you can use engine ignition sensor if available or just check if it is moving over the time:
if(exists('engine.ignition.status'), $engine.ignition.status, mileage()>0.005)
a month later
Currently analytics expressions support only double precision numbers as values. Even when you manipulate strings, we take their 32-bit hash as a value of string parameter.
But we are moving closer and closer to introduce various value types in analytics expressions. Already new expressions library is used in device plugins and sooner or later we will introduce it and in analytics expressions as well. Once introduced all values during expression evalution will have type attribute.
We will initially support next types: number (double precision floating point), boolean (true or false), string and null. It is not allowed to perform operations on the values of different type. For example you can not add true to 5 or 10 to "abc".
To simplify the transition period already in existing expression evaluations engine we introduced (dummy now) functions that will help you later. First of all this is type checks: isnumber(X), isboolean(X), isstring(X) and isnull(X) that will check if value is of given type. Also we introduced value cast functions: tonumber(X), toboolean(X), tostring(X) to convert value to different type.
And of course it is now also possible to use special keywords: null, true and false in expressions that in a new engine will have correct meaning and type. They are dummy now and true evaluates to one, while false and null evaluates to zero.
16 days later
- Edited
We have changed implementation of mileage() function:
- Previously if there were no position.latitude or position.longitude parameter in the current message function returned "impossible to calculate" status and now it will always return zero calculated mileage for all cases when not enough input information.
- Previously it used 3D calculation (with vertical dimension) if altitude data was available at all and now it will also validate altitude values in the range from -4 km up to 15 km and switch to 2D calculation (only horizontal dimension) if altitude is out of this range.
- Edited
Expressions engine in flespi analytics is ready to be upgraded to the next version. Current version of expressions processing engine is called legacy and to be replaced with next version on the next Friday, April 2nd.
The key difference is that we introduce different types of values except for simple number: nulls, strings, and booleans.
This will make different values for some expression evaluations between legacy and next versions. Take for example counter with type=expression and expression="engine.ignition.state". In legacy version it will store 0 or 1 in the interval message and in next version it will store false or true.
Another key difference is that reference to always-valid message parameter ($some.parameter.name) for parameters that are not present in the message will return 0 in legacy version and null in next.
To simplify the upgrade we did our best to synchronize type casts as much as possible and in the current version and for all comparison operators we do automatic type casts to numbers. null and false are cast to zero, true castes to one.
A special set of functions can be used to wrap value and convert it to another type: tonumber, toboolean, tostring.
So what you should do in order to continue to use analytics seamlessly starting next Friday, April 2nd?
We are now evaluating expressions by both versions of libraries and report any differences in internal logs. The only change for the last 48 hours of such operation that we detected is related to different types of counter values like on example above with engine.ignition.status or with expressions that contain tests like "position.speed>0" will evaluate to 1/0 in legacy version and to true/false in next.
We recommend to wrap all such counters with tonumber(X) function: "tonumber(engine.ignition.status)" and "tonumber(position.speed>0)" so that both legacy and next library versions will evaluate expression to the number. And once the expressions engine is upgraded (will be reported in its changelog) you may update your implementation to work with real booleans, strings, or nulls if required.
More information about the next version of the expressions library you may read in the Knowledge Base.
8 days later
- Edited
With a new expressions engine a few new functions are now possible to use in expressions:
- >> and << operators that perform bit shift for number values (remember that the maximum size of integer that can be used is 53 bits only).
- hex(X, [Y, Z]) function to convert a hexadecimal string to a number optionally extracting given amount of bits only.
- ~ and == operators can be used for matching string values to wildcards, case sensitive or case insensitive.
- sqrt(X) function to extract square root from X number.
- strftime(X, Y) function to format date into string similar to strftime.
More details about expressions, functions, and operators available in them can be read here.
7 days later
- Edited
We enhanced json(X, Y) function. Now it is possible to use json path as a second argument to access values inside complex structures.
Imagine you have the following JSON as a message:
{"f1":[{"x":"z", "z":"p"}, 102, {"e":[1, 2, 3]}]}
You may access some fields with expressions:
- json("f1", "/1") and json("f1", 1) will both return 102 (2nd element in array).
- json("f1", "/0/x") will return "z" (access 1st element in array and after that field with name "x").
- json("f1", "/2/1") will return 2 (access 1st element in array and after that 2nd element in next array).
19 days later
Analytics was enhanced with the new error() function. It will generate an error for expression evaluation when encountered.
And more significant change is the way we now handle if function: if(CONDITION, WHEN-TRUE, WHEN-FALSE)
Before this update both WHEN-TRUE and WHEN-FALSE were evaluated which was the result of such expressions like listed below to fail.
Now depending on the CONDITION only WHEN-TRUE or WHEN-FALSE will be evaluated further.
And expressions samples that are affected by the current update:
- if (exists('position.valid'), position.valid, true) - when position.valid is unknown whole expression failed. As a workaround we advised to use $position.valid. Now you may use pure logical expression.
- if (parameter != 0, something/parameter, 0) - this will always fail when the parameter is zero because something divided by zero is not possible at all. Now evaluation of something/parameter will be skipped if the parameter is zero.
