• shal

  • Post #14

We have installed new update to flespi analytics engine that modifies the behavior of active interval state - both active counter in generated interval and active MQTT topic/property for assigned device.

Now non-zero max_inactive property of interval selector can automatically trigger active interval recalculation due to timeout if no messages received within specified time and interval active state may reset to false.

In simple words it means that with active fields for assigned device it is now possible to control that device is in valid state within specified timeout - e.g. online (loss of GPRS connection) or have position information (loss of GPS visibility).

The control in realtime is done by subscribing via MQTT to the topic "flespi/state/gw/calcs/+/devices/+/active" and processing can be triggered once null message is received, meaning that interval for this device is no more active (e.g. timeouted).

  • shal

  • Post #15
  • Edited

flespi analytics engine today received a great new feature - the possibility to intersect intervals generated by multiple calculators or, in other words, inject intervals detected by one calculator into another calculator.

This is implemented via counter of type="calc" which contain the configuration of where to look for intervals for injection like this stops counter from trips calculator:

In order for this counter to work correctly, the same device should be assigned to a referenced calculator and once the referenced calculator (stops on the sample above) intervals change, they are automatically added to our calculator interval (trips):

It is possible to limit fields to include from another calculator, like begin and duration fields in the sample on the screenshot - we do not need other fields generated by stops calculator by default like end, id, timestamp.

allow_start_before and allow_finish_after configuration parameters for the counter define the selection of sourced intervals. By default, they are set to false meaning only intervals located exactly inside bounds of our interval are retrieved.

8 days later

And one more great feature for flespi analytics. We implemented new counter of type=accumulator. This counter can accumulate value between intervals and store accumulated resulting value in each interval.

For example you may have trips that contain counter to calculate mileage in each trip. Your next task is to create accumulating total_mileage and total_duration counters and also you want to calculate how much time you were driving since beginning of each day.

You can achieve this by adding to you counters 3 more of type=accumulator:

  • total_mileage: configured with name=total_mileage, counter=mileage.
  • total_duration: configured with name=total_duration, counter=duration.
  • daily_duration: configured with name=daily_duration, counter=duration, reset_interval=day.

With configuration like on the screenshot:

And you end up (in device toolbox)with pretty intervals, where each trip contain information about total mileage, total duration and total daily mileage as a parameter:

a month later

The experimental mark has been cleared from flespi analytics.
Now you may use it in production services and all blocking changes to the REST API are now informed in advance according to the flespi SLA.

    There is a certain limit on the size of each field generated by the counter into interval. Previously it was 32MB and today we installed the update that restricts maximum field size in interval to 8MB.
    There is also a limit to 32MB of total size of interval in binary representation. Once interval larger then 32MB is calculated flespi analytics system will disable further calculation for the specified device assigned to given calculator and update its state to "synced=false".

      3 months later

      New option is now available during configuration of expression/geofence interval selectors - max_active. It can restrict maximum active duration for the interval and create new interval if it's condition is still actual.

        11 days later

        Interval selector of type geofence now is able to control corridors - path of connected points of certain width.

        With flespi analytics now it is possible to control next types of geofences:

        • circle - single point of certain radius
        • polygon - for precise zonal control
        • corridor - for precise route control with certain deviation possibility
          6 months later

          • shal

          • Post #21
          • Edited

          New function can be used in expressions from now: exists('param-name') will check if parameter named param-name is exists in the message/interval and return 1 or 0.

          Typical usage can be the following. For example to detect movement state of the vehicle you can use engine ignition sensor if available or just check if it is moving over the time:
          if(exists('engine.ignition.status'), $engine.ignition.status, mileage()>0.005)

          a month later

          Currently analytics expressions support only double precision numbers as values. Even when you manipulate strings, we take their 32-bit hash as a value of string parameter.

          But we are moving closer and closer to introduce various value types in analytics expressions. Already new expressions library is used in device plugins and sooner or later we will introduce it and in analytics expressions as well. Once introduced all values during expression evalution will have type attribute.
          We will initially support next types: number (double precision floating point), boolean (true or false), string and null. It is not allowed to perform operations on the values of different type. For example you can not add true to 5 or 10 to "abc".

          To simplify the transition period already in existing expression evaluations engine we introduced (dummy now) functions that will help you later. First of all this is type checks: isnumber(X), isboolean(X), isstring(X) and isnull(X) that will check if value is of given type. Also we introduced value cast functions: tonumber(X), toboolean(X), tostring(X) to convert value to different type.

          And of course it is now also possible to use special keywords: null, true and false in expressions that in a new engine will have correct meaning and type. They are dummy now and true evaluates to one, while false and null evaluates to zero.

          16 days later

          • shal

          • Post #23
          • Edited

          We have changed implementation of mileage() function:

          • Previously if there were no position.latitude or position.longitude parameter in the current message function returned "impossible to calculate" status and now it will always return zero calculated mileage for all cases when not enough input information.
          • Previously it used 3D calculation (with vertical dimension) if altitude data was available at all and now it will also validate altitude values in the range from -4 km up to 15 km and switch to 2D calculation (only horizontal dimension) if altitude is out of this range.

          • shal

          • Post #24
          • Edited

          Expressions engine in flespi analytics is ready to be upgraded to the next version. Current version of expressions processing engine is called legacy and to be replaced with next version on the next Friday, April 2nd.

          The key difference is that we introduce different types of values except for simple number: nulls, strings, and booleans.

          This will make different values for some expression evaluations between legacy and next versions. Take for example counter with type=expression and expression="engine.ignition.state". In legacy version it will store 0 or 1 in the interval message and in next version it will store false or true.

          Another key difference is that reference to always-valid message parameter ($some.parameter.name) for parameters that are not present in the message will return 0 in legacy version and null in next.

          To simplify the upgrade we did our best to synchronize type casts as much as possible and in the current version and for all comparison operators we do automatic type casts to numbers. null and false are cast to zero, true castes to one.

          A special set of functions can be used to wrap value and convert it to another type: tonumber, toboolean, tostring.

          So what you should do in order to continue to use analytics seamlessly starting next Friday, April 2nd?

          We are now evaluating expressions by both versions of libraries and report any differences in internal logs. The only change for the last 48 hours of such operation that we detected is related to different types of counter values like on example above with engine.ignition.status or with expressions that contain tests like "position.speed>0" will evaluate to 1/0 in legacy version and to true/false in next.

          We recommend to wrap all such counters with tonumber(X) function: "tonumber(engine.ignition.status)" and "tonumber(position.speed>0)" so that both legacy and next library versions will evaluate expression to the number. And once the expressions engine is upgraded (will be reported in its changelog) you may update your implementation to work with real booleans, strings, or nulls if required.

          More information about the next version of the expressions library you may read in the Knowledge Base.

          8 days later

          shal We performed upgrade process for the analytics services and new expressions processing engine is in effect from now.

            • shal

            • Post #26
            • Edited

            With a new expressions engine a few new functions are now possible to use in expressions:

            • >> and << operators that perform bit shift for number values (remember that the maximum size of integer that can be used is 53 bits only).
            • hex(X, [Y, Z]) function to convert a hexadecimal string to a number optionally extracting given amount of bits only.
            • ~ and == operators can be used for matching string values to wildcards, case sensitive or case insensitive.
            • sqrt(X) function to extract square root from X number.
            • strftime(X, Y) function to format date into string similar to strftime.

            More details about expressions, functions, and operators available in them can be read here.

            7 days later

            • shal

            • Post #27
            • Edited

            We enhanced json(X, Y) function. Now it is possible to use json path as a second argument to access values inside complex structures.
            Imagine you have the following JSON as a message:
            {"f1":[{"x":"z", "z":"p"}, 102, {"e":[1, 2, 3]}]}
            You may access some fields with expressions:

            • json("f1", "/1") and json("f1", 1) will both return 102 (2nd element in array).
            • json("f1", "/0/x") will return "z" (access 1st element in array and after that field with name "x").
            • json("f1", "/2/1") will return 2 (access 1st element in array and after that 2nd element in next array).
            19 days later

            • shal

            • Post #28

            Analytics was enhanced with the new error() function. It will generate an error for expression evaluation when encountered.

            And more significant change is the way we now handle if function: if(CONDITION, WHEN-TRUE, WHEN-FALSE)
            Before this update both WHEN-TRUE and WHEN-FALSE were evaluated which was the result of such expressions like listed below to fail.

            Now depending on the CONDITION only WHEN-TRUE or WHEN-FALSE will be evaluated further.

            And expressions samples that are affected by the current update:

            • if (exists('position.valid'), position.valid, true) - when position.valid is unknown whole expression failed. As a workaround we advised to use $position.valid. Now you may use pure logical expression.
            • if (parameter != 0, something/parameter, 0) - this will always fail when the parameter is zero because something divided by zero is not possible at all. Now evaluation of something/parameter will be skipped if the parameter is zero.
            3 months later

            REST API requests for calculator management are served with a new internal framework. This shouldn't cause any problems, but if you encounter unexpected behavior, please let us know.
            This update allowed any device fields to be used as a selector when assigned to a calculator, i.e.
            POST gw/calcs/all/devices/connected=true - assign all connected devices to all available calculators.

            The following fields are available as a selector of already assigned devices: name, connected, configuration.
            GET gw/calcs/all/devices/name=test* - get all assigned devices whose name matches test * mask.

              2 months later

              • shal

              • Post #30
              • Edited

              Be aware that tomorrow we will start upgrade process and move devices messages store from seconds to microseconds granularity. At the same time flespi analytics system will continue to operate with devices on a seconds granularity and thus may not react to some state changes that may be visible in the source messages.

              Please read more information here.

              We plan to move analytics it to a microseconds granularity a little bit later (October). Please prepare your receivers for MQTT events and REST intervals from analytics system to receive double precision timestamps in begin, end and duration properties for each interval. Also it may be so that within one second there may be generated multiple intervals.
              Sample JSON for the old interval:

              {
	"begin": 1631516960,
	"end": 1631516963,
	"duration": 3,
	"timestamp": 1631516970.987203,
	...
}

              and same JSON for the new interval may look like this:

              {
	"begin": 1631516960.700201,
	"end": 1631516963.874323,
	"duration": 3.174122,
	"timestamp": 1631516970.987203,
	...
}
              10 days later

              • shal

              • Post #31
              • Edited

              We introduced the change in analytics behavior at the moment when you are assigning a device to the calculator.
              Previously only intervals within update_period were calculated. And from now all intervals from time_begin (by default - unlimited) are being generated. But then, when the calculator is changed, the assigned device is disabled/enabled or any new message from the device arrives - only intervals within the update_period are being modified.

              Thus with new behavior, once you assign a device to the calculator you end up with the whole device history analyzed and split into intervals. If this is not a behavior you want please use the time_begin attribute during assigning a device to the calculator specifying the exact time from which to calculate intervals.

              3 months later

              • shal

              • Post #32
              • Edited

              New function is now available to use in expressions (selectors, counters, plugins, etc) - distance(lat1, lon1, lat2, lon2).
              It will calculate distance between two points with specified latitude and longitude coordinates or return zero if unable to calculate or if one of the input parameters is zero or out of bounds.

              And a new REST API method GET /gw/calcs/XXX/devices/YYY/intervals/last is available to quickly access the last interval for the device YYY assigned to calculator XXX.

              2 months later

              • shal

              • Post #33
              • Edited

              We implemented and installed a set of optimizations to the analytic services. If you notice any strange behavior in the interval events generation, please let us know via HelpBox.