• shal

  • Post #34
  • Edited

We installed major update to the analytics engine.

There are two new features available for you now.

First of all - analytics, calculators, intervals and all events are now much easier to debug with toolbox. You can find in the logs & messages section of the corresponding calculator next log entries about corresponding events:

  • synced/unsynced - log event fires once the device assigned to the calculator changes its synced state. At the moment you assigned the device to the calculator one of the calculation jobs should pick it up. This can be monitored with the unsynced event. Once all initial calculations are made, synced event is generated. This event reflects the state of the calculation system - if it operates with device or is suspended for some reason.
  • created/updated/deleted - log event fires together with the same interval event. This can be used mostly to assist you in debugging your listeners for MQTT events.
  • activated/deactivated - this will reflect active the state change for the assigned device, see below regarding second new feature.

The second new feature is about two new events that should make application of flespi analytics in notification handling systems much easier. Initially flespi analytics was designed as a reports engine that provides updates in realtime. That's why we have information about created, updated, deleted intervals and can access the last or currently active interval. But our users often asked us about a simple system that will allow them to catch baisc events like enter/exit geofence, input activation/deactivation, speed control, and so on that are happening to the device right now. For such use cases analytics was a little cumbersome and didn't provide convenient tools. Until now.

So now there are two super-simple MQTT topics that you may use to receive update for the current state of the device:

  • flespi/interval/gw/calcs/+/devices/+/activated - to handle event when something has just happened (ON).
  • flespi/interval/gw/calcs/+/devices/+/deactivated - to handle event when something that was active and happening no more (OFF).

These topics will reflect the state of the device currently known to flespi. If you receive activated event, it means the interval configured in calculator's selector is active right now. If it was activated somewhere in the past (by uploading blackbox for example) and currently last interval is not active, you will not receive this event.

Some more information you can read here: https://flespi.com/kb/working-with-devices-assigned-to-calculator

If you want to catch these events in your solution you can wrap them as a flespi message in the MQTT channel and forward to your system with standard flespi streams. Here is a good guide for this: https://flespi.com/kb/how-to-stream-intervals

If you have questions, please contact our team via HelpBox - we will be glad to answer them.

2 months later

shal We finally installed pending update. Now analytics precision is the same as everywhere in the flespi - in microseconds.

    • shal

    • Post #36
    • Edited

    We implemented the new interval selector type="inactive". It can be used to detect connection/reporting problems with the device in realtime and historic mode.

    In contrast to all other selector types, in the "inactive" selector interval is created even if there is no data from the device. When device does not send the data for the specified period of time, a new "active" interval is created and its "end" time is automatically incremented every 60 seconds if there is still no data. Once a device is back online and delivers messages to the platform, the interval will be marked as non-active with updated and correct "begin" and "end" times of the inactivity.

    5 months later

    • shal

    • Post #37
    • Edited

    New feature is now available in calculators configuration - special boolean property update_onchange.

    This property controls the behavior of analytics system when calculator configuration is changed.

    With TRUE value (default) the system will synchronize all devices intervals according to a new calculator configuration within update_period (10 days by default). So if you change selectors, counters, or validation fields - you will immediately see the effect with intervals created, updated, deleted events. This is standard behavior of flespi analytics from the very beginning.

    While FALSE value will prevent any automatic synchronization to currently existent intervals and will only use the new configuration for new device messages processing. This is convenient to use in production level calculators with thousands of devices assigned to prevent flooding of your handlers with intervals update events.

    In order to provide our users with precise control we also added new method that will force analytics system to apply current configuration of calculator to a specific range of device messages: POST /gw/calcs/XXXX/devices/XXXX/recalculate. This can be convenient to force intervals synchronization for historical device messages beyond standard update_period.

    3 months later

    • shal

    • Post #38

    We tuned analytics system to use update_delay calculator's property to prevent the creation of false-positive short-living intervals.

    The system will delay intervals generation which begin time is less then current time minus update_delay.

    This can be effectively used in systems where data delivery can be delayed due to network issues or internal device's logic - for example to correctly delay processing of crash events from Teltonika (which are delivered out of ordinary data stream) or prevent false-positive notifications with only partial data sent by the device.

    a month later

    Two new features now available in analytics.

    1. Filter in the interval with counter type=parameter and method=distinct only distinct values of encountered parameter.
    2. New function available in expressions: previous("parameter-name"). It returns previous value of parameter when possible and operates in similar way as via #parameter-name notation however if the value is not yet known it will return null instead of error. You can check for availability of previous parameter for sequential calculation using not(isnull(previous("parameter-name"))). This function can be used in device plugins as well.

    And as a reminder, for those who works with plugins and calculators - we have nice expression testing tool that simplifies a lot expressions construction and evaluation.

    14 days later

    • shal

    • Post #40

    We changed geofences hit test implementation with new R-Tree index to improve the performance of such intervals selectors. Such change may introduce slightly different hit-test results between past and current algorithms which may trigger unexpected interval updates in some cases.

    This is just an informational message, but in case you will notice that your geofences selection in intervals generates something unexpected please contact us in the flespi built-in chat with all details.

    10 days later

    We have added new type of interval counter - message. It is used to store in the interval full JSON representation of message(-s) validated by some condition, with first, last or each method choices. For the first or last method the value of interval counter will be JSON object. For each method the value will be an array of JSON objects.

    a month later

    • shal

    • Post #42
    • Edited

    To improve expressions that are using json keys new notation has been developed.
    Instead of function json('field_name', 'json_path') you may now use field_name['json-path'].

    For example you have BLE beacons information in the message:

    {
"ble.beacons": [ 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:1000", 
			"rssi": -70 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:2000", 
			"rssi": -65 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:3000", 
			"rssi": -90 
		} 
	]
	}

    In order to access first beacon you may use construction: ble.beacons['/0/id'] in the expression. If unable to extract by this path it will return null.

    • ysv likes this.
    4 months later

    We added 2 additional features for message counter - the possibility to filter fields in the message(-s) added to the interval and possibility to find and add to the interval with some minimum or maximum extremum.

    Here is the configuration of counter you may use to add to the interval time and position information when maximum speed was reached within the interval:

    For each method counter will generate JSON array with all messages with the minimum/maximum value after expression calculation.

    2 months later

    • shal

    • Post #44

    We added validate_message option for selector of types expression, datetime, geofences. It allows to pass into selector for intervals markup operation only those that pass this filter while for interval counters all messages will be available.

    Previously similar functionality could be achieved by utilizing merge_unknown option which continues active interval if calculation was not possible. However with this special validate_message filter it is possible to perform much more advanced interval detection with even simpler and straightforward configuration.

    Now inside calculator configuration we have three places for validate_message filter:

    • inside calculator configuration - will filter all messages before passing to cauclator selector and counters;
    • inside selector configuration - will filter messages for this particular selector before passing them to intervals markup engine;
    • inside counter configuration - will filter messages for this particular counter during interval field calculation;

      • shal

      • Post #45

      We enhanced intervals generation system with reason_code field which defines the reason why this or that interval event was generated. Full actual list of all reasons is available in REST API:

      reason_code is attached as a user property (available for MQTT 5.0 protocol clients only) to all events and intervals generated by flespi analytics:

      At the same time it is appended as a JSON field into calculator log and visible in LOGS & MESSAGES tab of calculator to detect easily the reason of particular analytics event:

        6 days later

        • shal

        • Post #46

        We defined new type of interval selector - calc. It defines intervals based on the intervals created by referenced calculator. Special invert option can be enabled to create opposite intervals - for example if referenced calculator detects trips, opposite calculator will detect all parkings with the reversed logic. validate_interval option can be used to select very specific intervals in referenced calculator.

        Think of this interval selector as a building block to create complex logic with various calculators. It applications are vary from setting up specific and different counters for selected intervals generated by another (base, referenced) calculator up to creating matching inverted calculator to have consistent ON/OFF events with same begin/end timings - when parking is always starts on time when trip stopped and next trip always starts from time when parking stopped.

        When used in reverse mode it is desirable to match min_active value to max_inactive for both calculators to have maximum consistency.

          21 days later

          We implemented new function json_array_count(X) that can be used in expressions which will return number of elements in array X. For example the code: json_array_count(ble.beacons) will return quantity of BLE beacons in the device vicinity.

            Interval counter of type calc which can insert into one calculator intervals generated by another calculators (for example insert into trip associated stops) was enhanced with 2 new options: method and validate_interval:

            method defines how to insert referenced intervals into the counter field. Either each to generate the array (default method) or insert first or last referenced interval.

            validate_interval defines optional validation expression that can filter referenced intervals.

              6 days later

              We implemented a new counter type - variable. Value of this counter is dynamically updated from the left to the right with each message pass and it is possible to access it in expressions of all other counters and validate_interval, validate_message fields as well.

              This opens a totally new possibility to have some state during counters calculations.

              For example it is possible to ignore X count or Y seconds initial messages from the moment interval has started. Or ignore all other iButton/RFID codes except the very first.

              Variable value is accessed with variable("name") call. Check this artificial sample of 2 counters which will generate a timestamp of the 5th harsh acceleration event withing interval under name harsh.driving.started:

              Please note that variables are not available during intervals markup with selectors, but once interval is marked up and its calculation began variables can be used everywhere.

                3 months later

                In flespi analytics we updated the behavior of interval calculations for messages with timestamp.key parameters. timestamp.key is a special parameter that is usually constructed from the original timestamp sent by device with event.seqnum (record/event sequence number) added in fraction part. This method is used to store and process multiple messages with the same timestamp but different record sequence numbers.

                Previously the value inside timestamp.key was taken as interval begin and end which could result in intervals with fraction ranges and duration.

                Now we will always take just the original timestamp value as was reported by the device for the interval begin or end.

                This behavior change may result in a slightly higher level of events from the flespi analytics system towards your integration as some intervals will have timing changed.

                  2 months later

                  New function is now available in analytics expressions: weekday(timestamp). It will return day of week for its argument starting from Monday (1) to Sunday(7).

                    10 days later

                    • shal

                    • Post #52

                    We are currently upgrading the flespi analytics system with a new communication and storage architecture, moving its operational data out of RAM into MQTT and local cache. This process started on Monday March 18 and will take around a week.

                    The general architecture of the analytics system and its algorithms for device messages processing remained the same, we just changed the communication system between services and instead of a RAM cache we will use a distributed drive-backed mapped cache.

                    We do not expect any impact on the intervals calculation system - everything is covered with a very large set of cruel unit and runtime tests and we are keeping an eye on any possible problem that may popup. However if you notice something strange in the analytics system behavior please let us know in flespi chat.

                    Currently we have already upgraded 15% of nodes where analytics calculations are running and will slowly upgrade remaining nodes.

                    After the upgrade we will have a lot of new functional possibilities available inside analytics. And the very first in our features list is the possibility to use device metadata in calculators to configure flexible per-device algorithms with a single calculator. We will implement it shortly after the upgrade.

                      13 days later

                      We adjusted the contents of interval JSON generated by flespi calculators to always reflect in its JSON fields all defined counters.

                      Counters that were not able to calculate a value within interval messages will now store null values to the interval JSON. Previously such counters did not add any value to the interval which we consider inconsistent.

                      Please adopt your integration to that change if required.