• shal

  • Post #42
  • Edited

To improve expressions that are using json keys new notation has been developed.
Instead of function json('field_name', 'json_path') you may now use field_name['json-path'].

For example you have BLE beacons information in the message:

{
"ble.beacons": [ 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:1000", 
			"rssi": -70 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:2000", 
			"rssi": -65 
		}, 
		{ 
			"id": "0000000-00000-0000-0000-00000:B001:3000", 
			"rssi": -90 
		} 
	]
	}

In order to access first beacon you may use construction: ble.beacons['/0/id'] in the expression. If unable to extract by this path it will return null.

  • ysv likes this.
4 months later

We added 2 additional features for message counter - the possibility to filter fields in the message(-s) added to the interval and possibility to find and add to the interval with some minimum or maximum extremum.

Here is the configuration of counter you may use to add to the interval time and position information when maximum speed was reached within the interval:

For each method counter will generate JSON array with all messages with the minimum/maximum value after expression calculation.

2 months later

  • shal

  • Post #44

We added validate_message option for selector of types expression, datetime, geofences. It allows to pass into selector for intervals markup operation only those that pass this filter while for interval counters all messages will be available.

Previously similar functionality could be achieved by utilizing merge_unknown option which continues active interval if calculation was not possible. However with this special validate_message filter it is possible to perform much more advanced interval detection with even simpler and straightforward configuration.

Now inside calculator configuration we have three places for validate_message filter:

  • inside calculator configuration - will filter all messages before passing to cauclator selector and counters;
  • inside selector configuration - will filter messages for this particular selector before passing them to intervals markup engine;
  • inside counter configuration - will filter messages for this particular counter during interval field calculation;

    • shal

    • Post #45

    We enhanced intervals generation system with reason_code field which defines the reason why this or that interval event was generated. Full actual list of all reasons is available in REST API:

    reason_code is attached as a user property (available for MQTT 5.0 protocol clients only) to all events and intervals generated by flespi analytics:

    At the same time it is appended as a JSON field into calculator log and visible in LOGS & MESSAGES tab of calculator to detect easily the reason of particular analytics event:

      6 days later

      • shal

      • Post #46

      We defined new type of interval selector - calc. It defines intervals based on the intervals created by referenced calculator. Special invert option can be enabled to create opposite intervals - for example if referenced calculator detects trips, opposite calculator will detect all parkings with the reversed logic. validate_interval option can be used to select very specific intervals in referenced calculator.

      Think of this interval selector as a building block to create complex logic with various calculators. It applications are vary from setting up specific and different counters for selected intervals generated by another (base, referenced) calculator up to creating matching inverted calculator to have consistent ON/OFF events with same begin/end timings - when parking is always starts on time when trip stopped and next trip always starts from time when parking stopped.

      When used in reverse mode it is desirable to match min_active value to max_inactive for both calculators to have maximum consistency.

        21 days later

        We implemented new function json_array_count(X) that can be used in expressions which will return number of elements in array X. For example the code: json_array_count(ble.beacons) will return quantity of BLE beacons in the device vicinity.

          Interval counter of type calc which can insert into one calculator intervals generated by another calculators (for example insert into trip associated stops) was enhanced with 2 new options: method and validate_interval:

          method defines how to insert referenced intervals into the counter field. Either each to generate the array (default method) or insert first or last referenced interval.

          validate_interval defines optional validation expression that can filter referenced intervals.

            6 days later

            We implemented a new counter type - variable. Value of this counter is dynamically updated from the left to the right with each message pass and it is possible to access it in expressions of all other counters and validate_interval, validate_message fields as well.

            This opens a totally new possibility to have some state during counters calculations.

            For example it is possible to ignore X count or Y seconds initial messages from the moment interval has started. Or ignore all other iButton/RFID codes except the very first.

            Variable value is accessed with variable("name") call. Check this artificial sample of 2 counters which will generate a timestamp of the 5th harsh acceleration event withing interval under name harsh.driving.started:

            Please note that variables are not available during intervals markup with selectors, but once interval is marked up and its calculation began variables can be used everywhere.

              3 months later

              In flespi analytics we updated the behavior of interval calculations for messages with timestamp.key parameters. timestamp.key is a special parameter that is usually constructed from the original timestamp sent by device with event.seqnum (record/event sequence number) added in fraction part. This method is used to store and process multiple messages with the same timestamp but different record sequence numbers.

              Previously the value inside timestamp.key was taken as interval begin and end which could result in intervals with fraction ranges and duration.

              Now we will always take just the original timestamp value as was reported by the device for the interval begin or end.

              This behavior change may result in a slightly higher level of events from the flespi analytics system towards your integration as some intervals will have timing changed.

                2 months later

                New function is now available in analytics expressions: weekday(timestamp). It will return day of week for its argument starting from Monday (1) to Sunday(7).

                  10 days later

                  • shal

                  • Post #52

                  We are currently upgrading the flespi analytics system with a new communication and storage architecture, moving its operational data out of RAM into MQTT and local cache. This process started on Monday March 18 and will take around a week.

                  The general architecture of the analytics system and its algorithms for device messages processing remained the same, we just changed the communication system between services and instead of a RAM cache we will use a distributed drive-backed mapped cache.

                  We do not expect any impact on the intervals calculation system - everything is covered with a very large set of cruel unit and runtime tests and we are keeping an eye on any possible problem that may popup. However if you notice something strange in the analytics system behavior please let us know in flespi chat.

                  Currently we have already upgraded 15% of nodes where analytics calculations are running and will slowly upgrade remaining nodes.

                  After the upgrade we will have a lot of new functional possibilities available inside analytics. And the very first in our features list is the possibility to use device metadata in calculators to configure flexible per-device algorithms with a single calculator. We will implement it shortly after the upgrade.

                    13 days later

                    We adjusted the contents of interval JSON generated by flespi calculators to always reflect in its JSON fields all defined counters.

                    Counters that were not able to calculate a value within interval messages will now store null values to the interval JSON. Previously such counters did not add any value to the interval which we consider inconsistent.

                    Please adopt your integration to that change if required.

                      We've modified the logic inside inactivity detection selector in flespi analytics. The change in logic affected only the indicated end time of detected inactive interval. It was extended up to the moment of connectivity with the device was restored.

                      Previously the end time of such interval was a last message with latency beyond configured. This is indicated on a screenshot below from a inactivity detection calculator with 60 seconds configured allowed data latency. At the top of screenshot you can see interval and at the bottom last included messages (highlighted):

                      As you can see at 07:26:05 connectivity was restored, but the end of interval was indicated as 07:25:04 - it was the timestamp of last message with latency above configured 60 seconds.

                      The new logic will finish such interval on the last reported message before the connectivity was restored. Here is the same data calculated with a new logic:

                      In this logic message with timestamp 07:26:04 was included into inactivity period, because connectivity was restored at 07:26:05.

                        8 days later

                        • shal

                        • Post #55
                        • Edited

                        We added access to device metadata directly from expressions in calculators.

                        You may configure in device metadata any specific per-device configuration, for example maximum speed limit or controlling geofence and use this data in calculator selectors or counters.

                        To access metadata you use new function metadata(“field-path”). For example, let's imagine that you store in the device's metadata two fields: maximum speed limit and coordinates of locked point. You can do this by executing the following REST API call after which device metadata will be the following:

                        "metadata": {
  "speed_limit": 100,
  "geofence": {
   "lat": 50,
   "lon": 20
  }
 }

                        Now you can create a calculator that will handle speed and generate overspeeding intervals according to the currently configured speed limit. To do this use the following configuration of calculator:

                        {
  "selectors": [
	{
  	"expression": "position.speed > metadata(\"speed_limit\")",
  	"type": "expression"
	}
  ],
  "name": "Overspeeding",
  "validate_message": "not(isnull(metadata(\"speed_limit\")))",
  "update_onchange": true
}

                        Validate message parameter used to disable calculator when no speed_limit parameter is defined in device’s metadata.

                        Same you can achieve with geofence. In order to generate interval when device moves 100 meters outside of specified center point you use the following expression in selector:

                        distance(metadata("/geofence/lat"), metadata("/geofence/lon"), position.latitude, position.longitude) > 100

                        Important note. Update_onchange option in calculator properties will determine the calculator action when you changed the metadata - should it recalculate everything (update_onchange=true) or just ignore change in metadata and initiate minor recalculation only when the device sends a new message.

                        P.S. Access to metadata is provided only for devices assigned to calculators and not available using calculation on-demand API call.

                        5 days later

                        We enhanced parameter interval counter with method=distinct. Now it is able to add to the interval (e.g. trip) a list of all unique geofences visited, BLE beacons detected or DTC codes reported.
                        Of course It will also work with other parameter that you may have in device message.

                        Check the details in this KB article covering these use cases.

                        Now it is possible to add to the interval in one calculator aggregated data from intervals of another calculator. You can do this by utilizing two new methods for calculator counters of type=”calc”: aggregate and aggregate_by_field:

                        • “aggregate” will aggregate all intervals into a single resulting object.
                        • “aggregate_by_field” will take the first defined field as a key and aggregate all intervals with the same key into a single resulting object. The output of that counter will be an array of stats per each encountered key.

                        Also for counter type=”calc” we enabled the usage of flespi expressions in fields configuration. This is available for all methods, not only for aggregation.

                        You can read more information about the usage of this new functionality in a related KB article describing the method how to calculate aggregated time spent daily in each geofence.

                          19 days later

                          flespi analytics system received new intervals processing engine. This may affect how calculator's intervals are detected in real-time - created, updated and deleted.

                          We implemented a huge set of tests to ensure seamless upgrade of this engine however in such complex system as telematics analytics anything can go wrong.

                          We expect that it should now process data from your devices better and detect more complex edge cases but if you notice something strange with intervals detection please contact us in HelpBox with all details to look into your case.

                            8 days later

                            We enabled recently added geofences in analytics.

                            In order to activate them you need to assign geofences to the calculator either directly or via groups.

                            You have 2 options to use geofences in calculators.

                            Option 1

                            Use them to select intervals instead of explicitly defined geofences in calculator configuration:

                            Option 2

                            You may access the status of the device relative to geofence in any expression within the calculator. We implemented 2 new functions that you may use:

                            • geofence() - will return the name of the geofence with the highest priority device is in now or null otherwise.
                            • geofences() - will return the array of names device is currently in now.

                            You may enhance that usage by specifying other fields (or even a few fields) to return from the active geofence by using optional argument to these functions: geofence("metadata.max_speed") will return the maximum speed defined by geofence’s metadata.

                            For example to detect over-speeding intervals based on the maximum speed defined in geofence’s metadata you may use the following interval selector configuration:

                            Or if you want to count how much time vehicle moved above specified speed limit for the geofence you can use this inside interval counter as well:

                            Keep in mind that this functionality is not available in non-automated on-demand analytics mode - e.g. when you calculate intervals using REST API.

                              5 months later

                              We enabled geofences assigned directly to devices in flespi analytics.

                              So right now there are 3 variants how you can control geofences with calculators:

                              • assign geofences to calculator either directly or indirectly via groups;
                              • assign geofences to device;
                              • specify geofences geometry directly in interval selector (only to detect entrance/exit by the selector);

                              When you hit test current message with geofence() or geofences() function in calculators' expressions both geofences linked to the device and calculator are analyzed together, based on their actual priority and geometry.

                              To simplify the setup we recommend to use in your flespi-based project one of these methods only. However if both device and calculator have linked geofences you will have a combined hit-test result in expressions. Later we may add to calculators the configurable selection of geofences source (device, calculator or both) if there will be appropriate demand.

                              • shal

                              • Post #61
                              • Edited

                              We adjusted the behavior of interval activated and deactivated events in flespi analytics.

                              Now activated event will be additionally generated when active interval ID has changed. For previous interval if it had active status beforehand additional deactivated event will be generated correspondingly.

                              For example if you configured calculator to control multiple adjacent geofences with the same name and device is immediately transitioning from zone1 to zone2. Earlier activated event would be skipped because device state for calculator was already active (it was in zone1). Now it will generate deactivated event for interval related to zone1 and activated event for new interval related to zone2.

                              This makes notification systems on top of flespi analytics easier to setup with more straightforward binding to flespi/interval/gw/calcs/{calc-id}/devices/{device-id}/activated,deactivated topic for all most important events.