Place for information about telematics hub updates.

We supported special streaming protocol called nis . This protocol is used in Russia only, and although it is very stupidly designed - HTTP/SOAP based with just one message per HTTP request - it is quite popular on CIS space for inter-servers communications. Feel free to use it.

nis streaming protocol was updated to send values for Digital inputs, outputs and Analog inputs. The number of streamed Analog inputs is limited up to 8 for nis and wialon_retranslator streams.

We have removed deprecated fields " ident " and " phone " from the devices REST API. These fields was left for backward compatibility for a few months. In addition, we have removed old stream subscriptions REST API methods also marked as deprecated a few months ago. More details are placed in this article: https://flespi.com/blog/sync-up-with-changes-in-the-flespi-device-api

Platform has changed the way it accumulating logs when operating under subaccounts hierarchy . Say you have account root with subaccount sub and it branched another subaccount subsub . Tree looks like this: root<=sub<=subsub. Now imagine that you have log message for the subsub account, for example device was connected. Before the update : logs both in realtime (subscribing to flespi/logs/#) and in history mode (accessible via REST ) where placed by the system into an account subsub and all its parents: sub and root . Thus when operating with large volumes we have quite a big storage size of root top-account basically multiplied by the number of subbaccounts it has. After the update : the log message will be stored exactly and only at the account where it was generated. E.g if you have device or channel that belong to some subaccount you can access log messages for this entity only from its account. This is applied to both REST and MQTT. Toolbox was modified accordingly and now has account selector whenever you have some flespi accounts to switch between.

We increased telemetry expiration time up to 370 days.

We've added the "status" field for each device setting. It's indicating the source of the current setting value (if available) and the flespi platform intention to read the new current value or apply the pending setting value. The full documentation is available here: https://flespi.io/docs/#/gw/!/devices

Platform update affects devices with ident collisions. Collision occurs when two or more devices have the same ident, device type and share a common channel. The device with the lowest identifier remains unlocked, other devices will be blocked using the "bad configuration" code.

adsa We have reverted update until some problems being solved.

Next Monday, on October 14, we will install an update with the small change in our Telematics Gateway REST API. Once installed, the devices creation method ( POST , https://flespi.io/docs/#/gw/!/devices/post_devices ) will require the " device_type_id " and " configuration " fields to be specified. Also, the " configuration " object for all device types will require an " ident " field. Please review your integrations and correct them to always include such fields when you're creating devices. If you have any questions, please contact us in the chat embedded in the flespi.io panel or on forum.flespi.com

adsa The problems were resolved and the update was installed.

[Changelog] flespi telematics hub

shal

We are announcing the future upgrade for filter expression format in various REST API calls. The most important are the GET /devices/XXX/messages and GET /calcs/XXX/devices/YYY/intervals API calls. So all REST API calls where you can specify filter expression will be affected.

Once the whole upgrade is finished it will be possible to use extended expressions in requests with math functions, arithmetic operations, brackets for operations priority and so on. But so far we made simple compatibility mode that you need to adjust your requests to.

List of changes to the filters format

textual fields will require quotes. For example filter="email=123@gmail.com" should be upgraded to filter="email='123@gmail.com'".
instead of | or & separators please use || or &&. For example filter="field1=1|field1=100" should be upgraded to filter="field1=1||field1=100".
parameter existence check in form filter="param1" or filter="param1!=" will stop operating. Please use filter="param1!=''" instead (except the case when param1 may have empty string value).

If some parameter does not exists in the message the whole expression will be marked as invalid and thus message will be skipped. In that case you may already precede parameter name with dollar sign: filter="$param=1 || $param2 = 2"

There will be few stages for this update.

Stage 1: effective from April 26th

We will add warnings response to REST API requests that are still using old scheme. So the ordinary response from request will be:
{"result": [...], "warnings":[{"reason":"You are using REST API call format that will soon be deprecated"}]}
In most of the systems this will not change any kind of processing, because the whole result located under "result" nested object is unaffected.

Stage 2: effective from May 17th

We will switch filters processing in API calls to a new expressions implementation. In case you will be still using older format your requests will return error with corresponding information or even do not operate as expected in some cases. From that moment you will have the ability to use all the power of new expressions engine.

In between stages we are actively monitoring API usage and will personally inform users that will be affected by this change, especially if we already have a non-empty communication chat between flespi support and this user. But nevertheless please do the upgrade of your code on your own as soon as possible and be prepared for the possible changes in requests processing on May 17th.

shal

shal We postponed the upcoming update of filters processing to Wednesday, May 19th.

shal

shal Once again we postponed the update to Tuesday, May 25th. Some users are still using the older filter format and we want them to finish their updates (especially AppStore is slow on this) before we make the final change.

shal

shal we installed last part of the upgrade with filters. If you are noticing any unexpected behavior in your services operation please contact us via HelpBox.

shal

According to our release schedule today we marked method GET gw/devices/{device-selector}/telemetry as deprecated and now warnings are generated for those who are using it.

Also those who are using old scheme in filters (GET /gw/devices/xxx/messages, GET /gw/calcs/xxx/devices/xxx/intervals methods and so on) are also receive warning attached to the actual response of REST API call. Please see post above fore more details.

We are gathering statistics and will send personal notification to those users who are still using deprecated methods or format and will be affected by ongoing changed.

adsa

Along with the plugin filter, we added the same filter to the stream configuration. If a filter ("validate_message" field) is set, the stream will check any message before sending. If the message does not match the expression, it will be skipped.

adsa

Separate plugin field modes "publish", "telemetry" and "store" confused users and introduced inconsistencies between live messages and historical messages. Today we have combined these modes into a single "store" option. This option is enabled by default and adds a field to the message and telemetry.

adsa

The following fields will be added to the device telemetry:

ident
device.id
device.name
device.type.id
protocol.id

The update will take place on May 31.

shal

adsa This update has been installed.

shal

We are announcing the ongoing transition of device messages storage and related flespi subsystems granularity from seconds to microseconds.

The transition will take some time which we are unable to precisely calculate at the moment. It can take a few weeks or even a few months, depending on numerous factors. We will notify you once the transition is 100% finished.

Details about the upcoming change

The typical message that the flespi device generates consists of a timestamp and some additional parameters:
{ "timestamp": 1490347944.893743, "din": 5, "param1": 1, ... }
If you look at the timestamp value you will notice that it is already specified in microseconds granularity. For the sample above 1490347944 is the integer part which can be parsed as Friday, 24 March 2017, 9:32:24. And .893743 is the fractional part of that second - microseconds.
Right now this message will be stored under its integer part. And if for some reason another device message in the same second will arrive, for example:
{ "timestamp": 1490347944.900001, "din": 10, "param2": 2, ... }
Right now it will also be stored now under the same integer key - 1490347944.

Current behavior

In that case, once you will have these two messages registered for the device, if you will query its messages with REST API call right now you will end up with just a single message with combined parameters from both initial messages:
{ "timestamp": 1490347944.900001, "din": 10, "param1": 1, "param2": 2, ... }

New behavior

After we will make the transition of device messages storage from seconds to microseconds with the same REST API call you will get two messages because their fractional part (the storage key) is different and they will be stored separately:
{ "timestamp": 1490347944.893743, "din": 5, "param1": 1, ... }, { "timestamp": 1490347944.900001, "din": 10, "param2": 2, ... }

What should be adopted on the client side?

You should be aware that REST API call GET /gw/devices/XXX/messages now will return you all messages based on microseconds granularity.

This will also affect all other flespi systems - analytics will be able to generate multiple intervals per second and you should just be prepared for this. If possible - replace you your database keys from uint32_t to the double type and any change will be accepted smoothly.

According to our experience, most devices generate only a single message per second so in most cases you will not notice any difference. But for those who rely on multiple messages per one-second possibility - and there are some cases related to iButton on/off, digital/analog input change, and accelerometer data array on impact - just be prepared that you will be able to access all of them from device storage after the transition.

shal

shal Tomorrow on September 14th we will start the migration of devices messages to a new microseconds format.

We currently have 40 data clusters with device messages data independent of each other (we call it shares). Each data cluster is serviced by 3 mirrors located on different servers.

Initially, we will add 4th mirror with a new data format (in microseconds). Once synchronized all read operations from the database will be primarily serviced by these new mirrors and thus return messages with microseconds granularity. Once we ensure that everything is OK, we will stabilize each data cluster (share) with 1 data mirror in a new format and 2 data mirrors will remain as is in the old data format (seconds granularity), mostly maintained for fail-over and backup purposes.

After a day or two in such mode, we will move further and replace one old mirror with a new one resulting in 2 mirrors in microseconds granularity and 1 fail-over mirror in seconds granularity. In this mode we will operate few more days, up to a week, ensuring that everything is fine and smooth.

Once we make sure that everything is operating completely as expected, we will make the final transition and replace the last 3rd mirror with a new data format.

Basically for all users, the database will act on a microseconds granularity immediately once we synchronize the 1st mirror to a new data format, as this mirror will always be chosen as preferable for all read operations. Please be prepared for this. For more information about the changes you should expect please take a look at the quoted post above (posted on July 20).

We will indicate here in the changelog once the whole transitioning process is completed.

kial

Today we'll install an update for the commands and setting engine, to provide new features:

device-level commands: now you can send a command to the device without direct access to the channel where the device is connected. Added a new REST API interface for the device-level commands queue management, similar to the channel-level commands: you can POST command into the queue, examine queue content and check command execution results. The set of the available commands is the same as for the channel.
instant device-level commands sending: if your device is connected right now, you can send device-level command by HTTP REST request, which will return its response only when the command was executed on the device. To do this you can specify the optional "timeout" parameter and your request will be suspended until command completion.

Please be noted that during the update installation process you may face an insignificant delay in commands execution.

kial

In addition to kial and shal updates, we've added a command field to the GET
/gw/channel-protocols/{protocol-selector}/device-types/{devtypes-selector} request.
That field contains an array of all commands available to execute directly through the device (real-time, sms commands, and device-level commands queue).
The array of available commands are combined from the channel-level commands and get- and/or set-command per each setting defined for the device.
Now it's possible to execute one device setting several times with different parameters using the device commands queue, and all of the PUT requests will be delivered to the connected tracker.

shal

We modified the update with device commands and split various device remote control patterns into multiple methods.
Full listing of remote control methods you may find in the device REST API documentation.

There are four types of remote control methods over device now:

device settings - that can be used mostly for persistent device state configuration
device commands queue - used for all types of remote commands, where the caller want to asynchronously deliver command to the device once it is connected, commands in the queue are executed in serial mode, one by one
real-time command over connected device - REST API method that will check if device is connected at the moment and return false if not. If the device is connected it will deliver the command to the device within specified timeout and wait for response. Once command will be executed on the device, REST API method will return to the caller its actual response. It is also to post multiple commands in one single method and they will be executed one by one.
SMS command - this method will just send SMS to the device via available modem and return text of SMS.

All these commands report their status into device logs and all remote operations over device can be retrieved with GET /gw/devices/{device-selector}/commands-result method.

adsa

The general purpose streaming protocols http, mqtt, aws_iot, azure_iot, and google_iot have been leveraged with the new convert_message configuration parameter. This parameter is intended for flexible configuration of outgoing messages. This currently makes it possible to change the parameter names generated by flespi, such as 'position.latitude', using one of the available schemes:

CamelCase - PositionLatitude
lowCamelCase - positionLatitude
snake_case - position_latitude
{own_pattern} - position{own_pattern}latitude

shal

We enhanced channels with the possibility to post-process the message. The list of post-processing channels that are operating on special protocol type is available in the channel configuration:

It works such a way that message from normal device first being parsed by the original protocol which output standard message (JSON) for the channel. After that message is used as a source for input for any post-processing channel that may add or remove any field there or perform some special parsing for fields.

This feature open for us a whole world of new possibilities:

add or remove any parameters from the message
perform special parsing of 3rd party devices attached to serial port or CAN bus including SigFox and LoRaWAN protocols
attach geocoded position, from Google Maps for example
attach position detected via LBS service
perform any other logic we may need

Right now we are developing few protocols to parse data from peripheral devices attached to the serial port of standard GPS device. These devices are using standard GPS device just as a transport layer to deliver its data.

As usual we are open for our users to request to implement such protocols by our team. These protocols with custom logic can be installed in private mode, available only for particular flespi user.

kial

shal We've added a new public post-processing protocol parameters-filter that can be used to remove any unneeded parameters (except system ones) from the messages of any channel. For example, it can be used to reduce messages storage amount or reduce traffic when receiving a lot of messages through MQTT or a stream.

Another example to remove position information (latitude and longitude) from all messages:

screenshot

adsa

We are reporting minor changes to the Azure IOT stream configuration.
The previously introduced parameter convert_message has its own rules for working with periods in the names of message parameters. Therefore the special parameter replace_dots is deprecated and should be replaced with the convert_message parameter. Please uncheck replace_dots option, enable convert_message option and select snake_case for replace_dots option.

On September 16, the deprecated stream configuration will be fixed automatically.

adsa

If you change a non-existent device setting, you will not receive any errors. This is a bug, not a feature. We're going to change this behavior. Thus, any PUT/DELETE request to gw/devices/{devices-selector}/settings/{setting-name} to change a non-existent device setting will return 400 Bad Request.

The update will be installed on October 8th.

adsa

adsa The fix was released today.

kial

We've installed an update extending digital inputs and outputs (din and dout) parameters handling for the protocols, which sends them as a separated bits, not as a combined bitmask.
In that protocols in addition to parameters din and dout you will get the parameters din.N/dout.N where N is a 1-based bit index, and the parameter value will be true/false according to that bit state.

baja

SMS gateway updated: + symbol excluded from destination phone number of SMPP client when sending a message

shal

shal We completed 1st and 2nd steps described above and now for all 40 data clusters(shares) there are two mirrors providing messages with microseconds granularity.

If everything will be OK next week we will replace last 3rd mirror and finish transitioning to microseconds data format, but already starting you can enjoy this feature in flespi database as read operations are automatically routed only to new mirrors.

shal

shal We have migrated all our storage systems from seconds to microseconds granularity according to initially defined schedule.

We advice all our users using REST API to fetch devices messages to check for correctness of to field in their REST API queries. For example your message is the following:

{
"timestamp": 1490347944.893743,
"din": 5,
"param1": 1,
...
},

Legacy implementation of GET /gw/devices/XXX/messages for the query:

{
"from": 1490347940,
"to": 1490347944,
}

will correctly return this message, as the storage granularity was one second.
New (current) implementation will not return this message, because it's timestamp (1490347944.893743) is larger then right selection border (1490347944).
To avoid any discrepancy please either request 1490347944.999999 as right border (in from parameter) or just select up to next second: 1490347945 and filter out unsuitable values from result.