We are announcing the future upgrade for filter expression format in various REST API calls. The most important are the GET /devices/XXX/messages and GET /calcs/XXX/devices/YYY/intervals API calls. So all REST API calls where you can specify filter expression will be affected.

Once the whole upgrade is finished it will be possible to use extended expressions in requests with math functions, arithmetic operations, brackets for operations priority and so on. But so far we made simple compatibility mode that you need to adjust your requests to.

List of changes to the filters format

• textual fields will require quotes. For example filter="email=123@gmail.com" should be upgraded to filter="email='123@gmail.com'".
• instead of | or & separators please use || or &&. For example filter="field1=1|field1=100" should be upgraded to filter="field1=1||field1=100".
• parameter existence check in form filter="param1" or filter="param1!=" will stop operating. Please use filter="param1!=''" instead (except the case when param1 may have empty string value).

If some parameter does not exists in the message the whole expression will be marked as invalid and thus message will be skipped. In that case you may already precede parameter name with dollar sign: filter="$param=1 || $param2 = 2"

There will be few stages for this update.

Stage 1: effective from April 26th

We will add warnings response to REST API requests that are still using old scheme. So the ordinary response from request will be:
{"result": [...], "warnings":[{"reason":"You are using REST API call format that will soon be deprecated"}]}
In most of the systems this will not change any kind of processing, because the whole result located under "result" nested object is unaffected.

Stage 2: effective from May 17th

We will switch filters processing in API calls to a new expressions implementation. In case you will be still using older format your requests will return error with corresponding information or even do not operate as expected in some cases. From that moment you will have the ability to use all the power of new expressions engine.

In between stages we are actively monitoring API usage and will personally inform users that will be affected by this change, especially if we already have a non-empty communication chat between flespi support and this user. But nevertheless please do the upgrade of your code on your own as soon as possible and be prepared for the possible changes in requests processing on May 17th.

Along with the plugin filter, we added the same filter to the stream configuration. If a filter ("validate_message" field) is set, the stream will check any message before sending. If the message does not match the expression, it will be skipped.

We are announcing the ongoing transition of device messages storage and related flespi subsystems granularity from seconds to microseconds.

The transition will take some time which we are unable to precisely calculate at the moment. It can take a few weeks or even a few months, depending on numerous factors. We will notify you once the transition is 100% finished.

Details about the upcoming change

The typical message that the flespi device generates consists of a timestamp and some additional parameters:
{
"timestamp": 1490347944.893743,
"din": 5,
"param1": 1,
...
}
If you look at the timestamp value you will notice that it is already specified in microseconds granularity. For the sample above 1490347944 is the integer part which can be parsed as Friday, 24 March 2017, 9:32:24. And .893743 is the fractional part of that second - microseconds.
Right now this message will be stored under its integer part. And if for some reason another device message in the same second will arrive, for example:
{
"timestamp": 1490347944.900001,
"din": 10,
"param2": 2,
...
}
Right now it will also be stored now under the same integer key - 1490347944.

Current behavior

In that case, once you will have these two messages registered for the device, if you will query its messages with REST API call right now you will end up with just a single message with combined parameters from both initial messages:
{
"timestamp": 1490347944.900001,
"din": 10,
"param1": 1,
"param2": 2,
...
}

New behavior

After we will make the transition of device messages storage from seconds to microseconds with the same REST API call you will get two messages because their fractional part (the storage key) is different and they will be stored separately:
{
"timestamp": 1490347944.893743,
"din": 5,
"param1": 1,
...
},
{
"timestamp": 1490347944.900001,
"din": 10,
"param2": 2,
...
}

What should be adopted on the client side?

You should be aware that REST API call GET /gw/devices/XXX/messages now will return you all messages based on microseconds granularity.

This will also affect all other flespi systems - analytics will be able to generate multiple intervals per second and you should just be prepared for this. If possible - replace you your database keys from uint32_t to the double type and any change will be accepted smoothly.

According to our experience, most devices generate only a single message per second so in most cases you will not notice any difference. But for those who rely on multiple messages per one-second possibility - and there are some cases related to iButton on/off, digital/analog input change, and accelerometer data array on impact - just be prepared that you will be able to access all of them from device storage after the transition.

We modified the update with device commands and split various device remote control patterns into multiple methods.
Full listing of remote control methods you may find in the device REST API documentation.

There are four types of remote control methods over device now:

• device settings - that can be used mostly for persistent device state configuration
• device commands queue - used for all types of remote commands, where the caller want to asynchronously deliver command to the device once it is connected, commands in the queue are executed in serial mode, one by one
• real-time command over connected device - REST API method that will check if device is connected at the moment and return false if not. If the device is connected it will deliver the command to the device within specified timeout and wait for response. Once command will be executed on the device, REST API method will return to the caller its actual response. It is also to post multiple commands in one single method and they will be executed one by one.
• SMS command - this method will just send SMS to the device via available modem and return text of SMS.

All these commands report their status into device logs and all remote operations over device can be retrieved with GET /gw/devices/{device-selector}/commands-result method.

The general purpose streaming protocols http, mqtt, aws_iot, azure_iot, and google_iot have been leveraged with the new convert_message configuration parameter. This parameter is intended for flexible configuration of outgoing messages. This currently makes it possible to change the parameter names generated by flespi, such as 'position.latitude', using one of the available schemes:

• CamelCase - PositionLatitude
• lowCamelCase - positionLatitude
• snake_case - position_latitude
• {own_pattern} - position{own_pattern}latitude

shal Tomorrow on September 14th we will start the migration of devices messages to a new microseconds format.

We currently have 40 data clusters with device messages data independent of each other (we call it shares). Each data cluster is serviced by 3 mirrors located on different servers.

Initially, we will add 4th mirror with a new data format (in microseconds). Once synchronized all read operations from the database will be primarily serviced by these new mirrors and thus return messages with microseconds granularity. Once we ensure that everything is OK, we will stabilize each data cluster (share) with 1 data mirror in a new format and 2 data mirrors will remain as is in the old data format (seconds granularity), mostly maintained for fail-over and backup purposes.

After a day or two in such mode, we will move further and replace one old mirror with a new one resulting in 2 mirrors in microseconds granularity and 1 fail-over mirror in seconds granularity. In this mode we will operate few more days, up to a week, ensuring that everything is fine and smooth.

Once we make sure that everything is operating completely as expected, we will make the final transition and replace the last 3rd mirror with a new data format.

Basically for all users, the database will act on a microseconds granularity immediately once we synchronize the 1st mirror to a new data format, as this mirror will always be chosen as preferable for all read operations. Please be prepared for this. For more information about the changes you should expect please take a look at the quoted post above (posted on July 20).

We will indicate here in the changelog once the whole transitioning process is completed.