We are announcing the ongoing transition of device messages storage and related flespi subsystems granularity from seconds to microseconds.

The transition will take some time which we are unable to precisely calculate at the moment. It can take a few weeks or even a few months, depending on numerous factors. We will notify you once the transition is 100% finished.

Details about the upcoming change

The typical message that the flespi device generates consists of a timestamp and some additional parameters:
{
"timestamp": 1490347944.893743,
"din": 5,
"param1": 1,
...
}
If you look at the timestamp value you will notice that it is already specified in microseconds granularity. For the sample above 1490347944 is the integer part which can be parsed as Friday, 24 March 2017, 9:32:24. And .893743 is the fractional part of that second - microseconds.
Right now this message will be stored under its integer part. And if for some reason another device message in the same second will arrive, for example:
{
"timestamp": 1490347944.900001,
"din": 10,
"param2": 2,
...
}
Right now it will also be stored now under the same integer key - 1490347944.

Current behavior

In that case, once you will have these two messages registered for the device, if you will query its messages with REST API call right now you will end up with just a single message with combined parameters from both initial messages:
{
"timestamp": 1490347944.900001,
"din": 10,
"param1": 1,
"param2": 2,
...
}

New behavior

After we will make the transition of device messages storage from seconds to microseconds with the same REST API call you will get two messages because their fractional part (the storage key) is different and they will be stored separately:
{
"timestamp": 1490347944.893743,
"din": 5,
"param1": 1,
...
},
{
"timestamp": 1490347944.900001,
"din": 10,
"param2": 2,
...
}

What should be adopted on the client side?

You should be aware that REST API call GET /gw/devices/XXX/messages now will return you all messages based on microseconds granularity.

This will also affect all other flespi systems - analytics will be able to generate multiple intervals per second and you should just be prepared for this. If possible - replace you your database keys from uint32_t to the double type and any change will be accepted smoothly.

According to our experience, most devices generate only a single message per second so in most cases you will not notice any difference. But for those who rely on multiple messages per one-second possibility - and there are some cases related to iButton on/off, digital/analog input change, and accelerometer data array on impact - just be prepared that you will be able to access all of them from device storage after the transition.

We modified the update with device commands and split various device remote control patterns into multiple methods.
Full listing of remote control methods you may find in the device REST API documentation.

There are four types of remote control methods over device now:

• device settings - that can be used mostly for persistent device state configuration
• device commands queue - used for all types of remote commands, where the caller want to asynchronously deliver command to the device once it is connected, commands in the queue are executed in serial mode, one by one
• real-time command over connected device - REST API method that will check if device is connected at the moment and return false if not. If the device is connected it will deliver the command to the device within specified timeout and wait for response. Once command will be executed on the device, REST API method will return to the caller its actual response. It is also to post multiple commands in one single method and they will be executed one by one.
• SMS command - this method will just send SMS to the device via available modem and return text of SMS.

All these commands report their status into device logs and all remote operations over device can be retrieved with GET /gw/devices/{device-selector}/commands-result method.

The general purpose streaming protocols http, mqtt, aws_iot, azure_iot, and google_iot have been leveraged with the new convert_message configuration parameter. This parameter is intended for flexible configuration of outgoing messages. This currently makes it possible to change the parameter names generated by flespi, such as 'position.latitude', using one of the available schemes:

• CamelCase - PositionLatitude
• lowCamelCase - positionLatitude
• snake_case - position_latitude
• {own_pattern} - position{own_pattern}latitude

shal Tomorrow on September 14th we will start the migration of devices messages to a new microseconds format.

We currently have 40 data clusters with device messages data independent of each other (we call it shares). Each data cluster is serviced by 3 mirrors located on different servers.

Initially, we will add 4th mirror with a new data format (in microseconds). Once synchronized all read operations from the database will be primarily serviced by these new mirrors and thus return messages with microseconds granularity. Once we ensure that everything is OK, we will stabilize each data cluster (share) with 1 data mirror in a new format and 2 data mirrors will remain as is in the old data format (seconds granularity), mostly maintained for fail-over and backup purposes.

After a day or two in such mode, we will move further and replace one old mirror with a new one resulting in 2 mirrors in microseconds granularity and 1 fail-over mirror in seconds granularity. In this mode we will operate few more days, up to a week, ensuring that everything is fine and smooth.

Once we make sure that everything is operating completely as expected, we will make the final transition and replace the last 3rd mirror with a new data format.

Basically for all users, the database will act on a microseconds granularity immediately once we synchronize the 1st mirror to a new data format, as this mirror will always be chosen as preferable for all read operations. Please be prepared for this. For more information about the changes you should expect please take a look at the quoted post above (posted on July 20).

We will indicate here in the changelog once the whole transitioning process is completed.

We've added two useful methods to the flespi REST API.

The first method is POST gw/devices/{selector}/messages. It is designed to inject your own messages into the device storage. This method is useful for marking up device messages. For example, to mark up driver assignments, you need to register a message with a driver code and create a calculator based on this parameter. It is also suitable for simulating the message flow of a device.

The registered message is different from the message sent by the device in the following:

• has a rest.timestamp parameter
• has no server.timestamp and channel.id parameters.

The second method is GET gw/devices/{selector}/packets. Along with GET gw/channels/{selector}/packets, this method is useful for viewing the bi-directional traffic flow between a device and a gateway while debugging unexpected device behavior.

Two more useful features have been added:

1. Now you can configure the interval for polling the device settings using the options:

   • once, by default, query device setting values only once
   • never, never request settings from the device. This is useful for reducing device traffic.
   • daily, weekly, monthly - periodically update the settings
     

2. For the convenience of accessing available device commands, we have added a new device field - commands. This field contains an array with the definitions of all available commands.