[Changelog] flespi database

shal

flespi telematics storage system changelog.
Take a look also at blog articles to understand more about flespi storage system and the role it plays in our architecture.

shal

As you know if you want to add something you first need to think about what to remove. In order to simplify further development of flespi storage system we decided to remove its abque storage element. This abque is something we developed as asynchronous batch storage system with multiple writers and single reader and as our statistics says almost nobody except us using it in normal "production".

So, now we announce that in the nearest future abque will be removed from storage REST API. The dates are following:

September 4th: all abque REST API methods will be marked as deprecated and all operations with abque will normally operate, but also append error: method deprecated in result.
September 11th: we will cleanup and remove all existent abques and their REST API operations

Just in case you need to make longer terms for abque deprecation please contact us asap.

shal

We have marked all abque REST API methods as deprecated - they continue to operate, but are appending to returned result deprecated error with corresponding information.

shal

Finally today abque was completely removed from flespi storage system.

shal

Today flespi containers as a fundamental part of flespi storage system received few new features:

Now it is possible to change container flags with PUT request. Flags are used to specify the strategy for storing duplicate data under same key. New flag - 0x10 can be specified whenever container is expected to contain timeseries data and thus flespi will perform its automatic data generalization.

The rules for data generalization are predefined and not flexible at the moment:

messages up to 3 days are stored as is
messages from 3 to 10 days are generalized with 10 minutes interval
messages from 10 days up to 60 days are generalized with 1 hour period
messages later than 60 days are generalized with 1 day period

Generalization is done using average method. To experiment with the function you may request for data generalization using built-in GET /storage/containers/XXX/message API call.

At the same moment we have changed generalization call result slightly. Previously it always returned resulting average values as double, e.g. (1 + 2 + 2)/3 == 1.6666. Now it will use same type of number as original value, e.g. (1 + 2 + 2)/3 = 2.

We activated generalization in user statistics and enlarged flespi users statistics storage from 1 to 10 years.

shal

We have modified the behavior of the storage engine for containers and also for some additional system elements based on containers with multiple elements per key storage strategy: for REST API methods with any kind of logs.

The changed behavior is related to max_count parameter. Previously if you specify max_count it will return up to specified amount of messages in request. Now it can return slightly more elements by returning all messages with affected keys, e.g. if some key has 10 messages and we hit max_count limit at 2nd message all 8 messages for this will be returned. This is to provide the consistency working in historical mode and per accessing messages per page.

shal

This week we are installing new version of communication protocol between our storage nodes. Although we do not expect any impact on the storage system in general and ready for any kind of troubles including the ability to perform hot restore of telematics data there are some updates in storage system behavior for flespi users. This is related to storage system replies performance and is interesting only for those who are using containers directly.

Currently up to a certain level our storage system provides asynchronous buffering of the uploaded data and thus responses for REST API calls for data upload are very fast as the receiver just put them into internal buffer.

Once we will finish actual update the database engine won't buffer uploaded data and will wait for uploaded data acknowledgment on at least 2 of 3 database mirrors. This is to ensure data durability, but at the same time this may delay storage containers REST API calls up to a few milliseconds, please be aware.

Once protocol is upgraded we will move to updated storage format for the mirrors and re-synchronize all the data we have. But this should not be visible to the external world at all (looking from flespi side) - we do it quite often with zero impact on our users.

Once we will move to the new data format we are one step closer to installation of flespi database system in second datacenter.

shal

flespi containers now have new rotate property. If it is non-zero, then it specifies rotation size for container messages. Once messages consumes more storage then specified in rotate property, messages with lower keys will be automatically deleted until messages size will be equal or lower then value in rotate.

shal

Today we provisioned the next generation of REST API processing unit for flespi database service (replacement for mdbctl unit). It works in parallel with the older one. We will maintain this unit in such parallel mode for a certain period of time and later will completely shut the older service down.

This is just an information message. We would like our users to be aware of these changes and request you to test all your services that use flespi REST API which may be affected:

all storage API methods;
device messages access;
access and update to intervals generated by calculators;
access to logs and some other minor systems like channels traffic, account statistics, etc
MQTT Broker retained messages operations;

All our internal and external tests show that the system continues to operate seamlessly with both old and new units providing it in parallel mode.

If you will notice any inconsistencies from expected behavior please let us know immediately in flespi chat.

shal

shal Today we stopped the previous version of the REST API gateway for flespi database service and now all 100% of requests are being processed by the new version. Besides all the new possibilities, modern codebase, and so on, it uses 3 times less CPU and 4 times less RAM than the previous version, which gives us even more opportunities to handle millions of telematics devices with legacy server hardware.