flespi telematics storage system changelog.
Take a look also at blog articles to understand more about flespi storage system and the role it plays in our architecture.
[Changelog] flespi database
As you know if you want to add something you first need to think about what to remove. In order to simplify further development of flespi storage system we decided to remove its abque storage element. This abque is something we developed as asynchronous batch storage system with multiple writers and single reader and as our statistics says almost nobody except us using it in normal "production".
So, now we announce that in the nearest future abque will be removed from storage REST API. The dates are following:
- September 4th: all abque REST API methods will be marked as deprecated and all operations with abque will normally operate, but also append error: method deprecated in result.
- September 11th: we will cleanup and remove all existent abques and their REST API operations
Just in case you need to make longer terms for abque deprecation please contact us asap.
7 days later
We have marked all abque REST API methods as deprecated - they continue to operate, but are appending to returned result deprecated error with corresponding information.
7 days later
Finally today abque was completely removed from flespi storage system.
14 days later
Today flespi containers as a fundamental part of flespi storage system received few new features:
Now it is possible to change container flags with PUT request. Flags are used to specify the strategy for storing duplicate data under same key. New flag - 0x10 can be specified whenever container is expected to contain timeseries data and thus flespi will perform its automatic data generalization.
The rules for data generalization are predefined and not flexible at the moment:
- messages up to 3 days are stored as is
- messages from 3 to 10 days are generalized with 10 minutes interval
- messages from 10 days up to 60 days are generalized with 1 hour period
- messages later than 60 days are generalized with 1 day period
Generalization is done using average method. To experiment with the function you may request for data generalization using built-in GET /storage/containers/XXX/message API call.
At the same moment we have changed generalization call result slightly. Previously it always returned resulting average values as double, e.g. (1 + 2 + 2)/3 == 1.6666. Now it will use same type of number as original value, e.g. (1 + 2 + 2)/3 = 2.
We activated generalization in user statistics and enlarged flespi users statistics storage from 1 to 10 years.
23 days later
We have modified the behavior of the storage engine for containers and also for some additional system elements based on containers with multiple elements per key storage strategy: for REST API methods with any kind of logs.
The changed behavior is related to max_count parameter. Previously if you specify max_count it will return up to specified amount of messages in request. Now it can return slightly more elements by returning all messages with affected keys, e.g. if some key has 10 messages and we hit max_count limit at 2nd message all 8 messages for this will be returned. This is to provide the consistency working in historical mode and per accessing messages per page.
a month later
- Edited
This week we are installing new version of communication protocol between our storage nodes. Although we do not expect any impact on the storage system in general and ready for any kind of troubles including the ability to perform hot restore of telematics data there are some updates in storage system behavior for flespi users. This is related to storage system replies performance and is interesting only for those who are using containers directly.
Currently up to a certain level our storage system provides asynchronous buffering of the uploaded data and thus responses for REST API calls for data upload are very fast as the receiver just put them into internal buffer.
Once we will finish actual update the database engine won't buffer uploaded data and will wait for uploaded data acknowledgment on at least 2 of 3 database mirrors. This is to ensure data durability, but at the same time this may delay storage containers REST API calls up to a few milliseconds, please be aware.
Once protocol is upgraded we will move to updated storage format for the mirrors and re-synchronize all the data we have. But this should not be visible to the external world at all (looking from flespi side) - we do it quite often with zero impact on our users.
Once we will move to the new data format we are one step closer to installation of flespi database system in second datacenter.
a month later
flespi containers now have new rotate property. If it is non-zero, then it specifies rotation size for container messages. Once messages consumes more storage then specified in rotate property, messages with lower keys will be automatically deleted until messages size will be equal or lower then value in rotate.
8 months later
- Edited
Today we provisioned the next generation of REST API processing unit for flespi database service (replacement for mdbctl unit). It works in parallel with the older one. We will maintain this unit in such parallel mode for a certain period of time and later will completely shut the older service down.
This is just an information message. We would like our users to be aware of these changes and request you to test all your services that use flespi REST API which may be affected:
- all storage API methods;
- device messages access;
- access and update to intervals generated by calculators;
- access to logs and some other minor systems like channels traffic, account statistics, etc
- MQTT Broker retained messages operations;
All our internal and external tests show that the system continues to operate seamlessly with both old and new units providing it in parallel mode.
If you will notice any inconsistencies from expected behavior please let us know immediately in flespi chat.
8 days later
- Edited
shal Today we stopped the previous version of the REST API gateway for flespi database service and now all 100% of requests are being processed by the new version. Besides all the new possibilities, modern codebase, and so on, it uses 3 times less CPU and 4 times less RAM than the previous version, which gives us even more opportunities to handle millions of telematics devices with legacy server hardware.
8 months later
- Edited
We are announcing the future upgrade for filter expression format in various REST API calls. The most important are the GET /containers/XXX/messages and GET /cdns/XXX/files API calls. So all REST API calls where you can specify filter expression will be affected.
Once the whole upgrade is finished it will be possible to use extended expressions in requests with math functions, arithmetic operations, brackets for operations priority and so on. But so far we made simple compatibility mode that you need to adjust your requests to.
List of changes to the filters format
- textual fields will require quotes. For example filter="email=123@gmail.com" should be upgraded to filter="email='123@gmail.com'".
- instead of | or & separators please use || or &&. For example filter="field1=1|field1=100" should be upgraded to filter="field1=1||field1=100".
- parameter existence check in form filter="param1" or filter="param1!=" will stop operating. Please use filter="param1!=''" instead (except the case when param1 may have empty string value).
There will be few stages for this update.
Stage 1: effective from April 26th
We will add warnings response to REST API requests that are still using old scheme. So the ordinary response from request will be:
{"result": [...], "warnings":[{"reason":"You are using REST API call format that will soon be deprecated"}]}
In most of the systems this will not change any kind of processing, because the whole result located under "result" nested object is unaffected.
Stage 2: effective from May 17th
We will switch filters processing in API calls to a new expressions implementation. In case you will be still using older format your requests will return error with corresponding information or even do not operate as expected in some cases. From that moment you will have the ability to use all the power of new expressions engine.
In between stages we are actively monitoring API usage and will personally inform users that will be affected by this change, especially if we already have a non-empty communication chat between flespi support and this user. But nevertheless please do the upgrade of your code on your own and be prepared for the changes.
a month later
6 days later
4 months later
Tomorrow on the September 14th we will start migration of container messages to a new microseconds format.
This update mostly affects messages storage for telematics devices and is highlighted in flespi telematics hub changelog, please read more information about upcoming changes there.
3 months later
Small minor change in flespi storage REST API behavior.
We enhanced its subaccounts support up to a level of all other REST API systems. Now REST API calls for storage system from parent account will be able to operate with entities that belong to subaccount.
The behavior is documented here.
3 years later
