- Edited
Use cases, implementation features, changelog for flespi MQTT broker.
[Changelog] flespi MQTT broker
8 days later
We are migrating from 32-bit to 64-bit IDs of various broker elements. Broker was just upgraded with new binaries capable or working with both types of entities and in a hour we will upgrade the database schema. We do not expect any problems to the broker, may be short service interruptions in a second or two.
- Edited
Latest broker API changes:
'recursive' parameter added to GET mqtt/messages method. Parameter is optional and has value 'true' by default. If you want to fetch your own messages, i.e. skip subaccount's messages, just set parameter value to 'false'.
Method parameters, behavior and returning result are compatible with previous implementation.
19 days later
We introduced a new rule for processing overflowed sessions.
Broker keeps unacknowledged messages in the internal buffer wich has limit up to 1Gb per each MQTT session.
If session buffer is overflowed, the connection will be disconnected and undelivered messages will be deleted.
In flespi we are using timestamp user property automatically set by the broker as the time of message appearance.
We had some messages, like telematics messages telemetry, with expiry interval set that were outdated twice earlier then expected (twice shorter messages life). This was because broker, before delivering retained message marked expiry interval in seconds from current time, but at the same time it checked for message actuality starting from this timestamp time.
Now we fixed this behavior - broker will not decrement expiry interval based on current time, but will always count it from message timestamp. This not conforms to standard MQTT protocol and can be counted as flespi broker specifics.
2 months later
Rejected messages counter was added to the session log record upon disconnection.
Message can be rejected by one of these reasons:
- Permissions denied for publisher token.
- Message has reserved prefix flespi/
- Access denied to specified subaccount passed through cid property.
21 days later
One of our users has reported a security issue in flespi broker that allowed to use will message for publishing into topics protected by ACL (all topics except flespi/ were affected). We have installed a fix that prohibits session connections with invalid for given token at this moment of time will message topic.
14 days later
A bunch of internal updates have been deployed. Please contact us if you notice an incorrect broker behavior.
2 months later
- Edited
Update: please refer the post below
Some hardware MQTT implementations have limitations for username/password field length of authorization packet. The typical limitation is 32 bytes per field.
Thus regular authorization via 64-byte token is not possible.
So we have implemented an alternative way to authorize such MQTT connections.
To do it, use token id as username and at least 32 first bytes of the token key as a password. The token MUST be of ACL type.
Example:
username=123
password=at-least-32-first-bytes-of-token-key
It is now possible to specify for ACL tokens MQTT topics using flespi comma-separated topic selectors.
Imagine, you have this topics:
- path/1/field1
- path/1/field2
- path/1/field3
If you want to limit publish/subscribe access to to field1 and field2 only, leaving field3 protected you need to specify this in ACL section of ACL token.
Before today update you need to specify each topic separately, e.g. define two mqtt sections in the token:
path/1/field1 and path/1/field2.
After today update you may use flespi special topic selector for providing correct topics and have everything in just one rule:
path/1/field1,field2
Same applies if you have multiple identifiers in path, e.g. 1,2,3,4,5,6. You may describe full ACL with just one rule: path/1,2,3,4,5,6/field1,field2 Previously you need to define 12 MQTT ACL rules for this construction or use topic wildcards.
This is applicable for both publish and subscribe sections of mqtt ACL rule.
Dear flespi users!
The latest broker update, which proposed an alternative way to authorize the MQTT connection, also closed the security vulnerability. It was possible to log in with any string as a username and a valid token as a password.
Please refer Authorization section on the broker description page: https://flespi.com/mqtt-broker and review your connection settings and correct it if necessary.
6 days later
Two more properties were added to the MQTT session state.
Pending messages queue size in bytes:
flespi/state/mqtt/sessions/+/bytes_out_queue
In-flight, i.e. not ACK'ed QoS(1,2) messages count:
flespi/state/mqtt/sessions/+/puback_queue
The growth of these properties usually indicates insufficient bandwidth of the client network or the bottleneck of ACK processing.
5 months later
- Edited
Some hardware MQTT implementations have limitations for username/password field length of authorization packet (CONNECT). Sometimes limitation is set to 16 bytes.
Thus regular authorization via 64-byte token is not possible.
So we have changed previously implemented way to authorize such MQTT connections.
To do it, use token id as username and at least 16 first bytes of the token key as a password. The token MUST be of ACL type.
Example:
username=123
password=at-least-16-first-bytes-of-token-key
2 months later
- Edited
The parameter set for the POST https://flespi.io/mqtt/messages method has been extended with a full set of MQTT v5.0 message parameters such as user properties, correlation data, etc.
13 days later
- Edited
Broker REST API has moved to a new engine. This should not affect the functionality of the external API.
The sticky shared subscriptions have also been changed to make the messaging more balanced.
If you have experienced any issues, please contact us and we will fix them immediately.
5 months later
The REST API GET mqtt/messages method has been extended with the optional "fields" parameter, which allows you to customize the format of response messages.
19 days later
We are constantly trying to provide the flespi broker with new, useful features. Without breaking the standard, we have implemented additional methods for filtering messages.
Subscription filter format:
$filter/cid={cid filter}&modified_since={timestamp}&payload={expression}/{filter topic}
cid - message filter from the specified subaccount
modified_since - message filter by publication time
payload - message filter by content. The content of the message must comply with the JSON specification - be a number, a double-quoted string, null, true/false, JSON object, JSON array. Expression format: https://flespi.com/kb/expressions-the-way-to-analyze-message
Filters are listed with '&', the filter value must be urlencoded.
Payload filter added as an optional parameter to the REST API method GET /mqtt/messages.
Example 1, subscription to temperature sensors:
topic: sensor/1/temperature, payload: 21, filter: $>20
Subscription filter:
$filter/payload=%24%3E20/sensor/+/temperature
Example 2, subscription to active alarm sensors
topic: alarm/2/status, payload: true, fitler: $=true
Subscription filter:
$filter/payload=%24%3Dtrue/alarm/+/status
Example 3, telemetry subscription for generators with voltages outside the specified range
topic: generator/3, payload: {"name": "test", "voltage": 12}, filter: voltage<10||voltage>=12
Subscription filter:
$filter/payload=voltage%3C10%7C%7Cvoltage%3E12/generator/+
9 months later
We removed special processing of user property with name seqno. Previously this user property was used by the MQTT Broker and its retained storage in order to select latest message if timestamps are equal. We were able to get rid of it after modifying the way we are generate message timestamp. Now upon processing and caching messages on client side it is possible to compare timestamp property of message and ignore message with earlier timestamp.
6 months later
We are going to limit the maximum supported QoS for Flespi MQTT Broker subscriptions to QoS 1. This will increase the message throughput for both the clients and the broker. All existing subscriptions with QoS=2 will continue to operate as is, but new subscriptions will be downgraded to 1 upon subscription. We will set the Maximum QoS property in the CONNACK response. Publishing topics with QoS=2 is allowed. All of these steps should ensure that existing customers are not affected.
The update is scheduled for June 16th.
2 months later
For REST API calls over MQTT (very cool feature when using microservices architecture) additional user properties available in the response message:
- client_host - contains IP address of MQTT connection that initiated request
- client_id - contains client-id of MQTT connection that initiated request
Additionally each such REST API call towards flespi is enhanced with "x-flespi-app" header with urlencoded value of MQTT client-id which allow you to identify the source of such calls.
