Use cases, implementation features, changelog for flespi MQTT broker .

We are migrating from 32-bit to 64-bit IDs of various broker elements. Broker was just upgraded with new binaries capable or working with both types of entities and in a hour we will upgrade the database schema. We do not expect any problems to the broker, may be short service interruptions in a second or two.

Latest broker API changes: 'recursive' parameter added to GET mqtt/messages method . Parameter is optional and has value 'true' by default. If you want to fetch your own messages, i.e. skip subaccount's messages, just set parameter value to 'false'. Method parameters, behavior and returning result are compatible with previous implementation.

We introduced a new rule for processing overflowed sessions. Broker keeps unacknowledged messages in the internal buffer wich has limit up to 1Gb per each MQTT session. If session buffer is overflowed, the connection will be disconnected and undelivered messages will be deleted.

In flespi we are using timestamp user property automatically set by the broker as the time of message appearance. We had some messages, like telematics messages telemetry , with expiry interval set that were outdated twice earlier then expected (twice shorter messages life). This was because broker, before delivering retained message marked expiry interval in seconds from current time, but at the same time it checked for message actuality starting from this timestamp time. Now we fixed this behavior - broker will not decrement expiry interval based on current time, but will always count it from message timestamp. This not conforms to standard MQTT protocol and can be counted as flespi broker specifics .

Rejected messages counter was added to the session log record upon disconnection. Message can be rejected by one of these reasons: Permissions denied for publisher token. Message has reserved prefix flespi/ Access denied to specified subaccount passed through cid property.

One of our users has reported a security issue in flespi broker that allowed to use will message for publishing into topics protected by ACL (all topics except flespi/ were affected). We have installed a fix that prohibits session connections with invalid for given token at this moment of time will message topic.

A bunch of internal updates have been deployed. Please contact us if you notice an incorrect broker behavior.

Update: please refer the post below Some hardware MQTT implementations have limitations for username/password field length of authorization packet. The typical limitation is 32 bytes per field. Thus regular authorization via 64-byte token is not possible. So we have implemented an alternative way to authorize such MQTT connections. To do it, use token id as username and at least 32 first bytes of the token key as a password. The token MUST be of ACL type. Example: username=123 password=at-least-32-first-bytes-of-token-key

It is now possible to specify for ACL tokens MQTT topics using flespi comma-separated topic selectors . Imagine, you have this topics: path/1/field1 path/1/field2 path/1/field3 If you want to limit publish/subscribe access to to field1 and field2 only, leaving field3 protected you need to specify this in ACL section of ACL token . Before today update you need to specify each topic separately, e.g. define two mqtt sections in the token: path/1/field1 and path/1/field2 . After today update you may use flespi special topic selector for providing correct topics and have everything in just one rule: path/1/field1,field2 Same applies if you have multiple identifiers in path, e.g. 1,2,3,4,5,6. You may describe full ACL with just one rule: path/1,2,3,4,5,6/field1,field2 Previously you need to define 12 MQTT ACL rules for this construction or use topic wildcards. This is applicable for both publish and subscribe sections of mqtt ACL rule.

Dear flespi users! The latest broker update, which proposed an alternative way to authorize the MQTT connection, also closed the security vulnerability. It was possible to log in with any string as a username and a valid token as a password. Please refer Authorization section on the broker description page: https://flespi.com/mqtt-broker and review your connection settings and correct it if necessary.

[Changelog] flespi MQTT broker

shal

We removed special processing of user property with name seqno. Previously this user property was used by the MQTT Broker and its retained storage in order to select latest message if timestamps are equal. We were able to get rid of it after modifying the way we are generate message timestamp. Now upon processing and caching messages on client side it is possible to compare timestamp property of message and ignore message with earlier timestamp.

adsa

We are going to limit the maximum supported QoS for Flespi MQTT Broker subscriptions to QoS 1. This will increase the message throughput for both the clients and the broker. All existing subscriptions with QoS=2 will continue to operate as is, but new subscriptions will be downgraded to 1 upon subscription. We will set the Maximum QoS property in the CONNACK response. Publishing topics with QoS=2 is allowed. All of these steps should ensure that existing customers are not affected.
The update is scheduled for June 16th.

shal

For REST API calls over MQTT (very cool feature when using microservices architecture) additional user properties available in the response message:

client_host - contains IP address of MQTT connection that initiated request
client_id - contains client-id of MQTT connection that initiated request

Additionally each such REST API call towards flespi is enhanced with "x-flespi-app" header with urlencoded value of MQTT client-id which allow you to identify the source of such calls.

adsa

We have added a new feature unique to the flespi broker. You can get the number of subscribers for a topic you have published. Just add a count_subscribers user property with 1 payload and publish a QoS(1) message. You can find the subscribers_count user property in the received PUBACK.

adsa

We have increased the maximum number of allowed topics per SUBSCRIBE/UNSUBSCRIBE packet to 64.

adsa

We have implemented a new approach to delivering messages to shared subscribers who are all offline. Instead of delivering messages to the very first subscriber from the list, the broker now distributes messages among all subscribers.

adsa

We have added a new user property published by the broker in the CONNACK packet. The name of the property is session_id and it contains the ID assigned to the connection by the broker.

adsa

We have extended broker-side messages filtration with message field. Message is a JSON object with following keys:

topic - a topic string
topics - an array with topic words splited by '/'
cid - publisher id
timestamp - message timestamp
user_properties - an object with list of user properties
payload - a string with message payload

You can combine message filter with other filters like payload etc. Multiple filters are applied with AND logic.

Example:
We have following topics: temperature/kitchen, temperature/bedroom, temperature/hall. Message payload is a temperature. So we want to receive temperature from kitchen and hall if it is below 20 degrees.

Message filter is: (topics[1]=="kitchen"||topics[1]=="hall")&&tonumber(payload)<20
Subscription with encoded filter is: $filter/message=(topics%5B1%5D%3D%3D%22kitchen%22%7C%7Ctopics%5B1%5D%3D%3D%22hall%22)%26%26tonumber(%22payload%22)%3C20/temperature/+

adsa

We have included a meaningful description of the reasons for disconnecting MQTT sessions.

When the retained storage overflows, the PUBACK code 0x97 (Quota exceeded) will be sent for any PUBLISH packets with the retain flag. However, the broker will still deliver the message to the subscribers.

adsa

We've added two new event types to the MQTT broker logs. The code 530 (MQTT packet sent) represents CONNACK, SUBACK, and UNSUBACK packets, while the code 531 (MQTT packet received) represents CONNECT, SUBSCRIBE, and UNSUBSCRIBE packets.

adsa

Extended the MQTT broker REST API with additional fields for sessions and subscriptions.

Session fields added:

version — MQTT protocol version
expiry_interval — session expiry interval (seconds)
last_active — last activity timestamp
bytes_in — total bytes received
bytes_out — total bytes sent
pub_count — number of PUBLISH messages processed
sub_count — number of active subscriptions
bytes_out_queue — outgoing queue size (bytes)
puback_queue — number of messages awaiting PUBACK
inflight_msgs_size — in-flight messages size (bytes)

Subscription fields added:

cid — account ID of the subscription owner
recursive — include subaccount topics