Problem streaming json data to our server via http protocol

WilliamFeraux

Hello,

We are an electric bikes renting company, and we are currently working with teltonika to add their tracker TFT100 to some of our devices.

In order to do that, we use flespi to receive the data from the trackers and forward them to our server, in order to be able, for now, to display the positions of the bikes on our map on our website.

So for now, the data is well-received by flespi, and we created 3 devices, that are working well.

The problem is that when we try to forward the data to our server using http protocol and via a stream, the connection is made, the data is sent, but our server only receive a part of the data (the 10 first keys, sometimes duplicated), and sometimes no data either.

I tried to make a stream connection to a flespi channel to see if there was any problem, and there wasn't the flespi channel was receiving the data as it should.

So the fact is that I tried to ask help to the flespi support, we even upgraded our account to try to have access to better support, but it ended up that they just told me they had never encountered that issue with an other client, I also tried to talk with teltonika, but they weren't able to help me either.

So I'm a little bit worried now, cause I truly have no idea about where the problem is coming from, and I also want to say that our bikes possess other trackers made by other companies, and they are working well, their data are arriving on the same way it should be with flespi, in almost the same json format...

I will put the data we're receiving in the linked files

I hope someone will be able to help me.

Have a good day.

William Feraux

kial

So you have to identify where is the cause of your issue, right?
There are two options here:

flespi somehow sends messages with only first 10 keys
your server software somehow reduces each received message up to 10 keys

But you can easily test option 2 by sending a large message with 15 keys (for example) using cURL tool just like this:

curl -v -X POST --header 'x-api-key: X' -d '[{"k1":1,"k2":2,"k3":3,"k4":4,"k5":5,"k6":6,"k7":7,"k8":8,"k9":9,"k10":10,"k11":11,"k12":12,"k13":13,"k14":14,"k15":15}]' 'https://your.server.url.like.in.stream'

If you will debug print received messages from that cURL request and still see only 10 keys - so issue is caused by your software.
This is very simple but important test in your case.

WilliamFeraux

Sorry I forgot to specify that we used Postman to try to push the same data that flespi is supposed to push, and it was working well, our server received it well.
In my opinion, there is no reason for our server to reduce the messages up to 10 keys.

shal

WilliamFeraux can you show exactly how you tested it with Postman?
Also can you test it with curl like @kial posted above and let us know the result? curl here will be very precise emulator of what flespi is sending to your server.

And finally please take a look at point 3 from here and deploy any kind of proposed HTTP servers that are capable to receive the data.

kial

So then we have to check if flespi sends reduced messages.
To test it, we can use, for example, a https://ngrok.com/ service with a free account, to expose some local test HTTP server (like npx serve .) to the public internet.
So I did that test locally, and started ngrok inspector app shows me such an image:
screenshot

So flespi platform sends me more than 10 keys in the message.
I strongly suggest you to check your software using cURL tool I mentioned above in addition to Postman, to repeat your test.

WilliamFeraux

shal So for postman, I just tried an other post, and it worked, as I'll show you in the next pictures.

I think that's what is really surprising, cause neither flespi nor my server seems to reduce messages up to 10 keys, but when we link both, that's what happens...

WilliamFeraux

Hello,

So i tried to use curl with the right body and the right token, and this is what i got.

The problem is that I receive nothing on my server with that.

kial

WilliamFeraux Please note the wrong Content-Type sent and HTTP 301 redirect response on your curl screenshot. It may be an indication that your server software didn't even pass the request to a proper handler in the source code so debug gives you nothing.

So please:

Add parameter --header 'Content-Type: application/json' to your curl command
Add slash char trailer (/) to the curl uri to make proper URL as your server redirected (https://dms.velvyvelo.com/geolocalisation_flespi/)

And try again corrected curl command

WilliamFeraux

Okay so I did what you asked, so I'm gonna forward you the pictures.

So I have the same problem than with flespi, it's only receiving the first ten keys.

kial

WilliamFeraux Ok, now we have a strong evidence that your 10-keys issue is reproducing not only with flespi, so it's most likely in your code.
So the only suggestions from us is to check your code.
Also please note the 500 status code in your curl response, it usually indicating an uncaught exception somewhere. So check all logs related to handling requests on your endpoint.

WilliamFeraux

kial I think the error 500 is there because there isn't the keys that we need in the data, so our code return an error.
The thing I don't quite understand is that the same body and same header is working fine with postman.
And well maybe it's an error on our server, but the code that we used is really similar than the one we used for an other tracker supplier, that are pushing us json data with more than 10 keys, and it's working fine, so I don't really see why there would be a problem now with our server.

kial

WilliamFeraux Try to add the same headers to curl command as on your Postman screenshot one by one (note the minus chars replaced with underscore) and maybe switch to HTTP 1.0

WilliamFeraux

Hello again, so we tried the curl that the postman Post request that was working gave us, and it's working.

So what do I do now?

kial

WilliamFeraux Please look at your data - you're sending a JSON Object but flespi streams sending JSON Array with JSON Objects inside. You have to adapt your software to receive data in that form.

kial

I also strongly recommend you to change x-api-key value for your software as you posted it here in public.