Jul 17 2015
datacollection arduino-USB-RPI-WWW / IoT-cloud, new approach
Posted by kll · In Arduino Projects · 2711 Reads ·
- run measuring ( and PID control ) in Arduino
- usb connect to RPI
- a python service catch the data to ramdisk file
- it has a timer ( 30sec ) to send it on to a Google spread sheet ( called historic trend )
- a python GUI application ( with Tkinter ) did also a collection ( called current trend )
( and allow operation of arduino / faceplate slider.. )
-b- a different approach is the PoorManScope here
- some arduino boards or a max32 collect as batch
320 values ( 3 channel ) 75kHz or for fastest sample rate ( 1 channel ) >500kHz with the arduino due or MAX32
- in ascii / csv style / send 320 lines, 3 values ( plus one info line ) to a RPI or PC
- where processing or python code catch the batch data and
- show a oscilloscope like 3 channel graph about every second.
-c- from RPI forum i see several questions about catching a conti stream from sensors
and show graph continuously. But i had no real idea where the bottlenecks are?
in one of my first PMS tests arduino 2008 to PC i tested record transfer instead ascii ( print )
but as a batch scope has offline data transfer that speed improvement did not matter at all,
and data stream debugging gets difficult.
with a conti stream via USB that might matter again, unless the graphic tool can not deal with the data anyhow in that speed.
-d- the google spreadsheet is nice, my own php sql website for data collection did work basically ( in table view )
now i want test that free carriots service.
-e- with the RPI2 the python graphs might be faster, and as its easy for me to make it a PC python and RPI python code i will stick with it, if Tkinter again? have to play with the screen update rate?
processing is on HOLD until rev 3 using JDK8 hardfloat for RPI is ready
-f- now, to take samples, send them via usb, and do every 30sec a copy to internet is not much of a data reduction concept.
a new way i want try like:
__ the incoming data are catched to a ( adjustable ) array, and reduced by MIN AVG MAX as a sample for the current trend array. that array is reduced again ( MIN of MIN, AVG of AVG, MAX of MAX ) to a sample for the historic trend.
optimization for the historic data storage i know a little bit, but that would mean to write own sql code. and the retrieval and semi continuous trend view is a other big job, i am too old for.
but i just want to mention the basic idea about:
swinging door concept: instead of storing data based on a timing concept ( with or without reduction methods ) you define a range window ( like + - 2% ) when the value changes you wait until it exceeds that limit, compared to the last stored value, and store it ( with its time stamp )
so not changing values are not stored / not use database capacity.
in the trend you will just see a straight line. now that method is not that new, mpeg.. thinks like that too. for the data collection tools in the process industry it was a big boost in capacity ( but as you not know if will have many jumping process values difficult to predict, and tune limits (per signal) is too much work)
starting on the CARRIOTS service: see tutorial here and here
-1- registration FREE account here
( password need one capital , one number and 6 character BUT AAA1234 NOT WORK )
-2- get email and do verification
-3- login and edit default
there is a account, default group and device created,
timezone setting not come through from registration.
use menu MYSETTINGS MYACCOUNT or
on project group device
use [list] button for details
use [actions] [edit] button ...for change and [save] button at bottom
there is a default frequency for data and status of 1440???
i not understand!
from the free account limits here i see Max. accepted streams
15000 streams / day ( does that mean every 6 sec one sample? ) sample =?= stream
500 streams / minute
CARRIOTS web site operations feel difficult for me, but it could just mean that it is a powerful tool.
ok RPI2
what would be the minimal python program to talk to CARRIOTS
first i simulate data by a 0 .. 1023 ramp like a reading from arduino Ain 32 bit INT
for the API KEY, at menu MY SETTINGS / MY ACCOUNT see 3 keys,
i used 3. Automatic Apikey Stream Only
as i get HTTP error 401 UNauthorized i try
i used 1. Full Privileges Apikey
as i get "HTTP error 401 Unauthorized" try
i used 2. Automatic Apikey Read Only : same
for the DEVICE see Device management / devices / actions / show /
Id developer: defaultDevice@NAME.NAME
i use the RPI example with a 10 sec timer wrapper
but only get that Unauthorized error
i used from menu
Wizard Send Streams to simulate a own record and it worked, but i see in details that a v1 protocol was used
and a Id Developer like "xxxxxxxxxxxxxxxxx@NAME.NAME, i try to use that too.
at CARRIOTS forum i found same error, answer:
Where "YOUR DEVICE's ID_DEVELOPER HERE" must be replaced by something like "defaultDevice@myuser" and "YOUR APIKEY HERE" must be replaced by your Full privileges APikey (long alfanumeric string).
that's a more clear info, as there are 3 APIKEYs given.
again i search in tutorial and find a other example, with some more lines and some small different syntax, also give error, until i find mixture of that tutorial syntax examples what worked.
oh, by the way, even there i see not any note about that, but must use python2, not python3
code
under menu Data management / Data streams i now see all records ( with internet time correct localized )
with this data storage code "at":"now" the CARRIOTS server makes the timestamp, arduino, serial link, RPI not need to waste time on "time".
as i say i need 3 channel reduced to min,avg,max i would need 9 values per line / sample.
at CARRIOTS use menu Wizard Widget Graphs to show something.
when i send the 3,6,9 values i noticed that the data list was somehow resorted, changing the "names" did not help, but had a bad side effect, that changed "columns" could not be selected for GRAPHs.
when i wanted to clean up i found out that streams/records can not be deleted ( from control panel),
using API means i would have to write a extra python program to do that!
it just lists the newest record, if you set setdelete = 1 and start again that record will be deleted
start it for each record
now we go to the other end of the food chain: Arduino UNO
unless we find out later that the serial link is the bottle neck we just do a 3 channel analog read.
if we have to do a data reduction in arduino already we need ( like in PoorManScope ) a analog read to array
and a min avg max calc on that, before we send to RPI.
add we need the 2 options:
-- print values in ascii csv style ( what could be max "1023,1023,1023,CRLF" 17 byte
-- write values as record: 2 bytes, 2 bytes, 2 bytes, CR LF 8 bytes
and some timer routines for check code speed.
/* below usec info is related to 1000 loops
* NO SIGNAL OUTPUT to USB!
* use UNO, fast ADC
* 1 channel: usec 16208 Hz 61697
* 2 channel: usec 33136 Hz 30178
* 3 channel: usec 48900 Hz 20449
* use UNO, normal ADC
* 1 channel: usec 112068 Hz 8923
* 2 channel: usec 224144 Hz 4461
* 3 channel: usec 336140 Hz 2974
*
*
* ascci print to USB csv style 0 .. 1023 , use serial 115200 Bd
* ,1024,1024,1024,
* 0 channel: FADC usec 1529996 Hz 653.60 // for reference NO ADC AT ALL, only print long numbers
* ,707,
* 1 channel: FADC usec 595000 Hz 1680.67
* ,704,699,
* 2 channel: FADC usec 935000 Hz 1069.52
* ,696,701,696,
* 3 channel: FADC usec 1275000 Hz 784.31
*
*
* record write
*......
* 0 channel: FADC usec 679996 Hz 1470.60 // for reference NO ADC AT ALL, only print 6 byte
* §
* 1 channel: FADC usec 339996 Hz 2941.21
* §£
* 2 channel: FADC usec 509996 Hz 1960.80
* §œ“
* 3 channel: FADC usec 680000 Hz 1470.59
* so, a 20kHz sampling will go down to 1.5kHz using online data send ( instead of batch data send )
* definitely the serial ( coding and transport ) is a bottle neck, not the AD sampling
*
*
*/
i did this above tests ( and the short version with the minimal measure/send code ) on PC,
now install arduino IDE on RPI2B,
sudo apget install arduino
copy the sketches and connect, compile, upload to UNO.
start terminal see some data and then, at cpu 50%
( is that meter correct for RPI2? / install check htop possibly correct, 2 of 4 cpu full in use / switching)
it hung and i had to reboot / try again, same.
so, on RPI2 still same problems, i need that delay timer, after each line send, again.
settings 400usec not enough, 800 the [x] close terminal window worked again
now can start python work again to catch that stream
without any data reading, manipulation, just get the incoming lines ( netto 6 bytes ), again check on timing
now get ( in RPI2B python2 ) 1000 lines in 1.25sec?? 800 Hz, from the 1470 Hz we wanted before i added the 0.8msec delay per line, but possibly later can reduce the delay in arduino, might be that python serial is ?better? as arduino terminal.
above record sending would mean to write your own protocol,
i check on some structures, but most real protocols have overhead so my 8 bytes less per line ( ascii to 2byte int ) easy are used up, means lots of work and no speed.
just sending record lines [MSB][LSB][MSB][LSB][MSB][LSB][CR][LF] i tried
but it did not work when i used the readline in python, because depending on the int value i see short lines.
a more easy idea is to use FIRMATA, so i want do a speed check on that too.
i use arduino IDE 1.0.1 / example / firmata / simple analog / upload
RPI PYTHON2:
sudo apt-get install python-pip ? for python2.6 ?
sudo pip install pyfirmata
now again i do a timecheck ( in python read 1.000 .. 1.000.000 times
3 channel from firmata and end up with 47kHz, i doubt that that is the analog sample rate,
but for the serial protocol alone it would be impressive!
but i think that pinx.read() from firmata not samples or even talk USB, just get data from a variable,
where i not know its update rate. i did not test on the real speed.
