May 07 2018
RPI use MQTT
Posted by kll · In RPI · 2885 Reads ·
MQTT broker
Python
Python MQTT database
pack it into MYBLOG
also Cayenne can use mqtt
extra MYMQTT environment
intro
i wanted to test that MQTT (Message Queuing Telemetry Transport) already long time,
now play with a new board ESP8266 + OLED and that should be my first MQTT client.
also a other RPI could be a MQTT client just testing the MQTT python lib
MQTT broker
but anyhow i need a MQTT broker:
with node.js can use MQTT broker mosca: here
instead of the usual mosquitto and with python
and optional can use some service from the cloud esp8266 - phone and cloud broker
a very good online service info see review-iot-data-logging-services-with-mqtt
first a new setup: ( for hardware RPI3B )
that does not take long, as i have my MASTER 8GB SD card, a default setup
what just needs a update upgrade and a "SD card copier" action to a 16GB card.
boot it and change IP (203) and name RPI3_MQTT (reboot)
(20 min unless something goes wrong and you have to do it twice)
sudo apt install mosquitto mosquitto-clients
pi@RPI3_MQTT:~/projects $ mosquitto -v
1525698143: mosquitto version 1.4.10 (build date Fri, 22 Dec 2017 08:19:25 +0000) starting
1525698143: Using default config.
1525698143: Opening ipv4 listen socket on port 1883.
1525698143: Error: Address already in use
pi@RPI3_MQTT:~/projects $
so, check after reboot, and it is running already
i should have checked that first but no idea if it is still relevant.
should i do it again and follow this? possibly later, first play.
a test:
ps -ef | grep mosq && netstat -tln | grep 1883
pi@RPI3_MQTT:~ $ ps -ef | grep mosq && netstat -tln | grep 1883
mosquit+ 438 1 0 20:05 ? 00:00:00 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
pi 1372 1339 0 21:05 pts/0 00:00:00 grep --color=auto mosq
tcp 0 0 0.0.0.0:1883 0.0.0.0:* LISTEN
tcp6 0 0 :::1883 :::* LISTEN
pi@RPI3_MQTT:~ $
looks good
try from here i open 2 windows,
and type in -1-
mosquitto_sub –d –t armtronix_mqtt
type in -2-
mosquitto_pub –d –t armtronix_mqtt –m “Hello armtronixâ€
something is very wrong here
but not give up so easy ( as i am very lucky anyhow ( if i read all that chat about install problems ))
here i see different commands:
and type in -1-
mosquitto_sub -v -t "test/topic"
type in -2-
mosquitto_pub -t "test/topic" -m "Hello World!"
ok find also some documentation like mosquitto_sub but not sure related to what version ( as --help and -V not work here )
but
mosquitto_sub -v -t \$SYS/#
tells me that actually too much is going on there.
( even after reboot // who the hell is mosquitto talking to ? )
restart the ESP8266 with new IP + + + and get connected to RPI
but on RPI can't get past the connection refused for mosquitto_pub /_sub / -h 192.168.1.213
possibly YOU know already what is wrong?
the wrong IP
-h MQTT_BROKER_IP
so i not have to use the IP of the device i have connected ( MQTT knows that internally )
in this case i can just NOT use it, or use ( -h 127.0.0.1 ) local internal IP
or, and that is what this -h opt is for: connect to a remote MQTT broker.
it is all here, i just need to get it right in my little cpu
Python
now it is time to loadup PYTHON3
one question comes up, is MQTT a ascii/text protocol? how to do bool and floats?
because now i send "1"
other question, how i know i am not talking to a brick wall?
does MQTT even know that device is alive? listening?
well i better build in my own AKN acknowledgement, back to ESP.
until now i did not check what would be a usual app to use that MQTT
but for me the most obviouse would be a webserver - webpage with a BUTTON to switch on the light,
as making GUI desktop apps looks to me like a waste of time.
but you remember the big question about the light in the refrigerator, is it really OFF when the door is closed?
same here, even with the new AKN can not be sure.
and i am used to that problem, building MCC for big drives, each ( MCC only ) hundreds of $ ( with MODBUS link and up to 16bit word up and down ) still on a DCS it looks same even there was no Motor connected. I only trusted that info when i could see a current indication. So besides building a good feedback loop for the operation ( current-load > 0.0 ) i also do the most practical use of MQTT besides Lamp ON/OFF,
a measuring sensor / data collection.
while i was working arduino IDE / ESP8266 i think best would be the JSON format, basic
{ mysignal1: 123.45, mysignal2: 6.78 }
where the data come from is clear by the mqtt/topic
but also
{ pcu213: { mysignal1: 123.45, mysignal2: 6.78 }}
is possible, would be redundant / bandwidth related, anyhow that long text story take lots of
bandwidth compared to a unreadable record structure,
but could be more easy for later coding the data storage on a broker assuming there could be several sensor nodes connected ( doing the same job, just with different mqtt/topic )
i still think best is a CSV readable data transmission.
, 123.45 , 6.78 ,
the datacollection:
that part MUST be a different app from the operation / data visualisation.
( service at boot, after MQTT broker ready and client connected)
in this case here a python program using mqtt.subscribe, wait for incoming
and give device_ID and timestamp for the record and store ( append ) to file.
( as i only play here i will use RAM DISK of RPI : /run/shm/ ( possibly only 65MB big ))
yes, that is not professional, your into SQL, ok, can use sqlite3.
i just reread: above i write
coding the data storage on a broker
well with MQTT there is no limit, when the datacollection program uses MQTT.subscribe
and stores to 'file' you just give the IP of the server where the MQTT broker is running,
so no need to be same computer. but there are good arguments to do it directly,
never trust network or even internet connection.
and because you can read the data like for visualisation, also SQL remotely, you are free.
Python MQTT database
ok, lets start with a python3 MQTT datacollection:
from here i want test the Callback Example.
the first part was extremely easy:
but about JSON, tricky to take that string appart, esp. when you assume that you not know the names and structure
so i try byte array to string, string to JSON, JSON to column and column value, both needed for some SQL later.
but to do any SQL need first a database: now old style LAMP that is a big issue, need phpmyadmin installed...
here PYTHON SQLITE3 we just create a file, but as i want later use it from FLASK webserver must already follow some rules.
-a- subdir structure: /static/data/
if my programs are at
~/projects/py3_mqtt/
-b- i need 2 files, a initial database structure with data later used by webserver, and the from this created database
~/projects/py3_mqtt/static/data/schema.sql
-c- at first run ( or later when you deleted the database file ) you enable line
# init_db()
and then the run
~/projects/py3_mqtt $ ./py3_mqtt_to_database.py
will create
~/projects/py3_mqtt/static/data/myMQTT.db
-d- then you pls. disable that line again
basically, SQLITE3 creates its own database file as soon you try to use it.
now we have a database file, but no data tables for our data
we have incoming data, and we know its structure ( columns )
if the tables not exist we must create them first ( automatically )
and then write the data to the database file.
i use the feature that the timestamp is created automatically with the INSERT
but what time base it is using? great: UTC, here i find
timestamp DATE DEFAULT (datetime('now','localtime')), YES, good
to see the result i want use the sqlite3 command line interface
sudo apt update
sudo apt install -y sqlite3
and i find it all there, the file, the table
( note: the topicstructure a/b/c does not work for SQL table name,
so i needed to convert ( replace "/" with "_") it to a_b_c )
long version show also the database structure,
shorter version, show just the stored records.
as this worked i need to test if the sending of multiple floats would also work. for this database reset/delete required
and the ESP is now sending {"SIN":dirty sinus, "FIL":filtered value, "LED":1.0 }
and i changed the program that it creates the db file automatically if not exist.
and the table creation with all required columns did also work well,
BUT storing the data went wrong, makes 3 single records for each val instead one.
i wanted to make the SQLstring for loop easy but that was bad.
now after some small changes also this works
and i can switch OFF the diagnostic prints too.
but some of the above pls forget, change in plans: i want have a
separate database ( file ) for the MQTT ( data collection ) and the CMS ( webserver ),
so the init part i delete, SQLITE3 creates the file with the first MAKE TABLE.
sidestep:
install on RPI3 Arduino IDE nightly "1.8.6" with this, install ESP8266 board
copy ESP+OLED projects and libraries
compile upload from RPI3 to ESP8266
ok, also cleaned up my workspace, looks very good
so the RPI3 powers the ESP via USB, ( and power OFF at shutdown )
also with Arduino_IDE i can work on the code and get serial diagnostic.
but the MQTT communication is ESP--WIFI--myRouter--WIFI-RPI3 ( so in wifi-range it could be used as remote I/O)
// the RPI2 with the 2 cameras is a different playground //
pack it into MYBLOG
so i will not lead you trough the whole procedure to create a
PYTHON3 FLASK/JINJA2/WERKZEUG HTML5 website
with user login ( a must for the OPERATION part ) and a database driven mini CMS
and how to just install my example you could find at the BLOG and the Download area.
i will copy it inhere, so you later just/still have to do:
./install.sh makes YOUR start file and a desktop icon
./start
* Running on http://0.0.0.0:4567/
and from a PC call in browser (RPI _ webserver _ IP )
http://192.168.1.203:4567/
YES, it is that easy
now the default texts you see in that page
// title // header // main page // tile items //
are not in the source code, can modify online in the settings.
and as it is a CMS adding new articles with pictures... is easy.
but for a real new function must dig into python3
here i create a new menu entry / webpage for the MQTT
there are now 3 programs using settings about the MQTT ( broker / topics / databasename )
so i made a common file mqtt_settings.py they can use via
from mqtt_settings import *
the flask app need to connect to database, read the column names, read ( the last 100 ) data records
and hand this to the Jinja2 template MQTT.html, inthere find the list generation
usually it is much more easy, but here i again ( like with the data collection ) try to allow
a unknown list of variables from MQTT client ( names and count ),
+ you decide there ( arduino IDE like for ESP ) how many floats with what name to be send,
+ accepted by the MQTT broker mosquitto in RPI,
+ read by the python3 tool mqtt_to_database.py ( to early to be a service at boot )
and stored to SQLite3 database at the webserver path:
/static/data/myMQTT.db
+ read and presented in HTML web page by
flask_app.py using Jinja2 generator on
/templates/MQTT.html
called from browser myBLOG // menu // MQTT
the record list get a table style, hidden into a ( 3 line only ) scroll window
better use a DIY GRAPH: my LINE CHART style,
made by a HTML Jinja template using a inline SVG
you see mouse cursor "tool tip" show TAG and VALUE of each BULLET ( mouse over )
and at the end
the
operation part
( login required )NEXT STEP:
when i try to import the paho mqtt lib into this Flask project get a error,
while for 2 other stand alone python programs it works just fine.
looks like FLASK has its own wrapper for MQTT, see here.
but when i try:
from flask_mqtt import Mqtt
also error
check
pip3 search Flask-MQTT
Flask-MQTT (1.0.3) - Flask extension for the MQTT protocol
pip3 install Flask-MQTT
pip3 search Flask-MQTT
Flask-MQTT (1.0.3) - Flask extension for the MQTT protocol
INSTALLED: 1.0.3 (latest)
still same error!
Traceback (most recent call last):
File "/home/pi/projects/myblog/flask_app.py", line 17, in
from flask_mqtt import Mqtt
ImportError: No module named 'flask_mqtt'
File "/home/pi/projects/myblog/flask_app.py", line 17, in
from flask_mqtt import Mqtt
ImportError: No module named 'flask_mqtt'
also when i try that in a stand alone example it works too.
seems to be some kind of naming conflict i not get.
as i just need to build 2 send buttons [0] [1], i first go the easy way.
and failed again:
on a sys
call(['python3','mqtt_remote.py','0']) # invert logic again
in flask i get:
Traceback (most recent call last):
File "mqtt_remote.py", line 8, in
import paho.mqtt.publish as publish
ImportError: No module named 'paho'
File "mqtt_remote.py", line 8, in
import paho.mqtt.publish as publish
ImportError: No module named 'paho'
so i must very basic namespace problems from flask.
OK, thats it, if i change
nano start and erase the sudo of
sudo /usr/bin/python3 flask_app.py
or if i again install
sudo pip3 install paho-mqtt
in both cases it works.
( so that leaves me to check same way on the flask_mqtt way and recheck ALL
without sudo start. )
while the record has only a flat structure
on MQTT:
topic: pcu213/sensor/out
data: {"SIN":10.75,"FIL":13.03,"LED":0.00}
on the way from python to HTML template i needed like a record with 3 line array inside with: [{number,bulletposition,color,tooltip text},{},{}]
{ 'rec':100, 'Id': 5002, 'TS':timestamp, 'linesc':3, lines:[
{'lix':0, 'cix':317, 'col':'#77ff77', 'tit':'rec:100,line:0,Date:timestamp,Tag:Sin,value:23.6' }.
{..},{..}] }
after i found the trick to build that in
when i start with PYTHON FLASK Webserver
the idea was to show what is possible with a default RASPBIAN setup,
but using more features, also add installs are required.
now rework the setup ( use 'y' and it will do that installations for you ) and test a fresh start
start browser and test a OFF ON operation.
again test on a
new system setup
update upgarde customize RASPBIAN STRETCH SD card ( in RPI3B )
there i am per default in my ~/projects directory:
for your environment it comes very handy to have a
alias MQTTshow='mosquitto_sub -h 127.0.0.1 -t "#" -v'
like i make it in
nano .bash_aliases
you should copy its content inside your ?existing? ~.bash_aliases
for the ESP part pls. see my other BLOG
and for the code, that you can find at git
for the myBLOG webserver see here
and its own download
also Cayenne can use mqtt
i got from a very old account at MyDevices / Cayenne /
a email that my installs for Raspberry Pi are to be updated // using MQTT... now
Hello kll,
You are receiving this email because we show you have a legacy device either activated or with pending activation connected to Cayenne.
To strengthen security and reliability on devices connected to the Cayenne Cloud, we will be ending support for legacy devices not upgraded to our new Cayenne MQTT API starting July 1, 2018. After that date, legacy devices will no longer be able to connect to Cayenne.
Affected Devices:
Arduino devices not using the Cayenne MQTT library
Raspberry Pi devices not using RPi Agent 2.x
You are receiving this email because we show you have a legacy device either activated or with pending activation connected to Cayenne.
To strengthen security and reliability on devices connected to the Cayenne Cloud, we will be ending support for legacy devices not upgraded to our new Cayenne MQTT API starting July 1, 2018. After that date, legacy devices will no longer be able to connect to Cayenne.
Affected Devices:
Arduino devices not using the Cayenne MQTT library
Raspberry Pi devices not using RPi Agent 2.x
so i setup the RPI3B with a new (test) system (20min) and install cayenne again (10min)
will play for new features later...
extra MYMQTT environment
update 05/2019
add to the "myblog" webserver ( with mqtt db and its PYTHON service )
i now make a separate beginner environment
* MQTT
* Python
* Processing and Arduino
without all the webserver overhead.
/mymqtt/
read
Code
years later try MQTT mosquitto 2, please read Link
