Nov 18 2022
PICO W advanced webserver 2
Posted by kll · In RPI · 409 Reads ·
OSCI speed
MQTT
RPI3 new setup for MQTT and Node-Red
Circuit Python 8.0.0 beta4 to beta5
S31 AC Power Meter
MQTT BRIDGE
PICO_W problems
OSCI speed
still not happy with the OSCI speed i asked forum for help
and `anecdata` send me to bufio
nu bench
while now in my big code see 327680.1 Hz
the above bench suggests even more is possible???
but when i check with PICO puls 1/10 ON/OFF ?1ms signal?
i see
to keep it with the rest i need to change the ULTRA RANGE 4096 to * 16
PICO_W PMS1W ULTRA
# new speed buttons: | ULTRA | FAST | >>> | >> | > | SLOW | DEAD |
# 320 samples in 0.98 millis, that is 0.003 millis sample time or 327680.1 Hz using delay -1 loops
# 320 samples in 5.86 millis, that is 0.018 millis sample time or 54613.3 Hz using delay 0 loops,
# 320 samples in 17.58 millis, that is 0.055 millis sample time or 18204.5 Hz using delay 10 loops,
# 320 samples in 59.57 millis, that is 0.186 millis sample time or 5371.8 Hz using delay 45 loops,
# 320 samples in 306.64 millis, that is 0.958 millis sample time or 1043.6 Hz using delay 255 loops,
# 320 samples in 1187.50 millis, that is 3.711 millis sample time or 269.5 Hz using delay 1000 loops,
# 320 samples in 5912.11 millis, that is 18.475 millis sample time or 54.1 Hz using delay 5000 loops,
but now i get from forum info that this call might work:
#adcbuf = analogbufio.BufferedIn(board.GP26, osciH, rate) # TypeError: extra positional arguments given
#adcbuf = analogbufio.BufferedIn(board.GP26, osciH) # good
adcbuf = analogbufio.BufferedIn(board.GP26, osciH, sample_rate=rate) # thanks 'anecdata'
good, with this i can rewrite the whole sampling
#osci startt 5555507812500, stopt 5555508789062 ns, 320 samples in 0.98 millis, that is 0.003 millis sample time or 327680.1 Hz, called rate 500000 Hz
#osci startt 5586831054687, stopt 5586833984375 ns, 320 samples in 2.93 millis, that is 0.009 millis sample time or 109226.7 Hz, called rate 100000 Hz
#osci startt 5597339843750, stopt 5597346679687 ns, 320 samples in 6.84 millis, that is 0.021 millis sample time or 46811.4 Hz, called rate 50000 Hz
#osci startt 5606213867187, stopt 5606246093750 ns, 320 samples in 32.23 millis, that is 0.101 millis sample time or 9929.7 Hz, called rate 10000 Hz
#osci startt 5614054687500, stopt 5614119140625 ns, 320 samples in 64.45 millis, that is 0.201 millis sample time or 4964.9 Hz, called rate 5000 Hz
#osci startt 5620358398437, stopt 5620678710937 ns, 320 samples in 320.31 millis, that is 1.001 millis sample time or 999.0 Hz, called rate 1000 Hz
OSCI_ULTRA_bufio.zip
ok, that is a nice one-channel 6$ OSCI: "PMS1W"
and at home can call the view / operation interface / web server from mobile / PC / TV
but what you do when you are ?on the road?
this web server should have a "AP MODE" ( not only "STA" )
( or like the home devices SonOff / Tasmota / from AP to STA installation procedure )
so no need to login to a router.. just call it phone / wifi / AP ssid /
try to find a AP web server example
that is the problem with that MEGA projects,
you never know how far you can go..
and i had it with Arduino and ESP, you got erratic behavior, where a bigger code fails more often..
but you had to try..
MQTT
to push this on the limit i will try to also build in MQTT
JOB5 MQTT send ( means every 60 sec publish set of ( Volt Amp Watt ) data )
to a local broker RPI3 [192.168.1.103] ( what also has a Node-Red trend dash board
and a Sqlite3 historic data-collection
with the multi file structure add just a timer job is tricky already, esp about the en / disable of it by USB menu
hm.. i should build one EMPTY JOB6 USER spare ?
even the 1Mloop is at 70 sec already ( with all JOBs disabled ) ( 0.07 milli sec per loop but better as blocking... )
so: the timed JOB5 calls from file mqtt.py
send_MQTT()
what makes a counter and a JSON ( like string but use ' instead good " ) 'mqtts' to send.
for that string it needs the measuring data like from
* JOB1 PICOW_io A0 as volt and A1 as amp
or
* JOB3 INA volt amp watt
( only one should be enabled, default is board IO as i not know you have a INA )
at boot web_wifi.py
setup_webserver()
asks mqtt.py if useMQTT is True
so it also connects to broker, make topic, sends a 'hello' to broker..
see new details in '.env'
in
run_webserver() ( called by JOB0 and checks on web clients )
and when useMQTT is True
it gets the MQTT_count from mqtt.py
and if that increased it gets the new 'mqtts'
and publish it to broker..
i used RPI3 ( with ethernet cable ) at [192.168.1.103]
as broker and with mqttsub
alias mqttsub='mosquitto_sub -h localhost -u "u213" -P "p213" -v -t "P213/#"'
so again, mqtt.py does not measure or send data, just prepares a string, the above {...}
looks good so far, but when i call OSCI i get memory allocation error again.
( right where i make the string for html after i add the SVG code string)
well i should have seen that b4 already,
i did a
svg ="{"
( 320 lines .. of ) svg += "...."
htmldata += svg
so that means the svg needs double memory, first as svg string and them inside htmldata string.
so that was a easy clean up
the modularity with the many *.py files is something good,
but as each job calls just the main function ( from its own file.. picow_io, dht, ina, ads, mqtt ) and its under 15 lines,
now i go backward and pack all 'jobx.py' into one 'jobs.py'
also there each jobs 'update_' sec is defined,
import and shown at console (REPL )
and read by code from menu for the job execution
no what i copied already is the Node-Red dashboard operation to TUNE the
+ filter
+ change the mqtt data publish rate
when that is working ( again ) i can zip all user config of this project.
in the old project i used it to filter the INA_Volt, INA_Amp, INA_Watt
and adjusted a faster/fix sample rate to INA job3
-a- it normally not makes sense to filter Watt ( just filter Volt and Amp before you multiply )
BUT if Watt is coming from a sensor ( INA ) it needs filter too.
-b- i want to move the filter from INA to short before MQTT
+ + so in webpage you see the original readings,
+ + in MQTT send filtered values
+ + + from MQTT can remote adjust filter and MQTT sample rate ( not hardware sample rate )
( note: currently i check on a google workspace mqtt store and graph with remote data from a island ( a friends professional project ), much more professional as
this here, my node-red tools .. default NR trend...
but see that filter is missing...
and for a startup / test work a faster data rate as 1min might be nice )
ok, for the filter:
there is ( like for Volt ) a ( Va = 0.8 .. 0.9 ) and a ( Vb = 1.0 - Va )
and the filter math is:
vf = Va * vf + Vb * v
that is a recursive filter first order
with Va = 0.8 means we allow a 20% change of the filtered value
by a change in 'v', the new sensor value
as that would always start from 0 there is a firstrun setting ( vf = v ) as better init
and a MQTT ( remote ) operation of
* MQTT publish rate
Volt Amp Watt:
* Filter enable
* V A W filter tuning
- - the regarding code is unreadable, as my Python is poor, and i have to see / change values ( like update5 aka MQTT job timer )
over several .py files.. so mixing import values, get_ and set_ values across the involved .py files
RPI3 new setup for MQTT and Node-Red
just to show what i mean with using RPI3 ( as broker and Node-Red dashboard )
first setup a new OS drive ( this RPI3 can boot from USB )
RPI3 Install log
try to follow it for your own setup
current version of
Multi_Task_Ains_Temp_DHT22_INA_ADS_USB_MENU_dynamicWebserver_OSCI_ULTRA_bufio_MQTT_RPI3work
PICO_W-PMS1W_pp.zip
were you find:
* * code ( /lib/ ,env *.py )
* * cp800b4 ( pico pico_w uf2 and bundle and nuke )
* * NRflows ( Node-Red flow export ( Flow1, Flow2, Flow3 ) )
* * python_tools ( to test your broker / Node-Red without a active client )
Circuit Python 8.0.0 beta4 to beta5
when i see Circuit Python 8.0.0 beta 5 available
i try and failed miserably,
big changes in
* httpserver
* bufio
* and low memory problems.
so first i move out the PMS1W OSCI to a extra project, as the html-string ( with the inline SVG code ) is too long for the overloaded big project, but after a day work it runs as 'stand alone':
PMS1W
now i mod the rest of the project:
delete OSCI PMS1W
run new webserver
add cpu temp in MQTT ( old open point )
and pack it with its CP800b5 to
PICOW_MQTT_IO_pp
the PICOW has lots of flash / code space /
but my above combined project ( and new CP800b5 & webserver ) hit the memory limit
when i did a OSCI HTML page in a string.
so above i separated it into 2 projects...
but that is kind of annoying to handle
hmm, i try following idea what might make you scratch your head..
the OSCI PMS1W consists of /lib/htmlwebserver, .env file and
code.py
( but lib and .env same as old big MQTT_pp project with many *.py files. )
i copy the code.py from OSCI PMS1W back into the PICO_W but name it
pms1w.py
so a file what does nothing / unconnected from the rest of the project what is all based on / called from / code.py and
circuit python auto start ( and REPL [ctrl][d] )
how the hell can that be used?
when the main project is running
* open Mu editor
* open serial console REPL
* in there press [ctrl][c] and get the '>>>'
type:
import pms1w
see the OSCI project loading
and call from browser OSCI main page html://192.168.1.213
ha!
S31 AC Power Meter
here now comes a sidestep what might not be so obvious but follows 'a bigger plan'
use a smart home device 'SonOff S31' where i flashed a TASMOTA firmware here
and connect it also to the RPI3 MQTT broker ( like for PS ON / OFF and power consumption AC Volt Amp Watt )
there is normally no need to make a other MQTT user, but in my case it was required by S31 TASMOTA config not like too short passwords:
in RPI3 ( mosquitto broker ) ( -c creates first user / -b adds a other user )
sudo mosquitto_passwd -b /etc/mosquitto/pwfile u213S31
p213S31
p213S31
in device S31 / TASMOTA not have fixIP come up on 192.168.1.10 ( find by mobile FING )
setup MQTT for this project
and after change MQTT user password in S31, also need in RPI3 mqtt and Node-Red config AND PICO_W .env all works again.
- - problem was Node-Red MQTT only can do one mqtt connection ( with same user password config )
from S31 TASMOTA
the final TOPIC is P213/P01S31/SENSOR with
{ Time:..., ENERGY:{ Power: xx, Voltage: yy, Current: zz } }
every 5 minutes!
make a add S31 dashboard
using Flow4
the 5min timer is here:
i change 300 ( 5min ) to 60 ( 1 min )
MQTT BRIDGE
until now all LOCAL:
+ S31 smart AC feed ( WIFI MQTT )
+ power supply with a difficult job ( like charging? )
+ PICO W DC Power Meter ( WIFI MQTT )
+ + MQTT by RPI3 broker
+ + Node-Red dashboard for Trend and Operation ( filter tuning example, S31(tasmota) ON OFF )
+ + and SQLite Hist trend
now i want open that up to the world with a BRIDGE
OVVW:
while mosquitto can do that bridge directly ( map topic to external connection )
i want try a coded PYTHON version with full data remapping ( of the 2 sources PICOW and S31 )
to a ( existing online project ) topic/name structure.
and optional very-remote operation.
-0- i have a FREE online MQTT broker account at hivemq.com
( unused since i play only local RPI broker, but tested remote operation TH <> USA already )
-1- make a local client subscribe to see what PICO_W and S31 are sending
-2- make a remote client subscribe to what the bridge would send, to see its working
-3- make a remote client publish ( as start of the bridge ) and check it works ( see -2- )
-4- step by step subscribe to local broker to get the info to send ( bridge ) with -3-
-5- take the payload from 2 local publish to local python variables ( encode / json.loads / .get("?") )
-6- use the python variables ( and new "names" ) to make a new record string { } ( data marshaling ?)
for publish to remote broker
with local_sub we see the data from PICO_W and S31 to local broker
with remote_bridge send test and with remote_sub see test
running 2 pythons programs ( one publish one sub )
both with mqtt.loop() running the cpu shows 15% load..
that i replaced in the main loop with adding a time.sleep and cpu usage back low
time.sleep(0.5) # __________________________________________________ with 2 python tools run on RPI3 see 15% cpu load, now use sleep timer
# _________________________________________________________________ update both mqtt
mqttR.loop(.1)
mqttL.loop(.1)
already see the data from both devices
and now try to split it to python variables
and send new format to remote broker.
now that ( remote ) topic and format ( names..) is in hold,
as i still wait for info from this bigger data collection project
( where they use Google IoT Core for mqtt > database > trend-report ?EOL? )
meanwhile here play via HIVEMQ (free account) as remote broker
now this RPI3
python remote_client_bridge.py
must run as a service ( autostart best after mqtt broker started )
so i make a bash
start_bridge
and a
start_service.desktop file for that,
what could be on desktop for manual start or:
+++ will be copied to ~/.config/autostart/
and HOW TO use Node-Red again?
well we have the 4 page (Flow 1..4 ) Node-red app already inside the RPI3
with dashboard at http://192.168.1.103:1880/ui/
but Node-Red does not allow 2 mqtt broker / credentials, SO?
we, with the RPI3 are the LOCALS, can see more / do (operate) more?
but a REMOTE remote user just want to (could) see that remote data..
so idea is there is a www user with a different Node-Red app
what just use the data RPI3 sends out to the remote broker.
NOW, it is very good that i installed Node-Red with the 'project' option,
so i can use same RPI3 Node-Red also for the new REMOTE APP
( besides this project option also comes very handy for a test flow develop and copy/paste as import/export work )
PICO_W problems
now when i start the tasmota S31
- - but the RPI3 ( local broker ) was OFF
i see something smart in the S31 console
the mqtt broker connect failed and a smart timer does a retry: after 10s / 20s / 30s /... ( adding 10sec to the wait for reconnect )
+ that i need inside the PICO W too
- - - like if RPI3 and PICO W on same AC power ( ON ), RPI3 broker will be up too late and PICO W stay off line
- - - actually it is more bad, with RPI reboot the PICO_W LED goes OFF, ? run in to error ?
so need to check the MQTT connect_and_send code again!
update:
here now PICO_W code survives ( reconnects MQTT broker after ) a RPI reboot
( version with the optional pms1w.py project and a new added READ.ME )
python tools & node-red export & PICOW code
above you find already the PMS1W OSCI project as stand alone,
but i was thinking it could be of interest to have the
PICO_W_CP800b5_Multi_Task_Ains_Temp_USB_MENU.zip
also as stand alone framework,
should run on old PICO too ( but untested )
pls follow me more hardware
