2021-01-11

Progress on Project HausBus

Projects

My desk at home is hopelessly cluttered with /projects/ in various stages, ranging from contemplating to just about being finished --- its only one more thing ... you all know this, I guess.

So one of my long term projects is "collecting environmental data on premise".

It started out in maybe 2003 and it went a long way and several iterations of technology. State of affairs last summer was this:

• A handfull of AVR microcontrollers and sensors on printed circuit boards, distributed throughout the house, implement the sensor electronics..
• Each board is connected to a cable providing power (12V) and a data connection via RS485 bus.
• Each controller is programmed with assembly and Forth[a].
• The collection of sensors includes temperature and humidity (inside and outside the house), atmospheric pressure, use of electricity and water, the water level in the underground cistern[b] and a number of readings related to heating and solar thermal energy.
• A small /always on/ system[c] is running Linux and serveral services.
• Among Them A Perl Script, Which periodically addresses the microcontrollers one by one and collects their current sensor readings. The script thus is aptly called "collector.pl", you see?
• The /protocol/ on the bus consists of plain ascii text strings.
• The collector would dissect the data strings and feed them into a sqlite database.
• Another perl script, named "viewer.pl" would query said database, wrangle the resulting data a little and draw a diagram using perl/pgplot5! Repeat after me: pee-gee-plot-five. Very well!
• Most important feature: the whole thing ist /not/ /smart/! No! Not even a tiny bit!

This odd collection has served me well for more than 10 years! The WAF (wifes acceptance factor) was actually quite high, and we gained a bit of insight from this collection.

From the description some of you (assuming there are readers at all) will already have slight doubts whether this author is in a sane state of mind. perl? Wuttt? PGPLOT??? Yes well, pgplot itself used to be a Fortran library, if memory serves me well. It has been replaced with giza, which is not a drop in replacement for how I used it.

However, the pgplot based viewer remained working only, because all the neccessary pieces had survived the upgrade of the /viewing/ system to Debian 9 (stretch). It was clear, that on a newly installed system I would fail to piece everything together. Plus there was a number of shortcomings which were asking for more sophisticated tools.

So last summer I set out to make this whole thing more modern[d]. I placed another controller, with new firmware on my desk, together with a Pi running a new collector plus mqtt, influxdb, grafana and the MagicMirror3 software. Plus display, cables, power supply and whatever is needed for a happy controller's life. And the whole thing failed on me. Miserably so! The whole thing turned out to be unstable. It would run for maybe an hour, or for a whole week, and then it would just stall. Each component still blinking, but the communication would fail. And powercycle would /not/ make the system run again! Very strange indeed.

I spent hours and hours on this. No dice.

Then some day I put everything away and declared unconditional surrender.

But a tinkerer's brain will not give up that easily, right?

So over year end season I decided to keep as much as possible of the existing solution and try /the simplest thing, that could possibly work/ to copy the collected data into a newfangled influxdb. Keep it simple!

This /keeping it simple/ required three iterations nonetheless. But I did get out of this with small changes:

• The collector script will call "mosquitto_pub" externally for every good data point collected --- everything else remains in place.
• The collector script is now started from systemd replacing the old init.d script, journalctl replaces the screen hack used so far.

New in the odd collection are these:

• Mosquitto will forward the data telegrams to any interested subscriber.
• Telegraf will subscribe and feed the telegrams into influxdb.
• Grafana will query the database and make any webbrowser act as a display to visualize the data --- this acts as an additional viewer. Look Ma, no perl!

None of these new toys are self-explanatory to my sort of brain, but heck, I can show new diagrams without resorting to the perl viewer.

Mission accomplished!

And if I look at this, it is already so much easier to use:

• Any system in the network can look at this now! I can repurpose the viewer system, or at the very least upgrade its operating system!
• Annotations are one click away --- they didn't really exist before.
• Zooming into the data or out is a "mouse-down - move - mouse-up" sequence away --- this involved fiddly command lines before!
• Switching a given dataset on or off is a click or ctrl-click away!

Of course there are minor niggles:

• I have not found out yet how to calculate the dew point from different temperature and humidity time series.
• The cross hair is actually lacking the /cross/ feature.
• Unfortunately grafana currently cannot display more than 2 different y-axes on a planel. I would like to have tree, all of them standing free on the right hand side.

tl;dr: You have to start somewhere. And keep it simple!

Bartender? a Talisker 57North, please, with water at the side. Thank you so much!

Cheers,

~ew

[d] for some weird definition of modern, perhaps.