chemicaloliver

An MQTT/Websocket based Thermometer using the html5 meter tag

chemicaloliver — Sun, 05 Sep 2010 20:10:11 +0000

In this post I will show the latest in a line of mini examples using html5 and websockets – an example of how future home monitoring dashboards should be created. At present they commonly use a combination of flash dials and JavaScript canvasy bits and pieces to create representations of data obtained via ajax. This has the inherent disadvantage that the browser polls the server rather than the server pushing values to the browser when available. Also there is very little in the way of accessibility as far as canvas and flash are concerned. In this post I will show a simple example of using data via a web socket and displays it via the meter.

While nothing demonstrated here is rocket science it’s something that’s not being focused on enough, developers seem to be getting lost in the shinyness of the canvas and forgetting the built in meter functionality along with it’s inbuilt accessibility. Having said all that it is only displayed by the more bleeding edge browsers and things may pick up as the html spec matures.

HTML5 Meter

Of the new features introduced by the HTML5 spec one that particularly sparks my interest is the meter tag, in the words of the spec:

“The meter element represents a scalar measurement within a known range, or a fractional value; for example disk usage, the relevance of a query result, or the fraction of a voting population to have selected a particular candidate.”

However in simple terms (at present) it just produces a single bar meter with the length of the bar defined by the value attribute and maximum and minimum values defined thus, high and low values can also be defined which effect the colour of the bar depending on the value. They seem to have omited the possibility of dynamic content from their main definition. For browsers which don’t support the meter tag fall-back content can be inserted before the end tag.

The purpose of explaining this here is, as will most html elements, the attributes can be manipulated with javascript so here I will use a number sent via a web socket to set the value attribute of the meter (and also update the text).

Server Side

On the server I will be using pywebsocket and it’s included web server to subscribe to an MQTT broker topic which contains temperature data. The code is identical to that used in my previous example. The data could however come from any source which provides numerical data.

Client Side

The code used here again is very similar to my previous examples except values are take from the body of the message and then given to the value attribute of the meter element (using jquery). I have added some CSS to make it look a bit nicer too:









	Temperature Monitor

            
		0°
			20
		40°
	    

	    The current temperature is ° C

	

	Debug:

Moving On

Hopefully in the future developers both of browsers and web applications will make use of the meter element rather than non standard solutions especially once the html5 spec is finalised and there is (hopefully) going to be proper CSS hooks available to create more beautiful meters.

A simple MQTT to Websocket Bridge using Mosquitto and pywebsocket

chemicaloliver — Mon, 23 Aug 2010 18:07:38 +0000

UPDATE: I have now isolated the problems to just my particular configuration of apache, ubuntu and mod_python, the setup described here does work without using the standalone web server and using apache…just not on my main system!

Following on from my experiences installing and testing pywebsocket I now move to the main reason why I bothered…to create a bridge to allow a user to view a web page showing a live stream of MQTT messages.

It should be noted that despite making the effort to setup pywebocket with Apache this time I will be using the standalone web/websocket server provided with pywebsocket (standalone.py) as I’m still trying to debug why this code doesn’t work with apache!

Server Side Code

On the server it is a simple matter of writing a websocket handler file in python to subscribe to an mqtt topic then stream the messages through the web socket. Here I use the python MQTT client library included with mosquitto.

I defined the callbacks required by mosquitto then pynotify, web_socket_transfer_data(request) being where all the cool stuff happens, this function is called on connection to the web socket and then loops waiting for MQTT messages.

To use this code save it in your websocket handler folder was a file ending in _wsh.py then access it using the filename without the ending.

import mosquitto
from mod_pywebsocket import msgutil

#create global variable for the request object
#(Holds details of ws connection)
g_request = ""

#define callbacks for mosquitto
def on_connect(rc):
	print "connected"

def on_message(msg):
	global g_request
	#send mqtt message to socket using the connection specified in g_request
	msgutil.send_message(g_request, "" + msg.topic + " - " + msg.payload + "")

def on_subscribe(mid, granted_qos):
	print "subscribe"

#define callback for pywebsocket

#allows for extra handshaking
def web_socket_do_extra_handshake(request):
	pass  # Always accept.

#main web socket function called on starting connection
def web_socket_transfer_data(request):
	global g_request
	g_request=request

	#create mosquitto object
	mqttc = mosquitto.Mosquitto("python_sub")

	#assign callbacks
	mqttc.on_message = on_message
	mqttc.on_connect = on_connect
	mqttc.on_subscribe = on_subscribe

	#connect to mqtt broker on localhost
	mqttc.connect("127.0.0.1", 1883, 60, True)

	#subscribe to topic "test"
	mqttc.subscribe("test", 2)

	#keep web socket connected while mqtt is connected
	while mqttc.loop() == 0:
		pass

As already mentioned this code doesn’t work when using pywebsocket with apache – to launch the included standalone webserver launch standalone.py with the following command (when in pywebsocket/src/mod_pywebsocket/):

python standalone.py -p 8080 -d  -w

so in my case I use

python standalone.py -p 8080 -d /home/oliversmith/public_html/ -w /home/oliversmith/public_html/websock_handlers/

Front End

I used the same html file as detailed in my previous post, it’s simply a matter of browsing to the appropriate url not forgetting that it’s on port 8080 (or whatever port you define) to avoid conflicting with apache.

To send test MQTT messages I used the Java Client included in the ia92 package provided by IBM as pictured.

If anyone knows why this doesn’t work with apache I would appreciate them telling me! At the moment I suspect it’s some privileges thing but I may be wrong

Getting started web sockets using pywebsocket, mod_python and apache in Ubuntu

chemicaloliver — Sun, 22 Aug 2010 13:30:23 +0000

Since my initial posts on websockets I’ve moved to pywebsocket, this is a python based project which provides a websocket extension to apache via mod_python. It can also run a standalone web server/websocket server if required.

I made the move from node.js based systems as I was more comfortable developing using python and apache (even though I don’t know much python) and also most of the other systems I wish to integrate work well with python. I also feel that using a python extension to apache is a more sensible solution which is able to work on most webservers without major reconfiguration.

Installation

The install process for apache, mod_python and pywebsocket is quite trivial, I followed instructions written by Travis Glines, care should taken to make sure you use the correct apache config files, I didn’t and as a result spent ages wondering why it didn’t work!

A websocket compatible browser is also required (i.e. daily build of Firefox 4, Chrome/Chromium v 6.0.414+) earlier versions of chrome can be used however they use an older handshake protocol not enabled in pywebsocket by default, if you wish to use an older browser the you need to add:

PythonOption mod_pywebsocket.allow_draft75 On

to the appropriate apache conf file (the same one you added all the other stuff to!).

Testing

pywebsocket comes with an example in /src/example however these are aimed at the standalone mode, however the supplied handler echo_wsh.py can be used to test the apache version.

Place echo_wsh.py in the websocket handler folder defined during installation.
Check it has permission to be executed

Use the following html/JS to connect to your new handler, it will connect then on clicking the window will send a message with string “Click!” down the websocket and display anything returned, in this case the sent message is echoed so you should see “Click!” appended to #msg on every click:

Once that’s working you can go on to write your own handlers with the filenames being suffixed with _wsh.py eg: ws://localhost/foo will point to foo_wsh.py

Moving On

I’ve already got a few ideas in mind for what I’d like to implement using web sockets although at the moment debugging _wsh.py files is proving very tricky as there is no simple debugging available and I’m working blind with a minimal knowledge of python!

First steps using Python and MQTT (using pynotify on Ubuntu)

chemicaloliver — Sun, 15 Aug 2010 21:43:21 +0000

Until recently developing MQTT clients in anything but perl was a little tricky due to a selection of badly documented, restrictively licensed client libraries (in PHP, Java and C). After a lot of hard work on the part of Roger Light (@ralight developer behind the OSS MQTT Broker Mosquitto) there is now another solution available.

An MQTT pubsub client library written in C is now included along with a basic python wrapper. To me this greatly improves the situation as it now means it’s possible to quickly and simply develop real applications or web applications without wrestling with some of the more complex APIs traditionally used. After attempting to get to grips with the IBM ia92 java client library this was a breath of fresh air.

As an initial toe in the water exercise I looked at the included sub.py script (after compiling and installing mosquitto 0.8.2) and wrote the code below so that all messages on the “test” topic display a notification in ubuntu. While only a minor coding exercise this is a potentially useful application when MQTT is being used for monitoring systems.

Note: Recently others have noticed problems with installation of the python libraries, if you have such problems check to ensure that the appropriate files are in /usr/local/lib/python2.6/dist-packages/

To enable me to display notifications simply I chose the built in Ubuntu notifications system which can be hooked into with pynotify. Notifications are displayed as shown above.

Control of mosquitto is a simple matter of defining callbacks for the required events, in this case just connection and receiving a message.

The code is shown below and is really self explanatory:

#!/usr/bin/python

import pynotify
import mosquitto

#define what happens after connection
def on_connect(rc):
	print "Connected"

#On recipt of a message create a pynotification and show it
def on_message(msg):
	n = pynotify.Notification (msg.topic, msg.payload)
	n.show ()

#create a broker
mqttc = mosquitto.Mosquitto("python_sub")

#define the callbacks
mqttc.on_message = on_message
mqttc.on_connect = on_connect

#connect
mqttc.connect("", 1883, 60, True)

#subscribe to topic test
mqttc.subscribe("test", 2)

#keep connected to broker
while mqttc.loop() == 0:
	pass

Hopefully this short illustration will prove useful to those just getting started. This won’t be the end of working with mosquitto libraries…now just to work out python…

When web sockets won't work (Web socket handshaking updates)

chemicaloliver — Fri, 13 Aug 2010 12:01:49 +0000

This post serves as a update to my previous post: Getting started with Node.js and Web Sockets on Ubuntu 10.04 In which I stated various problems I had encountered. In this post I will explain why most of them occurred. In this post I refer mostly to the server side handshake, the client side processing is a 45 step process which looks pretty nightmarish to me!

After more in depth research I’ve discovered the cause of many the examples I tried in my previous post. On the 6th June 2010 the handshake for the web sockets protocol was changed and broke backwards compatibility with earlier implementations. This means that now you have to ensure the browser version you are using supports the same handshake as server. I suspect due to me not knowing this that many examples failled to work as I sporadically switched between builds of chromium and node.js – to guarantee support you need to use the latest version of Node.js and Chromium/Chrome 6.0.414.0 or above.

I also attempted to use the Arduino websockets library without success which I now find uses an old protocol, it is my intention to attempt to remedy this when I have time.

The new server > client handshake was summerised by Makoto in 7 steps:

Extract numbers at Key1(eg: 4 @1 46546xW%0l 1 5) and concatenate them
Count number of spaces at Key1
Devide #1 by #2
Change the format of #3 into “big-endian 32 bit integer”
Repeat #1 by #4 for Key2(eg: 12998 5 Y3 1 .P00)
Concatenate #4, #5, and the body(eg: ^n:ds[4U) of the request
Digest the result in MD5 format

The new server handshake is not complex but is a bit heavyweight for microcontrollers. A nice example of implementation of this in PHP can be found here:

http://webreflection.blogspot.com/2010/06/websocket-handshake-76-simplified.html

and the full, latest details of the handshake can be found here:

http://www.whatwg.org/specs/web-socket-protocol/ (currently on page 6)

I have mixed opinions on this update, on one hand it means the web sockets protocol is no longer strictly initiated by an http GET request as it has extra data in the header, on the other hand it seems sensible to incorporate some kind of validation mechanism for clients. What it may mean for Arduino implementations of the protocol I don’t yet know, it is my gut feeling that the server part may be too complex to easily implement. It is less likely for an Arduino client to be required but I would not like to attempt implementing the 45 step handshake required for a client on an Arduino.

Rather than using Node.js I have moved to using pywebsocket for my experiments as it seems to be used by the Google team working on web sockets and thus is very quickly updated in line with the newest developments. So far I have tried using both Chromium and Firefox 4 daily build as clients.

Getting started with Node.js and Web Sockets on Ubuntu 10.04

chemicaloliver — Tue, 03 Aug 2010 21:06:48 +0000

Having spent the past few hours taking my first steps into the glorious unknown world of web sockets I’ve drawn a few conclusions about web sockets:

The principals are very easy
It is an area of great interest and development
There are seeming 101 different implementations many of which are out of date and broken

So this post I want to document a simple demonstration of the use of web sockets with node.js which works now (August 2010) to serve as simple demonstration to those wanting to dip their toe in the water before going further. Many examples and demos were outdated and failed to run for a variety of reasons, mainly due to node.js being updated and the examples not being changed to reflect this. I did all this on ubuntu but there is no reason why it shouldn’t work on any other Linux system.

Requirements

For this demonstration you will need:

A computer running a recent version of linux
A web socket compatible browser (probably chrome/chromium v4 plus)
Appropriate compilers (Obtained on ubuntu/debian using apt-get install build-essential)
Git (Obtained on ubuntu/debian using apt-get install git-core)
A basic idea about compiling things, some basic JavaScript or a healthy sense of adventure!

Getting Node.js

Node.js is an event driven JavaScript framework for building server side applications such as web servers and web socket servers. A large proportion of web socket examples use it as a base for the server component of the system so the first thing we need to do is get it and compile it:

Get the latest code by cloning the git repo:
```
git clone git://github.com/ry/node.git
```
This will get the code for node.js and place it in a folder called node.
Now we’ve got the code we need to compile it which should go smoothly:
```
cd node
./configure
make
sudo make install
```
Now, assuming everything went well node.js will be launched if we type node in the terminal, if compiling hasn’t worked it’s probably worth looking at the output from the ./configure command as there may be some missing dependencies….I was fine though!

When required it can be invoked by

node