How to read out your smart gas meter with a raspberry pi
Quick guide on how to setup a raspberry pi to read out a smart gas meter and setup a website to plot gas consumption over time.
In the Netherlands, most houses are heated with natural gas and most people use gas for cooking. Over the last 10 years, the majority of houses switched to smart meters for gas and electricity. These meters make it possible for suppliers to remotely read out your consumption. Unfortunately, less than 10% of home owners actually read out their gas meter themselves. This is a shame as it could help reduce consumption.
I have a couple of raspberry pi’s laying around and I took up a project to read out my smart gas meter and show the readings in a graph on a website. I used AWS IoT core to process the messages from the smart meter and AWS dynamoDB to store the readings. Finally, the website is hosted on AWS and protected by AWS Cognito (not everyone needs to know how much gas I use). I assume you have some affinity with raspberry pi, Python and AWS to keep this quick guide relatively short.
The end result looks like this:
Lets get started
Step 1: Read out the smart gas meter with Python
Each smart meter comes with a P1 port (one of these old telephone connections), which you can connect to USB with the right cable. I used a pi zero without wifi, so had to use an USB to RJ45 ethernet adapter (full set up is shown in the picture above, the black cable is the P1-to-USB connector).
To read out the meter I used this Python script (heavily borrowed from http://gejanssen.com/howto/Slimme-meter-uitlezen/). Each meter type has a different message layout, which means you need to try different values in row 29 and row 43.
Next step is to setup a messaging service to push the readings to AWS IoT core.
Step 2: Setup AWS IoT core
AWS IoT core provides secure communication between internet-connected devices such as sensors, actuators, embedded micro-controllers, or smart appliances and the AWS Cloud. It uses the MQTT machine-to-machine connectivity protocol. To setup a new thing, follow these steps in the AWS IoT core developer guide.
Download the things’ certificates and store them on your pi. You also need to download the AWS IoT Python SDK. To send messages to AWS IoT core I enhanced the previous script with a MQTT messaging script, see below. This script sends a message with the gas meter’s reading to a so called ‘topic’. Within AWS IoT core you can subscribe to this topic and setup rules to process the incoming messages. Before we do that we need to create a dynamoDB database.
Step 3: Setup dynamoDB and a processing rule
As I want to plot a historic trend of gas consumption, I need a database. For this project, a dynamoDB is very suitable; It is easy to set up and cheap. Simply create a new table with datetime as the primary key. If you have multiple things you can use deviceID as primary key and datetime as sorting key and store everything in a single table.
Now that we have a database, we need to set up a rule in AWS IoT core to process incoming messages and push them to the database. In my case the rule is quite simple: send all messages from topic gas_reading to the dynamoDB database gas_meter. You can do this in the Act section of AWS IoT core.
Step 4: schedule python script with crontab
Now that the ‘pipeline’ is setup, the python script can be scheduled at the desired interval. The easiest way to do this is by using crontab. On your pi, run crontab -e and add the following line at the bottom. This will run the script every 5 minutes.
Now, every 5 minutes a reading is taken from the smart meter, published to a topic in AWS IoT and pushed into the dynamoDB.
Step 5: Setup a website to plot the graph
The last step is to plot the values in a graph on a website. I am pretty new to front end development, so bear with me on this one. I assume you have some affinity with node.js as you need to install two modules. The first one is chart.js, a powerful module for creating nice graphs and the second one is moment.js as you are working with date time values and these can be quite tricky. To be able to connect to the dynamoDB in AWS you need to add a link to the AWS SDK script (through CDN).
The most bear bone version of the html script is shown below.
The function to create the graph is shown below. The most tricky part is to work with AWS credentials as you don’t want everyone to just query your cloud-based database. For local development it’s fine to hard code AWS credentials (Access key ID and Secret access key ), but on a public website you need to secure access much more. I used AWS Cognito for that.
To setup AWS Cognito you need to define three things:
- user pool — who can login
- app client — to which application do they have access
- identify pool — which resources within AWS can be accessed for this application by these users
Within AWS, go to Amazon Cognito and click on ‘Manage User Pools’ first.
Give your user pool a name and proceed with entering the details. Note down the user pool ID as you need this later.
Once setup, go to ‘Users and Groups’ in the menu on the left and create a test user (using a working email address).
The next step is to define an App Client. Do this by clicking on App client settings on the menu on the left. Note down the App client ID as you need this later.
As part of the app client setup, you need to define sign in and sign out URLs. For development purposed you can use localhost on http, but all other URLs have to use https (keep this in mind when setting up your website).
As a final step in setting up the app client, you need to define the domain which is needed in the authentication process.
This sounds like a whole lot of work, but things make sense as soon as you integrate this with your website.
In order to trigger the authentication process you can define a button on your website with a URL that combines the parameters that you defined above.
The login URL is setup as follows:
https://<your domain prefix>.auth.<your region>.amazoncognito.com/login?client_id=<your app client id>&response_type=token&scope=aws.cognito.signin.user.admin+email+openid+phone&redirect_uri=<your callback url>Once logged in, you want to be able to access the dynamoDB with the gas meter readings. You do this by setting up a so called Identity Pool (i.e. Federated Identity).
Next step is to give the identity pool a name and specify the options (defaults work fine).
In the next screen take note of the identity pool ID (you need it later) and create new IAM roles for authenticated and unauthenticated users. When creating the roles, you need to specify to which AWS services they have access and to which extent (e.g. read or write). In this case, you want users to have read access to the dynamoDB table containing the gas readings.
Once this is setup, authenticated users get credentials to query the dynamoDB with the gas readings.
Now update the chart.js file for the website by entering the IdentityPoolId and userPoolId in the correct places. The end result will be the graph I showed in the beginning of the article.
The working website can be found at myhome.schrama.io. You will find some other features there as well, which I will write about in a next article.
You can find the full code on GitHub:
Summary
In this article I talked you through how you can use a Raspberry Pi to read out a smart gas meter, push the readings to a dynamoDB using AWS IoT and visualize the readings on a website with credentials.
For me this was a nice project to learn more about Python, Raspberry Pi, AWS, databases, IoT and front end development. I also learned a lot about our gas consumption at home. By being more conscious about how much we consume, we now limit the time we shower and set the thermostat a few degrees lower. So, something positive for the environment as well.
I hope you enjoyed reading this article and got enthusiastic to pick up a project like this yourself!
