Google Reader API (RSS) to Instapaper - Now on GitHub

I've been meaning on starting to use GitHub for my project and today I've finally commited my first project.

I wrote the Google Reader to Instapaper ages ago (see the original post for more details), but couldnt really make the full code available as my passwords etc were hard coded.

I spent a couple of hours refactoring the code yesterday and pushed it out to GitHub - if you're interested the full code can now be found at GitHub

The Raspberry Pi and Wireless RF (XRF) Temperature Loggers

Remote Temperature Logging - Why?

In our house one room has three exposed wall, as well as two double width windows fitted (one on each side of the room).  In the winter, this is a cold room. Very, very cold.

We're in the process of getting the central heating and double glazing replaced (they're both over 15-20 years old respectively) and are very ineffecient - I was however wondering what improvement we'd see once this work had been done. Really though what I wanted was an excuse to use as the basis for a project with my Pi, which I'd bought over 8 months ago and hadn't yet gotten aaround to playing with.

A Pi, wireless communication, python, Linux, soldering...now this sounded like fun!

First I searched for what I could use for this - what could I use that captured and could transmit temperature information - and most importantly, what could do this in a wireless way?

I found the answer using a combination of 2 Slice of Pi's, a 2 Temperature XRF Sensor's and 3 XRF Wireless RF Radios, all of which I got from CISECO.

The documentation of the above is pretty decent, but there were still a number of threads you have to tie together to get all of this to work, so here we go:

The XRF that will sit on the Pi will act as a pass-thru receiver, where any data received from the Serial line will be broadcast on PANID, and for our purposes, any data received on PANID or PANID2 will be sent to the Serial line. (more on PANID later).

The remote sensors XRF will need to be configured as Thermistor "personality" - this will capture the temperator from the onboard thermistor and transmit it.

Out of the box the XRF modules will have the serial pass-through personality, so the first thing we have to is upload the thermistor firmare to the two XRF modules that will be used for temperature capture.

At a high level, you'll need to do the following:

• Configure the Pi so that you can use the Serial Port
• Install pySerial onto your Pi
• Get the XRF Uploader downloaded and compiled onto your Pi
• Download the latest XRF Thermistor firmware
• Upload the Thermistor firmware to each of the remote modules
• Configure the remote devices
• Write software to read what's being sent!

Configure the Pi Serial Port

By default the stock raspbian (and debian I think) images use the UART for serial console communication - we need to change this so that we can use the serial port.

Note: For each of the system files modified here I suggest you back them up first, in case you need/want to roll the changes back

Modify /boot/cmdLine.txt to remove references to ttyAMA0 - this is the serial port

sudo vi /boot/cmdLine.txt

I needed to remove the following lines:

console=ttyAMA0,115200 kgdboc=ttyAMA0,115200  

and my cmdLine.txt now looks like this:

dwc_otg.lpm_enable=0 rpitestmode=1 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait

Next we need to edit /etc/inittab:

sudo vi /etc/inittab

and comment our (or remove) the following line:

T0:23:respawn:/sbin/getty -L ttyAMA0 115200 vt100

Install pySerial 

I used pySerial for my final program, but I also found it useful to use the included miniterm.py serial terminal for configuration and debugging.

Navigate to where you'd like to install pySerial - I'll assume you're installing into your home directory for now (/home/pi or the shorthand ~)


Download pySerial 2.5 (and use Python 2.x, not Python 3) from here and copy it to /home/pi (or your preferred location)
 

cd ~
gunzip pyserial-2.5.tar.gz tar - xvf pyserial-2.5.tar
cd pyserial-2.5
sudo python setup.py install

You can test your installation by doing the following:

pi@raspberrypi ~ $ python
Python 2.7.3rc2 (default, May  6 2012, 20:02:25)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import serial
>>> ser=serial.Serial('/dev/ttyAMA0',9600)
>>> ser.inWaiting()
0

If you don't get any errors with this you're probably ok - your serial port is ready for your use and pySerial is install correctly.

Thermistor Firmware - Download and Prepare

You need to do this on the remote sensors.

You can, optionally, also do follow the same steps for the local XRF module if you want to upload the very latest serial pass-through firmware too.  The same steps, but you'll download/upload the serial pass-through firmware instead.

The official process can be found on the OpenMicro site, but here are my abbreviated steps:

Note: All of these steps are to be done on your Pi

• Get the XRF Uploader from here: http://github.com/CisecoPlc/XRF-Uploader
• Copy this to your PI and run the following on it  g++ xfr_uploader.cpp -o xrf_uploader
• chmod +x xrf_uploader
• Download the appropriate firmware - you want the latest llapThermistor-XXX version

Firmware Upload

Ensure your Slice of Pi is fitted to your Pi first.

A Raspberry Pi, a Slice of Pi and an XRF module

For each of the XRF modules you want to update (the remote modules and optionally the local one, if you want to update the serial pass-through firmware), do the following:

• Switch off your Pi (you can do this by hot-swapping I've heard, but I wouldn't recommend it )
• Plug in your XRF module into your Slice of Pi
• Switch your Pi back on
• ./xrf_uploader -d /dev/ttyAMA0 -f llapThermistor-XXX

If you see something along the lines of:

pi@raspberrypi ~/xrf_loader $ ./xrf_uploader -d /dev/ttyAMA0 -f llapThermistor-V0.50-24MHz.bin
Writing new firmware file llapThermistor-V0.50-24MHz.bin to device /dev/ttyAMA0 with baud rate 9600...
Reading firmware file...
Read 1162 lines from firmware file
Opening device...
Setting serial parameters...
Waiting for device to settle...

<> Entering command modus
Timeout, no data received within 10 seconds

Either retry again, or switch off and reseat the XRF, and retry the process again.

A successful upload should look as follows:

pi@raspberrypi ~/xrf_loader $ ./xrf_uploader -d /dev/ttyAMA0 -f llapThermistor-V0.50-24MHz.bin
Writing new firmware file llapThermistor-V0.50-24MHz.bin to device /dev/ttyAMA0 with baud rate 9600...
Reading firmware file...
Read 1162 lines from firmware file
Opening device...
Setting serial parameters...
Waiting for device to settle...

<> Entering command modus
<- OK
-> ATVR
<- 0.63B XRF
<- OK
-> ATPG
<- OK
<> Sent 1162 of 1162 lines...

All OK, XRF successfully reprogrammed!

Waiting for device to settle...

<> Entering command modus
<- OK
-> ATVR
<- 0.50B APTHERM
<- OK

Optional! Set the PANID

The PANID is the ID that the XRF network will communicate on - if you leave it do the default everything will work just fine - I've changed mine mostly for security.  I'm probably just being over cautious - if you leave yours on the default PANID every else documented here will still work.

Note: If you do decide to set the PANID then the best time do to this is probably immediately after the firmware upload, and note too that you'll need to set the same PANID for every XRF module in your network

I'm using miniterm.py (included with pySerial, found under /pyserial-2.5/examples/miniterm.py) for these steps:

pi@raspberrypi ~ $ python ~/pyserial-2.5/examples/miniterm.py /dev/ttyAMA0
--- Miniterm on /dev/ttyAMA0: 9600,8,N,1 ---
--- Quit: Ctrl+]  |  Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

(Note: Entering Ctrl+T and then Ctrl+E will enable local echo, which can be useful when debugging)

The XRF should to each command with respond with "OK":

+++<no return>
ATID[4 hex chars, up to a max of EFFF]<return>
ATAC<return>
ATWR<return>

To Check the ID (but note, the output may be partly overwritten with OK):

Wait for 5 seconds
+++
ATID<return>

Test our new XRF network!

First a note on the two type of communication protocols available with the XRF modules - one is the the one we've used up until now, which the AT command mechanism and is the "direct", plugged in protocol.

The second is the over the air LLAP protocol - this is what we'll use to communicate between the XRF modules.

See here for the AT command reference and here for the LLAP command reference documentation.

Plug your remote XRF modules into the Thermistor boards - below are mine, one bare and one in the included enclosure (with holes drilled in by me for the XRF aerial and thermistor):

XRF Thermistor Modules

Ensure your Pi is setup with the Slice of Pi and your "local" XRF module plugged in.

Log into your Pi and using the serial terminal of your choice (I'm using miniterm.py) hop onto the serial port:

python ~/pyserial/pyserial-2.5/examples/miniterm.py /dev/ttyAMA0

Now plug the battery into one of your remote sensors (note: you probably want to hook up to a 3.3V power supply while testing as permanently on use while testing will drain the small 3V cells included in the remote modules really quickly)

You should see the following in your terminal:

a--STARTED--a--STARTED--a--STARTED--a--STARTED--a--STARTED--a--STARTED--

A quick note on the LLAP protocol (please see the command reference above for the full documentation), taken from the OpenMicro site:

The message is 12 characters long and in 3 distinct sections. To illustrate the 3 separate parts to the message, see this example:

aXXDDDDDDDD

1.  "a" is lower case and shows the start of the message
2.  XX is the device identifier (address A-Z & A-Z)
3.  DDDDDDDDDD is the data being exchanged.

So the a--STARTED message above is letting us know that the remote XRF module is talking to our local one - yay! Now lets see if we can communicate back to the remote XRF module:

Enter a--HELLO---- into your terminal - the remote XRF module should respond with HELLO - you can now communicate both way - fantastic.

Lets set the device ID of the remote XRF module - this is optional but if you're planning on more than 1 remote module I'd recommend it as you'll then be able to identify which module is talking.

In your serial terminal, issue the following commands:

a--CHDEVID[A-Z][A-Z]     (for example, a--CHDEVIDZZ will set the device ID to ZZ)
a--REBOOT---

The device should respond with the started messages again, but with the new ID:

aZZSTARTED--aZZSTARTED--aZZSTARTED--aZZSTARTED--aZZSTARTED--aZZSTARTED--

(assuming you used ZZ as your device ID)

Lets test it again:

Enter aZZHELLO---- into your terminal - the remote XRF module should respond with HELLO once more - you can now communicate using the new ID.

Finally, lets test that we can get a temperate reading back (which after all is what we're really after here!):

Enter the following command into your serial terminal:

aZZTEMP-----

You should get the temperature in Celcius back, along the lines of:

aZZTMPA18.78

You can get the thermistor module to send this periodically too, which is useful while testing:

aZZINTVL005S - this will get the device to send the temperature every 5 seconds

(the interval is 1 to 999 and the periods can be S(seconds), M(minutes), H(hours), D(days))

Once you're finished testing you can save the interval setting and make the device sleep between intervals by issuing the following command:

aZZCYCLE----

In addition to sending out the temperature at the set frequency, it will also send out the battery level every 10 cycles, which is useful.

Note however, that once you do this you'll be unable to communicate to the device while its sleeping, so only do this once you're happy that your network is working as desired (if you do however get it into a cycle and you decide you want to change the interval you can (again from the OpenMicro site) Ground pin 6 - this will force the device to be awake, you can then issue commands to change to interval or other settings. Note that unless you tell the device to stay awake (axxWAKE-----) then it will resume sleep as soon as you remove the ground from pin 6.)

Repeat the above exercise for each remote device - one at a time.

 

Write software to read what's being sent!

Below is a small program that will poll the serial port every minute for new messages (for this example I've set the XRF modules to send out a response once every 5 minutes):

import serial
from time import sleep, gmtime, strftime

DEVICE = '/dev/ttyAMA0'
BAUD = 9600

print (strftime("%a, %d %b %Y %H:%M:%S: Starting\n", gmtime()))

ser = serial.Serial(DEVICE, BAUD)
while True:
    print("%s: Checking..." % strftime("%a, %d %b %Y %H:%M:%S", gmtime()))
    n = ser.inWaiting()
    if n != 0:
        msg = ser.read(n)
        print("%s: %s" % (strftime("%a, %d %b %Y %H:%M:%S", gmtime()), msg))
    sleep(1 * 60)

An example of the output I get is:

Sat, 09 Feb 2013 21:00:14: Checking...
Sat, 09 Feb 2013 21:01:14: Checking...
Sat, 09 Feb 2013 21:01:14: aAATMPA3.242aABTMPA12.45
Sat, 09 Feb 2013 21:02:14: Checking...
Sat, 09 Feb 2013 21:03:14: Checking...

Note that I have two devices sending me data here, AA and AB.

That's it - I hope you find this useful!

iTunes blows

I've never really liked iTunes but I'm increasingly growing to loathe it.

This months issues:

• books dont fully sync to my iPad (some do, others get a black bar across the bottom and thats that - the book cant be opened)


• certains movies are copied across...every...single...time...

Sigh.

Dive Into Python

I recently finished Dive Into Python.

I was looking for a book that would provide a decent introduction to Python without having to wade through 3 to 4 "programming concepts" chapters - this fit the bill perfectly.

I'll probably be buying a follow up to cover advanced Python topics but as an introduction I can certainly recommend this book.

Open University MST121 Done!

I got the results of this module today - an overall average of 87%.  This inst what I was hoping for but I guess homelife, work and work related study got in the way!

Passing this gives me a Certificate in Mathematics (C46) which is nice, but I was really aiming for a BSc in Mathematics. I'll probably have to put that on hold indefinitely due to the recently introduced deadlines and price hikes. I would now need to complete my degree by 2017 (to be replaced by an equivalent degree) but as I was only doing 1 module a semester this is not really achievable.

After this date the fees switch to £1250 per module (so £5000 per year) and this is more than I'm willing to spend on something I was doing primarily for fun.

I understand that the funding has to come from somewhere after government funds were cut - I just wish they'd done it after 2017!

New Years Resolution

It's a little late, but I've finally decided on a New Years resolution: I'm aiming to read 24 books this year, 2 a month, one fiction one non-fiction.

Given that I used to devour books this should be achievable, but that was before kids...

So, for January:

• Credit Derivatives: A Primer on Credit Risk, by George Chacko et al


• The Subtle Knife, by Philip Pulman

I have a decent handle on Credit Risk gained over the last few years working in investment banking but wanted to gain a deeper understanding, especially around the models used.

The Subtle Knife is the sequel to Northern Lights which I finished last year. I confess I wasn't as blown away as I thought I'd be given all the positive reviews I'd heard but have decided to give the series another chance in the hope that it improves. Here's hoping!

Simple Property Retrieval/Storage with Google App Engine

After trying a few of the examples for storage in GAE I think that the simplest mechanism is as follows
First, create a key for your application data:

Key appDataKey = KeyFactory.createKey("AppData", "MyApp");

AppData and MyApp aren't fixed - change them according to your needs.

Saving a property:

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    
Entity invoiceEntity = new Entity(appDataKey);
invoiceEntity.setProperty("", );
datastore.put(invoiceEntity);

Reading a property:

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
try {
    Entity appData = datastore.get(appDataKey);
    return ((Long) appData.getProperty("")).intValue();
} catch (EntityNotFoundException notFoundException) {
    // handle the exception
}

In this case my property is a Long, but it can be any Object. You'll need to cast accordingly.
Note that for my purposes I don't need to worry about either load or concurrency, so the above example may not be suitable for all use cases, but it was the simplest option for me.