I have been giving some thought to making the process of creating maps form OpenStreetMap data easier. The problem is that to use the mapnik map renderer, you need to install quite a lot of tools and set different configuration files before you can even start to create a map.
This means you have to be pretty keen to persevere enough to get to an end result.
I am thinking of creating a service to allow you to bypass the tool installing bits, and concentrate on the setting up of the map configuration, so you can get to a result quicker - see http://wiki.openstreetmap.org/wiki/User:Grahamjones#Speciality%20Maps.
Quite a few of the responses that I got to that proposal were about the lack of good documentation to get people started on the process. I have had a go at addressing this by putting together a simple overview presentation to go through the main concepts and tools that you need to generate a map using mapnik. The presentation is on slideshare.net, and should be visible below:
Thursday, 24 February 2011
Friday, 18 February 2011
Mounting FTP Directory as a Filesystem
I just discovered a really useful trick to mount the directory structure of a remote FTP server as a directory on a linux computer. You use a program called curlftps. On Ubuntu do:
A solution is to create a .netrc file in the /root directory containing:
sudo apt-get install curlftpsThe directories on the remote ftp server are then mounted using:
curlftpfs ftp://[username]:[password]@[server address] [local directory mount point]This works fine (you do not even need to be root to do it), but the password will be present if you do 'ps -ef | grep ftp'.
A solution is to create a .netrc file in the /root directory containing:
machine [server address]
You can then add the file system to /etc/fstab by adding the following line to /etc/fstab:login [username]
password [password]
curlftpfs#[server address] [local directory mount point]You can then just dofuse allow_other,rw,user,noauto 0 0
sudo mount [local directory mount point]and you get the same effect with the server password invisible.
Sunday, 13 February 2011
OpenWRT Based Utility Meter Monitor
I have made some progress porting my MeterServ application (http://meterserv.webhop.net) to openWRT. I have managed to work out how the openWRT makefile system works enough to add it to the build system and produce an openWRT package for the MeterServ application.
To do the improvements that I want to make to the application though, it needs more work. In particular I don't like Perl, so want to convert the web interface part of it to Python. Rather than go for a completely DIY web framework, I have adopted webpy (http://webpy.org) - this seems like a simplified version of django - you define URLs that you want the application to work with, then point it to a python class that does the processing associated with the user requesting a particular URL.
I have started a seperate google code project for meterserv, because a couple of people expressed an interest in it. I have put the latest version of the software in its repository too - see http://code.google.com/p/meterserv.
To do the improvements that I want to make to the application though, it needs more work. In particular I don't like Perl, so want to convert the web interface part of it to Python. Rather than go for a completely DIY web framework, I have adopted webpy (http://webpy.org) - this seems like a simplified version of django - you define URLs that you want the application to work with, then point it to a python class that does the processing associated with the user requesting a particular URL.
I have started a seperate google code project for meterserv, because a couple of people expressed an interest in it. I have put the latest version of the software in its repository too - see http://code.google.com/p/meterserv.
Saturday, 29 January 2011
Utility Meter Monitor
I have had a simple utility meter monitoring system working in my parents and my house for a few years. It uses an NSLU2 single board computer connected to a simple USB input/output interface, as described at http://meterserv.webhop.net.
It worked fine for at least 3 years, but started to misbehave recently. I have not been able to determine if it is a hardware or software fault - I suspect that I may have worn out the USB memory sticks that I used for the root filesystems.
We have gone for a hardware change to see if that is the problem, but I want to develop the software a bit more too.
Future developments include:
It worked fine for at least 3 years, but started to misbehave recently. I have not been able to determine if it is a hardware or software fault - I suspect that I may have worn out the USB memory sticks that I used for the root filesystems.
We have gone for a hardware change to see if that is the problem, but I want to develop the software a bit more too.
Future developments include:
- Add support for cheap electricity meters (e.g. OWL USB energy monitors).
- Add support for temperature sensing.
- Better web interface (the current one uses perl based CGI scripts, which are awful to maintain).
- A simple (end user friendly) software upgrade process.
I want to get rid of the perl cgi scripts because I don't really speak perl, so they are hard to follow - will change them for an ajax based front end with python server side scripts.
The first stage is to get a sofware development environment working. The original version used the nslu2-linux slugos operating system. This has been changed quite a bit since I used it, so I decided that as I will need to do a complete re-build, I will standardise and use OpenWRT for all my little embedded system projects.
Compiling OpenWRT for the nslu2 was nice and easy - in the menuconfig just select the processor as an ixp4xxx and the sub-type as nslu2.
Edit target/linux/ixp4xxx/base_files/etc/config/network to give a default network configuration that will work on your system (the openwrt default ip address is 192.168.1.1, which clashes with my router....).
This creates a file called bin/ixp4xxx/openwrt-nslu2-squashfs.bin which can be flashed onto the nslu2 as follows:
- Put the NSLU2 into upgrade mode by switching off, then powering on while holding in the reset button for about 10 seconds. When the status LED changes colour (at about 10 sec), release the reset button. The status LED now flashes different colours (very subtly different colours if you are colour blind like me!).
- Use upslug2 (sudo apt-get install upslug2 on Ubuntu) to upgrade the NSLU2 using: upslug2 -d wlan0 --image=openwrt-nslu2-squashfs.bin
- The NSLU2 will re-boot, but then take a while (a few minutes) to set up a writeable overlay file system before being accessible over the network.
This showed that I could build software for the NSLU2 using openwrt, but now I need to port meterserv to OpenWRT, which will be a separate post....
Sunday, 23 January 2011
OpenWRT on Linksys NSLU2
I have had a couple of NSLU2's running a little application to monitor utility meter readings for a few years now (meterserv). It worked fine until the autumn when my Dad's one stopped recording readings. I have not been able to find out why - it looks ok. It may be that the interface card has broken, or I do wonder if I have worn out the USB flash drive that I am using for the root filesystem.
In my inability to find a fault, I decided to do a clean install on a usb disk to rule out software and flash drive problems. They had been using OpenSlug (now called SlugOsBE) from nslu2-linux, but it is now very out of date, because I have not updated them for a few years. This means that all the links to package directories are broken etc.
Given the recent success in building OpenWRT for the bifferboard, I decided to use OpenWRT for the re-build. These are my notes so I can do it again next time....
In my inability to find a fault, I decided to do a clean install on a usb disk to rule out software and flash drive problems. They had been using OpenSlug (now called SlugOsBE) from nslu2-linux, but it is now very out of date, because I have not updated them for a few years. This means that all the links to package directories are broken etc.
Given the recent success in building OpenWRT for the bifferboard, I decided to use OpenWRT for the re-build. These are my notes so I can do it again next time....
- In the MenuConfig system, set the target system to 'Intel IXP4xx', sub target to 'Generic' and target profile to 'Linksys NSLU2'.
- Because this is not my new router, I do not want it to use the default IP address of 192.168.1.1. This can be changed in the 'Image Configuration' bit of the configuration menu.
- I would like to re-code some of the meterserv software in python, so I need python and pyusb installed in the flash image - these were selected from the 'Languages' section of the openwrt menu.
- Make builds a firmware image in bin/ixp4xxx called openwrt-nslu2-squashfs.bin, which is an 8MB image - there is a 16MB one too, but I am not convinced that my nslu2 has that much flash memory.
- You have to put the nslu2 into 'recovery mode' by holding the re-set button in while you power up the machine - you have to release the reset button as soon as the status LED changes colour, after about 10 seconds. The status LED then flashes a couple of different colours.
- Flash the new firmware image onto the NSLU2 using upslug2 (sudo apt-get install upslug2 on ubuntu) - the command line was sudo upslug2 --target 00:14:bf:64:df:be --image openwrt-nslu2-squashfs.bin.
- The machine now appears on the network with the right IP address. Telnet into it and change the root password, then connect using SSH instead.
I have had a bit more of a look at it. There are two surprising things. The first is that if I write a simple 'hello world' script and execute it, it works, so it is just interactive python that is broken.
The second is that doing "ldd /usr/bin/python" on the nslu2 and on the bifferboard give very similar results, except that the bifferboard one is linked against libgcc and libc, but the nslu2 one is linked against libc.
I think this is telling me something, but I am not sure what....
OpenWRT on Linksys NSLU2
I have had a couple of NSLU2's running a little application to monitor utility meter readings for a few years now (meterserv). It worked fine until the autumn when my Dad's one stopped recording readings. I have not been able to find out why - it looks ok. It may be that the interface card has broken, or I do wonder if I have worn out the USB flash drive that I am using for the root filesystem.
In my inability to find a fault, I decided to do a clean install on a usb disk to rule out software and flash drive problems. They had been using OpenSlug (now called SlugOsBE) from nslu2-linux, but it is now very out of date, because I have not updated them for a few years. This means that all the links to package directories are broken etc.
Given the recent success in building OpenWRT for the bifferboard, I decided to use OpenWRT for the re-build. These are my notes so I can do it again next time....
In my inability to find a fault, I decided to do a clean install on a usb disk to rule out software and flash drive problems. They had been using OpenSlug (now called SlugOsBE) from nslu2-linux, but it is now very out of date, because I have not updated them for a few years. This means that all the links to package directories are broken etc.
Given the recent success in building OpenWRT for the bifferboard, I decided to use OpenWRT for the re-build. These are my notes so I can do it again next time....
- In the MenuConfig system, set the target system to 'Intel IXP4xx', sub target to 'Generic' and target profile to 'Linksys NSLU2'.
- Because this is not my new router, I do not want it to use the default IP address of 192.168.1.1. This can be changed in the 'Image Configuration' bit of the configuration menu.
- I would like to re-code some of the meterserv software in python, so I need python and pyusb installed in the flash image - these were selected from the 'Languages' section of the openwrt menu.
- Make builds a firmware image in bin/ixp4xxx called openwrt-nslu2-squashfs.bin, which is an 8MB image - there is a 16MB one too, but I am not convinced that my nslu2 has that much flash memory.
- You have to put the nslu2 into 'recovery mode' by holding the re-set button in while you power up the machine - you have to release the reset button as soon as the status LED changes colour, after about 10 seconds. The status LED then flashes a couple of different colours.
- Flash the new firmware image onto the NSLU2 using upslug2 (sudo apt-get install upslug2 on ubuntu) - the command line was sudo upslug2 --target 00:14:bf:64:df:be --image openwrt-nslu2-squashfs.bin.
- The machine now appears on the network with the right IP address. Telnet into it and change the root password, then connect using SSH instead.
I have had a bit more of a look at it. There are two surprising things. The first is that if I write a simple 'hello world' script and execute it, it works, so it is just interactive python that is broken.
The second is that doing "ldd /usr/bin/python" on the nslu2 and on the bifferboard give very similar results, except that the bifferboard one is linked against libgcc and libc, but the nslu2 one is linked against libc.
I think this is telling me something, but I am not sure what....
Friday, 21 January 2011
Mapnik Map Rendering
Someone asked a question on the osm-gb mailing list about viewing coverage of waterways - canals and navigable rivers.
We don't seem to have one, so I created one on my http://maps.webhop.net web site.
The trouble with that site is that although it is running on a nice powerful computer (my old laptop), it is behind my domestic broadband service from Virgin Media. Although this service has a nice fast download speed, the upload speed is very poor ~900 kbps compared to 9600 kbps download. This makes the web site very slow from the outside world.
I have a little virtual server provided by CloudNext - it is nowhere near as powerful as my own server, and has much less disk space, but at least it has a fast internet connection.
I loaded a british isles extract of OpenStreetMap data into the database - it took a very long time - maybe getting on for 12 hours (not sure when it finished!) - my server takes about 2 hours. Performance rendering maps was not too bad though - rendering down to zoom level 14 took about 2 hours, which is respectable.
Just need to get the database updating to keep it up to date, and re-generate changed tiles now....
You can see the difference at http://maps.webhop.net/canals (my home server) compared to http://maps2.webhop.net/canals (the virtual server)
We don't seem to have one, so I created one on my http://maps.webhop.net web site.
The trouble with that site is that although it is running on a nice powerful computer (my old laptop), it is behind my domestic broadband service from Virgin Media. Although this service has a nice fast download speed, the upload speed is very poor ~900 kbps compared to 9600 kbps download. This makes the web site very slow from the outside world.
I have a little virtual server provided by CloudNext - it is nowhere near as powerful as my own server, and has much less disk space, but at least it has a fast internet connection.
I loaded a british isles extract of OpenStreetMap data into the database - it took a very long time - maybe getting on for 12 hours (not sure when it finished!) - my server takes about 2 hours. Performance rendering maps was not too bad though - rendering down to zoom level 14 took about 2 hours, which is respectable.
Just need to get the database updating to keep it up to date, and re-generate changed tiles now....
You can see the difference at http://maps.webhop.net/canals (my home server) compared to http://maps2.webhop.net/canals (the virtual server)
Subscribe to:
Posts (Atom)