Using VectorMapDistrict with PostGIS

Well, vmd2pgsql took about 9 hours to import the entire UK into my postgresql database...No too bad.
Now I need to do something with it, because the mapnik stylesheets that I set up to use the shapefiles will not work - need to change it to use the postgresql database instead.
The things that need changing are:

1. Add the layers and datasources in XML (no need to do it in python now I don't have hundreds of separate datasources).
2. Convert the styles to use lowercase letters for the fields ('featcode' rather than 'FEATCODE') - postgresql seems to be case sensitive, and they have gone into the database lower case, but were upper case in the shapefiles.

Sounds easy,  but I decided I don't want it to look like the OSM stylesheet, which is a bit garbled - I want a nice structure to make it easy to maintain.  I could have used XML entities, but that seems a bit crude - define an entity, then 'use' it to actually 'call' it.   Includes seem much more suitable....But to get includes working I need mapnik2.  I have put a basic structure at code.google.com/p/ntmisc/vmdmap, but still have a few issues:

• I have had to define the database parameters manually because entities do not seem to be working in the include files, which is no good!
• The output is transparent background with black lines (see picture below), which is not what I asked for - not sure if this is a problem with the style file, or my build of mpanik2 - I'll have to run some tests on mapnik2 to make sure it works right.
• But it's too late to fix it now - job for tomorrow!

Using VectorMapDistrict with PostGIS

Well, vmd2pgsql took about 9 hours to import the entire UK into my postgresql database...No too bad.
Now I need to do something with it, because the mapnik stylesheets that I set up to use the shapefiles will not work - need to change it to use the postgresql database instead.
The things that need changing are:

1. Add the layers and datasources in XML (no need to do it in python now I don't have hundreds of separate datasources).
2. Convert the styles to use lowercase letters for the fields ('featcode' rather than 'FEATCODE') - postgresql seems to be case sensitive, and they have gone into the database lower case, but were upper case in the shapefiles.

Sounds easy,  but I decided I don't want it to look like the OSM stylesheet, which is a bit garbled - I want a nice structure to make it easy to maintain.  I could have used XML entities, but that seems a bit crude - define an entity, then 'use' it to actually 'call' it.   Includes seem much more suitable....But to get includes working I need mapnik2.  I have put a basic structure at code.google.com/p/ntmisc/vmdmap, but still have a few issues:

• I have had to define the database parameters manually because entities do not seem to be working in the include files, which is no good!
• The output is transparent background with black lines (see picture below), which is not what I asked for - not sure if this is a problem with the style file, or my build of mpanik2 - I'll have to run some tests on mapnik2 to make sure it works right.
• But it's too late to fix it now - job for tomorrow!

More Rendering of VectorMapDistrict Data

The nice people from Ordnance Survey have sent me a complete set of VectorMapDistrict data (which is on 6 DVDs!).   Surprisingly they didn't even charge me for the DVDs or postage.
Now I have all that data I thought I'd better do something with it!

First step was to get it onto a nice fast disk, so I copied it onto the hard disk of my server...which took a while...

Then I tried to use my vmdmap.py program to render it all, but mapnik bombed out with an error as it was adding the various shapefile layers.  I suspect that this is because every single shapefile is in its own layer, as it is a completely separate datasource, and I think I either ran out of memory or hit some mapnik internal limit.   This means I can't use vmdmap.py to render the whole country, which is a bit of a shame.

To get around this I think I need to merge them into a single datasource.  I don't know enough about manipulating shapefiles to do this, so instead am making use of shp2pgsql which allows you to import a shapefile into a postgresql database.   I have written another program, based on vmdmap.py called vmd2pgsql which will scan through a directory tree looking for the various shapefiles in the vectormap district dataset, and importing them into postgresql.   This gives a much lower number of tables - just one per shapefile name, but each one will have a lot of data.

It is importing now, so will see how long the import takes, then how well it renders.  I suppose it should render ok because I have the whole UK OSM dataset in a single database and that works, but we'll see over the weekend if it ever finishes!  The code is at my google code site.

More Rendering of VectorMapDistrict Data

The nice people from Ordnance Survey have sent me a complete set of VectorMapDistrict data (which is on 6 DVDs!).   Surprisingly they didn't even charge me for the DVDs or postage.
Now I have all that data I thought I'd better do something with it!

First step was to get it onto a nice fast disk, so I copied it onto the hard disk of my server...which took a while...

Then I tried to use my vmdmap.py program to render it all, but mapnik bombed out with an error as it was adding the various shapefile layers.  I suspect that this is because every single shapefile is in its own layer, as it is a completely separate datasource, and I think I either ran out of memory or hit some mapnik internal limit.   This means I can't use vmdmap.py to render the whole country, which is a bit of a shame.

To get around this I think I need to merge them into a single datasource.  I don't know enough about manipulating shapefiles to do this, so instead am making use of shp2pgsql which allows you to import a shapefile into a postgresql database.   I have written another program, based on vmdmap.py called vmd2pgsql which will scan through a directory tree looking for the various shapefiles in the vectormap district dataset, and importing them into postgresql.   This gives a much lower number of tables - just one per shapefile name, but each one will have a lot of data.

It is importing now, so will see how long the import takes, then how well it renders.  I suppose it should render ok because I have the whole UK OSM dataset in a single database and that works, but we'll see over the weekend if it ever finishes!  The code is at my google code site.

Mapnik and OS_OpenData

I decided to compare the quality of OpenStreetMap mapping to that which has been released by Ordnance Survey OpenData.  
I started with VectorMapDistrict, which is a dataset provided as ESRI Shapefiles.   It is provided as a large number of shapefiles, so setting up a mapnik stylesheet manually would have been a pain.  Instead I separated the styles into an XML file, but added the layers using a python program which scans down the directory tree to add the various shapefiles to the map.  The python code is vmdmap.py and the stylesheet is styles.xml.

Then I tried Meridian2.  This is much simpler - just one shapefile for each sort of feature (A-Road, river etc.).  I kept the same structure with the styles defined in an XML file, and the layers in python, but it could have all been done in a single XML file this time.  The python code is md2map.py and the stylesheet is styles_md2.xml.

You can see the results below.

The most notable things are that Meridian2 is much simpler geometry, but it includes more road names than Vector Map District.  The OpenStreetMap rendering is fancier because I used the standard OSM style, rather than the home made ones I used for the OS Data.   I do like the rocks that appear in the VectorMap District rendering though - I will have to import it into OSM!.

VectorMapDistrict

Meridian2

OpenStreetMap

Mapnik and OS_OpenData

I decided to compare the quality of OpenStreetMap mapping to that which has been released by Ordnance Survey OpenData.  
I started with VectorMapDistrict, which is a dataset provided as ESRI Shapefiles.   It is provided as a large number of shapefiles, so setting up a mapnik stylesheet manually would have been a pain.  Instead I separated the styles into an XML file, but added the layers using a python program which scans down the directory tree to add the various shapefiles to the map.  The python code is vmdmap.py and the stylesheet is styles.xml.

Then I tried Meridian2.  This is much simpler - just one shapefile for each sort of feature (A-Road, river etc.).  I kept the same structure with the styles defined in an XML file, and the layers in python, but it could have all been done in a single XML file this time.  The python code is md2map.py and the stylesheet is styles_md2.xml.

You can see the results below.

The most notable things are that Meridian2 is much simpler geometry, but it includes more road names than Vector Map District.  The OpenStreetMap rendering is fancier because I used the standard OSM style, rather than the home made ones I used for the OS Data.   I do like the rocks that appear in the VectorMap District rendering though - I will have to import it into OSM!.

VectorMapDistrict

Meridian2

OpenStreetMap

Mapping - OSM and Mapnik

I have been playing with mapnik quite a bit lately - I decided I needed to learn how to use it better to improve my townguide program.
I have been very impressed with Lonvia's Hiking Map, which is an overlay showing hiking routes on top of a standard OpenStreetMap map, and thought I could do something similar.

My first effort was a map showing the locations of supermarkets (handy when we go on holiday so we know where to go to stock up on provisions) - It can be seen here.
Someone has imported a lot of powerline data from Ordnance Survey into my local area, so I tried a Power Station Map.  These maps all have an 'about' link showing how they work.

I have produced a similar one to show the use of Ordnance Survey OpenData in the OpenStreetMap map of the UK, which can be seen here.

More recently I have wanted to produce maps that include contours, and also allow me to plot GPX traces to show where we have been.   To do this I have imported the SRTM data for the UK into my postgresql database using the technique described here.   This allows me to render an Openstreetmap map with contours, which is useful for the countryside.

Adding a GPX trace over the top was a bit difficult - I wanted to do it in python to save hard coding the path to the GPX file in the mapnik style file, but I haven't managed to do that - I ended up adding a GPX layer to my mapnik style file using the mapnik ogr plugin.   The result can be seen here.