Python geospatial development [2nd ed.] 178216152X, 9781782161523, 1782161538, 9781782161530

Citation preview

---

Python Geospatial Development Second Edition

Learn to build sophisticated mapping applications from scratch using Python tools for geospatial development

Erik Westra

BIRMINGHAM - MUMBAI

Python Geospatial Development Second Edition Copyright © 2013 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: December 2012 Second Edition: May 2013

Production Reference: 1170513

Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78216-152-3 www.packtpub.com

Cover Image by Karl Moore ([email protected])

Credits Author Erik Westra

Proofreaders Stephen Silk Katherine Tarr

Reviewers Will Cadell

Indexers

Richard Marsden

Hemangini Bari

Silas Toms

Rekha Nair Tejal Daruwale

Acquisition Editor Kartikey Pandey

Graphics Ronak Dhruv

Lead Technical Editor

Abhinash Sahu

Susmita Panda Production Coordinator Technical Editors

Manu Joseph

Sharvari Baet Meenakshi Gupta Chirag Jani Project Coordinator Arshad Sopariwala

Cover Work Manu Joseph

About the Author Erik Westra has been a professional software developer for over 25 years, and has worked almost exclusively in Python for the past decade. Erik's early interest in graphical user-interface design led to the development of one of the most advanced urgent courier dispatch systems used by messenger and courier companies worldwide. In recent years, Erik has been involved in the design and implementation of systems matching seekers and providers of goods and services across a range of geographical areas. This work has included the creation of real-time geocoders and map-based views of constantly changing data. Erik is based in New Zealand, and works for companies worldwide. I would like to thank Ruth, the love of my life, for all of her support and encouragement.

About the Reviewers Richard Marsden has over 15 years of professional software development

experience. After starting in the ield of geophysics and oil exploration, he has spent the last ten years running the Winwaed Software Technology LLC, an independent software vendor. Winwaed specialize in geospatial tools and applications including web applications, and operate the http://www.mapping-tools.com website for tools and add-ins for Microsoft's MapPoint product. Richard also manages the technical aspects of the EcoMapCostaRica.com project for the Biology Department at the University of Dallas. This includes the website, online ield maps, ield surveys, and the creation and comparison of panoramic photographs. Richard is also active in the ield of natural language processing, especially with Python's NLTK package.

Will Cadell is a principal consultant with Sparkgeo.com. He builds next generation web mapping applications, primarily using Google Maps, geoDjango, and PostGIS. He has worked in academia, government, and natural resources but now mainly consults for the start-up community in Silicon Valley. His passion has always been the implementation of geographic technology and with over a billion smart, mobile devices in the world it's a great time to be working on the geoweb. Will lives in Prince George, Northern British Columbia, and when he's not writing code or talking about geographic web technology you can ind him on a ski hill with his family.

Silas Toms is a GIS programmer with ICF International. His main professional interests are programming in Python and automation of large-scale environmental impact analyses. He has lived in San Francisco for 6 years while inishing a masters in GIS at San Francisco State University on temperature interpolation. He wishes to thank his girlfriend and his family for being supportive of his many interests.

www.PacktPub.com Support iles, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support iles and downloads related to your book. Did you know that Packt offers eBook versions of every book published, with PDF and ePub iles available? You can upgrade to the eBook version at www.PacktPub. com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. TM

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why Subscribe? • • •

Fully searchable across every book published by Packt Copy and paste, print and bookmark content On demand and accessible via web browser

Free Access for Packt account holders If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Table of Contents Preface Chapter 1: Geospatial Development Using Python Python Geospatial development Applications of geospatial development Analyzing geospatial data Visualizing geospatial data Creating a geospatial mash-up Recent developments Summary

Chapter 2: GIS

Core GIS concepts Location Distance Units Projections Cylindrical projections Conic projections Azimuthal projections The nature of map projections

Coordinate systems Datums Shapes GIS data formats Working with GIS data manually Summary

1 7

7 9 12 12 14 16 17 19

21 21 22 25 27 29

29 31 32 33

34 36 37 39 41 49

Table of Contents

Chapter 3: Python Libraries for Geospatial Development Reading and writing geospatial data GDAL/OGR GDAL design GDAL example code OGR design OGR example code

51 51 52

52 55 56 57

Documentation Availability Dealing with projections pyproj Design

58 58 59 59 60

Proj Geod

60 61

Example code Documentation Availability Analyzing and manipulating geospatial data Shapely Design Example code Documentation Availability Visualizing geospatial data Mapnik Design Example code Documentation Availability Summary

62 63 64 66 66 66 68 69 69 70 71 72 74 76 76 77

Chapter 4: Sources of Geospatial Data

79

Data format Obtaining and using OpenStreetMap data

81 82

Sources of geospatial data in vector format OpenStreetMap TIGER

80 80 84

Data format Obtaining and using TIGER data

86 87

Natural Earth

88

Data format Obtaining and using Natural Earth vector data

[ ii ]

89 89

Table of Contents

Global, self-consistent, hierarchical, high-resolution shoreline database (GSHHS) Data format Obtaining the GSHHS database

90 91 92

World Borders Dataset

92

Data format Obtaining World Borders Dataset

93 93

Sources of geospatial data in raster format Landsat Data format Obtaining Landsat imagery

93 94 95 95

Natural Earth

98

Data format Obtaining and using Natural Earth raster data

Global Land One-kilometer Base Elevation (GLOBE) Data format Obtaining and using GLOBE data

99 99

100 100 101

National Elevation Dataset (NED)

102

Data format Obtaining and using NED data

103 103

Sources of other types of geospatial data GEOnet Names Server Data format Obtaining and using GEOnet Names Server data

Geographic Names Information System (GNIS) Data format Obtaining and using GNIS Data

105 106 107 107

107 108 109

Choosing your geospatial data source Summary

Chapter 5: Working with Geospatial Data in Python Pre-requisites Reading and writing geospatial data Task – calculate the bounding box for each country in the world Task – save the country bounding boxes into a shapeile Task – analyze height data using a digital elevation map Changing datums and projections Task – change projections to combine shapeiles using geographic and UTM coordinates Task – change datums to allow older and newer TIGER data to be combined Representing and storing geospatial data Task – deine the border between Thailand and Myanmar

[ iii ]

109 110

111 111 112 112 114 119 126 127 132 135 136

Table of Contents

Task – save geometries into a text ile Performing geospatial calculations Task – identify parks in or near urban areas Converting and standardizing units of geometry and distance Task – calculate the length of the Thai-Myanmar border Task – ind a point 132.7 kilometers west of Soshone, California Exercises Summary

Chapter 6: GIS in the Database

Spatially-enabled databases Spatial indexes Open source spatially-enabled databases MySQL PostGIS Installing and coniguring PostGIS Using PostGIS Documentation Advanced PostGIS Features

SpatiaLite

140 141 141 146 147 154 156 158

159

159 160 163 163 168

168 170 172 173

174

Installing SpatiaLite Installing pysqlite Accessing SpatiaLite from Python Documentation Using SpatiaLite SpatiaLite capabilities

174 175 175 176 177 178

Commercial Spatially-enabled database Oracle MS SQL Server Recommended best practices Using the database to keep track of spatial references Using the appropriate spatial reference for your data Option 1 – using a database that supports geographies Option 2 – transform features as required Option 3 – transform features from the outset When to use unprojected coordinates

Avoiding on-the-ly transformations within a query Don't create geometries within a query Using spatial indexes appropriately

179 180 180 181 181 183 184 185 185 185

186 187 188

Knowing the limits of your database's query optimizer MySQL PostGIS SpatiaLite

189 189 191 193

[ iv ]

Table of Contents

Working with geospatial databases using Python Prerequisites Working with MySQL Working with PostGIS Working with SpatiaLite Comparing the databases Summary

Chapter 7: Working with Spatial Data About DISTAL Designing and building the database Downloading the data World Borders Dataset GSHHS GNIS GEOnet Names Server Importing the data World Borders Dataset GSHHS US place name data Worldwide places' name data Implementing the DISTAL application The shared "database" module The "select country" script The "select area" script Calculating the bounding box Calculating the map's dimensions Setting up the data source Rendering the map image

195 195 195 199 201 204 205

207

207 211 216 216 216 216 217 217 218 220 222 227 229 232 233 236 241 242 245 246

The "show results" script

248

Identifying the clicked-on point Identifying features by distance Displaying the results

249 250 259

Application review and improvements Usability Quality Place name issues Lat/Long coordinate problems

262 262 264 264 264

Performance

266

Finding the problem Improving performance Calculating the tiled shorelines

266 269 271

[v]

Table of Contents Using tiled shorelines Analyzing performance improvement

280 282

Summary

Chapter 8: Using Python and Mapnik to Generate Maps Introducing Mapnik Creating an example map Mapnik in depth Data sources Shapeile PostGIS Gdal Ogr SQLite OSM MemoryDatasource

282

285 286 292 296 296

297 298 300 301 302 303 304

Rules, ilters, and styles

304

Filters Scale denominators "Else" rules "Also" rules

305 307 309 309

Symbolizers

310

Drawing lines Drawing polygons Drawing labels Drawing points Drawing raster images Using colors

310 316 319 329 333 335

Maps and layers

336

Map attributes and methods Layer attributes and methods

337 338

Map rendering MapGenerator revisited The MapGenerator interface Creating the main map layer Displaying points on the map Rendering the map What the map generator teaches us Map deinition iles Summary

Chapter 9: Putting It All Together – a Complete Mapping System About ShapeEditor Designing ShapeEditor Importing a shapeile

[ vi ]

339 341 342 343 344 345 346 346 350

353

353 357 358

Table of Contents

Selecting a feature Editing a feature Exporting a shapeile Prerequisites The structure of a Django application Models Views URL dispatching Templates Setting up the database Setting up the ShapeEditor project Deining ShapeEditor's applications Creating a shared application Deining data models Shapeile Attribute Feature AttributeValue The models.py ile Playing with the admin system Summary

Chapter 10: ShapeEditor – Implementing List View, Import, and Export Implementing the "list shapeiles" view Importing shapeiles The "import shapeile" view function Extracting the uploaded shapeile Importing the shapeile's contents Open the shapeile Add the Shapeile object to the database Deine the shapeile's attributes Store the shapeile's features Store the shapeile's attributes

Cleaning up Exporting shapeiles Deining the OGR shapeile Saving the features into the shapeile Saving the attributes into the shapeile Compressing the shapeile Deleting temporary iles

[ vii ]

360 362 362 362 363 365 366 366 369 371 373 375 375 377 378 378 379 379 380 384 391

393

393 398 399 403 405 405 406 407 408 410

413 414 415 416 418 420 420

Table of Contents

Returning the ZIP archive to the user Summary

Chapter 11: ShapeEditor – Selecting and Editing Features Selecting a feature to edit Implementing Tile Map Server Setting up the base map Tile rendering

420 422

423 424 425

435 438

Using OpenLayers to display the map Intercepting mouse clicks Implementing the ind feature view Editing features Adding features Deleting features Deleting shapeiles Using ShapeEditor Further improvements and enhancements Summary

Index

[ viii ]

444 449 454 460 468 471 472 474 474 475

477

Preface With the explosion of map-based websites and spatially-aware devices and applications, geospatial development is becoming increasingly important. The geospatial market is growing rapidly, and as a Python developer you can't afford to be left behind. In today's location-aware world, all commercial Python developers can beneit from an understanding of geospatial concepts and development techniques. Working with geospatial data can get complicated because you are dealing with mathematical models of the Earth's surface. Since Python is a powerful programming language with high-level toolkits, it is well-suited to geospatial development. This book will familiarize you with the Python tools required for geospatial development. It introduces basic geospatial concepts with a clear, detailed walkthrough of the key concepts such as location, distance, units, projections, datums, and geospatial data formats. We then examine a number of Python libraries and use these with freelyavailable geospatial data to accomplish a variety of tasks. The book provides an indepth look at the concept of storing spatial data in a database and how you can use spatial databases as tools to solve a variety of geospatial problems. It goes into the details of generating maps using the Mapnik map-rendering toolkit, and helps you to build a sophisticated web-based geospatial map editing application using GeoDjango, Mapnik, and PostGIS. By the end of the book, you will be able to integrate spatial features into your applications and build a complete mapping application from scratch. This book is a hands-on tutorial. It teaches you how to access, manipulate, and display geospatial data eficiently using a range of Python tools for GIS development.

Preface

What this book covers Chapter 1, Geospatial Development Using Python, gives an overview of the Python programming language and the concepts behind geospatial development. Major use-cases of geospatial development and recent and upcoming developments in the ield are also covered. Chapter 2, GIS, introduces the core concepts of location, distance, units, projections, shapes, datums, and geospatial data formats, before discussing the process of working with geospatial data manually. Chapter 3, Python Libraries for Geospatial Development, explores the major Python libraries available for geospatial development, including the available features, how the library is organized, and how to install and use it. Chapter 4, Sources of Geospatial Data, investigates the major sources of freely-available geospatial data, what information is available, the data format used, and how to import the downloaded data. Chapter 5, Working with Geospatial Data in Python, uses the libraries introduced earlier to perform various tasks using geospatial data, including changing projections, importing and exporting data, converting and standardizing units of geometry and distance, and performing geospatial calculations. Chapter 6, GIS in the Database, introduces the spatial capabilities of PostGIS, MySQL, and SQLite. It discusses best practices for storing different types of spatial data, and looks at how to access these databases from Python. Chapter 7, Working with Spatial Data, works through the design and implementation of a complete geospatial application called DISTAL, using freely-available geospatial data stored in a spatial database. It investigates the performance of this application and then works to optimize it using best-practice techniques. Chapter 8, Using Python and Mapnik to Produce Maps, gives an in-depth look at the Mapnik map-generation toolkit, and how to use it to produce a variety of maps. Chapter 9, Putting it all Together: a Complete Mapping Application, introduces the "ShapeEditor", a complete and sophisticated web application built using PostGIS, Mapnik and GeoDjango. We start by designing the overall application, and then build the ShapeEditor's database models.

[2]

Preface

Chapter 10, ShapeEditor: Implementing List View, Import, and Export, continues the implementation of the ShapeEditor system, concentrating on displaying a list of imported shapeiles, along with the logic for importing and exporting shapeiles via a web browser. Chapter 11, ShapeEditor: Selecting and Editing Features, concludes the implementation of the ShapeEditor, adding logic to let the user select and edit features within an imported shapeile. This involves the creation of a custom Tile Map Server, and the use of the OpenLayers JavaScript library to display and interact with geospatial data. Bonus chapter, Web Frameworks for Geospatial Development, examines the concepts of web application frameworks, web services, JavaScript UI libraries, and slippy maps. It introduces a number of standard web protocols used by geospatial applications, and inishes with a survey of the tools and frameworks available for building geospatial applications that run via a web interface. You can download this chapter from: http://www.packtpub.com/sites/default/ files/downloads/1523OS_Bonuschapter.pdf

What you need for this book To follow through this book, you will need to have Python Version 2.5 to 2.7. You will also need to download and install the following tools and libraries; full instructions are given in the relevant sections of this book: • • • • • • • • • • • • • •

GDAL/OGR GEOS Shapely Proj pyproj MySQL

MySQLdb SpatiaLite pysqlite

PostgreSQL PostGIS pyscopg2

Mapnik Django

[3]

Preface

Who this book is for This book is aimed at experienced Python developers who want to get up to speed with open source geospatial tools and techniques in order to build their own geospatial applications, or to integrate geospatial technology into their existing Python programs.

Conventions In this book, you will ind a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning. Code words in text are shown as follows: " The pyproj Geod class allows you to perform various geodetic calculations based on points on the Earth's surface." A block of code is set as follows: import mapnik symbolizer = mapnik.PolygonSymbolizer( mapnik.Color("darkgreen")) rule = mapnik.Rule() rule.symbols.append(symbolizer)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold: import mapnik symbolizer = mapnik.PolygonSymbolizer( mapnik.Color("darkgreen")) rule = mapnik.Rule() rule.symbols.append(symbolizer)

Any command-line input or output is written as follows: python setup.py build sudo python.setup.py install

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "clicking the Next button moves you to the next screen".

[4]

Preface

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of. To send us general feedback, simply send an e-mail to [email protected], and mention the book title via the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code iles for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the iles e-mailed directly to you.

[5]

Preface

Errata Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you ind a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you ind any errata, please report them by visiting http://www.packtpub. com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are veriied, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.

Piracy Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors, and our ability to bring you valuable content.

Questions You can contact us at [email protected] if you are having a problem with any aspect of the book, and we will do our best to address it.

[6]

Geospatial Development Using Python This chapter provides an overview of the Python programming language and geospatial development. Please note that this is not a tutorial on how to use the Python language; Python is easy to learn, but the details are beyond the scope of this book. In this chapter, we will cover: •

What the Python programming language is, and how it differs from other languages

•

An introduction to the Python Standard Library and the Python Package Index

•

What the terms "geospatial data" and "geospatial development" refer to

•

An overview of the process of accessing, manipulating, and displaying geospatial data

•

Some of the major applications for geospatial development

•

Some of the recent trends in the ield of geospatial development

Python Python (http://python.org) is a modern, high level language suitable for a wide variety of programming tasks. It is often used as a scripting language, automating and simplifying tasks at the operating system level, but it is equally suitable for building large and complex programs. Python has been used to write web-based systems, desktop applications, games, scientiic programming, and even utilities and other higher-level parts of various operating systems.

Geospatial Development Using Python

Python supports a wide range of programming idioms, from straightforward procedural programming to object-oriented programming and functional programming. While Python is generally considered to be an "interpreted" language, and is occasionally criticized for being slow compared to "compiled" languages such as C, the use of byte-compilation and the fact that much of the heavy lifting is done by library code means that Python's performance is often surprisingly good. Open-source versions of the Python interpreter are freely available for all major operating systems. Python is eminently suitable for all sorts of programming, from quick one-off scripts to building huge and complex systems. It can even be run in interactive (command-line) mode, allowing you to type in commands and immediately see the results. This is ideal for doing quick calculations or iguring out how a particular library works. One of the irst things a developer notices about Python compared with other languages such as Java or C++ is how expressive the language is: what may take 20 or 30 lines of code in Java can often be written in half a dozen lines of code in Python. For example, imagine that you wanted to print a sorted list of the words that occur in a given piece of text. In Python, this is trivial: words = set(text.split()) for word in sorted(words): print word

Implementing this kind of task in other languages is often surprisingly dificult. While the Python language itself makes programming quick and easy, allowing you to focus on the task at hand, the Python Standard Libraries make programming even more eficient. These libraries make it easy to do things such as converting date and time values, manipulating strings, downloading data from websites, performing complex maths, working with e-mail messages, encoding and decoding data, XML parsing, data encryption, ile manipulation, compressing and decompressing iles, working with databases—the list goes on. What you can do with the Python Standard Libraries is truly amazing. As well as the built-in modules in the Python Standard Libraries, it is easy to download and install custom modules, which can be written in either Python or C.

[8]

Chapter 1

The Python Package Index (http://pypi.python.org) provides thousands of additional modules which you can download and install. And if that isn't enough, many other systems provide python "bindings" to allow you to access them directly from within your programs. We will be making heavy use of Python bindings in this book. It should be pointed out that there are different versions of Python available. Python 2.x is the most common version in use today, while the Python developers have been working for the past several years on a completely new, non-backwardscompatible version called Python 3. Eventually, Python 3 will replace Python 2.x, but at this stage most of the third-party libraries (including all the GIS tools we will be using) only work with Python 2.x. For this reason, we won't be using Python 3 in this book.

Python is in many ways an ideal programming language. Once you are familiar with the language itself and have used it a few times, you'll ind it incredibly easy to write programs to solve various tasks. Rather than getting buried in a morass of type-deinitions and low-level string manipulation, you can simply concentrate on what you want to achieve. You end up almost thinking directly in Python code. Programming in Python is straightforward, eficient, and, dare I say it, fun.

Geospatial development The term "geospatial" refers to information that is located on the earth's surface using coordinates. This can include, for example, the position of a cell phone tower, the shape of a road, or the outline of a country:

[9]

Geospatial Development Using Python

Geospatial data often associates some piece of information with a particular location. For example, the following is an interactive map from the http://www.bbc.co.uk/ website, showing the percentage of people in each country with access to the Internet in 2008:

Geospatial development is the process of writing computer programs that can access, manipulate, and display this type of information. Internally, geospatial data is represented as a series of coordinates, often in the form of latitude and longitude values. Additional attributes such as temperature, soil type, height, or the name of a landmark are also often present. There can be many thousands (or even millions) of data points for a single set of geospatial data. For example, the following outline of New Zealand consists of almost 12,000 individual data points:

[ 10 ]

Chapter 1

Because so much data is involved, it is common to store geospatial information within a database. A large part of this book will be concerned with how to store your geospatial information in a database, and how to access it eficiently. Geospatial data comes in many different forms. Different Geographical Information System (GIS) vendors have produced their own ile formats over the years, and various organizations have also deined their own standards. It is often necessary to use a Python library to read iles in the correct format when importing geospatial data into your database. Unfortunately, not all geospatial data points are compatible. Just like a distance value of 2.8 can have a very different meaning depending on whether you are using kilometers or miles, a given latitude and longitude value can represent any number of different points on the earth's surface, depending on which projection has been used. A projection is a way of representing the curved surface of the earth in two dimensions. We will look at projections in more detail in Chapter 2, GIS, but for now just keep in mind that every piece of geospatial data has a projection associated with it. To compare or combine two sets of geospatial data, it is often necessary to convert the data from one projection to another. Latitude and longitude values are sometimes referred to as unprojected coordinates. We'll learn more about this in the next chapter.

[ 11 ]

Geospatial Development Using Python

In addition to the prosaic tasks of importing geospatial data from various external ile formats and translating data from one projection to another, geospatial data can also be manipulated to solve various interesting problems. Obvious examples include the task of calculating the distance between two points, or calculating the length of a road, or inding all data points within a given radius of a selected point. We will be using Python libraries to solve all of these problems, and more. Finally, geospatial data by itself is not very interesting. A long list of coordinates tells you almost nothing; it isn't until those numbers are used to draw a picture that you can make sense of it. Drawing maps, placing data points onto a map, and allowing users to interact with maps are all important aspects of geospatial development. We will be looking at all of these in later chapters.

Applications of geospatial development Let's take a brief look at some of the more common geospatial development tasks you might encounter.

Analyzing geospatial data Imagine that you have a database containing a range of geospatial data for San Francisco. This database might include geographical features, roads, the location of prominent buildings, and other man-made features such as bridges, airports, and so on. Such a database can be a valuable resource for answering various questions. For example: •

What's the longest road in Sausalito?

•

How many bridges are there in Oakland?

•

What is the total area of the Golden Gate Park?

•

How far is it from the Pier 39 to the Moscone Center?

Many of these types of problems can be solved using tools such as the PostGIS spatially-enabled database. For example, to calculate the total area of the Golden Gate Park, you might use the following SQL query: select ST_Area(geometry) from features where name = "Golden Gate Park";

[ 12 ]

Chapter 1

To calculate the distance between two places, you irst have to geocode the locations to obtain their latitude and longitude. There are various ways to do this; one simple approach is to use a free geocoding web service, such as this: http://nominatim.openstreetmap.org/search?q=Pier 39, San Francisco,CA

This returns a latitude value of 37.82 and a longitude value of -122.42. These latitude and longitude values are in decimal degrees. If you don't know what these are, don't worry; we'll talk about decimal degrees in Chapter 2, GIS.

Similarly, we can ind the location of the Moscone Center using this query: http://nominatim.openstreetmap.org/search?q=Moscone Center, San Francisco,CA

This returns a latitude value of 37.80 and a longitude value of -122.44. Now that we have the coordinates for the two desired locations, we can calculate the distance between them using the Proj Python library: import pyproj lat1,long1 = (37.82,-122.42) lat2,long2 = (37.80,-122.44) geod = pyproj.Geod(ellps="WGS84") angle1,angle2,distance = geod.inv(long1, lat1, long2, lat2) print "Distance is %0.2f meters" % distance

This prints the distance between the two points: Distance is 2833.64 meters Don't worry about the "WGS84" reference at this stage; we'll look at what this means in Chapter 2, GIS.

[ 13 ]

Geospatial Development Using Python

Of course, you wouldn't normally do this sort of analysis on a one-off basis like this—it's much more common to create a Python program that will answer these sorts of questions for any desired set of data. You might, for example, create a web application that displays a menu of available calculations. One of the options in this menu might be to calculate the distance between two points; when this option is selected, the web application would prompt the user to enter the two locations, attempt to geocode them by calling an appropriate web service (and display an error message if a location couldn't be geocoded), then calculate the distance between the two points using Proj, and inally display the results to the user. Alternatively, if you have a database containing useful geospatial data, you could let the user select the two locations from the database rather than typing in arbitrary location names or street addresses. However you choose to structure it, performing calculations like this will usually be a major part of your geospatial application.

Visualizing geospatial data Imagine that you wanted to see which areas of a city are typically covered by a taxi during an average working day. You might place a GPS recorder into a taxi and leave it to record the taxi's position over several days. The results would be a series of timestamps, latitude and longitude values as follows: 2010-03-21 2010-03-21 2010-03-21 2010-03-21 ...

9:15:23 9:15:27 9:15:34 9:15:39

-38.16614499 -38.16608632 -38.16604198 -38.16601507

176.2336626 176.2335635 176.2334771 176.2333958

By themselves, these raw numbers tell you almost nothing. But when you display this data visually, the numbers start to make sense:

[ 14 ]

Chapter 1

You can immediately see that the taxi tends to go along the same streets again and again. And if you draw this data as an overlay on top of a street map, you can see exactly where the taxi has been:

[ 15 ]

Geospatial Development Using Python

(Street map courtesy of http://openstreetmap.org). While this is a very simple example, visualization is a crucial aspect of working with geospatial data. How data is displayed visually, how different data sets are overlaid, and how the user can manipulate data directly in a visual format are all going to be major topics of this book.

Creating a geospatial mash-up The concept of a "mash-up" has become popular in recent years. Mash-ups are applications that combine data and functionality from more than one source. For example, a typical mash-up may combine details of houses for rent in a given city, and plot the location of each rental on a map, as follows:

This example comes from http://housingmaps.com. The Google Maps API has been immensely popular in creating these types of mash-ups. However, Google Maps has some serious licensing and other limitations—as does Google's main competitor, Bing. Fortunately, these are not the only options; tools such as Mapnik, Openlayers, and MapServer, to name a few, also allow you to create mash-ups that overlay your own data onto a map.

[ 16 ]

Chapter 1

Most of these mash-ups run as web applications across the Internet, running on a server that can be accessed by anyone who has a web browser. Sometimes the mashups are private, requiring password access, but usually they are publicly available and can be used by anyone. Indeed, many businesses (such as the housing maps site shown in the previous image) are based on freely-available geospatial mash-ups.

Recent developments A decade ago, geospatial development was vastly more limited than it is today. Professional (and hugely expensive) Geographical Information Systems were the norm for working with and visualizing geospatial data. Open source tools, where they were available, were obscure and hard to use. What is more, everything ran on the desktop—the concept of working with geospatial data across the Internet was no more than a distant dream. In 2005, Google released two products that completely changed the face of geospatial development. Google Maps and Google Earth made it possible for anyone with a web browser or a desktop computer to view and work with geospatial data. Instead of requiring expert knowledge and years of practice, even a four-year old could instantly view and manipulate interactive maps of the world. Google's products are not perfect: the map projections are deliberately simpliied, leading to errors and problems with displaying overlays; these products are only free for non-commercial use; and they include almost no ability to perform geospatial analysis. Despite these limitations, they have had a huge effect on the ield of geospatial development. People became aware of what was possible, and the use of maps and their underlying geospatial data has become so prevalent that even cell phones now commonly include built-in mapping tools. The Global Positioning System (GPS) has also had a major inluence on geospatial development. Geospatial data for streets and other man-made and natural features used to be an expensive and tightly controlled resource, often created by scanning aerial photographs and then manually drawing an outline of a street or coastline over the top to digitize the required features. With the advent of cheap and readilyavailable portable GPS units, anyone who wishes to can now capture their own geospatial data. Indeed, many people have made a hobby of recording, editing, and improving the accuracy of street and topological data, which are then freely shared across the Internet. All this means that you're not limited to recording your own data, or purchasing data from a commercial organization; volunteered information is now often as accurate and useful as commercially-available data, and may well be suitable for your geospatial application.

[ 17 ]

Geospatial Development Using Python

The open source software movement has also had a major inluence on geospatial development. Instead of relying on commercial toolsets, it is now possible to build complex geospatial applications entirely out of freely-available tools and libraries. Because the source code for these tools is often available, developers can improve and extend these toolkits, ixing problems and adding new features for the beneit of everyone. Tools such as PROJ.4, PostGIS, OGR, and GDAL are all excellent geospatial toolkits which are benefactors of the open source movement. We will be making use of all these tools throughout this book. As well as standalone tools and libraries, a number of geospatial Application Programming Interfaces (APIs) have become available. Google have provided a number of APIs, which can be used to include maps and perform limited geospatial analysis within a website. Other services, such as the OpenStreetMap geocoder we used earlier, allow you to perform various geospatial tasks that would be dificult to do if you were limited to using your own data and programming resources. As more and more geospatial data becomes available, from an increasing number of sources, and as the number of tools and systems which can work with this data also increases, it has become increasingly important to deine standards for geospatial data. The Open Geospatial Consortium, often abbreviated to OGC (http://www. opengeospatial.org) is an international standards organization which aims to do precisely this: to provide a set of standard formats and protocols for sharing and storing geospatial data. These standards, including GML, KML, GeoRSS, WMS, WFS, and WCS, provide a shared "language" in which geospatial data can be expressed. Tools such as commercial and open source GIS systems, Google Earth, web-based APIs, and specialized geospatial toolkits such as OGR are all able to work with these standards. Indeed, an important aspect of a geospatial toolkit is the ability to understand and translate data between these various formats. As GPS units have become more ubiquitous, it has become possible to record your location data as you are performing another task. Geolocation, the act of recording your location as you are doing something, is becoming increasingly common. The Twitter social networking service, for example, now allows you to record and display your current location as you enter a status update. As you approach your ofice, sophisticated To-do list software can now automatically hide any tasks which can't be done at that location. Your phone can also tell you which of your friends are nearby, and search results can be iltered to only show nearby businesses. All of this is simply the continuation of a trend that started when GIS systems were housed on mainframe computers and operated by specialists who spent years learning about them.

[ 18 ]

Chapter 1

Geospatial data and applications have been "democratized" over the years, making them available in more places, to more people. What was possible only in a large organization can now be done by anyone using a handheld device. As technology continues to improve, and the tools become more powerful, this trend is sure to continue.

Summary

In this chapter, we briely introduced the Python programming language and the main concepts behind geospatial development. We have seen: •

That Python is a very high-level language eminently suited to the task of geospatial development.

•

That there are a number of libraries which can be downloaded to make it easier to perform geospatial development work in Python.

•

That the term "geospatial data" refers to information that is located on the earth's surface using coordinates.

•

That the term "geospatial development" refers to the process of writing computer programs that can access, manipulate, and display geospatial data.

•

That the process of accessing geospatial data is non-trivial, thanks to differing ile formats and data standards.

•

What types of questions can be answered by analyzing geospatial data.

•

How geospatial data can be used for visualization.

•

How mash-ups can be used to combine data (often geospatial data) in useful and interesting ways.

•

How Google Maps, Google Earth, and the development of cheap and portable GPS units have "democratized" geospatial development.

•

The inluence the open source software movement has had on the availability of high quality, freely-available tools for geospatial development.

•

How various standards organizations have deined formats and protocols for sharing and storing geospatial data.

•

The increasing use of geolocation to capture and work with geospatial data in surprising and useful ways.

[ 19 ]

Geospatial Development Using Python

In the next chapter, we will look in more detail at traditional GIS, including a number of important concepts which you need to understand in order to work with geospatial data. Different geospatial formats will be examined, and we will inish by using Python to perform various calculations using geospatial data.

[ 20 ]

GIS The term GIS generally refers to Geographical Information Systems, which are complex computer systems for storing, manipulating, and displaying geospatial data. GIS can also be used to refer to the more general Geographic Information Sciences, which is the science surrounding the use of GIS systems. In this chapter we will look at: •

The central GIS concepts you will have to become familiar with: location, distance, units, projections, datums, coordinate systems, and shapes

•

Some of the major data formats you are likely to encounter when working with geospatial data

•

Some of the processes involved in working directly with geospatial data

Core GIS concepts Working with geospatial data is complicated because you are dealing with mathematical models of the earth's surface. In many ways it is easy to think of the earth as a sphere on which you can place your data. That might be easy, but it isn't accurate—the earth is more like an oblate spheroid than a perfect sphere. This difference, as well as other mathematical complexities we won't get into here, means that representing points, lines, and areas on the surface of the earth is a rather complicated process. Let's take a look at some of the key GIS concepts you will have to become familiar with as you work with geospatial data.

GIS

Location Locations represent points on the surface of the earth. One of the most common ways to measure location is through the use of latitude and longitude coordinates. For example, my current location (as measured by a GPS receiver) is 38.167446 degrees south and 176.234436 degrees east. What do these numbers mean, and how are they useful? Think of the earth as a hollow sphere with an axis drawn through the middle:

For any given point on the earth's surface, you can draw a line that connects that point with the centre of the earth, as shown in the following image:

[ 22 ]

Chapter 2

The point's latitude is the angle that this line makes in the north-south direction, relative to the equator:

In the same way, the point's longitude is the angle that this line makes in the eastwest direction, relative to an arbitrary starting point (typically the location of the Royal Observatory in Greenwich, England):

[ 23 ]

GIS

By convention, positive latitude values are in the northern hemisphere, while negative latitude values are in the southern hemisphere. Similarly, positive longitude values are east of Greenwich, and negative longitude values are west of Greenwich. Thus, latitudes and longitudes cover the entire earth as shown in the following image:

The horizontal lines, representing points of equal latitude, are called parallels, while the vertical lines, representing points of equal longitude, are called meridians. The meridian at zero longitude is often called the prime meridian. By deinition, the parallel at zero latitude corresponds with the earth's equator. There are two things to remember when working with latitude and longitude values: 1. Western longitudes are generally negative, but you may ind situations (particularly when dealing with US-speciic data) where western longitudes are given as positive values. 2. The longitude values wrap around at the ±180 degrees point. That is, as you travel east, your longitude will go 177, 178, 179, 180, -179, -178, -177, and so on. This can make basic distance calculations rather confusing if you are doing them yourself rather than relying on a library to do the work for you. A latitude and longitude value refers to what is called a geodetic location. A geodetic location identiies a precise point on the earth's surface, regardless of what might be at that location. While much of the data we will be working with involves geodetic locations, there are other ways of describing a location which you may encounter.

[ 24 ]

Chapter 2

For example, a civic location is simply a street address, which is another perfectly valid (though scientiically less precise) way of deining a location. Similarly, jurisdictional locations include information about which governmental boundary (such as an electoral ward, borough, or city) the location is within, as this information is important in some contexts.

Distance The distance between two points on the earth's surface can be thought of in different ways. For example: •

Angular distance: This is the angle between two rays going out from the centre of the earth through the two points:

Angular distances are commonly used in seismology, and you may encounter them when working with geospatial data. •

Linear distance: This is what people typically mean when they talk of distance: how far apart two points on the earth's surface are:

This is often described as an "as the crow lies" distance. We'll discuss this in more detail shortly, though be aware that linear distances aren't quite as simple as they might appear.

[ 25 ]

GIS

•

Traveling distance: Linear ("as the crow lies") distances are all very well, but very few people can ly like crows. Another useful way of measuring distance is to measure how far you would actually have to travel to get from one point to another, typically following a road or other obvious route:

Most of the time, you will be dealing with linear distances. If the earth was lat, linear distances would be trivial to calculate—you simply measure the length of a line drawn between the two points. Of course, the earth is not lat, which means that actual distance calculations are rather more complicated:

[ 26 ]

Chapter 2

Because we are working with distances between points on the earth's surface, rather than points on a lat surface, we are actually using what is called the great circle distance. The great circle distance is the length of a semicircle between two points on the surface of the earth, where the semicircle is centered around the middle of the earth:

It is relatively straightforward to calculate the great circle distance between any two points if you assume that the earth is spherical; the Haversine formula is often used for this. More complicated techniques which more accurately represent the shape of the earth are available, though in many cases the Haversine formula is suficient.

We will learn more about the Haversine formula later in this chapter.

Units In September 1999, the Mars Climate Orbiter reached the outer edges of the Martian atmosphere, after having traveled through space for 286 days and costing a total of $327 million to create. As it approached its inal orbit, a miscalculation caused it to ly too low, and the Orbiter was destroyed. The reason? The spacecraft's thrusters calculated force using imperial units, while the spacecraft's computer worked with metric units. The result was a disaster for NASA, and a pointed reminder of just how important it is to understand which units your data is in. Geospatial data can come in a variety of different units. Distances can be measured in metric and imperial, of course, but there are actually a lot of different ways in which a given distance can be measured, including: •

Millimeters

•

Centimeters [ 27 ]

GIS

•

Inches

•

International feet

•

U.S. Survey feet

•

Meters

•

Yards

•

Kilometers

•

International miles

•

U.S. survey (statute) miles

•

Nautical miles

Whenever you are working with distance data, it is important that you know which units those distances are in. You will also often ind it necessary to convert data from one unit of measurement to another. Angular measurements can also be in different units: degrees or radians. Once again, you will often have to convert from one to the other. While these are not strictly speaking different units, you will often ind yourself dealing with different ways of representing longitude and latitude values. Traditionally, longitude and latitude values have been written using degrees, minutes, and seconds notation, as follows: 176° 14' 4''

Another possible way of writing these numbers is to use degrees and decimal minutes notation: 176° 14.066'

Finally, there is the decimal degrees notation: 176.234436°

Decimal degrees are quite common now, mainly because these are simply loatingpoint numbers you can put directly into your programs, but you may well need to convert longitude and latitude values from other formats before you can use them. Another possible issue with longitude and latitude values is that the quadrant (east, west, north, south) can sometimes be given as a separate value rather than using positive or negative values. For example: 176.234436° E

[ 28 ]

Chapter 2

Fortunately, all these conversions are relatively straightforward. But it is important to know which units, and which format your data is in—your software may not crash a spacecraft, but it will produce some very strange and incomprehensible results if you aren't careful.

Projections Creating a two-dimensional map from the three-dimensional shape of the earth is a process known as projection. A projection is a mathematical transformation that "unwraps" the three-dimensional shape of the earth and places it onto a twodimensional plane. Hundreds of different projections have been developed, but none of them are perfect. Indeed, it is mathematically impossible to represent the three-dimensional earth's surface on a two-dimensional plane without introducing some sort of distortion; the trick is to choose a projection where the distortion doesn't matter for your particular use. For example, some projections represent certain areas of the earth's surface accurately, while adding major distortion to other parts of the earth; these projections are useful for maps in the accurate portion of the earth, but not elsewhere. Other projections distort the shape of a country while maintaining its area, while yet other projections do the opposite. There are three main groups of projections: cylindrical, conical, and azimuthal. Let's look at each of these briely.

Cylindrical projections An easy way to understand cylindrical projections is to imagine that the earth is like a spherical Chinese lantern, with a candle in the middle:

[ 29 ]

GIS

If you placed this lantern-earth inside a paper cylinder, the candle would "project" the surface of the earth onto the inside of the cylinder:

You can then "unwrap" this cylinder to obtain a two-dimensional image of the earth:

Of course, this is a simpliication—in reality, map projections don't actually use light sources to project the earth's surface onto a plane, but instead use sophisticated mathematical transformations to achieve the same effect. Some of the main types of cylindrical projections include the Mercator Projection, the Equal-Area Cylindrical Projection, and the Universal Transverse Mercator Projection. The following map, taken from Wikipedia, is an example of a Mercator projection:

[ 30 ]

Chapter 2

Conic projections A conic projection is obtained by projecting the earth's surface onto a cone:

[ 31 ]

GIS

The cone is then "unwrapped" to produce the inal map. Some of the more common types of conic projections include the Albers Equal-Area Projection, the Lambert Conformal Conic Projection, and the Equidistant Projection. The following is an example of a Lambert Conformal Conic Projection, again taken from Wikipedia:

Polar-aligned conic projections are particularly useful when displaying areas that are wide but not very high, such as a map of Russia.

Azimuthal projections An azimuthal projection involves projecting the earth's surface directly onto a lat surface:

[ 32 ]

Chapter 2

Azimuthal projections are centered around a single point, and don't generally show the entire earth's surface. They do, however, emphasize the spherical nature of the earth. In many ways, azimuthal projections depict the earth as it would be seen from space. Some of the main types of azimuthal projections include the Gnomonic Projection, the Lambert Equal-Area Azimuthal Projection, and the Orthographic Projection. The following example, taken from Wikipedia, shows a Gnomonic projection based around the north pole:

The nature of map projections As mentioned earlier, there is no such thing as a perfect projection—every projection distorts the earth's surface in some way. Indeed, the mathematician Carl Gausse proved that it is mathematically impossible to project a three-dimensional shape such as a sphere onto a lat plane without introducing some sort of distortion. This is why there are so many different types of projections: some projections are more suited to a given purpose, but no projection can do everything. Whenever you create or work with geospatial data, it is essential that you know which projection has been used to create that data. Without knowing the projection, you won't be able to plot data or perform accurate calculations.

[ 33 ]

GIS

Coordinate systems Closely related to map projection is the concept of a coordinate system. There are two types of coordinate systems you will need to be familiar with: projected coordinate systems, and unprojected coordinate systems. Latitude and longitude values are an example of an unprojected coordinate system. These are coordinates that refer directly to a point on the earth's surface:

Unprojected coordinates are useful because they can accurately represent a desired point on the earth's surface, but they also make it quite dificult to perform distance and other geospatial calculations. Projected coordinates, on the other hand, are coordinates which refer to a point on a two-dimensional map that represents the surface of the earth:

[ 34 ]

Chapter 2

A projected coordinate system, as the name implies, makes use of a map projection to irst convert the earth into a two-dimensional Cartesian plane, and then places points onto that plane. To work with a projected coordinate system, you need to know which projection was used to create the underlying map. For both projected and unprojected coordinates, the coordinate system also implies a set of reference points that allow you to identify where a given point will be. For example, the unprojected lat/long coordinate system represents the longitude value of zero by a line running north-south through the Greenwich observatory in England. Similarly, a latitude value of zero represents a line running around the equator of the earth. For projected coordinate systems, you typically deine an origin and the map units. Some coordinate systems also use false northing and false easting values to adjust the position of the origin, as shown in the following image:

[ 35 ]

GIS

To give a concrete example, the Universal Transverse Mercator (UTM) coordinate system divides the world up into 60 different "zones", each zone using a different map projection to minimize projection errors. Within a given zone, the coordinates are measured as the number of meters away from the zone's origin, which is the intersection of the equator and the central meridian for that zone. False northing and false easting values are then added to the distance in meters away from this reference point to avoid having to deal with negative numbers. As you can imagine, working with projected coordinate systems like this can get quite complicated. The big advantage of projected coordinates, however, is that it is easy to perform geospatial calculations using these coordinates. For example, to calculate the distance between two points that both use the same UTM coordinate system, you simply calculate the length of the line between them, which is the distance between the two points, in meters. This is ridiculously easy, compared with the work required to calculate distances using unprojected coordinates. Of course, this assumes that the two points are both in the same coordinate system. Since projected coordinate systems are generally only accurate over a relatively small area, you can get into trouble if the two points aren't both in the same coordinate system (for example, if they are in two different UTM zones). This is where unprojected coordinate systems have a big advantage: they cover the entire earth.

Datums Roughly speaking, a datum is a mathematical model of the earth used to describe locations on the earth's surface. A datum consists of a set of reference points, often combined with a model of the shape of the earth. The reference points are used to describe the location of other points on the earth's surface, while the model of the earth's shape is used when projecting the earth's surface onto a two-dimensional plane. Thus, datums are used by both map projections and coordinate systems. While there are hundreds of different datums in use throughout the world, most of these only apply to a localized area. There are three main reference datums which cover larger areas, and which you are likely to encounter when working with geospatial data: •

NAD 27: This is the North American Datum of 1927. It includes a deinition of the earth's shape (using a model called the Clarke Spheroid of 1866), and a set of reference points centered around Meades Ranch in Kansas. NAD 27 can be thought of as a local datum covering North America.

[ 36 ]

Chapter 2

•

NAD 83: The North American Datum of 1983. This datum makes use of a more complex model of the earth's shape (the 1980 Geodetic Reference System, GRS 80). NAD 83 can be thought of as a local datum covering the United States, Canada, Mexico, and Central America.

•

WGS 84: The World Geodetic System of 1984. This is a global datum covering the entire earth. It makes use of yet another model of the earth's shape (the Earth Gravitational Model of 1996, EGM 96) and uses reference points based on the IERS International Reference Meridian. WGS 84 is a very popular datum. When dealing with geospatial data covering the United States, WGS 84 is basically identical to NAD 83. WGS 84 also has the distinction of being used by Global Positioning System satellites, so all data captured by GPS units will use this datum.

While WGS 84 is the most common datum in use today, a lot of geospatial data makes use of other datums. Whenever you are dealing with a coordinate value, it is important to know which datum was used to calculate that coordinate. A given point in NAD 27, for example, may be several hundred feet away from that same coordinate expressed in WGS 84. Thus, it is vital that you know which datum is being used for a given set of geospatial data, and convert to a different datum where necessary.

Shapes Geospatial data often represents shapes in the form of points, paths, and outlines:

A point, of course, is simply a coordinate, described by two or more numbers within a projected or unprojected coordinate system. Downloading the example code You can download the example code iles for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub. com/support and register to have the iles e-mailed directly to you

[ 37 ]

GIS

A path is generally described using what is called a linestring:

A linestring represents a path as a connected series of line segments. A linestring is a deliberate simpliication of a path, a way of approximating the curving path without having to deal with the complex maths required to draw and manipulate curves. Linestrings are often used in geospatial data to represent roads, rivers, contour lines, and so on. Linestrings are also sometimes referred to as polylines. When a linestring is closed (that is, the last line segment inishes at the point where the irst line segment starts), the linestring is often referred to as a linear ring.

An outline is often represented in geospatial data using a polygon:

Polygons are commonly used in geospatial data to describe the outline of countries, lakes, cities, and so on. A polygon has an exterior ring, deined by a closed linestring, and may optionally have one or more interior rings within it, each also deined by a closed linestring. The exterior ring represents the polygon's outline, while the interior rings (if any) represent "holes" within the polygon:

[ 38 ]

Chapter 2

These holes are often used to depict interior features such as islands within a lake.

GIS data formats

A GIS data format speciies how geospatial data is stored in a ile (or multiple iles) on disk. The format describes the logical structure used to store geospatial data within the ile(s). While we talk about storing information on disk, data formats can also be used to transmit geospatial information between computer systems. For example, a web service might provide map data on request, transmitting that data in a particular format.

A GIS data format will typically support: •

Geospatial data describing geographical features.

•

Additional metadata describing this data, including the datum and projection used, the coordinate system and units that the data is in, the date this ile was last updated, and so on.

•

Attributes providing additional information about the geographical features that are being described. For example, a city feature may have attributes such as "name", "population", "average temperature", and others.

•

Display information such as the color or line style to use when a feature is displayed.

[ 39 ]

GIS

There are two main types of GIS data: raster format data, and vector format data. Raster formats are generally used to store bitmapped images, such as scanned paper maps or aerial photographs. Vector formats, on the other hand, represent spatial data using points, lines, and polygons. Vector formats are the most common type used by GIS applications as the data is smaller and easier to manipulate. Some of the more common raster formats include: •

Digital Raster Graphic (DRG): This format is used to store digital scans of paper maps

•

Digital Elevation Model (DEM): Used by the US Geological Survey to record elevation data

•

Band Interleaved by Line, Band Interleaved by Pixel, Band Sequential (BIL, BIP, BSQ): These data formats are typically used by remote sensing systems

Some of the more common vector formats include: •

Shapeile: An open speciication, developed by ESRI, for storing and exchanging GIS data. A Shapeile actually consists of a collection of iles, all with the same base name, for example, hawaii.shp, hawaii.shx, hawaii.dbf, and so on.

•

Simple features: An OpenGIS standard for storing geographical data (points, lines, polygons) along with associated attributes.

•

TIGER/Line: A text-based format previously used by the US Census Bureau to describe geographic features such as roads, buildings, rivers, and coastlines. More recent data comes in the Shapeile format, so the TIGER/Line format is only used for earlier Census Bureau datasets.

•

Coverage: A proprietary data format used by ESRI's ARC/INFO system.

In addition to these "major" data formats, there are also so-called "micro-formats" which are often used to represent individual pieces of geospatial data. These are often used to represent shapes within a running program, or to transfer shapes from one program to another, but aren't generally used to store data permanently. As you work with geospatial data, you are likely to encounter the following micro-formats: •

Well-known Text (WKT): This is a simple text-based format for representing a single geographic feature such as a polygon or linestring

•

Well-known Binary (WKB): This alternative to WKT uses binary data rather than text to represent a single geographic feature

[ 40 ]

Chapter 2

•

GeoJSON: An open format for encoding geographic data structures, based on the JSON data interchange format

•

Geography Markup Language (GML): An XML-based open standard for exchanging GIS data

Whenever you work with geospatial data, you need to know which format the data is in, so that you can extract the information you need from the ile(s), and, where necessary, transform the data from one format to another.

Working with GIS data manually Let's take a brief look at the process of working with GIS data manually. Before we can begin, there are two things you need to do: •

Obtain some GIS data

•

Install the GDAL Python library so that you can read the necessary data iles

Let's use the US Census Bureau's website to download a set of vector maps for the various US states. The main site for obtaining GIS data from the US Census Bureau can be found at: http://www.census.gov/geo/www/tiger

To make things simpler though, let's bypass the website and directly download the ile we need from the following link: http://www2.census.gov/geo/tiger/TIGER2012/STATE/tl_2012_us_state.zip

The resulting ile, tl_2009_us_state.zip, should be a ZIP-format archive. After uncompressing the archive, you should have the following iles: •

tl_2012_us_state.dbf

•

tl_2012_us_state.prj

•

tl_2012_us_state.shp

•

tl_2012_us_state.shp.xml

•

tl_2012_us_state.shx

These iles make up a Shapeile containing the outlines of all the US states. Place these iles together in a convenient directory.

[ 41 ]

GIS

We next have to download the GDAL Python library. The main website for GDAL can be found at: http://gdal.org

The easiest way to install GDAL onto a Windows or Unix machine is to use the FWTools installer, which can be downloaded from the following site: http://fwtools.maptools.org

If you are running Mac OS X, you can ind a complete installer for GDAL at: http://www.kyngchaos.com/software/frameworks

After installing GDAL, you can check that it works by typing import osgeo into the Python command prompt; if the Python command prompt reappears with no error message, GDAL was successfully installed and you are all set to go: >>> import osgeo >>>

Now that we have some data to work with, let's take a look at it. You can either type the following directly into the command prompt, or else save it as a Python script so that you can run it whenever you wish (let's call this analyze.py): import osgeo.ogr shapefile = osgeo.ogr.Open("tl_2012_us_state.shp") numLayers = shapefile.GetLayerCount() print "Shapefile contains %d layers" % numLayers print for layerNum in range(numLayers): layer = shapefile.GetLayer(layerNum) spatialRef = layer.GetSpatialRef().ExportToProj4() numFeatures = layer.GetFeatureCount() print "Layer %d has spatial reference %s" % (layerNum, spatialRef) print "Layer %d has %d features:" % (layerNum, numFeatures) print for featureNum in range(numFeatures): feature = layer.GetFeature(featureNum)

[ 42 ]

Chapter 2 featureName = feature.GetField("NAME") print "Feature %d has name %s" % (featureNum, featureName)

The previous example assumes you've placed this script in the same directory as the tl_2012_us_state.shp ile. If you've put it in a different directory, change the osgeo.ogr. Open() command to include the path to your Shapeile. If you are running MS Windows, don't forget to use double backslash characters (\\) as directory separators.

This gives us a quick summary of how the Shapeile's data is structured: Shapefile contains 1 layers Layer 0 has spatial reference +proj=longlat +datum=NAD83 +no_defs Layer 0 has 56 features: Feature 0 has name Hawaii Feature 1 has name Arkansas Feature 2 has name New Mexico Feature 3 has name Montana ... Feature 53 has name Arizona Feature 54 has name Nevada Feature 55 has name California

This shows us that the data we downloaded consists of one layer, with 56 individual features corresponding to the various states and protectorates in the USA. It also tells us the "spatial reference" for this layer, which tells us that the coordinates are projected as latitude and longitude values using the NAD 83 datum. As you can see from the previous example, using GDAL to extract data from Shapeiles is quite straightforward. Let's continue with another example. This time, we'll look at the details for Feature 2, New Mexico: import osgeo.ogr shapefile = osgeo.ogr.Open("tl_2012_us_state.shp")

[ 43 ]

GIS layer = shapefile.GetLayer(0) feature = layer.GetFeature(2) print "Feature 2 has the following attributes:" print attributes = feature.items() for key,value in attributes.items(): print " %s = %s" % (key, value) geometry = feature.GetGeometryRef() geometryName = geometry.GetGeometryName() print print "Feature's geometry data consists of a %s" % geometryName

Running this produces the following: Feature 2 has the following attributes: DIVISION = 8 INTPTLAT = +34.4346843 NAME = New Mexico STUSPS = NM FUNCSTAT = A REGION = 4 LSAD = 00 INTPTLON = -106.1316181 AWATER = 756438507.0 STATENS = 00897535 MTFCC = G4000 STATEFP = 35 ALAND = 3.14161109357e+11 Feature's geometry data consists of a POLYGON

[ 44 ]

Chapter 2

The meaning of the various attributes is described on the US Census Bureau's website, but what interests us right now is the feature's geometry. A geometry object is a complex structure that holds some geospatial data, often using nested geometry objects to relect the way the geospatial data is organized. So far, we've discovered that New Mexico's geometry consists of a polygon. Let's now take a closer look at this polygon: import osgeo.ogr def analyzeGeometry(geometry, indent=0): s = [] s.append(" " * indent) s.append(geometry.GetGeometryName()) if geometry.GetPointCount() > 0: s.append(" with %d data points" % geometry.GetPointCount()) if geometry.GetGeometryCount() > 0: s.append(" containing:") print "".join(s) for i in range(geometry.GetGeometryCount()): analyzeGeometry(geometry.GetGeometryRef(i), indent+1) shapefile = osgeo.ogr.Open("tl_2012_us_state.shp") layer = shapefile.GetLayer(0) feature = layer.GetFeature(2) geometry = feature.GetGeometryRef() analyzeGeometry(geometry)

The analyzeGeometry() function gives a useful idea of how the geometry has been structured: POLYGON containing: LINEARRING with 7550 data points

In GDAL (or more speciically the OGR Simple Feature library we are using here), polygons are deined as a single outer "ring" with optional inner rings that deine "holes" in the polygon (for example, to show the outline of a lake).

[ 45 ]

GIS

New Mexico is a relatively simple feature in that it consists of only one polygon. If we ran the same program over California (feature 55 in our Shapeile), the output would be somewhat more complicated: MULTIPOLYGON containing: POLYGON containing: LINEARRING with 10105 data points POLYGON containing: LINEARRING with 392 data points POLYGON containing: LINEARRING with 152 data points POLYGON containing: LINEARRING with 191 data points POLYGON containing: LINEARRING with 121 data points POLYGON containing: LINEARRING with 93 data points POLYGON containing: LINEARRING with 77 data points

As you can see, California is made up of seven distinct polygons, each deined by a single linear ring. This is because California is on the coast, and includes six outlying islands as well as the main inland body of the state. Let's inish this analysis of the US state Shapeile by answering a simple question: what is the distance from the northernmost point to the southernmost point in California? There are various ways we could answer this question, but for now we'll do it by hand. Let's start by identifying the northernmost and southernmost points in California: import osgeo.ogr def findPoints(geometry, results): for i in range(geometry.GetPointCount()): x,y,z = geometry.GetPoint(i) if results['north'] == None or results['north'][1] < y: results['north'] = (x,y) if results['south'] == None or results['south'][1] > y: results['south'] = (x,y)

[ 46 ]

Chapter 2 for i in range(geometry.GetGeometryCount()): findPoints(geometry.GetGeometryRef(i), results) shapefile = osgeo.ogr.Open("tl_2012_us_state.shp") layer = shapefile.GetLayer(0) feature = layer.GetFeature(55) geometry = feature.GetGeometryRef() results = {'north' : None, 'south' : None} findPoints(geometry, results) print "Northernmost point is (%0.4f, %0.4f)" % results['north'] print "Southernmost point is (%0.4f, %0.4f)" % results['south']

The findPoints() function recursively scans through a geometry, extracting the individual points and identifying the points with the highest and lowest y (latitude) values, which are then stored in the results dictionary so that the main program can use it. As you can see, GDAL makes it easy to work with the complex geometry data structure. The code does require recursion, but is still trivial compared with trying to read the data directly. If you run the previous program, the following will be displayed: Northernmost point is (-122.3782, 42.0095) Southernmost point is (-117.2049, 32.5288)

Now that we have these two points, we next want to calculate the distance between them. As described earlier, we have to use a great circle distance calculation here to allow for the curvature of the earth's surface. We'll do this manually, using the Haversine formula: import math lat1 = 42.0095 long1 = -122.3782 lat2 = 32.5288 long2 = -117.2049 rLat1 = math.radians(lat1)

[ 47 ]

GIS rLong1 = math.radians(long1) rLat2 = math.radians(lat2) rLong2 = math.radians(long2) dLat = rLat2 - rLat1 dLong = rLong2 - rLong1 a = math.sin(dLat/2)**2 + math.cos(rLat1) * math.cos(rLat2) \ * math.sin(dLong/2)**2 c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a)) distance = 6371 * c print "Great circle distance is %0.0f kilometres" % distance

Don't worry about the complex maths involved here; basically, we are converting the latitude and longitude values to radians, calculating the difference in latitude/ longitude values between the two points, and then passing the results through some trigonometric functions to obtain the great circle distance. The value of 6371 is the radius of the earth, in kilometers. More details about the Haversine formula and how it is used in the previous example can be found at http://mathforum.org/library/drmath/view/51879.html. If you run the previous program, your computer will tell you the distance from the northernmost point to the southernmost point in California: Great circle distance is 1149 kilometres

There are, of course, other ways of calculating this. You wouldn't normally type the Haversine formula directly into your program, as there are libraries which will do this for you. But we deliberately did the calculation this way to show just how it can be done. If you would like to explore this further, you might like to try writing programs to calculate the following: • • • •

The easternmost and westernmost points in California. The midpoint in California. Hint: you can calculate the midpoint's longitude by taking the average of the easternmost and westernmost longitude. The midpoint in Arizona. The distance between the middle of California and the middle of Arizona.

As you can see, working with GIS data manually isn't too onerous. While the data structures and maths involved can be rather complex, using tools such as GDAL makes your data accessible and easy to work with.

[ 48 ]

Chapter 2

Summary In this chapter, we discussed many of the core concepts that underlie GIS development, looked briely at the history of GIS, examined some of the more common GIS data formats, and got our hands dirty exploring US state maps downloaded from the US Census Bureau website. We have learned the following: •

Locations are often, but not always, represented using coordinates

•

Calculating the distance between two points requires you to take into account the curvature of the earth's surface

•

You must be aware of the units used in geospatial data

•

Map projections represent the three-dimensional shape of the earth's surface as a two-dimensional map

•

There are three main classes of map projections: cylindrical, conic and azimuthal

•

Datums are mathematical models of the earth's shape

•

The three most common datums in use are called NAD 27, NAD 83, and WGS 84

•

Coordinate systems describe how coordinates relate to a given point on the earth's surface

•

Unprojected coordinate systems directly represent points on the earth's surface

•

Projected coordinate systems use a map projection to represent the earth as a two-dimensional Cartesian plane, onto which coordinates are then placed

•

Geospatial data can represent shapes in the form of points, linestrings, and polygons

•

There are a number of standard GIS data formats you might encounter. Some data formats work with raster data, while others use vector data

•

Using Python to manually perform various geospatial calculations on Shapeile data

In the next chapter, we will look in more detail at the various Python libraries which can be used for working with geospatial data.

[ 49 ]

Python Libraries for Geospatial Development This chapter examines a number of libraries and other tools which can be used for geospatial development in Python. More speciically, we will cover: •

Python libraries for reading and writing geospatial data

•

Python libraries for dealing with map projections

•

Libraries for analyzing and manipulating geospatial data directly within your Python programs

•

Tools for visualizing geospatial data

Note that there are two types of geospatial tools which are not discussed in this chapter: geospatial databases and geospatial web toolkits. Both of these will be examined in detail later in this book.

Reading and writing geospatial data While you could in theory write your own parser to read a particular geospatial data format, it is much easier to use an existing Python library to do this. We will look at two popular libraries for reading and writing geospatial data: GDAL and OGR.

Python Libraries for Geospatial Development

GDAL/OGR Unfortunately, the naming of these two libraries is rather confusing. Geospatial Data Abstraction Library (GDAL), was originally just a library for working with raster geospatial data, while the separate OGR library was intended to work with vector data. However, the two libraries are now partially merged, and are generally downloaded and installed together under the combined name of "GDAL". To avoid confusion, we will call this combined library GDAL/OGR and use "GDAL" to refer to just the raster translation library. A default installation of GDAL supports reading 116 different raster ile formats, and writing to 58 different formats. OGR by default supports reading 56 different vector ile formats, and writing to 30 formats. This makes GDAL/OGR one of the most powerful geospatial data translators available, and certainly the most useful freely-available library for reading and writing geospatial data.

GDAL design GDAL uses the following data model for describing raster geospatial data:

Let's take a look at the various parts of this model: •

A dataset holds all the raster data, in the form of a collection of raster "bands", along with information that is common to all these bands. A dataset normally represents the contents of a single ile.

[ 52 ]

Chapter 3

•

A raster band represents a band, channel, or layer within the image. For example, RGB image data would normally have separate bands for the red, green, and blue components of the image.

•

The raster size speciies the overall width and height of the image, in pixels.

•

The georeferencing transform converts from (x, y) raster coordinates into georeferenced coordinates—that is, coordinates on the surface of the earth. There are two types of georeferencing transforms supported by GDAL: afine transformations and ground control points. °

An affine transformation is a mathematical formula allowing the following operations to be applied to the raster data:

More than one of these operations can be applied at once; this allows you to perform sophisticated transforms such as rotations. Affine transformations are sometimes referred to as linear transformations.

[ 53 ]

Python Libraries for Geospatial Development

°

Ground Control Points (GCPs) relate one or more positions within the raster to their equivalent georeferenced coordinates, as shown in the following figure:

Note that GDAL does not translate coordinates using GCPs— that is left up to the application, and generally involves complex mathematical functions to perform the transformation. •

The coordinate system describes the georeferenced coordinates produced by the georeferencing transform. The coordinate system includes the projection and datum, as well as the units and scale used by the raster data.

•

The metadata contains additional information about the dataset as a whole.

Each raster band contains the following (among other things): •

The band raster size: This is the size (number of pixels across and number of lines high) for the data within the band. This may be the same as the raster size for the overall dataset, in which case the dataset is at full resolution, or the band's data may need to be scaled to match the dataset.

•

Some band metadata providing extra information speciic to this band.

•

A color table describing how pixel values are translated into colors.

•

The raster data itself.

GDAL provides a number of drivers which allow you to read (and sometimes write) various types of raster geospatial data. When reading a ile, GDAL selects a suitable driver automatically based on the type of data; when writing, you irst select the driver and then tell the driver to create the new dataset you want to write to.

[ 54 ]

Chapter 3

GDAL example code

A Digital Elevation Model (DEM) ile contains height values. In the following example program, we use GDAL to calculate the average of the height values contained in a sample DEM ile. In this case, we use a DEM ile downloaded from the GLOBE elevation dataset: from osgeo import gdal,gdalconst import struct dataset = gdal.Open("data/e10g") band = dataset.GetRasterBand(1) fmt = "100000") for row in cursor: print row[0],row[1]

Because Psycopg conforms to Python's DB-API speciication, using PostgreSQL from Python is relatively straightforward, especially if you have used databases from Python before. Here is how you might create a new spatially-enabled table using PostGIS: import psycopg2 connection = psycopg2.connect("dbname=... user=...") cursor = connection.cursor() cursor.execute("DROP TABLE IF EXISTS cities") cursor.execute("CREATE TABLE cities (id INTEGER," + "name VARCHAR(255), PRIMARY KEY (id))") cursor.execute("SELECT AddGeometryColumn('cities', 'geom', " + "-1, 'POLYGON', 2)") cursor.execute("CREATE INDEX cityIndex ON cities " + "USING GIST (geom)") connection.commit()

Let's take a look at each of these steps in more detail. We irst get a cursor object to access the database, and then create the nonspatial parts of our table using standard SQL statements: connection = psycopg2.connect("dbname=... user=...") cursor = connection.cursor() cursor.execute("DROP TABLE IF EXISTS cities") cursor.execute("CREATE TABLE cities (id INTEGER," + "name VARCHAR(255), PRIMARY KEY (id))")

Once the table itself has been created, we have to use a separate PostGIS function called AddGeometryColumn() to deine the spatial columns within our table: cursor.execute("SELECT AddGeometryColumn('cities', 'geom', " + "-1, 'POLYGON', 2)")

[ 171 ]

GIS in the Database

Recent versions of PostGIS support two distinct types of geospatial data, called geometries and geographies. The geometry type (which we are using here) uses Cartesian coordinates to place features onto a plane, and all calculations are done using Cartesian (x, y) coordinates. The geography type, on the other hand, identiies geospatial features using angular coordinates (latitude and longitude values) positioning the features onto a spheroid model of the Earth. The geography type is relatively new, much slower to use, and doesn't yet support all the functions that are available for the geometry type. Despite having the advantages of being able to accurately calculate distances which cover a large portion of the Earth and not requiring knowledge of projections and spatial references, we will not be using the geography type in this book.

Finally, we create a spatial index so that we can eficiently search using the new geometry column: cursor.execute("CREATE INDEX cityIndex ON cities " + "USING GIST (geom)")

Once you have created your database, you can insert geometry features into it using the ST_GeomFromText() function, like this: cursor.execute("INSERT INTO cities (name,geom) VALUES " + "(%s, ST_GeomFromText(%s)", (cityName, wkt))

Conversely, you can retrieve a geometry from the database in WKT format using the ST_AsText() function: cursor.execute("select name,ST_AsText(geom) FROM cities") for name,wkt in cursor:

Documentation Because PostGIS is an extension to PostgreSQL, and you use Psycopg to access it, there are three separate sets of documentation that you will need to refer to: •

The PostgreSQL manual: http://postgresql.org/docs

•

The PostGIS manual: http://postgis.refractions.net/docs

•

The Psycopg documentation: http://initd.org/psycopg/docs

[ 172 ]

Chapter 6

Of these, the PostGIS manual is probably going to be the most useful, and you will also need to refer to the Psycopg documentation to ind out the details of using PostGIS from Python. You will probably also need to refer to the PostgreSQL manual to learn the nonspatial aspects of using PostGIS, though be aware that this manual is huge and extremely complex, and relects the complexity of PostgreSQL itself.

Advanced PostGIS features PostGIS supports the following features that not available with MySQL: •

On-the-ly transformations of geometries from one spatial reference to another.

•

The ability to edit geometries by adding, changing, and removing points, and by rotating, scaling, and shifting entire geometries.

•

The ability to read and write geometries in GeoJSON, GML, KML, and SVG formats, in addition to WKT and WKB.

•

A complete range of bounding-box comparisons, including A overlaps B, A contains B, A is to the left of B, and so on. These comparison operators make use of spatial indexes to identify matching features extremely quickly.

•

Proper spatial comparisons between geometries, including intersection, containment, crossing, equality, overlap, touching, and so on. These comparisons are done using the true geometry rather than just their bounding boxes.

•

Spatial functions to calculate information such as the area, centroid, closest point, distance, length, perimeter, shortest connecting line, and so on. These functions take into account the geometry's spatial reference, if known.

•

Support for both vector and raster format geospatial data.

•

An optional geocoder based on TIGER/Line data, allowing you to convert from street addresses to a list of matching locations (US addresses only).

PostGIS has a reputation for being a geospatial powerhouse. While it is not the only option for storing geospatial data, and is certainly the most complex database discussed in this book, it is worth considering if you are looking for a powerful spatially-enabled database to use from within your Python geospatial programs and can deal with the complexity of setting up and administering a PostgreSQL database.

[ 173 ]

GIS in the Database

SpatiaLite As the name suggests, SpatiaLite is a "lightweight" spatial database, though the performance is surprisingly good and it doesn't skimp on features. Just like PostGIS is a spatial extension to PostgreSQL, SpatiaLite is a spatial extension to the serverless SQLite database engine. To access SQLite (and SpatiaLite) from Python, you need to use the pysqlite database adapter:

SpatiaLite pysqlite

+

Your Python program

SQLite

Installing SpatiaLite Before you can use SpatiaLite in your Python programs, you need to install SQLite, SpatiaLite, and pysqlite. How you do this depends on which operating system your computer is running. •

Mac OS X If you're using a Mac OS X-based system, you're in luck. The framework build of sqlite3 can be downloaded from: http://www.kyngchaos.com/software/frameworks

This will install everything you need, and you won't have to deal with any coniguration issues at all. •

MS Windows For MS Windows based systems, you can download binary installers from the following site: http://gaia-gis.it/gaia-sins

Near the bottom of this page is the MS Windows Binaries section, where you can download the appropriate installer. •

Linux For Linux, you can download the source code to libspatialite from the SpatiaLite website: https://www.gaia-gis.it/fossil/libspatialite/index

You can then follow the build instructions to compile libspatialite yourself. [ 174 ]

Chapter 6

Installing pysqlite After installing the libspatialite library and its dependencies, you'll need to make sure you have a workable version of pysqlite, the Python database adapter for SQLite. Mac users are once again in luck; the sqlite3 framework you downloaded already includes a suitable version of pysqlite, so you can ignore this section.

A version of pysqlite comes bundled with Python Version 2.5 and later, in the form of a standard library module named sqlite3. This standard library module, however, may not work with SpatiaLite. Because SpatiaLite is an extension to SQLite, the pysqlite library must be able to load extensions—a feature that was only introduced in pysqlite Version 2.5, and is often disabled by default. To see if your version of Python includes a usable version of sqlite3, type the following into the Python command line: >>> import sqlite3 >>> conn = sqlite3.connect(":memory:") >>> conn.enable_load_extension(True)

If you get an AttributeError, your built-in version of sqlite3 does not support loading extensions, and you will have to download and install a different version. The main website for pysqlite is: http://code.google.com/p/pysqlite

You can download binary versions for MS Windows, and source code packages for Linux, which you can compile yourself.

Accessing SpatiaLite from Python Now that you have all the libraries installed, you are ready to start using pysqlite to access and work with SpatiaLite databases. There is, however, one inal thing to be aware of; because pysqlite is a database adapter for SQLite rather than SpatiaLite, you will need to load the libspatialite extension before you can use any of the SpatiaLite functionality in your Python program.

[ 175 ]

GIS in the Database

Mac users don't need to do this, because the version of sqlite3 you downloaded comes with the libspatialite extension built in. If you are running on MS Windows, you may need to copy the SpatiaLite DLLs into the SYSTEM32 folder, or add the folder containing the SpatiaLite DLLs to the system path.

To load the libspatialite extension, add the following highlighted statements to your Python program: from pysqlite2 import dbapi as sqlite conn = sqlite.connect("...") conn.enable_load_extension(True) conn.execute('SELECT load_extension("libspatialite-2.dll")') curs = conn.cursor()

For Linux users, make sure you use the correct name for the libspatialite extension. You may also need to change the name of the pysqlite2 module you're importing depending on which version you downloaded.

Documentation With all these different packages, it can be quite confusing knowing where to look for more information. First off, you can learn more about the SQL syntax supported by SQLite (and SpatiaLite) by looking at the SQL as Understood by SQLite page: http://sqlite.org/lang.html

Then, to learn more about SpatiaLite itself, check out the main SpatiaLite web page: https://www.gaia-gis.it/fossil/libspatialite/index

You can access the SpatiaLite online documentation, as well as read through various tutorials, though these aren't Python-speciic. Finally, to learn more about using pysqlite to access SQLite and SpatiaLite from Python, see: http://pysqlite.googlecode.com/svn/doc/sqlite3.html

[ 176 ]

Chapter 6

Using SpatiaLite In many ways, SpatiaLite has been modeled after PostGIS. Before using SpatiaLite for your database, you need to initialize SpatiaLite's internal metadata tables. You also need to explicitly deine your spatial columns by calling the AddGeometryColumn() function, just like you do in PostGIS. Let's see how all this works by creating a SpatiaLite database and creating an example database table. As described earlier, the irst step in using SpatiaLite is to connect to the database and load the SpatiaLite extension, like this: from pysqlite2 import dbapi2 as sqlite db = sqlite.connect("myDatabase.db") db.enable_load_extension(True) db.execute('SELECT load_extension("libspatialite.dll")') cursor = db.cursor()

Note that because SQLite is a serverless database, the myDatabase.db database is simply a ile on your hard disk. Also, if you are running on Mac OS X, you can skip the enable_load_extension/SELECT load_extension dance and remove or comment out these two lines. You next need to initialize the SpatiaLite metadata tables in your database. In previous versions of SpatiaLite, you had to import these tables by hand. It's now much easier—simply execute the following within your Python script: cursor.execute('SELECT InitSpatialMetaData()')

If the metadata tables already exist, InitSpatialMetaData() will do nothing. This means you can safely call this function whenever you open the database, regardless of whether or not the database has already been initialized.

After initializing the metadata, you can create your own database table to hold your geospatial data. As with PostGIS, this is a two-step process; you irst create the nonspatial parts of your table using standard SQL statements: cursor.execute("DROP TABLE IF EXISTS cities") cursor.execute("CREATE TABLE cities (" + "id INTEGER PRIMARY KEY AUTOINCREMENT, " + "name CHAR(255))")

[ 177 ]

GIS in the Database

You then use the SpatiaLite function AddGeometryColumn() to deine the spatial column(s) in your table: cursor.execute("SELECT AddGeometryColumn('cities', 'geom', " + "4326, 'POLYGON', 2)")

The number 4326 is the spatial reference ID (SRID) used to identify the spatial reference this column's features will use. The SRID number 4326 refers to a spatial reference using latitude and longitude values and the WGS84 datum; we will look at SRID values in more detail in the Recommended Best Practices section.

You can then create a spatial index on your geometries using the CreateSpatialIndex() function, like this: cursor.execute("SELECT CreateSpatialIndex('cities', 'geom')")

Now that you have set up your database table, you can insert geometry features into it using the GeomFromText() function: cursor.execute("INSERT INTO cities (name, geom)" + " VALUES (?, GeomFromText(?, 4326))", (city, wkt))

And you can retrieve geometries from the database in WKT format using the AsText() function: cursor.execute("SELECT name,AsText(geom) FROM cities") for name,wkt in cursor:

SpatiaLite capabilities Some highlights of SpatiaLite include: •

The ability to handle all the major geometry types, including Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon and GeometryCollection.

•

Experimental support for topology-based datatypes (nodes, edges, faces, and so on) as an alternative to the above geometry types.

•

Every geometry feature has a spatial reference identiier (SRID) which tells you the spatial reference used by this feature.

[ 178 ]

Chapter 6

•

Geometry columns are constrained to a particular type of geometry and a particular SRID. This prevents you from accidentally storing the wrong type of geometry, or a geometry with the wrong spatial reference, in a database table.

•

Support for translating geometries to and from various microformats, including WKT, WKB , GML, KML, and GeoJSON.

•

Support for geometry functions to do things such as calculate the area of a polygon, to simplify polygons and linestrings, to calculate the distance between two geometries, to calculate intersections, differences, and buffers.

•

Functions to transform geometries from one spatial reference to another, and to shift, scale, and rotate geometries.

•

Support for fast spatial relationship calculations using minimum bounding rectangles.

•

Support for complete spatial relationship calculations (equals, touches, intersects, and so on) using the geometry itself rather than just the bounding rectangle.

•

The use of R-Tree indexes, which can (if you use them correctly) produce impressive results when performing spatial queries. Calculating the intersection of 500,000 linestrings with 380,000 polygons took just nine seconds, according to one researcher.

•

An alternative way of implementing spatial indexes, using in-memory MBR caching. This can be an extremely fast way of indexing features using minimum bounding rectangles, though it is limited by the amount of available RAM and so isn't suitable for extremely large datasets.

While SpatiaLite is considered to be a lightweight database, it is indeed surprisingly capable. Depending on your application, SpatiaLite may well be an excellent choice for your Python geospatial programming needs.

Commercial Spatially-enabled databases While we will be concentrating on the use of open source databases in this book, it is worth spending a moment exploring the commercial alternatives. There are two major commercial databases which support spatial operations: Oracle and Microsoft's SQL Server.

[ 179 ]

GIS in the Database

Oracle Oracle provides one of the world's most powerful and popular commercial database systems. Spatial extensions to the Oracle database are available in two lavors. Oracle Spatial provides a large range of geospatial database features, including spatial data types, spatial indexes, the ability to perform spatial queries and joins, and a range of spatial functions. Oracle Spatial also supports linear referencing systems, spatial analysis, and data-mining functions, geocoding, and support for raster-format data. While Oracle Spatial is only available for the Enterprise edition of the Oracle database, it is one of the most powerful spatially-enabled databases available anywhere. A subset of the Oracle Spatial functionality, called Oracle Locator, is available for the Standard edition of the Oracle database. Oracle Locator does not support common operations such as unions and buffers, intersections, area and length calculations. It also excludes support for more advanced features such as linear referencing systems, spatial analysis functions, geocoding, and raster format data. While being extremely capable, Oracle does have the disadvantage of using a somewhat non-standard syntax compared with other SQL databases. It also uses non-standard function names for its spatial extensions, making it dificult to switch database engines or use examples written for other databases.

MS SQL Server Microsoft's SQL Server is another widely-used and powerful commercial database system. SQL Server supports a full range of geospatial operations, including support for both geometry and geography data types, and all of the standard geospatial functions and operators. Because Microsoft has followed the Open Geospatial Consortium's standards, the data types and function names used by SQL Server match those used by the open source databases we have already examined. The only difference stems from SQL Server's own internal object oriented nature; for example, rather than ST_ Intersects(geom, pt), SQL Server uses geom.STIntersects(pt). Unlike Oracle, all of Microsoft's spatial extensions are included in every edition of the SQL Server; there is no need to obtain the Enterprise edition to get the full range of spatial capabilities.

[ 180 ]

Chapter 6

There are two limitations with MS SQL Server that may limit its usefulness as a spatially-enabled database. Firstly, SQL Server only runs on Microsoft Windows based computers. This limits the range of servers it can be installed on. Also, SQL Server does not support transforming data from one spatial reference system to another.

Recommended best practices In this section, we will look at a number of practical things you can do to ensure your geospatial databases work as eficiently and effectively as possible.

Using the database to keep track of spatial references As we've seen in earlier chapters, different sets of geospatial data use different coordinate systems, datums, and projections. Consider, for example, the following two geometry objects:

The geometries are represented as a series of coordinates, which are nothing more than numbers. By themselves, these numbers aren't particularly useful—you need to position these coordinates onto the Earth's surface by identifying the spatial reference (coordinate system, datum and projection) used by the geometry.

[ 181 ]

GIS in the Database

In this case, the POLYGON is using unprojected lat/long coordinates in the WGS84 datum, while the LINESTRING is using coordinates deined in meters using the UTM Zone 12N projection. Once you know the spatial reference, you can place the two geometries onto the Earth's surface. This reveals that the two geometries actually overlap:

In all but the most trivial databases, it is recommended that you store the spatial reference for each feature directly in the database itself. This makes it easy to keep track of which spatial reference is used by each feature. It also allows the queries and database commands you write to be aware of the spatial reference, and enables you to transform geometries from one spatial reference to another as necessary in your spatial queries. Spatial references are generally referred to using a simple integer value called a Spatial Reference Identiier (SRID). While you could choose arbitrary SRID values to represent various spatial references, it is strongly recommended that you use the European Petroleum Survey Group (EPSG) numbers as standard SRID values. Using this internationally-recognized standard makes your data interchangeable with other databases, and allows tools such as OGR and Mapnik to identify the spatial reference used by your data. To learn more about EPSG numbers, and SRID values in general, please refer to: http://epsg-registry.org

You have seen SRID values before. For example, in the Using SpatiaLite section of this chapter, we encountered the following SQL statement: SELECT AddGeometryColumn('cities','geom',4326,'POLYGON',2)

The value 4326 is the SRID used to identify a particular spatial reference, in this case the WGS84 Long Lat reference (unprojected lat/long coordinates using the WGS84 datum).

[ 182 ]

Chapter 6

Both PostGIS and SpatiaLite add a special table to your spatially-enabled database called spatial_ref_sys. This table comes preloaded with a list of over 3,000 commonly-used spatial references, all identiied by EPSG number. Because the SRID value is the primary key into this table, tools that access the database can refer to this table to perform on-the-ly coordinate transformations using the PROJ.4 library. Even if you are using MySQL, which doesn't provide a spatial_ref_sys table or other direct support for coordinate transformations, you should be using standard EPSG numbers for your spatial references. Note that all three open source spatial databases allow you to associate an SRID value with a geometry when importing from WKT: •

MySQL: GeometryFromText(wkt, [srid])

•

PostGIS: ST_GeometryFromText(wkt, [srid])

•

SpatiaLite: GeometryFromText(wkt, [srid])

While the SRID value is optional, you should use this wherever possible to tell the database which spatial reference your geometry is using. In fact, both PostGIS and SpatiaLite require you to use the correct SRID value if a column has been set up to use a particular SRID. This prevents you from using mixing spatial references within a table.

Using the appropriate spatial reference for your data When you import spatial data into your database, it will be in a particular spatial reference. This doesn't mean, though, that it has to stay in that spatial reference. In many cases, it will be more eficient and accurate to transform your data into the most appropriate spatial reference for your particular needs. Of course, "appropriate" depends on what you want to achieve. With the exception of PostGIS and its new geography type, all three spatial databases assume that coordinates exist on a Cartesian plane—that is, that you are using projected coordinates. If you store unprojected coordinates (latitude and longitude values) in the database, you will be limited in what you can do. Certainly, you can use unprojected geographic coordinates in a database to compare two features (for example, to see if one feature intersects with another), and you will be able to store and retrieve geospatial data quickly. However, any calculation that involves area or distance will be all but meaningless.

[ 183 ]

GIS in the Database

Consider, for example, what would happen if you asked MySQL to calculate the length of a LINESTRING geometry: mysql> SELECT GLength(geom) FROM roads WHERE id=9513; +-------------------+ | GLength(geom) | +-------------------+ | 192.3644911426572 | +-------------------+

If your data was in unprojected lat/long coordinates, the resulting "length" would be a number in decimal degrees. Unfortunately, this number is not particularly useful. You can't assume a simple relationship between the decimal degree "length" and the actual length on the Earth's surface, for example multiplying by some constant to yield the true length in meters. The only thing is that this so-called "length" value could be used for is to give a very rough estimate of the true length, as we did in the previous chapter to ilter out features obviously too far away from a desired point. If you do need to perform length and area calculations on your geospatial data (and it is likely that you will need to do this at some stage), you have three options: • • •

Using a database that supports unprojected coordinates Transform the features into projected coordinates before performing the length or distance calculation Store your geometries in projected coordinates from the outset

Let's consider each of these options in more detail.

Option 1 – using a database that supports geographies Of the open source databases we are considering, only PostGIS has the ability to work directly with unprojected coordinates, through the use of the relatively-new geography type. Unfortunately, the geography type has some major limitations which make this a less than ideal solution: •

• •

Performing calculations on unprojected coordinates takes approximately an order of magnitude longer than performing the same calculations using projected (Cartesian) coordinates The geography type only supports lat/long values on the WGS84 datum (SRID 4326) Many of the functions available for projected coordinates are not yet supported by the geography type

[ 184 ]

Chapter 6

For these reasons, as well as the fact that they are only supported by PostGIS, we will not be using geography columns in this book.

Option 2 – transforming features as required Another possibility is to store your data in unprojected lat/long coordinates, and transform the coordinates into a projected coordinate system before you calculate the distance or area. While this will work, and will give you accurate results, you should beware of doing this because (a) you may well forget to transform into a projected coordinate system before making the calculation, and (b) performing on-the-ly transformations of large numbers of geometries is very time-consuming. Despite these problems, there are situations where storing unprojected coordinates makes sense. We will look at this shortly.

Option 3 – transforming features from the outset Because transforming features from one spatial reference to another is rather timeconsuming, it often makes sense to do this once, at the time you import your data, and store it in the database already converted to a projected coordinate system. Doing this, you will be able to perform your desired spatial calculations quickly and accurately. However, there are situations where this is not the best option, as we will see in the next section.

When to use unprojected coordinates As we saw in Chapter 2, GIS, projecting features from the three-dimensional surface of the Earth onto a two-dimensional Cartesian plane can never be done perfectly. It is a mathematical truism that there will always be errors in any projection. Different map projections are generally chosen to preserve values such as distance or area for a particular portion of the Earth's surface. For example, the Mercator projection is accurate at the tropics but distorts features closer to the poles. Because of this inevitable distortion, projected coordinates work best when your geospatial data only covers a part of the Earth's surface. If you are only dealing with data for Austria, then a projected coordinate system will work very well indeed. But if your data includes features in both Austria and Australia, then using the same projected coordinates for both sets of features will once again produce inaccurate results.

[ 185 ]

GIS in the Database

For this reason, it is generally best to use a projected coordinate system for data that covers only part of the Earth's surface, but unprojected coordinates will work best if you need to store data covering large parts of the Earth. Of course, using unprojected coordinates leads to problems of its own, as discussed earlier. This is why it is recommended that you use the appropriate spatial reference for your particular needs; what is appropriate for you depends on what data you need to store and how you intend to use it. The best way to ind out what is appropriate would be to experiment; try importing your data in both spatial references, and write some test programs to work with the imported data. This will tell you which is the fastest and easiest spatial reference to work with, rather than having to guess.

Avoiding on-the-ly transformations within a query Imagine that you have a cities table with a geom column containing POLYGON geometries in UTM 12N projection (EPSG number 32612). Being a competent geospatial developer, you have set up a spatial index on this column. Now, imagine that you have a variable named pt that holds a POINT geometry in unprojected WGS84 coordinates (EPSG number 4326). You might want to ind the city that contains this point, so you would issue the following reasonable-looking query: SELECT * FROM cities WHERE Contains(Transform(geom, 4326), pt);

This would give you the right answer, but would take an extremely long time. Why is it that? Because the Transform(geom, 4326) expression is converting every geometry in the table from UTM 12N to WGS84 coordinates before the database can check to see if the point is inside the geometry. The spatial index is completely ignored, as it is in the wrong coordinate system. Compare this with the following query: SELECT * FROM cities WHERE Contains(geom, Transform(pt, 32612));

[ 186 ]

Chapter 6

A very minor change, but a dramatically different result. Instead of taking hours, the answer should come back almost immediately. Can you see why? The transformation is being done on a variable that does not change from one record to the next, so the Transform(pt, 32612) expression is being called just once, and the Contains() call can make use of your spatial index to quickly ind the matching city. The lesson here is simple: be aware of what you are asking the database to do, and make sure you structure your queries to avoid on-the-ly transformations of large numbers of geometries.

Don't create geometries within a query While we are discussing database queries that can cause the database to perform a huge amount of work, consider the following (where poly is a polygon): SELECT * FROM cities WHERE NOT ST_IsEmpty(ST_Intersection(outline, poly));

In a sense this is perfectly reasonable: identify all cities which have a nonempty intersection between the city's outline and the given polygon. And the database will indeed be able to answer this query—it will just take an extremely long time to do so. Hopefully you can see why; the ST_Intersection() function creates a new geometry out of two existing geometries. This means that for every row in the database table, a new geometry is created, and is then passed to ST_IsEmpty(). As you can imagine, these types of operations are extremely ineficient. To avoid creating a new geometry each time, you can rephrase your query like this: SELECT * FROM cities WHERE ST_Intersects(outline, poly);

While this example may seem obvious, there are many cases where spatial developers have forgotten this rule, and have wondered why their queries were taking so long to complete. A common example is to use the ST_Buffer() function to see if a point is within a given distance of a polygon, like this: SELECT * FROM cities WHERE ST_Contains(ST_Buffer(outline, 100), pt);

Once again, this query will work, but will be painfully slow. A much better approach would be to use the ST_DWithin() function: SELECT * FROM cities WHERE ST_DWithin(outline, pt, 100);

[ 187 ]

GIS in the Database

As a general rule, remember that you never want to call any function which returns a Geometry object (or one of its subclasses) within the WHERE portion of a SELECT statement.

Using spatial indexes appropriately Just as ordinary database indexes can make an immense difference to the speed and eficiency of your database, spatial indexes are also a very powerful tool for speeding up your database queries. Like all powerful tools, though, they have their limits: •

If you don't explicitly deine a spatial index, the database can't use it. Conversely, if you have too many spatial indexes, the database will slow down because each index needs to be updated every time a record is added, updated or deleted. Thus, it is crucial that you deine the right set of spatial indexes: index the information you are going to search on, and nothing more.

•

Because spatial indexes work on the geometries' bounding boxes, the index itself can only tell you which bounding boxes actually overlap or intersect; they can't tell you if the underlying points, lines, or polygons have this relationship. Thus, they are really only the irst step in searching for the information you want. With PostGIS and SpatiaLite, the database itself can further reine the search by comparing the individual geometries for you; with MySQL, you have to do this yourself, as we saw earlier.

•

Spatial indexes are most eficient when dealing with lots of relatively small geometries. If your polygons cover a large area, the polygon bounding boxes are going to be so large that they will intersect with many other geometries, and the database will have to revert to doing full polygon calculations rather than just the bounding box. Furthermore, if your geometries consist of many thousands of vertices, these calculations can be very slow indeed—the entire polygon will have to be loaded into memory and processed one vertex at a time. If you have polygons that are both large and complex, your spatial queries will be slow. If possible, it is generally better to split large and complex polygons (and multipolygons) into smaller pieces so that the spatial index can work with them more eficiently. We will revisit this issue in Chapter 7, Working with Spatial Data, where we'll split large polygons into smaller ones to speed up our program.

[ 188 ]

Chapter 6

Knowing the limits of your database's query optimizer When you send a query to the database, it automatically attempts to optimize the query to avoid unnecessary calculations and to make use of any available indexes. For example, if you issued the following (nonspatial) query: SELECT * FROM people WHERE name=Concat("John ","Doe");

The database would know that Concat("John ","Doe") yields a constant, and so would only calculate it once before issuing the query. It would also look for a database index on the name column, and use it to speed up the operation. This type of query optimization is very powerful, and the logic behind it is extremely complex. In a similar way, spatial databases have a spatial query optimizer that looks for ways to precalculate values and make use of spatial indexes to speed up the query. For example, consider this spatial query from the previous section: select * from cities where ST_DWithin(outline, pt, 12.5);

In this case, the PostGIS function ST_DWithin() is given one geometry taken from a table (outline), and a second geometry that is speciied as a ixed value (pt), along with a desired distance (12.5 "units", whatever that means in the geometry's spatial reference). The query optimizer knows how to handle this eficiently, by irst precalculating the bounding box for the ixed geometry plus the desired distance (pt ±12.5), and then using a spatial index to quickly identify the records which may have their outline geometry within that extended bounding box. While there are times when the database's query optimizer seems to be capable of magic, there are many other times when it is incredibly stupid. Part of the art of being a good database developer is to have a keen sense of how your database's query optimizer works, when it doesn't—and what to do about it. Let's see how you can ind out more about the query optimization process in each of our three spatial databases.

MySQL MySQL provides a command, EXPLAIN SELECT, that tells you how the query optimizer will process your query. For example: mysql> EXPLAIN SELECT * FROM cities WHERE MBRContains(geom, GeomFromText(pt))\G

[ 189 ]

GIS in the Database *********************** 1. row *********************** id: 1 select_type: SIMPLE table: cities type: range possible_keys: geom key: geom key_len: 34 ref: NULL rows: 1 Extra: Using where 1 row in set (0.00 sec)

Don't worry about the \G at the end of the command; this just formats the output in a way which makes it easier to read.

This command tells you that this query involves a simple search against the cities table, searching for a range of records using the geom spatial index to speed up the results. The rows:1 tells you that the query optimizer thinks it only needs to read a single row from the table to ind the results. This is good. Compare it with the following: mysql> EXPLAIN SELECT * FROM cities WHERE MBRContains(Envelope(geom), GeomFromText(pt))\G *********************** 1. row *********************** id: 1 select_type: SIMPLE table: cities type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL

[ 190 ]

Chapter 6 rows: 34916 Extra: Using where 1 row in set (0.00 sec)

This query uses the Envelope() function to create a new geometry, which is then checked to see if it contains the given point. As explained in the Don't Create Geometries Within a Query section, the database has to create a new geometry for every row in the table. In this case the query optimizer cannot use an index, as shown by the NULL value for possible_keys and key. It also tells you that it would have to scan through 34,916 records to ind the matching points—not exactly an eficient query. Indeed, running this query could take several minutes to complete.

PostGIS MySQL uses a theoretical approach to query optimization, looking only at the query itself to see how it could be optimized. PostGIS, on the other hand, takes into account the amount of information in the database and how it is distributed. In order to work well, the PostGIS query optimizer needs to have up-to-date statistics on the database's contents. It then uses a sophisticated genetic algorithm to determine the most effective way to run a particular query. Because of this approach, you need to regularly run the VACUUM ANALYZE command, which gather statistics on the database so that the query optimizer can work as effectively as possible. If you don't run VACUUM ANALYZE, the optimizer simply won't be able to work. Here is how you can run the VACUUM ANALYZE command from Python: import psycopg2 connection = psycopg2.connect("dbname=... user=...") cursor = connection.cursor() old_level = connection.isolation_level connection.set_isolation_level(0) cursor.execute("VACUUM ANALYZE") connection.set_isolation_level(old_level)

Don't worry about the isolation_level logic here; that just allows you to run the VACUUM ANALYZE command from Python using the transaction-based psycopg2 adapter.

[ 191 ]

GIS in the Database

It is possible to set up an autovacuum daemon that runs automatically after a given period of time, or after a table's contents has changed enough to warrant another vacuum. Setting up an autovacuum daemon is beyond the scope of this book.

Once you have run the VACUUM ANALYZE command, the query optimizer will be able to start optimizing your queries. As with MySQL, you can see how the query optimizer works using the EXPLAIN SELECT command: psql> EXPLAIN SELECT * FROM cities WHERE ST_Contains(geom,pt); QUERY PLAN -------------------------------------------------------Seq Scan on cities

(cost=0.00..7.51 rows=1 width=2619)

Filter: ((geom && '010100000000000000000000000000000000000000'::geometry) AND _st_ contains(geom, '010100000000000000000000000000000000000000'::geometry)) (2 rows)

Don't worry about the Seq Scan part; there are only a few records in this table, so PostGIS knows that it can scan the entire table faster than it can read through an index. When the database gets bigger, it will automatically start using the index to quickly identify the desired records. The cost= part is an indication of how much this query will "cost", measured in arbitrary units that by default are relative to how long it takes to read a page of data from disk. The two numbers represent the "start up cost" (how long it takes before the irst row can be processed), and the estimated total cost (how long it would take to process every record in the table). Since reading a page of data from disk is quite fast, a total cost of 7.51 is very quick indeed. The most interesting part of this explanation is the Filter. Let's take a closer look at what the EXPLAIN SELECT command tells us about how PostGIS will ilter this query. The irst part is given here: (geom && '010100000000000000000000000000000000000000'::geometry)

[ 192 ]

Chapter 6

This makes use of the && operator, which searches for matching records using the bounding box deined in the spatial index. The second part of the ilter condition is: _st_contains(geom, '010100000000000000000000000000000000000000'::geometry)

This uses the ST_Contains() function to identify the exact geometries which actually contain the desired point. This two-step process (irst iltering by bounding box, then by the geometry itself) is exactly what we had to implement manually when using MySQL. As you can see, PostGIS does this for us automatically, resulting in a quick and accurate search for geometries that contain a given point.

SpatiaLite One of the disadvantages of using a lightweight database such as SpatiaLite is that the query optimizer is rather naive. In particular, the SpatiaLite query optimizer will only make use of B*Tree indexes; you can create a spatial R-Tree index, but it won't be used unless you explicitly include it in your query. For example, consider the following SQL statements: CREATE TABLE cities (id INTEGER PRIMARY KEY AUTOINCREMENT, name CHAR(255)); SELECT AddGeometryColumn('cities','geom',4326,'POLYGON',2); INSERT INTO cities (name,geom) VALUES ('London', GeomFromText(wkt, 4326);

This creates a cities table, deines a spatial index and inserts a record into the table. Because SpatiaLite uses triggers to automatically update the spatial index as records are added, updated, or deleted, the preceding statements would correctly create the spatial index and update it as the new record is inserted. However, if we then issue the following query: SELECT * FROM cities WHERE Contains(geom, pt);

The SpatiaLite query optimizer won't know about the spatial index, and so will ignore it. We can conirm this using the EXPLAIN QUERY PLAN command, which shows the indexes used by the query: sqlite> EXPLAIN QUERY PLAN SELECT * FROM cities WHERE id < 100;

[ 193 ]

GIS in the Database 0|0|0|SEARCH TABLE cities USING INTEGER PRIMARY KEY (rowid





Your Python code would then look like this: import mapnik map = mapnik.Map(800, 400) mapnik.load_map(map, "sampleXMLStylesheet.xml")

[ 349 ]

Using Python and Mapnik to Generate Maps datasource = mapnik.Shapefile(file="TM_WORLD_BORDERS-0.3/" + "TM_WORLD_BORDERS-0.3.shp") layer = mapnik.Layer("layer") layer.datasource = datasource layer.styles.append("My Style") map.layers.append(layer) map.zoom_to_box(mapnik.Box2d(-180, -90, +180, +90)) mapnik.render_to_file(map, "map.png")

Note how we simply exclude the

Edit Shapefile

[ 445 ]

ShapeEditor – Selecting and Editing Features Please choose a feature to edit





Cancel



This is only the basic outline for this template, but it gives us something to work with. With the Django development server running (python manage.py runserver in a terminal window), go to http://127.0.0.1:8000/editor click on the Edit hyperlink for a shapeile. You should see the basic outline for the select feature page:

[ 446 ]

Chapter 11

Notice that we created a element to hold the OpenLayers map, and we use a CSS stylesheet to give the map a ixed size and border. The map itself isn't being displayed yet, because we haven't written the JavaScript code needed to launch OpenLayers. Let's do that now. Add the following

Also, change the tag deinition to look like this:

Notice that there are two

Edit Feature

{{ form.as_table }}

[ 464 ]

Chapter 11

	{% for attr in attributes %} {% endfor %} {{ attr.0 }} {{ attr.1 }}
	Cancel

This template uses an HTML table to display the form, and uses the form.as_table template function call to render the form as HTML table rows. We then display the list of feature attributes within a sub-table, and inally include Save and Cancel buttons at the bottom.

[ 465 ]

ShapeEditor – Selecting and Editing Features

With all this code written, we are inally able to edit features within the ShapeEditor:

[ 466 ]

Chapter 11

Within this editor, you can make use of a number of GeoDjango's built-in features to edit the geometry: •

You can click on the Edit Geometry tool (

• •

You can click on the Add Geometry tool ( ) to start drawing a new geometry. When a geometry is selected, you can click on a dark circle and drag it to move the endpoints of a line segment. When a geometry is selected, you can click on a light circle to split an existing line segment in two, making a new point which can then be dragged. If you hold the mouse down over a dark circle, you can press the Delete key (or type D) to delete that point. Note that this only works if the geometry has more than three points. You can click on the Delete all Features hyperlink to delete the current feature's geometries. We'll look at this hyperlink in more detail shortly.

• •

•

) to select a feature for editing.

Once you have inished editing the feature, you can click on the Save button to save the edited features, or the Cancel button to abandon the changes. While this is all working well, there is one rather annoying quirk: GeoDjango lets the user remove the geometries from a map by using a hyperlink named Delete all Features. Since we're currently editing a single feature, this hyperlink is rather confusingly named: what it actually does is delete the geometries for this feature, not the feature itself. Let's change the text of this hyperlink to something more meaningful. Go to the copy of Django that you downloaded, and navigate to the contrib/gis/ templates/gis/admin directory. In this directory is a ile named openlayers. html. Take a copy of this ile, and move it into your editor application's templates directory, renaming it to openlayers-custom.html. Open your copy of this ile, and look near the bottom for the text Delete all Features. Change this to Clear Feature's Geometry, and save your changes. So far so good. Now we need to tell the GeoDjango editing widget to use our custom version of the openlayers.html ile. To do this, edit your utils.py module and ind your deinition of the get_map_form() function. Replace the line which deines the admin_instance variable with the following highlighted lines: def get_map_form(shapefile): geometry_field = calc_geometry_field(shapefile.geom_type) class CustomGeoModelAdmin(admin.GeoModelAdmin): map_template = "openlayers-custom.html"

[ 467 ]

ShapeEditor – Selecting and Editing Features adminInstance = CustomGeoModelAdmin(Feature, admin.site) field = Feature._meta.get_field(geometry_field) widget_type = admin_instance.get_map_widget(field) class MapForm(forms.Form): geometry = forms.CharField(widget=widget_type(), label="") return MapForm

If you then try editing a feature, you'll see that your customized version of the openlayers.html ile is being used:

By replacing the template, and by creating your own custom subclass of GeoModelAdmin, you can make various changes to the appearance and functionality of the built-in editing widget. If you want to see what is possible, take a look at the modules in the django.contrib.gis.admin directory.

Adding features We'll next implement the ability to add a new feature. To do this, we'll put an Add Feature button onto the edit shapefile view. Clicking on this button will call the "edit feature" URL, but without a feature ID. We'll then modify the edit feature view so that if no feature ID is given a new Feature object is created. Open the editor application's views.py module, ind the edit_shapefile() function, and add the following highlighted lines to this function: def editshapefile(request, shapefile_id): try: shapefile = Shapefile.objects.get(id=shapefile_id) except Shapefile.DoesNotExist: raise Http404 tms_url = "http://"+request.get_host()+"/tms/" find_feature_url = "http://" + request.get_host() \ + "/editor/find_feature" add_feature_url = "http://" + request.get_host() \ + "/editor/edit_feature/" \ + str(shapefile_id)

[ 468 ]

Chapter 11 return render(request, "select_feature.html", {'shapefile' : shapefile, 'find_feature_url' : find_feature_url, 'add_feature_url' : add_feature_url, 'tms_url' : tms_url})

Then edit the select_feature.html template and add the following highlighted lines to the body of this template:

Edit Shapefile Please choose a feature to edit





Add Feature

Cancel



This will place an Add Feature button onto the "select feature" page. Clicking on that button will call the URL http://127.0.0.1:8000/editor/edit_feature/N (where N is the record ID of the current shapeile). We next need to add a URL pattern to support this URL. Open the editor application's urls.py module and add the following entry to the URL pattern list: url(r'^edit_feature/(?P\d+)$', 'edit_feature'), # feature_id = None -> add.

Then go back to views.py and edit the function deinition for the edit_feature() function. Change the function deinition to look as follows: def editFeature(request, shapefile_id, feature_id=None):

Notice that the feature_id parameter is now optional. Now ind the following block of code: try: feature = Feature.objects.get(id=feature_id)

[ 469 ]

ShapeEditor – Selecting and Editing Features except Feature.DoesNotExist: return HttpResponseNotFound()

You need to replace this block of code with the following: if feature_id == None: feature = Feature(shapefile=shapefile) else: try: feature = Feature.objects.get(id=feature_id) except Feature.DoesNotExist: return HttpResponseNotFound()

This will create a new Feature object if the feature_id is not speciied, but still fail if an invalid feature ID was speciied. With these changes, you should be able to add a new feature to the shapeile. Go ahead and try it out: run the Django web server if it's not already running and click on the Edit hyperlink for your imported shapeile. Then click on the Add New Feature hyperlink, and try creating a new feature. The new feature should appear on the Select Feature view:

[ 470 ]

Chapter 11

Deleting features We next want to let the user delete an existing feature. To do this, we'll add a Delete Feature button to the edit feature view. Clicking on this button will redirect the user to the delete feature view for that feature. Edit the edit_feature.html template, and add the following highlighted lines to the section of the template:

{{ form.as_table }} ...

Notice that we've used for this button. This will submit the form, with an extra POST parameter named delete. Now go back to the editor application's views.py module again, and add the following to the top of the edit_ feature() function: if request.method == "POST" and "delete" in request.POST: return HttpResponseRedirect("/editor/delete_feature/" + shapefile_id+"/"+feature_id)

We next want to create the delete feature view. Open the editor application's urls.py and add the following to the list of URL patterns: url(r'^delete_feature/(?P\d+)/' + r'(?P\d+)$', 'delete_feature'),

Next, create a new ile named delete_feature.html in the templates directory, and enter the following text into this ile:

ShapeEditor

[ 471 ]

ShapeEditor – Selecting and Editing Features Delete Feature

Are you sure you want to delete this feature?

Delete   Cancel



This is a simple HTML form that conirms the deletion. When the form is submitted, the POST parameter named confirm will be set to 1 if the user wishes to delete the feature. Let's now implement the view which uses this template. Open the editor application's views.py and add the following new view function: def delete_feature(request, shapefile_id, feature_id): try: feature = Feature.objects.get(id=feature_id) except Feature.DoesNotExist: return HttpResponseNotFound() if request.method == "POST": if request.POST['confirm'] == "1": feature.delete() return HttpResponseRedirect("/editor/edit/" + shapefile_id) return render(request, "delete_feature.html")

As you can see, deleting features is quite straightforward.

Deleting shapeiles

The inal piece of functionality we'll need to implement is the Delete shapefile view. This will let the user delete an entire uploaded shapeile. The process is basically the same as for deleting features; we've already got a Delete hyperlink on the main page, so all we have to do is implement the underlying view.

[ 472 ]

Chapter 11

Go to the editor application's urls.py module and add the following entry to the URL pattern list: url(r'^delete/(?P\d+)$', 'delete_shapefile'),

Then edit views.py and add the following new view function: def delete_shapefile(request, shapefile_id): try: shapefile = Shapefile.objects.get(id=shapefile_id) except Shapefile.DoesNotExist: return HttpResponseNotFound() if request.method == "GET": return render(request, "delete_shapefile.html", {'shapefile' : shapefile}) elif request.method == "POST": if request.POST['confirm'] == "1": shapefile.delete() return HttpResponseRedirect("/editor")

Notice that we're passing the Shapefile object to the template. This is because we want to display some information about the shapeile on the conirmation page. Remember that shapefile.delete()doesn't just delete the Shapefile object itself; it also deletes all the objects associated with the Shapefile through ForeignKey ields. This means that the one call to shapefile.delete() will also delete all the Attribute, Feature, and AttributeValue objects associated with that shapeile.

Finally, create a new template named delete_shapefile.html, and enter the following text into this ile:

ShapeEditor

Delete Shapefile

Are you sure you want to delete the

[ 473 ]

ShapeEditor – Selecting and Editing Features "{{ shapefile.filename }}" shapefile?

Delete   Cancel



You should now be able to click on the Delete hyperlink to delete a shapeile. Go ahead and try it; you can always re-import your shapeile if you need it.

Using ShapeEditor

Congratulations! You have just inished implementing the last of the ShapeEditor's features, and you now have a complete working geospatial application built using GeoDjango. Using the ShapeEditor, you can import shapeiles, view their features and the feature attributes, make changes to the feature geometries, add and delete features, and then export the shapeile again. This is certainly a useful application. Even if you don't have a full-blown GIS system installed, you can now make quick and easy changes to a shapeile's contents using the ShapeEditor. And, of course, the ShapeEditor is a great starting point for the development of your own geospatial applications.

Further improvements and enhancements As with any new application, there are a number of ways in which the ShapeEditor could be improved. For example: •

Adding user signup and login, so that each user has his or her own private set of shapeiles, rather than every user seeing the entire list of all the uploaded shapeiles.

•

Adding the ability to edit a feature's attribute values.

•

Using a higher resolution base map. An obvious candidate for this would be the GSHHS high-resolution shoreline database.

[ 474 ]

Chapter 11

•

Adding a tile cache for our TMS server.

•

Using JavaScript to add a please wait pop-up message while a shapeile is being imported or exported.

•

Improving the reusability of the ShapeEditor's codebase. We've concentrated on learning how to use GeoDjango to build a working system, but with a suitable redesign the code could be made much more generic so that it could be used in other applications as well.

Feel free to make these improvements; you will learn a lot more about GeoDjango, and about geospatial development in general. As you work with the ShapeEditor, you'll probably come up with your own list of things you'd like to improve.

Summary

In this chapter, we inished implementing a sophisticated geospatial web application using GeoDjango, Mapnik, PostGIS, OGR, and pyproj. This application is useful in its own right, as well as being a springboard to developing your own geospatial web applications. We have learned: •

That we can easily create our own Tile Map Server using Mapnik and GeoDjango.

•

That we can include OpenLayers on our own web pages, independent of GeoDjango, and display map data from our Tile Map Server.

•

How to create a custom "click handler" to respond to mouse-clicks within an OpenLayers map.

•

That we can use AJAX calls to have the server respond to events within the web browser.

•

That GeoDjango provides a powerful query language for performing spatial queries without writing a single line of SQL.

•

How to "borrow" geometry editing widgets from GeoDjango and use them within your own web application.

•

That you can create your own GeoModelAdmin subclass to change the appearance and functionality of GeoDjango's geometry editing widgets.

•

That you can use a simple HTML form to conirm the deletion of a record.

[ 475 ]

ShapeEditor – Selecting and Editing Features

This completes our exploration of GeoDjango, and also completes this book. Hopefully you have learned a lot about geospatial development, and how to create geospatial applications using Python. With these tools at your disposal, you are now ready to start developing your own complex geospatial systems. Have fun!

[ 476 ]

Index Symbols __unicode__ method 388

authentication system, Django application 364 azimuthal projection 32

A

B

add_dash() method 313 AddGeometryColumn() function 177 AddPoint() 116 admin interface, Django application 364 admin system, Django application about 384 adding 384 admin URLs, adding 385 database objects, adding 384 database, resynchronizing 384 working with 386-391 afine transformation, DEM 121 also rules 309, 310 angular distance 252 25 Application Programming Interfaces (APIs) 18 application review about 262 performance 266 performance improvement, analyzing 282 performance, improving 269-271 problem, searching 266-268 quality 264 tiled shorelines, calculating 271-279 tiled shorelines, using 280-282 usability 262, 263 aspect ratio 244 AsText() function 178 attributes 10

Band Interleaved by Line (BIL) format 40 Band Interleaved by Pixel (BIP) format 40 band.ReadRaster() method 122 Band Sequential (BSQ) format 40 base layer 438 base map setting up 435, 437 best practices, geospatial databases about 181 appropriate spatial reference, using 183 databases, using supporting geographies 184 features, transforming 185 features, transforming outset 185 limits, spatial query optimizer 189 on-the-ly transformations, avoiding within query 186, 187 spatial indexes, using 188 spatial references, monitoring 181, 182 unprojected coordinates, using 185 Bing 16 bounding 82

C calc_search_radius() function 457 changesets 83 civic location 25 colors using 335, 336

common spatially-enabled databases about 179 MS SQL Server 180 Oracle 180 conditions, rules 305 conic projection 31, 32 coordinates 10 coordinate system about 34 projected coordinate systems 34 unprojected coordinate systems 34 coordinate transformation 128 Core Based Statistical Areas (CBSAs) 141 country bounding boxes, geospatial data calculating 112-114 saving, into shapeile 114-118 coverage format 40 CreateSpatialIndex() function 178 crosses() method 68 cursor.execute() method 165 cylindrical projections 29, 30

D dash segments 313 data downloading 216 GEOnet Names Server 217 GNIS 216, 217 GSHHS 216, 220, 221 importing 217 places' name data 222-227 World Borders Dataset 216-220 worldwide places' name data 227-229 data models attribute 378 AttributeValue object 379 deining 377 feature 379 models.py ile 380, 381 shapeile, importing 378 Datasource object 73 datasources Gdal 300 MemoryDatasource 304 Org 301, 302 OSM 303

PostGIS 298, 299 shapeile 297 setting up 245 SQLite 302, 303 datum about 36, 126 changing 132-134 NAD 27 36 NAD 83 37 reference points 36 WGS 84 37 decimal degrees 13 defaultHandlerOptions dictionary 450 design, Mapnik Layers 73 Styles 73 Symbolizers 73 Destroy() method 116, 417 digital elevation maps (DEM) 94 about 119 afine transformation 121 used, for analyzing height data 119-121 Digital Elevation Model (DEM) format 40 Digital Raster Graphic (DRG) format 40 DISTAL about 207 basic worklow 207-210 database, building 211 database, designing 211 DISTAL application about 229 implementing 229-235 select area script 236 select country script 233-235 shared database module 232, 233 show result script 248 DISTAL database designing 211-215 distance about 25 angular distance 25 linear distance 25 travelling distance 26 Distance-based Identiication of Shorelines, Towns And Lakes. See DISTAL distance features angular distance, using 253 [ 478 ]

hybrid approach 255-259 identifying, manually 252 projected coordinates, using 254 result, displaying 259-261 Django administration working with 386-391 Django application admin interface 364 authentication system 364 data-entry forms 370 downloading 363 markup application 364 messages framework 364 model 365 sessions system 364 settings ile 364 sitemaps framework 364 structure 363 syndication system 364 templates 365, 369-371 URL dispatching 366-368 view 365, 366 DSG (Feature Designation Code) ield 227

features adding 468-470 deleting 471, 472 editing 460-468 ields 113 ilter 290 Filter() constructor 305 ilters, Mapnik about 305, 306 scale denominators 307, 308 ind_feature() function 455 ind_feature_url parameter 453 ind feature view implementing 454-460 ind_places_within() function 257, 259 indPoints() function 47 ixture 438 form.as_table template function 465 forms 400 FWTools installer URL 42

E

gamma correction 317 GDAL 18, 102, 333 Gdal data source 300 gdaldem utility 105 GDAL design about 52 afine transformation 53 band metadata 54 band raster size 54 color table 54 coordinate system 54 dataset 52 drivers 54 georeferencing transform 53 Ground Control Points (GCPs) 54 metadata 54 raster band 53 raster data 54 raster size 53 GDAL example code 55 GDAL, for Mac OS X 42 GDAL/OGR about 52

G

edit_feature() function 462 edit_shapeile() function 453 edit_shapeFile() view function 452 else rules 309 envelope 113 ESRI format 102 European Petroleum Survey Group (EPSG) 182 EveryBlock URL 71 example map, Mapnik creating 292-296 expandRect() function 273 export_data() function 414, 418 export_shapeile() view function 415

F FC (Feature Classiication) ield 227 Feature Classiication (FC) 107 Feature Designation Code (DSG) 107 feature layer 438

[ 479 ]

availability 58 documentation 58 GDAL Python library downloading 42 Generalized Search Tree (GiST) 160 generateMap() function 345 342 Generic Mapping Tools (GMT) about 91 URL 91 geocode 13 geocoder 173 Geod about 61 fwd() method 61 inv() method 61 npts() method 62 geodetic location 24 GeoDjango 363 Geographical Information System (GIS) vendors 11 Geographic Names Information System. See GNIS geographies 172 Geography Markup Language (GML) format 41 GeoJSON format 41 Geolocation 18 geometries 172 geometry 113 GeometryCollection class 68 geometry types GeometryCollection 455 LineString 455 MultiLineString 455 MultiPoint 455 MultiPolygon 455 Point 455 Polygon 455 geometry units converting 146 Shoshone latitude, calculating 154, 155 Shoshone longitude, calculating 154, 155 standardizing 146 Thai-Myanmar border length, calculating 147-154 GEOnet Names Server about 106, 211, 217

data format 107 obtaining 107 screenshot 106 using 107 GeoRSS 18 geospatial 9 geospatial calculations parks, identifying in or near urban areas 141-145 performing 141 geospatial data about 10 analyzing 12-14, 66 attributes 10 coordinates 10 country bounding boxes, calculating 112-114 country bounding boxes, saving into shapeile 114-118 GDAL 52 GDAL design 52 GDAL example code 55 geometries, saving into text ile 140, 141 height data, analyzing with DEM 119-125 manipulating 66 OGR 52 OGR design 56 OGR example code 57 pre-requisites 111, 112 reading 112 representing 135 sources 105 storing 135, 136 Thailand and Myanmar border, calculating 136-140 visualizing 14-70 writing 112 Geospatial Data Abstraction Library. See GDAL geospatial databases best practices 181 geospatial databases, Python used comparing 204 MySQL, working with 195-198 PostGIS, working with 199, 200 prerequisites 195 SpatiaLite, working with 201-204 [ 480 ]

geospatial data source. See sources, geospatial data geospatial development about 10 overview 17 geospatial development applications geospatial data, analyzing 12-14 geospatial data, visualizing 14-16 geospatial mash-up, creating 16, 17 geospatial mash-up creating 16 GeoTIFF iles 105 getattr() function 464 get_country_datasource() function 260 get_datasource() function 246 get_map_form() function 461 get_map_widget() 461 GetNoDataValue() method 125 get_ogr_feature_attribute() function 418 get_shoreline_datasource() function 280 GIS common spatially-enabled databases 179 open source spatially-enabled databases 163 spatial indexes 160 spatially-enabled databases 159 GIS concepts about 21 coordinate system 34 datums 36 distance 25 location 22 projection 29 shapes 37 units 27 GIS data working manually 41-48 GIS data format about 39 micro-formats 40 raster format data 40 vector format data 40 Global Land Cover Facility 97 Global Land One-kilometer Base Elevation. See GLOBE Global Positioning System (GPS) 17 Global self-consistent, hierarchical,

high-resolution shoreline database. See GSHHS GLOBE about 100 data format 100 data, obtaining 101, 102 GLOBE DEM data 100 GML 18 GNIS about 107, 216 data format 108 obtaining 109 screenshot 108 using 109 GNIS Database 211 Google Earth 17, 18 Google Maps 17 Google Maps API 16 great circle distance calculation 47 GSHHS about 90, 216 data format 91, 92 obtaining 92 screenshot 90

H handleResponse() callback function 452 Haversine formula URL 48 height data, geospatial data analyzing, DEM used 119-125 HTML Forms 235

I import_data() function 399, 403 import shapeile view function 399-403 imposm URL 84

J jurisdictional locations 25

K KML 18

[ 481 ]

L labels drawing 319 Landsat about 94 data format 95 Landsat imagery obtaining 95-97 Layer objects 287 layers 336 Layers, Mapnik 73 libspatialite extension loading 176 linear distance 25 linear ring 38 LinearRing class 68 line-drawing options dashed and dotted lines 313 line caps 312 line color 311 line joins 312 line width 312 opacity 312 LinePatternSymbolizer 315 lines drawing, onto map 310 linestring 38 LineString class 67 LineSymbolizer about 288, 310 using 311 Linux SpatiaLite, installing 174 list_countries() function 235 list shapeiles view implementing 393-397 list_shapeiles() view function 394-396 locations about 22 measuring 22-24 LULC datailes 127

M Mac OS X SpatiaLite, installing 174

map deinition ile 346-350 Map Deinition File 291 MapGenerator about 341 interface 342 main map layer, creating 343, 344 map, rendering 345 points, displaying on map 344, 345 mapGenerator.generateMap() function 246 mapGenerator.py module 342 Mapnik about 16, 71, 285, 286 availability 76 data sources 296, 297 design 72 documentation 76 example code 74, 75 example map, creating 292-296 features 71, 287 ilters 304 layers 336 map, generating 287 map rendering 339, 341 maps 336 Polygons layer 290 Python documentation 296 rules 304 styles 304 symbolizers 310 URL 76, 286 Mapnik Datasource object setting up 296, 297 mapnik.Layer class methods 338 mapnik.Map class attributes 337, 338 methods 337, 338 mapnik.Shapeile() function 297 mapnik.SQLite() function 302 Mapnik Wiki URL 76 map rendering 339 maps 336 MapServer 16 markup application, Django application 364 MBRContains() function 165 [ 482 ]

MemoryDatasource 304 meridians 24 messages framework, Django application 364 micro-formats Geography Markup Language (GML) 41 GeoJSON 41 Well-known Binary (WKB) 40 Well-known Text (WKT) 40 minimum bounding rectangle 161, 166 models, Django 365 models.py ile editing 380, 381 mouse clicks, ShapeEditor application intercepting 449-453 MS SQL Server 180 MS Windows SpatiaLite, installing 174 MultiLineString class 68 MultiPoint class 68 MultiPolygon class 68 MySQL about 163-166 accessing, from Python programs 164 disadvantages 167 downloading 163 MySQLdb URL 165 MySQL-Python driver 164 164 MySQL query optimizer 189

N NAD 27 36 NAD 83 37 National Elevation Dataset. See NED National Map Viewer 103, 104 Natural Earth, raster-format data about 98 data format 99 obtaining 99, 100 raster maps 98 using 100 Natural Earth, vector-format data about 88 cultural map data 88 data format 89

data, obtaining 89 data, using 89 physical map data 88, 89 URL 88 nature of map projections 33 NED about 102 data format 103 data, obtaining 103 data, using 104, 105 no data value 125 NT (Name Type) ield 227

O OGR 18 Ogr data source 301, 302 OGR design about 56 attributes 57 data source 56 feature 56 geometry 57 layers 56 spatial reference 56 OGR example code 57 OGR Shapeile deining 415 onClick() function 450 Open Geospatial Consortium about 18 URL 18 OpenLayers used, for displaying map 444-449 OpenLayers.Control.Click class 450 OpenLayers.Request.GET function 452 open source spatially-enabled databases about 163 MySQL 163 PostGIS 168 SpatiaLite 174 OpenStreetMap about 80 data format 81 data, working with 84 geospatial data, obtaining 82 geospatial data, using 82 [ 483 ]

screenshot 80 URL 71, 80 OpenStreetMap API 82, 83 OpenStreetMap geocoder 18 Oracle 180 Oracle Locator 180 Oracle Spatial 180 orthorectiication 95 osm2pgsql tool 84 OsmApi 83 OSM data source 303 os.path.join() function 298 overlay 15

P painters algorithm 287 parallels 24 parameters 368 Planet.osm about 83 extracts 84 mirror site 84 point 37 Point class 67 points drawing 329 PointSymbolizer 330, 331 polygon 38 Polygon class 68 polygon.contains(point) method 166 polygon-drawing options attribute 317 ill color 316 gamma correction 317 PolygonPatternSymbolizer 318 polygons drawing 316 PolygonSymbolizer 288, 316 polylines 38 PostGIS about 12, 18, 168 coniguring 169, 170 documentation 172, 173 downloading 168 features 173 installing 168

using 170, 171 PostGIS database setting up, for ShapeEditor application 371, 372 PostGIS datasource 298 PostGIS manual URL 172 PostGIS query optimizer 191 PostgreSQL database 168 PostgreSQL manual URL 172 prime meridian 24 Proj 60 PROJ.4 18, 65 projected coordinate system 35 projection about 11, 29, 126 availability 64 azimuthal projection 32 changing 127-131 conic projection 31 cylindrical projections 29 dealing with 59 design 60 documentation 63 example code 62 nature of map projections 33 pyproj 59 Proj Python library 13 Psycopg installing 169 Psycopg database 168 Psycopg documentation URL 172 pyproj library about 60 for Linux 64 for Macintosh 64 for MS Windows 64 Geod 61, 62 Proj 60 pysqlite installing 175 URL 175 Python about 7 features 7, 8 [ 484 ]

URL 7 Python Database API 165 Python Database Programming Wiki page URL 165 Python Package Index about 9 geospatial development 9 URL 9 Python Standard Libraries 8

Q quality Lat/Long coordinate problems 264-266 place name issues 264 query optimization process MySQL 189, 190 PostGIS 191, 192 SpatiaLite 193

R raster format data about 40 BIL 40 BIP 40 BSQ 40 Digital Elevation Model (DEM) 40 Digital Raster Graphic (DRG) 40 raster images drawing 333, 335 raster maps about 98 Cross-Blended Hypsometric Tints 98 Natural Earth 1 98 Natural Earth 2 98 Ocean Bottom dataset 98 Shaded Relief imagery 98 RasterSymbolizer about 333 uses 335 ReadRaster() method 122, 123 reference points 35 roads drawing 314 root() function 427 R-Tree data structures 160-163 R-Tree indexes 161

rule 290 rules, Mapnik also rules 309, 310 conditions 305 else rules 309 symbolizers 305

S scale denominators 307, 308 select area script about 236-240 bounding box, calculating 241 data source, setting up 245, 246 map image, rendering 246, 247 map's dimension, calculating 242-244 select country script 233-235 select_feature.html template 469 service() function 429 sessions system, Django application 364 setattr() function 464 SetField() method 418 set_ogr_feature_attribute() function 418 shaded relief imagery 94 ShapeEditor application about 353 admin system 384-388 building 353 database, setting up 371, 372 data models, deining 377 deining 375 designing 357 enhancements 474 feature, editing 362 features 356, 357 features, adding 468 features, deleting 471, 472 features, editing 460 feature, selecting 360, 361 feature, selecting for edit 424 list shapeiles view, implementing 393, 394 mouse clicks, intercepting 449-453 prerequisites 362 setting up 373, 374 shapeile, exporting 362, 414 shapeile, importing 355-398 shapeiles, deleting 472 [ 485 ]

shared application, creating 375-377 Tile Map Server, implementing 425 using 474 web interface 355 worklow 354, 424 shapeile datasource 297 Shapeile format 40 Shapeile object adding, to database 406 shapeiles attributes, deining 407 attributes, storing 410-413 cleaning up 413 contents, importing 405 deleting 472, 473 features, storing 408, 409 importing 398, 399 opening 405 shapeiles, exporting about 414 attributes, saving into shapeile 418 features, saving into shapeile 416, 417 OGR Shapeile, deining 415 shapeile, compressing 420 temporary iles, deleting 420 ZIP archive, returning to user 420 Shapely about 66 availability 69 design 66, 67 documentation 69 example code 68, 69 GeometryCollection class 68 LinearRing class 68 LineString class 67 MultiLineString class 68 MultiPoint class 68 MultiPolygon class 68 Point class 67 Polygon class 68 shapes about 37 linestring 38 point 37 polygon 38 shared database module 232, 233 shared.utils module 456

ShieldSymbolizer 331, 332 show result script about 248 clicked-on point, identifying 249, 250 distance features, identifying 250, 251 showResults.py script 282 simple features format 40 sitemaps framework, Django application 364 source code format, Linux 164 source code format, Mac OS X 164 sources, geospatial data about 105 GEOnet Names Server 106 GNIS 107 selecting 109 sources, raster-format geospatial data about 93 GLOBE 100 Landsat 94 National Elevation Dataset (NED) 102 Natural Earth 98 sources, vector-format geospatial data GSHHS 90 Natural Earth 88 OpenStreetMap 80 TIGER 84 World Borders Dataset 92 spatial datatypes 159 spatial functions 160 spatial indexes about 160 R-Tree indexes 161 SpatiaLite about 174 accessing, from Python 175 capabilities 178, 179 documentation 176 installing 174 installing, on Linux 174 installing, on Mac OS X 174 installing, on MS Windows 174 online documentation 176 pysqlite, installing 175 URL 176 using 177, 178 SpatiaLite query optimizer 193 [ 486 ]

spatial joins 160 spatially-enabled databases about 159, 160 functioning 159 spatial queries 160 spatial query functions 454 spatial reference 73, 114, 136, 181 Spatial Reference Identiier (SRID) 182 SQLite data source 302, 303 ST_AsText() function 172 ST_CONTAINS() function 258 ST_DWITHIN() function 258 ST_GeomFromText() function 172 Styles, Mapnik 73, 290 subselect query about 299 using 299 Symbolizers, Mapnik about 73, 288 labels, drawing 319 lines, drawing 310 LineSymbolizer 288 points, drawing 329 polygons, drawing 316 PolygonSymbolizer 288 TextSymbolizer 288 syndication system, Django application 364

T template, Django 369-371 terminology, TMS protocol Tile 425 Tile Map 425 Tile Map Server 425 Tile Map Service 425 Tile Set 425 TextSymbolizer about 288, 319 advanced text placement and formatting 328, 329 capitalization, controlling 328 character spacing, controlling 327 font, selecting 320 labels, repeating 322, 323 labels, splitting across multiple lines 326 line spacing, controlling 327

position, adjusting of text 325 semi-transparent text, drawing 320 text, drawing on dark background 324, 325 text overlap, controlling 323, 324 text placement, controlling 320, 321 text, selecting for display 320 TIGER about 84, 85 data format 86 data, obtaining 87 data, using 88 TIGER/Line format 40 Tile 425 Tile Map 425 tileMap() function 430 432 Tile Map Server completing 443, 444 Tile Map Server (TMS) 361 Tile Map Service 425 tilePolys array 274 tile rendering about 438 base layer, deining 440, 441 feature layer, deining 441, 442 map, setting up 440 map tile, rendering 442 query parameters, parsing 439, 440 Tile Set 425 TMS protocol about 425 base map, setting up 435-438 error handling 429 implementing 425-434 map, displaying with OpenLayers 444-449 terminology 425 tile, rendering 438 Topologically Integrated Geographic Encoding and Referencing System. See TIGER travelling distance 26, 27 triggers 193

U units 27, 28 Universal Transverse Mercator (UTM) coordinate system 36 [ 487 ]

Universal Transverse Mercator (UTM) projection 127 unprojected coordinates 11 unprojected coordinate system 34 unwrap_geos_geometry() function 417 uploaded shapeile extracting 403, 404 URLConf 367 URL dispatching, Django 366, 367 usability 262 US Census Bureau URL 41 utils.calc_geometry_ield() 464 utils.get_map_form() 464 utils.get_ogr_feature_attribute() function 418

V vector format data about 40 coverage 40 shapeile 40 simple features 40 TIGER/Line 40 vector-format geospatial data about 80 sources 80 view, Django 366 Virtual Datasource (VRT) format 301

W WCS 18 WebGIS website URL 127 Well-known Binary (WKB) format 40, 136 Well-known Text (WKT) format 40, 135 WFS 18 WGS 84 37 WMS 18 World Borders Dataset about 92, 93, 216, 218 data format 93 downloading 112 obtaining 93 World Data Bank II 90 world reference system (WRS) 96 World Vector Shoreline 90

X XCode about 65 installing 65

[ 488 ]

Thank you for buying

Python Geospatial Development: Second Edition

About Packt Publishing

Packt, pronounced 'packed', published its irst book "Mastering phpMyAdmin for Effective MySQL Management" in April 2004 and subsequently continued to specialize in publishing highly focused books on speciic technologies and solutions. Our books and publications share the experiences of your fellow IT professionals in adapting and customizing today's systems, applications, and frameworks. Our solution based books give you the knowledge and power to customize the software and technologies you're using to get the job done. Packt books are more speciic and less general than the IT books you have seen in the past. Our unique business model allows us to bring you more focused information, giving you more of what you need to know, and less of what you don't. Packt is a modern, yet unique publishing company, which focuses on producing quality, cutting-edge books for communities of developers, administrators, and newbies alike. For more information, please visit our website: www.packtpub.com.

About Packt Open Source In 2010, Packt launched two new brands, Packt Open Source and Packt Enterprise, in order to continue its focus on specialization. This book is part of the Packt Open Source brand, home to books published on software built around Open Source licences, and offering information to anybody from advanced developers to budding web designers. The Open Source brand also runs Packt's Open Source Royalty Scheme, by which Packt gives a royalty to each Open Source project about whose software a book is sold.

Writing for Packt We welcome all inquiries from people who are interested in authoring. Book proposals should be sent to [email protected]. If your book idea is still at an early stage and you would like to discuss it irst before writing a formal book proposal, contact us; one of our commissioning editors will get in touch with you. We're not just looking for published authors; if you have strong technical skills but no writing experience, our experienced editors can help you develop a writing career, or simply get some additional reward for your expertise.

Python Geospatial Development ISBN: 978-1-849511-54-4

Paperback: 508 pages

Build a complete and sophisticated mapping application from scratch using Python tools for GIS development 1.

Build applications for GIS development using Python

2.

Analyze and visualize Geo-Spatial data

3.

Comprehensive coverage of key GIS concepts

4.

Recommended best practices for storing spatial data in a database

Programming ArcGIS 10.1 with Python Cookbook ISBN: 978-1-849694-44-5

Paperback: 304 pages

Over 75 recipes to help you automate geoprocessing tasks, create solutions, and solve problems for ArcGIS with Python 1.

Learn how to create geoprocessing scripts with ArcPy

2.

Customize and modify ArcGIS with Python

3.

Create time-saving tools and scripts for ArcGIS

Please check www.PacktPub.com for information on our titles

OpenStreetMap ISBN: 978-1-847197-50-4

Paperback: 252 pages

Be your own Cartographer 1.

Collect data for the area you want to map with this OpenStreetMap book and eBook

2.

Create your own custom maps to print or use online following our proven tutorials

3.

Collaborate with other OpenStreetMap contributors to improve the map data

GeoServer Beginner’s Guide ISBN: 978-1-849516-68-6

Paperback: 350 pages

Share and edit geospatial data with this open source software server 1.

Learn free and open source geospatial mapping without prior GIS experience

2.

Share real-time maps quickly

3.

Learn step-by-step with ample amounts of illustrations and usable code/list

Please check www.PacktPub.com for information on our titles