The hills are alive with the sound of shadows

I rode the train from New York to Boston, last October.

Because I haven't yet implemented a local tile cache in ModestMaps (well, I have actually but that's another story) I decided to work on the index pages for the pocketMMap of Montreal I'd offered to make for Design Engaged. What started out as a casual suggestion to generate some simple, possibly annotated, paper maps that people could walk around with during the weekend quickly turned in to a gargantuan MyMaps database spanning neighbourhoods that would never be visited and completely unrelated random events from my past.

But why the hell not. I've needed to put the giant list of things to do in Montréal in one place for a while now and there's nothing like having too much of something to demonstrate how the thing that will hold it is going to break. Specifically:

• If you have too many points spread over too wide an area (the bounding box used the determine the actual map image for a pocketMMap) then you'll end up with an image that spans to many sheets of paper to fold and nest. Experience has shown that three sheets is the upper limit which really means that four is possible and two is borderline manageable. A single sheet of paper remains the ideal and I'll touch on that more below.
• If you decide to limit yourself to a fixed number of sheets then the only way you can fit the entirety of an otherwise oversized viewing area is to zoom out. This has two consequences. First you reduce the fidelity of your underlying map. This is a visualization problem in and of itself exacerbated by the mundane realities of consumer grade printers. Secondly, if you have a lot of items to render then you also collapse the physical paper-space on which they can be rendered meaning you have to play tricks to make all the markers fit in a limited space. This usually involves floating markers on some kind of visual anchor unless of course that means the anchor bleeds off the page in which case you have to get your hands dirty with collision detection algorithms and hope that you don't then end up covering the entire map surface with markers. (That would be depressing, boring and ultimately defeat the purpose.)

Since I'd already decided to aim for fail I settled on four sheets. So, Monday morning came and I was on a train sitting at a table with three lawyers from what I guessed was a big New York City real estate firm deep in talk of the end days and other water cooler banter with no network connection.

pocketMMaps are created as PDF files using the ReportLab Python libraries, a handy set of helper tools for generating the actual PostScript code that make up a PDF file. PostScript was created by Adobe and remains the Wintermute of programming languages. The great unresolved question is whether Damian Conway's talk about programming in Klingon served as the design document for the language or whether the presentation is merely a reflection of what passes for rational thought at Adobe. Which is to say: No one ever writes plain old PostScript when they can avoid it.

Which is to say: We're all share-cropping on the helper libraries we can't be bothered to write ourselves.

Which is to say: ReportLab doesn't have an option to rotate text; a problem when creating an index of table of contents for a PocketMod-ish document what with the individual pages laid out 180 degrees from one another. The FPDF library, written in PHP, does and long-time readers will remember that all of the original PocketMod and Papernet related work was built on top of it. But ModestMaps is written in Python and I didn't feel like porting all that code to PHP which led, instead, to me writing a bare-bones webservices interface to ModestMaps which led to a lot more tinkering in Python-land as a way to create sexy vector based pinwins for other map-loving visualizations which finally led me to a train somewhere in Connecticut with no Internet connection and a wrapper library that doesn't rotate text. And three lawyers discussing inter-office strategery.

Which is to say: I have now written even more code that I don't feel like porting back to PHP.

So, in the absence of the actual data I needed to plot, I pretended that people and phone numbers were to my address book what the place names and page/marker indicators were to a table of contents in a pocketMMap.

And rendered that text as a rasterized image. And rotated it.

Please, tell it to the hand.

By the time I got to Boston I not only had the workings to build the index pages for a pocketMMap I also had an easy to create phone book that doesn't require electricity to operate which, I know, is hard to imagine anymore. I couldn't tell you what my home phone number is without looking at my phone.

Here's another unpleasant truth: Even when the network is always on, and even if you are comfortable sharing the contents of your address book with complete strangers like Google sync still sucks. It's a genuinely hard problem but I have mostly given up trying to understand or care how and why programs like iSync sometimes work and more often don't. I take for granted the horror stories told by the iPhone users who have watched helplessly as their address books are deleted by the happy magic MobileMe cloud fairies. I try to mitigate the inevitable synchronization disaster or database failures by routinely exporting everything to nothing more sophisticated that a text file and shuttling it between a variety of devices.

Which includes paper, now:

from pocketphone import pocketphone

width = 8.5
height = 11
margin = 0.5

number = {'name':'my name', 'number': '999-999-9999'}
out = "/path/to/my/pocketphone.pdf"

ph = pocketphone(width, height, margin)
ph.add_number(number)
ph.draw()
ph.save(out)

Or if you are like me and use the totally excellent ab2vcard application to periodically dump the contents your address book database to a bunch of text files, you can do this:

from pocketphone import vcard
                        
vcards = "/path/to/a/directory/fullof/vcards"

v = vcard(width, height, margin)
v.add_directory(vcards)
v.draw()
v.save(out)

And then automate the whole thing with the magic of cron:

# Every hour dump the contents of the address book database
# At ten past every hour render the contents of that as a pocketphone book

0 * * * * /usr/local/bin/ab2vcard -d /path/to/yer/abook/
10 * * * * /usr/local/bin/vbook.py -v /path/to/yer/abook -p /path/to/yer/abook/vbook.pdf

Meanwhile.

A few weeks later, in Montréal, I stood and watched as Raphael typed his number in to my phone. A month and some unremembered number of syncs later I went to call Raphael to coordinate a dinner plan only to discover that his number had disappeared from my phone. It's one thing when a rogue wave attacks you on the beach and soaks your phone before you've remembered to do a backup but otherwise it's genuinely irritating to have to be left wondering whether suddenly vanished data is due to a logic bug in your laptop's sync application or your phone's weird-ass and uncustomizable public/private sync settings or just a random attack of suck. Life is too short for this sort of thing. Nevermind trying to guess at the short list of phone numbers you might need to scribble down on a piece of paper just before the battery in your phone dies.

I mention all this not only because I was finally able to call Raphael having a pocketphone with me at the time or because the actual bits that created the pocketphone are the same bits used to the index pages for the Design Engaged pocketMMaps but because they're also a way to transform any old key-value list in to a handy booklet complete with (optional) categories.

The very creatively named pocketindex:

from pocketindex import pocketindex                    

# hey look! pixels instead of inches!!
width = 300
height = 500

legend = {'top_right':'vocabulary'}

data = [
	{'label': 'challenge', 'value': 'O HAI', 'tags':['lol', 'cat']},
        {'label': 'response', 'value': 'O RLY',}
]

idx = pocketindex(width, height)
pages = idx.create(data, legend)
pages[0].save("/home/asc/Desktop/test.png")

Which is exciting only in that it's a tool. Not at all exciting. Until you need it.

In theory, markers on a map should be the same. In practice there are always too many of them and they're always clustered too closely together. We are creatures of habit and given the chance we never shut up. This is reflected in our maps. It is a problem made worse by the tools used to build those maps not being able to adapt and keep pace with opportunities they create. The classic illustration of the problem remains red-dot fever when a map contains so many markers that by showing them all the map itself is complete hidden.

I don't have a working solution for this problem but I do know that instead of yet another pre-rerendered data layer that applies the same degree, or lack of, detail across all space I'd like to be able to generate map tiles that reflect the data being added to them. To stop faking it, as Mike says.

As usual, Mike is doing all the actual hard work particularly with tools like Cascadenik which is all about hiding the pain of generating Mapnik's XML-based stylesheets, from users and replacing it with a friendlier and more intuitive CSS-like syntax. (Mapnik is the rendering engine that the Open Street Maps crew uses to produce their maps and the XML stylesheets describe the look and feel of the map tiles.) I would love it if I could generate those stylesheets on the fly passing them along with a request to a tool like mod_tile and get back a freshly minted tile whose detail and definition are a function of the information I am trying to display.

11:24:16 AM Mike Migurski: so like, driven by DPI? that sort of thing? 
11:24:25 AM Aaron Straup Cope: driven by DPI?
11:24:39 AM Mike Migurski: detail-based
11:24:53 AM Mike Migurski: or you mean more subtly controlled than that
11:25:00 AM Mike Migurski: a detail knob =)
11:25:01 AM Aaron Straup Cope: stuff like : show all the large roads and
connectors but only draw street names and other details in areas with a
cluster of markers
11:25:08 AM Mike Migurski: ah
11:25:14 AM Mike Migurski: a detail spray can
11:25:19 AM Aaron Straup Cope: yes :-)
11:25:25 AM Mike Migurski: detail hillshading
11:25:38 AM Aaron Straup Cope: I am so copy-pasting this conversation in to the
post
11:25:49 AM Mike Migurski: hehe
11:26:01 AM Mike Migurski: detail brush
11:26:29 AM Aaron Straup Cope: yeah, like what maciej did but for the actual
tiles

What I really want is for someone to build an on-demand EC2-style instance running that service which I could fire up and point my code (read: Modest Maps) at to get new tiles. I don't actually care whether it's EC2 or not. I'd prefer something I could run locally, like a stand alone Java daemon, but given the dependency chain required to render tiles it seems that you still need a full-on fake computer/operating system rather than a sandboxed web application framework.

Which is to say: Markers in pocketMMaps still only sort of work as a chicken-and-egg proof of concept. They get drawn to the correct spot on the page and there is some rudimentary repositioning done to ensure that markers don't overlap each other on the y axis. What's still missing, entirely, is code to handle markers that bleed off the top or side of a page (often as a result of being repositioned) or in to the crease between two pages. Because there's literally no room to do postermap-style raining markers I will probably need to sit down and brush up on labelling algorithms before any clever solutions present themselves.

Which is to say: If you display a modicum of restraint selecting the points to include, it all Just Works. In fact, you can generate a fancy pocketMMap with markers and a table of contents from any old GeoRSS feed.

Like this:

from pocketMMap import GeoRSS

mm = GeoRSS(width, height, margin)
mm.draw_feed('http://maps.google.com/maps/blah/blah/blah')
mm.save(out)

Which produces this: pocketmmontreal.pdf !

The code will automagically calculate the size of the bounding box of the map to draw based on the set of points in the feed and parse each description element looking for Stikkit-style tag as magic words. Each tag will be treated as a category and all the places associated with it will be included under that header in the index at the end of the pocketMMap. This way you can look up a restaurant, for example, by both its cuisine and its neighbourhood. The exception, or magic magic word, is the skip tag which will cause that item to be excluded from the pocketMMap entirely. Here's an example:

<item>
    <guid isPermaLink="false">00045657b554aed0b9388</guid>
    <pubDate>Mon, 08 Sep 2008 00:43:31 +0000</pubDate>
    <title>Cosmos</title>
    <description>
    <![CDATA[<div dir="ltr">
    Best greasy breakfast in town. Tiny and full of hangover-fighting
    grease.<br><br>
    5843 Sherbrooke St West<br><br>

    tag as skip resto breakfast ndg cheapeats

    </div>]]>
    </description>
    <author>aaronofmontreal</author>
    <georss:point>45.469440 -73.618057</georss:point>
</item>

There is no good reason to continue (ab)using the description element for tags other than I started by using MyMaps and the wunderkids at Google still haven't added a proper tagging interface for the stuff you create there. Now that Dopplr has launched their new city pages I expect to see proper geo-enabled feeds for the corresponding tips pages any day now (damn you, Matt Jones!) and I will add support for whatever goofy element they think is appropriate for tags.

Were the pocketMMaps a raging success, in Montreal?

Not at all, but they were a start and they helped prove some suspicions about how to proceed from here. One thing that was immediately apparent is that it's not at all obvious to most people how they are supposed to fold a PocketMod book. The PocketMod site seems to address this problem by making people watch a video which is helpful but not terribly useful for a printed document. I seemed to remember that the Mapufacture kids had included a how-to diagram with their pocketmaps but in the end I never found anything to ask permission to use. So I made my own.

A handy how-to infographic is now included on the last page of both pocketMMap and pocketphone books and the source Omnigraffle file is included should anyone like to make improvements. I'm quite pleased with the illustration but then again I've been folding these things almost non-stop for two years so I won't pretend to know where the stumbling blocks are anymore.

Like everything else, the actual mechanics of how the diagram gets rendered are ugly and make you feel ashamed. Because PocketMod derived books can be printed on any sheet of paper than can be divided in eight there's no way to know what size a how-to diagram will need to be ahead of time. Originally, I had planned to include a copy of the diagram exported as SVG with the source code that could then be scaled and rendered, at runtime, using the svglib package. I'd still like to do this but that means first figuring out why the library always renders my SVG file as a blank image.

In the meantime there are two kludgetastic workarounds and a third (hopefully) 80/20 default solution. If you have a copy of ws-raster (a standalone HTTP wrapper around the Batik SVG rendering engine) you can use that to create a nice and crisp rasterized diagram. If you don't and are printing on non US-letter sized sheets of paper there's a base64 encoded copy of a large PNG version of the image that will be scaled accordingly producing a workable but fuzzier diagram. The default, assuming that most people are going to be printing on either US-letter or A4 paper, was just to include a letter-sized base64 encoded PNG of the diagram produced by ws-raster and get on with it.

Like I said, talk to the hand. Or submit a patch.

You may have noticed that by now there's a pocketindex class, a pocketphone class and a pocketMMap class floating around. The first was always designed to be used by the others but since the last two were built in the margins of the day eyeing each other suspiciously there was a lot of code duplication (and even more badly named variables). The main reason it's taken so long to release anything since getting back from Montréal (aside from the day job) has been a desire to reconcile all that code and move the boring bits common to all PocketMod thingies like page layout, numbering and instructions in to a single shared package. You guessed it: pocketmod.py.

Another code sample seems a little silly at this point but for thoroughness' sake if you wanted to generate a blank PocketMod book (with page numbers and the how-to infographic) you would do this:

from pocketmod import pocketmod

pm = pocketmod(width, height, margin)

# this is the magic method that all
# the other pm-derived packages call
pm.layout()

pm.save(out)

With the exception of ModestMaps which you'll still need to install separately (there's a proper installable version of the Python libraries in the works) all the packages now use the magic of setuptools which means any other dependencies should magically install themselves without any extra work from you.

The shiny-shiny:

• pocketMMap.py 0.3
• pocketphone.py 1.1

The gorey details:

• pocketmod.py 0.2
• pocketindex.py 0.1

What's next, then?

Documentation, first of all. I've tried to make the user-facing code as brain-dead stupid as possible with sensible defaults but like all things it is starting to grow knobs so they should be recorded soon if only to save people having to read stuff like this just to know where to start.

After that, there's some long overdue house-keeping to do for ws-modestmaps including adding support for pinwins that contain arbitrary chunks of text instead of an image.

This was another by-product of the Design Engaged pocketMMap: Boris, Mouna and I were sitting around the night before the conference started and I lishly suggested that we print a really big version of the pocketMMap with the names and descriptions of places in pinwins. That plan didn't work either because in my haste I chose poorly when trying to cluster places by proximity in order to produce a bunch of smaller maps, to compensate for Python running out of memory creating one monster image, and the aspect ratios were all out of whack.

Oh well. I'll probably poke at that problem too eventually.

Then maybe a brief detour to make some turkishMMaps for lack of any better name since people are excited by the Turkish Map fold these days and it shouldn't be very hard. I'm not in love with them but I'm hoping it will be a useful exercise in, dare I say it... brevity.

It's (always been) clear that a four-page pocketMMap doesn't work but it's equally clear that any collection of personal or shared history will eventually grow beyond the upper limits of a single sheet of paper. So what to do? One idea I'd like to play around with is clustering the list of points by some measure of proximity — say by distance or mapping lat/long to a corresponding WOE neighbourhood ID — and then generating a smaller, shorter pocketMMap for each. That probably makes for more individual sheets to fold but also defers doing so until it's actually necessary and in the end, better maps to the way that we hold the overlapping facets and stories of a place in our mind.

I think.