much history so remember

This is the story I want to remember. See also: and see also-er.

privatesquare two dot oh

There's a new version of privatesquare.

I guess it's safe to call it version "two dot oh". It is sufficiently different and sufficiently backwards incompatible with earlier versions that I created a snapshot of the project's first two-years worth of work and called it version 1.0. Installing privatesquare from scratch is pretty much the same as always but upgrading an existing installation is a thing that will demand some time and attention. It's not ideal but privatesquare remains a mornings-and-weekends project.

Upgrading requires attention because there are some pretty fundamental changes to the underlying database structure. The changes mean that it's now possible to check-in to other kinds of places that aren't Foursquare venues. As of this writing you can check-in to three other types of places.

The first is a canned list of funny-haha "states of mind" like "on a bike" or "on a plane" or "on a telco" or "kill me now". I say funny-haha because on the surface it just seems sort of silly but is actually really useful. It's a fixed list and the criteria for what get's included is pretty arbitrary. At the moment it's very much biased towards modes of travel and office life but both are situations which often demand a kind of non-specific outlet, whether it's in the moment or in the re-telling.

Just in case you still thought artisanal integers were a joke...

Second, people now have the ability to create and check-in to "user-defined" places. This is basically what it sounds like. I could have just written a bespoke interface to add things to Foursquare via their API but like the original motivation for writing privatesquare there are always going to be places that you don't necessarily want to share with strangers or a third-party service. It is aggressively simple, partly because I wondered how quickly I could add it to this round of changes (after getting "states of mind" working) without it ballooning in to a sinkhole of feature creep.

User-defined places have names and nothing else, yet. You can choose to check-in to a place at the time it's created but that's about it. The other notable feature, which user-defined places share with states of mind is that they may not have fixed geographies. Just think about being able to check in to "hungover" for a minute and you can see the attraction of this feature. It's not required but it is possible.

It's not possible to share user-defined places yet and it probably won't be for a while. Back in the Flickr days we used to talk about user-defined places a lot. We never did anything about it but that was only because of organizational minutiae and a lack of time. All the geo place-based stuff was built using the Where On Earth (WOE) gazetteer packaged up as an an enormous (like 12GB by the time I left) "data pack" which was then poked and prodded by an HTTP interface. This is where WOE IDs came from.

I always hoped that we could supplement the data pack approach in two ways. First, I wanted to be able to create new data packs specific to a service (where service means a thing that people used, like Flickr or Upcoming or Yahoo Autos) and the ability to pair them with the master data pack and to then weight the results for the associate data services (like geocoding) by source. Second, I wanted to be able to bias geographic results using a ranked list of data sources or providers which in Flickr's case would have inevitably meant user-defined places. The closest we ever got to this were the flickr.people.geoBookmarks API methods but they never got the attention they deserved and were sort of stillborn from the moment they were released.

I suppose I could (will have to... ?) add photo sharing to privatesquare but in the meantime most of the "pipe" has been laid to start doing all the yummy things around user-defined places that we never did at Flickr.

As an aside, one of the interesting things about creating venues that don't have fixed geographies is that it's not hard to start thinking about privatesquare as a kind of Dotspotting client. One of the early design decisions with Dotspotting was that it be a document-based system which means it's never been possible to update a sheet once it's been uploaded in to the site. Variable geography check-ins in privatesquare don't actually solve this problem but you could imagine using privatesquare to create a user-defined place (for example Eric's coloured-dots-on-drains collection) and doing all your collection over time and then exporting it as a CSV file in to Dotspotting at the end. Between user-defined places and a proper public-facing API (more on that below) the number of places that privatesquare and Dotspotting could finally be made to hold hands is actually pretty exciting but there are enough other things going on right now that I will just have to look at that place longingly for the moment. The possibilities are delicious to think about, though.

Finally, it's possible to check in to buildings from the New York Public Library's 1852 Perris Atlas. This is a feature that was started way back in the summer during the NYPL's Maps Hack workshop and which then got steamrollered by the rest of life. It's not really of much use to anyone outside of New York City and even for those people who live it's not entirely clear how or why you'd use it. The API used to look up buildings is (I think) not completely worked out and it doesn't completely map to the idea of venues or check-ins so in that way it's sort of like a proof-of-concept or reference implementation that's been pushed in to the spotlight without having had time to put its pants on.

But who cares! It's working code that proves it can be done. The code now supports four distinct types of places you can check-in to and while none of it is very elegant (or documented anywhere yet) it does work. It means that it's possible to think about adding support for other things. It's possible to think about a way to check-in to the 37, 000 works of art at MoMA (not a metaphor). It means that, with a bit of work, it ought to be possible to rebuild the Upcoming.org database from the Archive Team rescue and use that a source for venues. Or, going back to the NYPL buildings, imagine using the Library of Congress' historical gazetteer (which isn't yet public so far as I know) as a check-in source.

I was there, indeed.

All of which feels like progress to me and a nice place to stop for a while and let things settle down. privatesquare remains a thing that you build and install yourself. This is not the service-for-strangers that I am ready or want to run yet. The code and the installation (or upgrade) instructions are available on the GitHubs, over here:

https://github.com/straup/privatesquare/

Things that got finished.

• You can check-in to states of mind.
• You can create your own custom places which may or may not have fixed geographies.
• You can check-in to historic buildings in New York City.
• All of the templates and stylesheets were updated to use Bootstrap 3.0. On the subject of things like Bootstrap John Allsop's blog post The Other Web We Lost, about the slippery slope of frameworks, is worth reading. I am not sure that I am as suspicious of things that ultimately boil down to HTML + CSS + JavaScript (which is already a moon language so it's hard to make it any weirder...) but he raises some valid points. When it comes to using Bootstrap, though, I can only say the same things I said at the end of the blog post about the beta release of the Cooper-Hewitt collection website.
• A bunch of experimental features were removed since they either weren't being actively developed or didn't actually work very well.

Things that I didn't quite finish.

• Exporting a series of check-ins as an iCal file. The code is all there but I appear to be generating invalid calendar documents. I thought I could get this working on a couple of long-haul flights but that didn't happen so I'm just going to set this aside for the time being and let it be a tiny release of its own.

That that didn't get addressed at all.

• Timezones. OMGWTFTZ... timezones. I so desperately have to fix this. I know this is what I said the last time and I haven't done anything about it. This ought to be pretty straightforward to do for Foursquare check-ins since a venue's timezone is typically returned with the metadata for that venue. For the other place types it gets a little more complicated. On the one hand it's not complicated at all. I could simply load the whereonearth-timezone dataset in to Matt Biddulph's handy reverse geocoder and call that for every not-Foursquare check-in. The problem with this approach is that privatesquare has always tried to follow the rule that the Dotspotting project set for itself: That it be possible to install and run on the plainest of vanilla web-hosting services. This actually introduced a world of (still) never-ending pain from a development point of view but the hope was that we could build a genuinely useful tool without the burden (or cutting edge) of this week's dependency chain. Which pretty much rules out long-running Java daemons. Geonames has a handy reverse geocoding timezone API so that's a possibility. The number of timezones, though, is finite and so there's probably a way to filter the number of actual possibilities down to a manageable level using simple point-in-bounding-box queries and then final checking doing point-in-polygon queries with simplified shapefiles, keeping the whole thing in the local database and application code. That's what I am thinking on a Saturday morning, anyway...
• A proper public facing API, which really means folding in all the work from flamework-api. This also means moving privatesquare over to using HTTPS exclusively because... grumble grumble, OAuth2. One of these days I will get around to writing the Epic Rant about the piss-poor state and delusional pony-thinking that surrounds delegated authentication but today is not that day. One of the problems with using OAuth2 is that is introduces a massive chunk of bloat and burden in to the dependency chain (see above). It's not that SSL is bad or wrong. It is plain as day that it's the correct thing to do. In principle. In practice, it's fiddly and confusing and potentially expensive depending on who you're using as a certificate authority or hosting provider. So, a proper public facing API is coming but only once I make sure that the code to support it can be used to replicate the same not-really-secure private API hooks that the site already uses.

Things that come next, pretty much in this order.

• Timezones. For all the reasons mentioned above, this has to be the next serious chunk of work. If only to make iCal export not completely stupid and to make it possible to add trips. Trips?
• "Trips", aka Dopplr-mode. I'm not sure what this will look like yet. It's not going to be a complete Dopplr clone but to the extent that I had to get out my laptop and check the site during my clearance interview for Government Club in order to remember why I went to Canada in the summer of 2007 it feels like functionality that needs to be added, in light of Dopplr's recent demise.
• A public-facing API. See above.
• Profit?

Things that happened between then and now.

• Apparently, Foursquare removed the ability to add private check-ins. I haven't had a chance to confirm whether this means you can't check in off the grid or whether it means that you can no longer scope check-ins to only followers or what. So I bet there are some API calls which privatesquare makes that will fail...

Good times.