I’d like to take a moment to acknowledge and to thank our colleagues in the library community. You have been thinking about machine learning and so-called artificial intelligence since before it was cool. You have been trying to understand what these technologies mean for our collective work outside of, and beyond, the hype cycles and press releases. While not alone in those efforts you have been leading the pack and the rest of the cultural heritage sector, museums in particular, owe you a debt of gratitude. We follow in the paths you have tread, so thank you.

This is not going to be a talk about the bleeding edge of technology. I am going to talk about some work that, on its own, is really pretty simple but which I think points to some larger, and more important, issues.

If you happen to be the recipient of juicy subsidies from a frontier lab to tokenmaxx your collections or have secured a fat cheque from an extra-national organization to build a niche data center, that’s great. We need those projects but those projects, alone, are not enough.

These kinds of projects are so far outside the operational and financial realities of most of our small and mid-sized peers as to be rendered functionality irrelevant. That’s not a dynamic we should allow to continue because it’s clear that there is something – no one is quite sure what, but something – to the changes and opportunities made manifest by machine learning technologies. We owe it ourselves and to our peers to better understand how they can be adopted with agency and within our respective but unequal means.

I work at SFO Museum. We are a department at the San Francisco International Airport which is owned and operated as a business unit by the City and County of San Francisco. This means we have a lot of rules, especially about spending money. We are subject to very stringent regulations about how AI technologies can be used and deployed. Because we are operating in an airport environment there is a very real requirement that external software platforms, especially remote and cloud-based software platforms, be vetted.

Even if we wanted to sign up for every new 20-200$ a month AI service, an abundance of policy and procedure prevents us from doing so. The silver lining here is that it has forced us to think about how we approach these technologies beyond the latest bells-and-whistles demo released by all the competing frontier labs. It has forced us to think hard about what is left in the wake of those demos, which can run locally on infrastructure that is not data-center-shaped or that doesn’t require a recurring monthly fee.

One of the things I’ve noticed in the years since AI has become all the rage is that Apple has been quietly making more and more machine learning models, and system-level interfaces to use them, available on consumer-grade hardware. These are the interfaces to user-facing features like live text which is actually fantastically good OCR software able to interpret a wide variety of multi-lingual texts, fonts and writing styles, at weird angles and in low-to-terrible lighting conditions. Or high quality image segmentation, mostly used by your friends and family to make goofy stickers, but just as easily applicable to a variety of cultural heritage needs and uses.

SFO Museum has been able to integrate both of these things in to our collections workflow by writing custom software to wrap those system-level interfaces and deploying that software as internal network services. Although we have since upgraded to a newer Mac Mini this work began using an old recycled 2019 MacBook Pro that lives in a closet.

Apple is a not unproblematic vendor but there is no denying that they produce exceptionally capable hardware, hardware well-suited for machine learning tasks, at consumer-grade prices. They also enjoy a familiarity and an ease-of-use which can’t be overstated, especially when the alternative is trying to explain to a non-technical peer how to do something like setting up a Python virtual environment on a Linux host.

For operational and security reasons this is not hardware we can deploy with persistent credentials, the sort of thing you need for complex and sophisticated software development environments. This has forced us to make sure our tools can be signed and notarized – in effect blessed by Apple – and deployed with one-click installers. The happy side effect of this requirement is that we have also made tools that are easier to share with, and be used by, less technically-minded peers.

Earlier this year we started working with Apple’s open-weight MobileCLIP vision models; the same models used to enable visual and text based search of photos on your phone. As before, we wrote a simple wrapper service, deployed it via a one-click installer and accessed its functionality over the network: In this case a service to generate vector embeddings for images in our aviation collection, specifically to enable image similarity on our website.

In order to store and query vector embeddings you need a specialized vector database. For reasons I don’t have time to get in to here we wrote a custom database system to do this. Like all the other tools we made sure that it had a one-click installer. For more reasons that I don’t have time for, this was even harder than all the other installers. But we did it.

The reason I am mentioning it is because I believe this work – the boring, tedious and unrewarded work to package and distribute tools targeting clever but non-technical audiences – is something we have all collectively avoided. If we have any hope for an equitable adoption of these tools, it a task (and yes, an additional task) that we need to take on. Those who can, should.

If specialized vector databases weren’t enough there is the additional, compounding, problem of trying to integrate querying those vector embeddings in to already overtaxed production search infrastructures struggling to keep up with the daily onslaught of AI bots.

SFO Museum addressed this challenge by simply not engaging with it. Image similarity does not need to happen in real-time so we pre-calculated all the associations offline and stored those lookup tables alongside their corresponding object and image records.

In this way we have been able to introduce an entirely new avenue for exploring the collection. Image similarity derived from vector embeddings is not like traditional search and it is definitely not like the kind of directed search so often used by scholars and researchers to find something they already know they are looking for.

The results, associative in nature at best, are not easily classified. They operate in a space best described as “not right, but not wrong either”. They operate in a space of mystery and surprise which does not replace traditional directed search but does, meaningfully, supplement it.

What distinguishes vector-based image similarity is that it affords an efficient technical architecture by which people might notice the proverbial trees for the forest; to find those objects that for editorial or cataloging reasons might have otherwise remained unseen.

We have done this with minimal changes to our existing infrastructure and for, basically, the cost of a Mac Mini.

Once launched it didn’t take long to imagine doing the same for our other sources of imagery. We have been pro-active about archiving our social media presence and have almost 15 years of Instagram posts. We have 40 years of installation photos and 20 years of studio-quality photographs of exhibition objects. Why wouldn’t we do what we have already done with object imagery for all these other image sources? Why wouldn’t we mix-and-match these disparate sources as much for our own curiousity internally as for our visitors.

Once you start thinking about what image similarity makes visible when you pair internal collections it is not a stretch to imagine doing the same with external collections.

So we did. We harvested all the images referenced in the open data releases published by the NGA, MoMA and four different museums at the Smithsonian. We generated vector embeddings for these images and tossed them in to the same database used to store SFO Museum embeddings. We’ve generated embeddings for these organizations using the three different MobileCLIP models in addition to two of Google’s larger SigLIP2 models.

For anyone who needs reminding: The ability to do this is, and has always been, the promise of open data.

All of this has necessitated more tooling and more packaging notably for containerized virtual machines to hide the details, and the pain, of setting up the computing environment necessary to use models like SigLIP2. Container-based environments are not new and recently Apple has released their own virtualization system. What makes their system novel is the ability to save, share and re-use a container as though it were a traditional disk image. If not a one-click installer then a fewer-clicks-than-before installer.

Once we had done the work to generate all these embeddings for different collections, using different models, it didn’t take long to realize that there is little benefit to doing that work over and over again. We can generate static data files mapping embeddings to canonical identifiers and iterate over them, as needed, in a fraction of the time it takes to create those embeddings from scratch.

And once we had done that work it didn’t take long to ask: What if we published those embeddings on the internet for anyone to download and use as their circumstances, or imagination, saw fit? So we did. We have published embeddings for each source and model-pair as Parquet files and made them available to download on the web. We chose Parquet because it compresses well, enjoys lots of tooling and can be read locally or remotely over the wire.

The files themselves contain the embeddings and the least amount of common metadata possible. Put another way, any given file MAY contain as much metadata as it wants but MUST contain the minimum amount of common metadata to convey suitable attribution for the thing for which embeddings have been generated. We have established some conventions for what constitutes the “common” but are very much open to feedback and suggestions.

Since doing this, the Metropolitan Museum of Art has done the same. they have published vector embeddings for 500,000 objects, using a number of different models, available from their Hugging Face account.

Going forward we are investigating how we might assert provenance for the embeddings we generate using digital signatures. These data are enormous sets of numbers which are basically incomprehensible to the naked eye making them easy prey to subtle, but meaningful, manipulation. This is a hard problem still because none of the existing options, for asserting provenance, are an elegant fit. Most of the developing options aren’t well suited to large datasets while the well-known and well-established options still remain too difficult for most people to use.

I am telling you all of this because what precompiled vector embeddings, shared publicly and in a spirit of generousity, makes possible is this:

Something approaching.

Lightweight and functional.

Cross-institutional search.

We all know that the landscape of cultural heritage is littered with well-intentioned battleship-sized attempts to do cross-institutional search. Most of them have failed. Shared vector embeddings endeavour to enable cross-institutional search tailored to the needs and goals of a single organization with as little as infrastructure as possible.

There is, for example, more overlap between the SFO Museum’s aviation collection and MoMA’s collection than you might imagine. We probably won’t display those associations publicly because vector search has an uncanny ability to surface a lot of the sexually-themed work in MoMA’s collection. However, it is easy to imagine displaying associated objects from, say, the National Air and Space Museum.

I recognize that the Europeanas and DPLAs of the world may bristle at the suggestion they are being lumped in with the failures of the past. That is not my intent. What I am proposing does nothing to prevent those larger organizations from continuing their efforts or even supplementing them with shared vector embeddings. No one needs to stop participating in any of these larger umbrella projects.

Shared vector embeddings simply makes it possible to do something similar at a smaller, more targeted, scale with fewer resources and less coordination.

There is a both an online demo and a one-pager for everything I’ve been talking about today. For the next week, or so, I have enabled on-demand vector generation so you can test things out using real-time text and image-based searches. I will show these links again at the end of the talk.

This talk is an invitation for participation; an invitation for feedback. I think these sorts of efforts are important because I also believe there are two uncomfortable realities that we, the cultural heritage sector, would do well not to ignore:

First, we are all basically passive consumers when it comes to AI and machine learning systems. This presents a long-term power imbalance between the work we aim to do and the tools we use to do that work.

Second, no one – and I mean no one – really understands why or how large language models return the results they do. We may understand in broad strokes how they work but there is still no meaningful introspection in to how decisions are made by these systems.

I do not want to suggest that shared vector embeddings, alone, will solve or fully address these problems. They will not. Shared vector embeddings are, rather, an exercise in practicing agency in our work. They are an exercise in producing tangible and reusable artifacts which have a longer shelf life than a recurring monthly fee; artifacts that we can catalog and then use to start reasoning how these models work with relevant data (ours) and evidence.

Many, if not all, of the organizations attending this conference are already producing vector embeddings for one or more initiatives. Most other institutions, I believe, would produce vector embeddings if it could be done easily. The sum total of what these data make possible, at both a micro and macro level, is greater than the cost to any one organization. We could, each of us, produce embeddings for one another’s collections, duplicating the same work unnecessarily.

Or, we could simply publish what we already produce practicing the act of sharing in the process.

My own personal definition of digital transformation is decidedly unromantic. I understand it as: The process by which issues and challenges that were always present, but otherwise able to remain unseen, are exposed by the commercial availability and consumer adoption of digital technologies.

This can take many forms and I would challenge you to consider that one of them, which existed before AI but which AI makes nearly impossible to deny, is that the distinction between libraries, archives and museums is collapsing in the minds of most of the general public. If e-commerce has taught us anything in the last twenty years it’s that the distinction between the store front and the warehouse has all but disappeared.

If you squint your eyes and consider machine learning models as a kind of warehouse you can imagine where things are going; particularly the public’s expectations around search, retrieval and discovery as they relate to our collections. People know that other ways of accessing and interacting with our collections are possible because they live it every day in other aspects of their lives.

The question I want to leave you with is whether we will be able to address those changing expectations with ability and agency or only though vendor contracts subject to the whim, folly and manufactured limitations of the marketplace. Agency and ability not so much through shared vector embeddings themselves but through the collective practice of producing, sharing and using them.

Thank you.