tilting towards action
I spoke about SFO Museum's work to promote the idea of shared vector embeddings, at this year's Museum Computer Network conference, in Seattle. The theme of the talk was the same as the talk I did at the AI4LAM conference, in September. I tried to make it the same, but different
because I appear to be constitutionally incapable of doing the same talk twice. This is what I said, including an important update to some factually incorrect claims I made about half-way through the talk.
If we haven’t always organized information dimensionally then it’s been a very long time since we haven’t. For example, the two-dimensional Cartesian grid of latitudinal and longitudinal coordinates. These allow us to represent, in a consistent manner, both the location of two different places in space but also the distance between them; their relationship in space. Add a third dimension and you have another distance: altitude or depth.
The manner by which we measure the relationship between any two or three points is often just as, if not more, important than the coordinates themselves.
The determination of those measures, though, is not always obvious. Consider two objects, a stone and a feather, falling to the ground from the same height; the same distance. In a vacuum, those two objects, despite their significant differences in weight, will fall at exactly the same rate. Likewise in four dimensions where Einstein’s theories of relativity demonstrate that time itself can bend and warp.
A measure, then, is not always constant but contextual and subject to its environment. Now, take these basic ideas as they apply to three or four dimensions, in particular the ambiguity and squishiness with which the relationship between any two properties is determined, and apply them to billions or even trillions of dimensions.
That is basically a large language model.
Now, consider that these billions of dimensions and their relationships to one another can be packaged into files which, while undeniably large compared to say a Word document or photos of your cat, can still be stored on consumer-grade storage devices, like an SD card. Those files, in turn, can be queried in literal seconds if not faster.
A lot of attention is paid to the generative and conversational features of large language models. While impressive I also think they are just brute force applications of what is really the foundational big deal of these technologies: Namely, multi-faceted bucketing and retrieval at a scale and a speed that quite literally we don’t have any mental models to perceive, let alone understand or intuit. This is one reason I often refer to large language models as the blob
.
So far, the best way I’ve managed to visualize the blob in my own mind is as being something not unlike a swarm of small drones, equipped with lights, organizing themselves into a constantly changing shape in the sky.
Imagine, then, that each drone is actually a many-sided magnet; one where each side has a different charge such that the combination of those charges relative to all the other magnets, and their charges, hold all the magnets in place. The force of those charges, though, is not determined by so-called natural laws but instead by the human condition.
If you’re starting to feel a little queasy trying to imagine all of this, that is a natural reaction.
None of the physics I am describing here actually works. These magnets are only a conceptual device to help understand that the individual properties in a model, or blob, are organized in ways that group them by their relative proximity - their distance - to one another based the statistical probability, or in magnet-speak the charge
- of their similarity (the definition of which is just as fraught as that of distance).
When you query a large language model you are essentially converting your input - some text or an image - into a smaller representative blob of magnets. These smaller blobs are referred to as vector embeddings. To query a large language model with vector embeddings, then, is to create two or more smaller blobs derived from a larger blob (a large language model) and then bring them together; allowing the magnets in these smaller blobs to find one another and snap into place.
To say that things are snapping into place is a misnomer because it suggests that similar things find one another exactly. They don’t. Instead similar things find each other approximately. But remember: We’re talking about blobs of data with billions, sometimes trillions, of inter-related properties.
Being able to mechanize even approximately-good retrieval based on similarity at machine speeds, at this scale, is demonstrably crazy-impressive.
It is also important to recognize that the cost to produce and operate these models can be just as crazily impressive, but not in a good way. The debate around the costs and benefits of these technologies has not been settled. All things being equal, at least for the sake of the argument today, let us accept that structured storage and retrieval at this scale enlarges the surface area of opportunities which were previously so fiddly or time-consuming as to be functionally impossible.
What is now possible that previously wasn’t?
In a nutshell this: This is the ArtButMakeItSports social media account. It posts screenshots of contemporary sporting events alongside similar looking works of capital-A art.
This is made possible by converting all of these images into their vector embedding representations, storing those embeddings in a specialized database and querying them. I think this is genius.
It would be irresponsible not to note that, at this point in the talk, I misspoke. I said This is made possible by...
when the author of the social media account has repeatedly claimed they do not use any AI technologies to make these associations. I was negligent in not pointing that out. I think my central argument still holds, though. If we accept that the author of the ArtButMakeItSports account is an unusually encyclopedic fount of both art and sports-related imagery the fact remains that most people, not to mention most museums, are not. Vector embeddings are the mechanical means to do the same thing in an automated fashion.
I read Homer’s Odyssey when I was in high school, but didn’t read the Iliad until much later. When I did I remember thinking that it read a bit like every action-adventure movie ever made. At which point I realized it was, obviously, the other way around.
What I love about these images is that they act as a kind of bridge between the contemporary language and imagery of sports, a language spoken and understood by many if not most people, and the language and concerns of cultural heritage, a language spoken and understood by fewer and fewer people.
Lamine Yamal need not be the only vehicle for discovering the work of Georgia O’Keefe but I challenge anyone to tell me why it’s an inappropriate vehicle; especially when everything else we do seems to fall on deaf ears, these days.
Peter Leonard has said Embeddings do violence to the original image
and indeed vector-based image similarity does not afford answers so much as associations. Image similarity operates in that funny know-it-when-you-see-it space where something may not be right but it is simultaneously not wrong either.
These are definitely not the kinds of search results that most scholars and researchers have come to expect. Image similarity does not, can not and should not replace traditional directed search but that does not invalidate similarity-based search either. Or rather, we invalidate it at our peril.
What image similarity can do is increase the surface area of both discoverability and, crucially, approachability for all those other people out there doing something awesome with their lives at the expense of having time to study art history. Which is, we would well to remember, most people.
If this is all still a bit hard to wrap your head around, take a moment to remember that there is another practice which also traffics in collecting, sorting, bucketing and organizing things in to similar spaces, often with different but equally dubious moral and ethical costs: Museums.
One of the problems we have is that many of the things in our collections can only exist, physically, in one place at a time; we only have three dimensions to work with.
Consider the work of Dane artist Brian Jungen who creates traditional First Nations artifacts produced out of Western (or colonial) consumer sporting goods, like sneakers and golf bags.
To find these objects in a museum you might need to look for them in the “First Nations” wing. Or maybe the more expansive “North American” wing. Or the “Modern Art” wing. Or the “Product Design” wing. Few if any museums would deign to have a “Sports” wing but perhaps you can see where this is going.
I spoke earlier about surface areas and I think this bears repeating: What image similarity can do is increase the surface area – the dimensionality – of both discoverability and approachability for people who don’t already know what they are looking for.
What distinguishes vector-based image similarity is that it affords an efficient technical architecture by which people might notice the proverbial trees for the forest; to find those objects that for editorial or cataloging reasons might have otherwise remained unseen.
This has a side effect of shining light on the fact that museums are, by and large, both hostile to alternative narratives but also largely unwilling to admit, or own up, to that hostility. I happen to be squarely in the alternative narratives camp so I’ll just leave it at that, for the time being.
SFO Museum has generated vector embeddings for every published object in its aviation collection. We have used this work to enable image similarity, with little to no infrastructure changes, for every collection object on our website. We have generated embeddings for 15 years of Instagram photos, in addition to the 40 years of installation photos and studio photography we have produced for exhibitions.
We have produced embeddings using a variety of different open-weight models; different collections of magnets with different charges yielding different definitions of approximate. Large language models are not truth machines so much as machines which tilt towards statistical consensus through an interpretive lens which, very often, is both unseen and unknowable. Gathering evidence from a multiplicity of models, then, becomes a way to give shape and understanding to those lenses.
We have also done this work for objects in the collections of the National Gallery of Art, the Museum of Modern Art, the Cleveland Museum of Art and parts of the Smithsonian. We are currently in the process of generating vector embeddings for the whole of the Flickr Commons. We have published all of these datasets on our website for anyone to download. Not the images themselves but just the vector embeddings alongside the least amount of common metadata to allow for suitable attribution.
Perhaps you are noting the absence of the Metropolitan Museum of Art in that list. That’s because I didn’t need to generate embeddings on their behalf. They did it themselves publishing embeddings data for half a million objects to their Hugging Face account.
We have also been developing and documenting a suite of open source tools to do this work.
We are endeavouring to make it possible to deploy these tools if not with a one-click
installer then with a as-few-clicks-as-possible
installer. The point is to craft a set of tools that organizations lacking technical resources might be able to use to produce their own vector embeddings. I am happy to speak to anyone about these tools after this talk but the salient point here is that all of this work has been done using a four-year old Mac Mini without any specialized developer environments.
We have done these things because we want to demonstrate what shared vector embeddings make possible, and ultimately, to encourage you to want to produce and share these artifacts for your collections on your own. So, what does it make possible?
Lightweight and functional cross-institutional search. Not necessarily the search of academics but a different, equally meaningful, search of discovery and association.
This is a safe space. We can be honest with one another and admit that the landscape of cultural heritage is littered with, and possibly defined by, well-intentioned and battleship-sized attempts to do cross-institutional search which have failed. Seen in that light, shared vector embeddings are the plumbing which allow cross-institutional search tailored to the needs and goals of a single organization with as little as infrastructure as possible.
I recognize that the Europeanas and Artstors and probably even the Google Art Projects of the world may bristle at the suggestion they are being lumped in with the failures of the past. What I am proposing does nothing to prevent those larger organizations from continuing their efforts or even supplementing them with shared vector embeddings. No one needs to stop participating in any of these larger umbrella projects.
The evergreen challenge of cross-institutional search has been that very few organizations have the time, temperament or money to run it. A lot of effort is required to coordinate participation, to manage the harvesting and storage of other’s people data, to sustain the infrastructure necessary to enable access, public or not, to all that data. There are typically as many different terms of use as there are data sources. It is easy to see why cross-institutional search never seems to work.
Shared vector embeddings do not, can not, solve all these problems but they do represent an important shift in how we might approach things.
Shared vector embeddings start to make it possible to do something that larger centralized projects have aimed to do but at smaller, more targeted, scales with fewer resources and less coordination.
Shared vector embeddings are a way for individual organizations to harvest their own data, on their own time and their own terms, and then, in effect, publish the search index specific to their holdings for others to use. To say you are publishing the search index
is not entirely accurate, though.
Really what you’re doing is the work of converting your holdings in to little blobs of so-called magnets that others can download and associate with their own blobs of magnets, large and small. Computing those associations doesn’t magically happen for free. If you want to index and query millions and millions of embeddings from every museum in the world that will continue to cost you real money, but probably still less than you think.
The point is that you don’t have to. If you want to mix-and-match a small, purpose-fit, search index from the institutions of your choosing these are the means, technical and financial, which allow that to happen.
Last year, at this conference, I pointed out an uncomfortable truth about our modern time: That the infrastructure of search as a common good, an infrastructure which had perhaps naively been taken for granted, was actively and willfully being dismantled.
There is another uncomfortable truth worth recognizing when it comes to machine learning or AI-related technologies: Namely that we, the cultural heritage sector, are, at present, little more than passive consumers of these technologies. What little agency we might have consists largely of choosing one vendor over another.
The value of shared vector embeddings, whether deployed individually or collectively, is not that they will remedy all that ails us. We have dug ourselves a hole deeper than any one project like this will get us out of.
The value is in demonstrating that we have the capacity to be producers, even if it is just at the margins, rather than idle consumers. To demonstrate that the hole we presently find ourselves in is not inevitable and that, with time and shared effort, we might conceivably crawl out of it.
Thank you.
Some notes about the demo:
- Depending on when you read this
real-time
search will probably be disabled. Generating on-demand vector embeddings remains resource, and therefore cost, intensive. In a world of negligent or malicious, and not-infrequently both, AI bots crawling the web it can be hard to justify leaving this kind of service exposed as a public good. - The service itself is run off a lightweight AWS Lambda Function URL with the embeddings themselves stored in a (AWS) S3 Vectors bucket. I mention that because if you have already made peace with using AWS services, this can be a low-cost way to deploy vector embeddings (and is part of the reason we wrote our own modular database system).
This blog post is full of links.
#mcn26






