The Application of AI for Museum Collections Online

What excites me the most about Generative Artificial Intelligence (GenAI) application to museum work is its potential to improve discoverability of museum collections online.

Continuing from last week’s post on the AI application for museum exhibitions, this week we will explore how GenAI can enhance collection search and discovery for everyone, and especially our users. With the rise of AI use in the general population, it’s no surprise that during an Ask Me Anything (AMA) webinar (in December 2023) I was asked about the impact of AI on Collections Management System (CMS) use. I wrote a post on my response (Ask Me Anything (AMA): The Impact of AI on CMS Data Creation and Enhancement) where I offer the following:

With this kind of power and source checking in mind, here’s how I see AI supporting our data creation and cleanup work:

• Helpful for writer’s block when creating an object description or artist biography.
• Instructive for data management or cleanup approaches (as demonstrated in the prompt example above). As well as specific areas of data to cleanup via Excel Power Query or Open Refine.
• Pattern recognition and identification of areas for data for review.
• Data organization and remixing.
• Script creation to run with (mostly open-source) collections management systems to extract and/or transform data.

In reading it now, just 8 months later, I realize I wasn’t thinking big enough. My initial response was specific to how I saw AI supporting our work as collection stewards. However, the real power here is how GenAI can improve the search and discovery of our collections without the corollary of increasing the size of human labor required.

GenAI is a Game Changer for Collection Discovery

It’s becoming clear to practitioners that GenAI can do what we (often) cannot, and that’s to describe the collections in plain English and in a way that’s more intuitive to the general user. In other words, GenAI is better at humanizing the descriptive process, which is a bit of beautiful irony. Even if we think we’re capturing data that helps general users in search and discovery, the truth is that we’re biased humans who are highly specialized in our field and very familiar with our collection. This makes it incredibly difficult for us to shift out of that mode and essentially re-catalog an entire museum collection. With GenAI as our partner, we don’t have to try and catalog in two different modes, nor do we need to re-catalog the collection—that labor can be done by GenAI.

AI Explorer at Harvard Art Museum

Harvard Art Museum is an interesting example for us to view. They applied a few different GenAI tools to their catalog and you can view exactly how each tool interpreted the work.

As you’ll see in the examples below, it is not a perfect science (yet), but it is promising. It’s essentially asking the AI to look at the art and describe it. The GenAI data is then added to the catalog record and can be searched on. Here’s how the AI Explorer at Harvard Art Museum is explained:

Artificial intelligence when united with art advances how humans understand and interact with artworks. Specifically, the Harvard Art Museums is using computer vision to categorize, tag, describe, annotate, and humanize its collections. We start by giving each AI an image of an artwork. We then ask it to describe what it sees. The machine views and describes the museum’s collection with no additional context. It’s as if the machine is walking into an art museum for the first time. Since the computer lacks context or artistic knowledge beyond what is depicted in each image, the machine’s interpretations of the artworks lean closer to reflecting the public rather than experts. By folding in an AI’s perspective, browsing and searching art becomes more accessible and intuitive to non-museumgoers.

Reviewing an Example: AI Explorer

For a quick show-and-tell, we’ll review two examples from the AI Explorer at Harvard Art Museum catalog.

Exploring Collections: Starting on the Home page

The following screenshot is of the landing page for AI Explorer at Harvard Museum. Already I’m interested in the words underneath the search bar. Those would never be terms I consider adding to a catalog record, and yet I can see how our users may want images that represent “love” or are “shiny”.

I clicked on “shiny” to see the results. You can see a few of the items are not necessarily shiny, and of interest is the level of confidence assigned to each item by the AI. I know I’ve often wished there was a way for us to indicate “We think it’s this, but also aren’t super confident, so maybe we won’t use that descriptor or term.”

Search Results for “Work”

This time I chose my own word to search: work.

My first impression is that most of these images are pretty spot on, especially as some of the images without people aren’t immediately obvious that it could be in a work context. The one that stands out as an outlier is the more abstract man with dog image.

Here’s a view of the record (below). The top half is the typical catalog data, with the GenAI-provided data below. In this record we can see five different AI tools, the words they picked to describe the art, and their confidence levels. Some of the descriptors are spot on: art, pet, illustration. Whereas others you can see why the AI may have struggled to interpret the abstract aspects; such as: child art, jaw, fruit.

A record of Richard Lindner’s Man’s Best Friend with a mix of human- and machine-generated data.

Also of interest is the emotional analysis done by the AI. Another descriptor I know I wouldn’t have thought to include, and yet I can see how our general users would like to search on feelings. While the AI still struggles with the abstract nature of the art, I can see why it’s picking up angry, sad, and disgusted.

AI-generated emotional analysis of subject in Lindner’s work, Man’s Best Friend.

And for a final amusement, we can see how the AI interpreted the dog as a map given the colorization of the dog.

The pink dog in Lindner’s Man’s Best Friend is misidentified as a map by AI.

For a more accurate (as I interpret it) example, let’s review the item below, which had a similar confidence-level as the previous piece of art. While scissors, weapon, blade, laptop, and keyboard are inaccurate, the remaining tags generated are correct.

Unlike the “It’s a map” captions, these captions aren’t what I would have chosen, but they’re also another fairly accurate way to caption something.

Conclusion

It’s exciting to see the near future of GenAI application and how we can work with it to create robust and intuitive catalog records in a way that doesn’t increase demand for our labor. I also love that GenAI is helping to humanize our records by applying a layer of human interpretation that I otherwise wouldn’t have thought of. In the end, a GenAI partnership in data creation will improve our ability to search in collections and significantly increase collection discovery.

Additional Reading

• Ask Me Anything (AMA): The Impact of AI on CMS Data Creation and Enhancement
• Museum Forecast 2024
• Museum TrendsWatch 2024: AI Adolescence in Museums