Vector databases

Building a library on AI and work

Suppose you want to know how AI changes what workers do. One paper might discuss “output per hour” and another “labor productivity.” A searchable library should help you find both. Below, I build such a library, retrieve evidence for a question, and use a paper tracker to find new work worth adding. The same passages can later supply evidence for an answer in a RAG system.

1 Prepare sources

Start by collecting PDFs on a specific question. For the example, take Generative AI at Work, which studies productivity among customer-support agents, and AI and Jobs: Evidence from Online Vacancies, which studies hiring. Both concern AI and work, but they measure different outcomes. Keeping the citation attached to each passage matters as soon as we want to use it as evidence.

A PDF is laid out for a reader. Columns, footnotes and equations can come out in the wrong order when a program extracts the text. I used Marker from a local Python script to convert the papers to Markdown, a text format that preserves headings and can represent equations. I preferred Marker for the equations in this collection. If you have the original LaTeX manuscript, Pandoc can convert that source to Markdown too.

The worked example uses short, independently written notes in place of full papers so that every passage and search result fits on the page. A figure description can also be searchable: a result about falling wages could then take the reader to the chart, where the axes and units can be checked.

2 Split the text into chunks

Once a paper is in Markdown, what size piece should a search return? A whole paper may cover theory, data and findings. A chunk is a passage small enough to point the reader to one of those discussions. One option is to split at paragraph or section boundaries. Another is to choose a maximum number of characters and decide how many characters two neighboring passages share.

In the paper tracker I used a target of 768 characters per chunk, with 50 characters of overlap, preferring a nearby sentence ending where possible. Overlap keeps some context on either side of the break. For a visible example, the notebook uses 102-character chunks on a short teaching note about the customer-support study. The highlighted sentence is exactly 50 characters long.

If a figure or table is included, its description or headers travel with it. A number such as “14 percent” is hard to interpret once it loses its outcome, population and comparison group.

3 Embed and index

An embedding assigns a passage a point in a numerical space. The capital example used two coordinates. For the paper tracker, I used SPECTER2, a model developed for scientific literature. It represents a passage with 768 coordinates. We interpret how two points relate to each other; a single coordinate has no simple economic label such as “wages.”

I stored each vector in Chroma, alongside the passage, paper ID, version and location in the text. This builds the index. Given a point, Chroma can return the nearby passages and the information the reader needs to open the source.

A second view of the library

Vector search works on its own. I also built a knowledge graph to record explicit links among papers and economic concepts. A passage might be near a question about wages in vector space; the graph can separately tell us which papers discuss both wages and automation. The tracker gives matches to graph concepts 10% of its screening score. The other 90% comes from two text comparisons described in step 5.

4 Retrieve the nearest passages

How does giving customer support workers an AI assistant affect their productivity?

To encode this question means to translate its words into a point in the same space as the passages. Using the same model gives the question and the papers a common set of coordinates. We then compare their directions using cosine similarity. A higher score means a closer match in meaning. It says nothing by itself about the size of an economic effect.

The first passage offers a measure of worker productivity: issues resolved per hour. The second also concerns AI and labor, but its outcome is hiring. Its score of 0.880 illustrates a limit of vector search: related subject matter is not enough to answer the question.

5 Screen new papers

Now reverse the search. Instead of looking up a question in papers you already have, take the title and abstract of a newly announced paper and ask whether it belongs in the collection. The paper tracker did this for a saved batch of 628 announcements from arXiv, CEPR and NBER. Its guiding question was “How does AI affect labor markets and employment?”

See an excerpt from the saved announcements

6 Review and expand

After reading Equalizer or amplifier?, suppose we decide to keep it. We record the reason, prepare its text and add its passages to the index. Now ask a question the first two studies could hardly answer: Which study discusses AI use in coursework and exams?

The first result is about support agents, despite its high score. The new passage asks how schools should assess students when AI is allowed in coursework and exams. This is something we can check by opening the paper. Keeping a nearby passage is useful only when it supplies the evidence we were looking for.

To run the tracker weekly, a scheduled job can collect new announcements, skip versions seen before and prepare a shortlist. The researcher records which papers to keep. A subsequent run adds those versions and uses the expanded collection in the next search. The example replays a saved batch so readers can reproduce the result at any time.