This podcast episode is all about semantic search and using embeddings to analyse text and social media data.
Dominik Weckmüller, a researcher at the Technical University of Dresden, talks about his PhD research, where he looks at how to analyze text with geographic references.
He explains hyperloglog and embeddings, showing how these methods capture the meaning of text and can be used to search big databases without knowing the topics beforehand.
Here are the main points discussed:
- Intro to Semantic Search and Hyperloglog: Looking at social media data by counting different users talking about specific topics in parks, while keeping privacy in mind.
- Embeddings and Deep Learning Models: Turning text into numerical vectors (embeddings) to understand its meaning, allowing for advanced searches.
- Application Examples: Using embeddings to search for things like emotions or activities in parks without needing predefined keywords.
- Creating and Using Embeddings: Tools like transformers.js let you make embeddings on your computer, making it easy to analyze text.
- Challenges and Innovations: Talking about how to explain the models, deal with long texts, and keep data private when using embeddings.
- Future Directions: The potential for using embeddings with different media (like images and videos) and languages, plus the ongoing research in this fast-moving field.
Connect with Dominik Weckmüller here https://geo.rocks/
Stay up to date with AI here
https://huggingface.co/ Try searching for “map” here https://huggingface.co/spaces
Check out this project I am working on
In Conversation
From Hyperloglog to Semantic Search
Daniel: We met a couple of years ago at the FOSS4G conference in Florence, where you were talking about semantic search, something called hyperloglog, and scraping a lot of data from social media. Could you introduce yourself?
Dominik: I’m Dominik, currently doing my PhD at the Technical University of Dresden, investigating how to analyze textual data that has a geospatial reference attached to it. Back in Florence we used an algorithm called hyperloglog to analyze social media data. It counts cardinalities — the distinct number of social media users who posted about a certain topic. We were analyzing urban green spaces: if a user talked about trees, grass, bushes, and flowers, that could be one topic, and we counted the distinct users at each urban park. We used a framework called cultural ecosystem services to understand what users actually do in parks — socializing, eating, studying, playing sport. Hyperloglog let us do this while minimizing the detrimental privacy effects. But back then we had to know the topics beforehand — you can’t see anything in the data you’re not aware of in advance. That’s why I moved on and developed different strategies.
What Social Media Data Looks Like
Daniel: Help us understand this social media data. In my mind it’s very structured.
Dominik: It depends on the perspective. Platforms like Instagram, Flickr, or Twitter exchange data in a structure called JSON, which is inherently structured — there’s a node for the caption text, a datetime, and a location. Instagram really nudges users to add a geospatial reference; on Twitter it’s a little hidden. So social media data is simple: some text, maybe an image, and a geospatial reference — at its simplest, a lat-long and a datetime. For our research we have a text and a geospatial reference. But text is hard for computers to process — they don’t think in characters, just zeros and ones — so we need a way to make computers understand what we’re talking about. That’s where embeddings come in, and that’s where the magic happens.
Embeddings: Turning Text into Meaning
Dominik: An embedding is nothing but the numerical representation of any input data — in our case a piece of text, like the caption a user posted. We use a deep learning model to convert that text into a vector: a list of numbers.
Daniel: Would it be fair to describe that vector — that list of numbers — as dimensions, each number representing a dimension?
Dominik: Exactly. You could think of a dimension as having a meaning — a model could count the number of legs of something: a human has two, a dog four, a centipede a thousand, and the first dimension could be the number of legs. But dimensions can also represent more complex things — an attitude, a positive or negative vibe, the meaning of a whole sentence. These vectors are great at grasping the meaning of something instead of just looking at keywords. You can boil down any input — a word, a sentence, a paragraph, a whole document — to a fixed number of numbers, one embedding. Then you perform a similarity calculation: imagine two points in space, draw lines to the origin, and the angle between those two vectors gives you the cosine distance, which tells you how similar the two texts are. You can run a user query against a whole database of vectors, rank by distance, and the most similar posts come out on top.
Searching by Meaning, Not Keywords
Daniel: So now you don’t have to define topics in advance — you just need the geographic reference and the text turned into an embedding, and you can query against the database later without prior knowledge of the topics?
Dominik: That’s correct. The main advantage over a full-text search is that you capture the meaning of the input data. The embeddings condense all that information in themselves, so you don’t need to know anything in advance — you just create the embeddings and run any query against them. These deep learning models are trained on very large corpuses — the whole internet, all of Reddit, all of Wikipedia — so they’re good at capturing the meaning of basically anything. They’re very generalized, so they might not capture a niche research topic well, but they do an amazing job with everyday language, which is what we use on social media.
Daniel: Give me an example of what’s possible now.
Dominik: You can search for emotions, individual words, or whole sentences. If your query is “enjoying my book at this urban green park,” it would also match someone who posted about reading lying on the grass without mentioning a book — the meaning is almost the same even though the words differ. If your query is just “sports” and someone posted exactly “sports,” the similarity is 100%; football or tennis would be high but maybe 90%; two people tossing a ball around would still register as sport-related. Emotions are subjective, though — a group of young people partying at the central station at night could read as positive to one person and as a threat to another — so you need to be careful and explain what you’re referring to.
Building It in the Browser with Transformers.js
Daniel: How did you create this database? My understanding is it involved something called Transformers.js.
Dominik: Transformers is the latest state-of-the-art architecture for building deep learning models, and there’s a plethora of models you can download and run yourself. Transformers.js does a fantastic job because you can run almost any model on your phone or computer — even without a GPU, even in your browser. You don’t need to be GPU-rich with setups that cost tens of thousands of dollars. I calculated the embeddings for my whole database of social media posts on my own computer, and I don’t even have a GPU — maybe 20 to 40 embeddings per second on a really old CPU, which is enough, because you only encode the database once. For the demo I’m writing a paper about, the moment a user inputs a query the embedding is created in the browser — a tiny amount of work, done in about 100 milliseconds — and compared against my pre-calculated database. You don’t even need an expensive server; GitHub Pages is enough to host applications that previously required complex setups of elastic search and fuzzy search.
Vector Dimensions and Chunking Strategies
Daniel: Do you decide how many dimensions a vector has when you create it?
Dominik: It depends on the model. Until a couple of months ago, a trained model would always output a vector of a fixed length — 384 dimensions, 800, 1,000. Larger, higher-quality models use more — 4,000, or GPT models maybe 16,000. More dimensions can capture more meaning, but the longer the vector, the harder it is to handle: a million posts times 16,000 numbers is a huge amount of gigabytes. So you keep vectors as short as possible but as long as needed. The massive text embedding leaderboard is worth looking at — small models with 300 to 500 dimensions work almost as well as the huge ones. There are also new techniques — quantization, binary quantization, and Matryoshka embeddings — that let you flexibly reduce vector size without losing too much of the meaning. The one rule: your query vector and your database vectors must be the same length to calculate the cosine distance.
Daniel: Can you tell me about chunking strategies?
Dominik: Embedding models have a fixed input length. Social media posts are short and fit fine, but if I input a thousand PDF pages, the model can’t handle it. So we chunk the text into smaller parts. A Wikipedia article — the article about Berlin even has a geospatial reference — has paragraphs and headings you can split by. For an unstructured report with no headings, you might chunk by a fixed length, say 300 words — but then you risk splitting sentences and losing the meaning, so you want to chunk semantically, into coherent passages. You hand each chunk to the model, get one embedding per chunk, then average them into a single high-level embedding that represents the whole text. It’s actually similar to a tiling strategy for large satellite images.
Embeddings vs. SQL — and the Privacy Problem
Daniel: Is this going to replace SQL and structured databases?
Dominik: No — a definite no. With keyword search you have a binary, explainable result: the keyword is in the text or it isn’t. Embeddings give you a fuzzy score between zero and one — it becomes a bit of a black box, not as explainable. So use embeddings when you’re looking for something fuzzy or doing data exploration; use simple SQL keyword queries when you want binary results. You can also combine them — start with embeddings to explore the data, see which keywords rank highest, then search for those keywords.
Daniel: How do you deal with that explainability problem?
Dominik: The easiest way, ignoring privacy entirely, would be to persist the whole database — the raw text a user posted, attached to the similarity score, so you can verify directly. We don’t do that, because saving the raw data would heavily harm user privacy — that’s data retention we want to avoid. Instead we save an abstraction of the geolocation and verify against the platform: look at the most recent posts at that Instagram location, and there’s a high probability you’ll see other people doing something similar. And let me be clear: embeddings do not preserve your privacy yet. Google released a research paper reconstructing the input data from embeddings — so if you have the exact embedding, you can recreate the original input. We’re still researching how much to compress the vectors so they become too coarse to reconstruct a single post, while staying meaningful enough to query against.
Multimodal, Multilingual, and Keeping Up
Daniel: Can we use images to help verify what’s happening in the text?
Dominik: Yes — and that’s the beauty of embeddings. You can use not only text but imagery, videos, audio, any input data you can imagine, and convert it to embeddings. You could create image embeddings and ask the database what’s most similar. Embeddings can also be multilingual — you can query in German against an English database, or English against a German database. Social media posts are inherently mixed: German users use English hashtags, tourists post in Spanish, French, or Chinese, and these models are great at handling any language and even mixed languages.
Daniel: How do you keep on top of this fast-moving field?
Dominik: I follow what’s going on at Hugging Face — it’s like GitHub for anyone interested in AI. You can look at the best-performing embedding models, the leading text models, and dive into the communities training models to see what input data and architecture they use. A good starting point is following some staff from Hugging Face, who repost the latest research. The development is so rapid you can develop a fear of missing out — every day something is released — so don’t let yourself be stressed. If you have a high-level overview of how these things work and code one small example yourself — get back one embedding, calculate a similarity once — you’ll realise the potential. Transformers.js is a great place to start, with amazing demos that run entirely in your browser, privately on your computer.
