Embeddings: a map where words live

Nani has lived in the same Lahore mohalla for fifty years. Ask her where to buy anything and she points without thinking. Bangles? Anarkali. Car parts? McLeod Road. Books? Urdu Bazaar. Fabric for a wedding? Liberty or Ichhra.

In her head, every shop has a place. And shops that sell similar things sit close to each other. If you found a new shop next to three cloth merchants, you would guess it sells cloth too, without even looking.

That mental map is what AI calls an embedding space. And it is how computers finally learned to work with meaning.

Computers need numbers, words are not numbers

A computer can't do maths with the word "biryani". An old trick was to give every word an ID number: biryani = 1, pulao = 2, tractor = 3. But those numbers say nothing. Is biryani closer to pulao or to tractor? The IDs can't tell you.

An embedding solves this by giving every word not one number, but a list of numbers, a set of coordinates. Just like a shop in Lahore has a location on a map, each word gets a location in a "meaning space".

Real embeddings don't use two coordinates like a paper map. They use hundreds or thousands. Nobody can picture that many dimensions, but the idea is the same: similar meanings sit close together.

Where do the coordinates come from?

Here is the beautiful part. Nobody places the words by hand. The model learns the map from one simple observation, often summed up as:

"You shall know a word by the company it keeps." — J. R. Firth, 1957

"Biryani" and "pulao" show up in similar sentences: with "rice", "dinner", "masala", "plate". "Tractor" shows up with "field", "wheat", "diesel". A model reading millions of sentences keeps adjusting the coordinates so that words used in similar ways end up near each other.

Directions mean something too

In 2013, a model called word2vec showed something surprising. On the learned map, the direction between words carried meaning. The famous example:

king − man + woman ≈ queen

Walking from "man" to "woman" on the map is roughly the same journey as walking from "king" to "queen". The model had learned a direction that roughly means "female", without anyone telling it.

Why this matters for businesses

Once meaning has coordinates, a computer can measure how close two things are. That unlocks a lot:

• Smart search: find the answer about "refund policy" even when the customer typed "can I get my money back".
• Grouping: cluster thousands of customer complaints into common themes.
• Recommendations: customers who liked this product are close to these other ones.
• Knowledge-base answers: find the right paragraph in your documents before an AI writes a reply. We'll come back to this in Part 9.

The maths, briefly

An embedding is a vector $\mathbf{v} \in \mathbb{R}^d$, where $d$ is often in the hundreds or thousands. Closeness is usually measured with cosine similarity, the cosine of the angle between two vectors:

$$\text{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}$$

It is 1 when two vectors point the same way and close to 0 when they are unrelated. The famous analogy is plain vector arithmetic:

$$\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}$$

What to remember

• An embedding is a list of numbers that places a word (or sentence, image, product) on a "meaning map".
• Things with similar meanings are close together.
• The map is learned from how words are used, not written by hand.
• Measuring distance on this map powers search, grouping and AI answers.

But before a word can be placed on the map, the AI has to decide what counts as a "word" in the first place. That's Part 6.