Applications that process large amounts of text often run into duplicate content. While exact duplicates are easy to detect using hashes, identifying documents that say the same thing with different wording is much harder.

I’m talking about documents like these:

“Apple unveils the new iPhone 17 at WWDC.

and

“At WWDC, Apple announced its latest iPhone 17.”

These sentences are semantically identical, i.e they convey the same thing, but traditional string comparison techniques treat them as different. Methods like hashes, edit distance, or regular expressions only compare the text itself, not its meaning.

A much better approach is semantic deduplication using embeddings.

What are embeddings?

An embedding is a numerical representation of text generated by an AI model. Instead of representing a document as words, it is represented as a list of floating-point numbers, commonly called a vector.

For example, a sentence might be converted into something like:

[0.127, -0.443, 0.982, ...]

The numbers themselves are not meaningful to humans, but together they capture the semantic meaning of the text.

Documents discussing similar topics are placed close together in this high-dimensional vector space, while unrelated documents are placed farther apart.

For example:

"The meeting starts at 9 AM."
[0.12, -0.44, 0.98, ...]

"The conference begins at nine in the morning."
[0.11, -0.46, 0.97, ...]

"I ordered pizza for dinner."
[-0.81, 0.63, -0.24, ...]

Notice that the first two vectors are very similar because both sentences describe the same event. The third vector is very different because it talks about an unrelated topic.

Instead of comparing the original text, applications compare these vectors. If two vectors are close together, the documents are likely to have similar meanings, even if they use different words.

These numbers aren’t random. The embedding model has been trained on billions of pieces of text, allowing it to learn that certain words and phrases have similar meanings. For example, it learns that “starts” and “begins” are closely related, and that “9 AM” and “nine in the morning” refer to the same time.

This allows applications to compare meaning rather than exact wording.

Generating embeddings

Once the concept of embeddings is understood, the next step is generating them.

In my project, I had used OpenAI’s text embedding model is used through OpenRouter.

An embedding model is different from a chatbot like ChatGPT. Rather than generating text, its only job is to read the input and convert it into a numerical vector that captures its meaning. The model has been trained on massive amounts of text, allowing it to recognize relationships between words, phrases, and concepts. As a result, pieces of text with similar meanings produce similar vectors.

When a document is received, its text is sent to the embedding API, which returns a vector representation of that document.

Document
      │
      ▼
OpenAI Text Embedding Model
(via OpenRouter)
      │
      ▼
Embedding Vector

For example, if the document contains:

Apple announced the iPhone 17 during WWDC.

the embedding model might return a vector containing hundreds or thousands of floating-point numbers:

[0.127, -0.443, 0.982, ...]

The actual values aren’t important, they’re simply the mathematical representation of the document’s meaning.

This embedding is generated only once and stored alongside the original document in the database. Later, when another document arrives, its embedding is generated and compared against the stored embeddings to determine whether it is semantically similar to any existing document.

OpenRouter is an API gateway that provides access to AI models from multiple providers through a single API. Instead of integrating directly with OpenAI’s API, requests are sent to OpenRouter, which routes them to the selected model.

Storing embeddings in a vector database

Once an embedding has been generated, it needs to be stored somewhere so it can be compared against future documents.

While a regular database can store the vector as an array of numbers, it isn’t designed to efficiently answer questions like:

“Which stored embedding is most similar to this new one?”

Finding the answer by comparing the new vector with every vector in the database would work for a few hundred records, but it quickly becomes too slow as the dataset grows.

A vector database is designed specifically for storing embeddings and performing similarity searches. Instead of looking for exact matches, it finds vectors that are mathematically closest to a query vector. This makes it ideal for semantic search, recommendation systems, image search, and duplicate detection.

Some popular vector databases include:

• Pinecone — A fully managed cloud vector database built specifically for similarity search.
• Milvus — An open-source vector database designed for large-scale AI applications.
• Qdrant — An open-source vector database with filtering and metadata support.
• Weaviate — A vector database that combines semantic search with structured data.
• Chroma — A lightweight vector database commonly used in local AI applications and prototypes.

For applications that already use PostgreSQL, however, there is often no need to introduce another database. The pgvector extension turns PostgreSQL into a vector database by adding support for vector data types, similarity operators, and specialized indexes.

A simple table might look like this:

CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,
    content TEXT,
    embedding VECTOR(1536)
);

Here, the embedding column stores the vector returned by the embedding model alongside the original document.

Making similarity search fast

If the database only contained a few hundred vectors, PostgreSQL could compare every stored embedding against a new one. But with thousands or millions of documents, that approach becomes too slow.

pgvector solves this by supporting Approximate Nearest Neighbor (ANN) indexes, such as HNSW (Hierarchical Navigable Small World).

CREATE INDEX idx_documents_embedding
ON documents
USING hnsw (embedding vector_cosine_ops);

Rather than comparing every vector in the table, the HNSW index organizes embeddings into a graph that allows PostgreSQL to quickly navigate toward the closest matches. This dramatically reduces the number of comparisons needed, making similarity searches fast even on very large datasets.

With the index in place, PostgreSQL can efficiently answer questions instead of scanning the entire table every time a search is performed.

How similarity search works

Once every document has an embedding, duplicate detection becomes a nearest-neighbor search.

When a new document arrives:

1. Generate its embedding.
2. Search for the closest existing embedding.
3. If the similarity exceeds a chosen threshold, treat it as a duplicate.

The SQL query is straightforward.

SELECT
    id,
    content,
    embedding <=> $1 AS distance
FROM documents
ORDER BY embedding <=> $1
LIMIT 5;

The <=> operator computes the cosine distance between vectors.

Cosine distance measures the angle between two vectors rather than their numerical values. Documents with similar meanings tend to point in nearly the same direction, resulting in a very small distance.

For example:

Rather than checking every word, the database simply finds vectors that are closest together.

Choosing a similarity threshold

The threshold determines what counts as a duplicate.

A very high threshold only catches nearly identical documents.

A lower threshold begins grouping paraphrases and summaries together.

There isn’t a universal value because different embedding models produce different vector distributions. The best approach is to test several thresholds using real data and manually inspect the matches.

Why this approach works well

Compared to traditional text matching techniques, semantic embeddings offer several advantages.

They understand meaning rather than exact wording, making them robust against paraphrasing and minor edits.

They require very little custom logic, since the embedding model captures semantic relationships automatically.

Finally, they scale efficiently. Combined with pgvector’s HNSW indexing, PostgreSQL can perform similarity searches across millions of embeddings in milliseconds.