# Vector & Semantic Search in the Lakehouse: Faster Insight from Unstructured Data

Engineering · Jun 11, 2025 · Adishesh Kishore, Srikanth Venugopalan

> Vector and semantic search let lakehouse teams unlock 80% of enterprise data that lives in free-form text, images, and other unstructured formats. By embedding high-dimensional vectors directly into the SQL optimizer, e6data’s “Unify, Don’t Migrate” engine runs cosine-similarity joins faster without extra ETL or a separate vector database. The result: lower costs, simpler pipelines, and far richer insights from customer voice, support chats, docs, and more.

By 2025, the world will generate **175 zettabytes of data, and ≈80% of it will be unstructured;** yet **90% of that trove is never analyzed**[(Forbes Tech Council, 2022)](https://www.forbes.com/councils/forbestechcouncil/2022/02/03/the-unseen-data-conundrum/?utm_source=chatgpt.com). Conventional SQL predicates (LIKE '%price%') were designed for rows and columns, not emojis, paraphrases, multilingual nuances, or numeric representations. **Vector search, semantic search, and similarity search** provide a first scalable way, via high-dimensional vector embeddings, to make these dark corners of the lakehouse searchable without exporting petabytes to a specialist silo.

‍

## From Neat Tables to Messy Reality

Early “data warehouse” stories read like urban planning fables: define schema, enforce contracts, rinse, repeat. Then reality intruded. Modern ingest tiers happily dump semi-structured or fully opaque blobs into S3, Delta, and Iceberg: Pick your poison.

![A reminder that unstructured data is outgrowing structured data](/images/blog/KQi0dx5FFjbBSjpQK3hN2f1h8.png)

*A reminder that unstructured data is outgrowing structured data*

‍

Text fields become data swamps. For example, a text field labeled ‘Customer Notes’ doesn’t really express what notes it may contain. There is no semantic around it. There are just blobs of text stored in this text field. The dashboard simply ignores sentiment, intent, and sarcasm hidden within text fields.  
  
Text fields are stored in SQL, but it’s challenging to analyze because there is no built-in semantic understanding. Traditional predicates (`LIKE '%price%'`) miss paraphrases, negations, emojis, and multilingual synonyms. The result is an analytics blind spot that grows in lock-step with your customer’s voice.

‍

## Why Keyword Search Isn’t Enough

Human language relies on meaning, not token coincidence. The image below compares **Keyword Search** and **Similarity Search**: the former highlights identical strings, while the latter clusters semantically proximate phrases, even when their stems diverge.

Keyword match = token coincidence; human understanding = **semantic proximity**.

semantic proximity ∝ 1 – cosine θ

![Keyword search matches only the exact term, while similarity (vector) search matches semantic intent](/images/blog/8Qto9uqQ4Wh1jf0HEgHC5TM30.png)

*Keyword search matches only the exact term, while similarity (vector) search matches semantic intent*

‍

If you type ‘enterprise plan’ in the keyword search, it ends up looking for the characters in the phrase. But an enterprise plan could mean a premium tier, guaranteed uptime, and white-glove support.  These are not evident when you want to do a textual lookup. A semantic search, in contrast, allows you to do that. When you look up ‘enterprise plan,’ you land on something that gives you details on the premium tier, guaranteed uptime, and white-glove support. Semantic search searches by meaning and not just words.

**Example:** Think of a support scenario. An agent tries to find “refund frustration” tickets, but the keyword engine surfaces only literal matches. In contrast, a vector engine bubbles up “product didn’t meet expectations,” “want my money back,” and a dozen more paraphrases that the agent would otherwise miss.

| Query | Keyword Search Finds | Similarity Search Also Surfaces |
| --- | --- | --- |
| “enterprise plan” | Strings containing “enterprise plan” | “premium tier”, “guaranteed uptime”, “white-glove support” |
| “refund frustration” | Literal phrase only | “want my money back”, “didn’t meet expectations” |

  
Vector embeddings map every phrase into a **high-dimensional space**; **cosine similarity** then retrieves nearest neighbors regardless of wording, each vector being a numeric representation of linguistic context.

‍

## The Lakehouse Dilemma

Analytical databases excel at predicate push-down, column pruning, vectorized execution—*on structured columns*. Vector databases bloom on the side, optimized for ANN indices but alien to OLAP joins.

![The Lakehouse Dilemma](/images/blog/vdWT4VDKcY1Y6kaz3zl7PRXopPg.png)

Operational DBs hold a slice, ETL copies into warehouses, specialized vector DBs fork yet another silo. Each hop costs latency, dollars, and governance headaches.  
‍

| Option | Strength | Pain |
| --- | --- | --- |
| Standalone Vector DB | Millisecond ANN search | New cluster, new governance, ETL hop |
| Traditional OLAP | Joins, predicate push-down | No semantic awareness |
| Lakehouse-native vector search | Shares parquet metadata, cache & RBA | Requires deep engine integration |

‍

## Enter e6data’s “Unify, Don’t Migrate” Motto

e6data’s philosophy—“Unify, Don’t Migrate”—embeds vector functions into the same optimizer that already handles column pruning and distributed scans. One planner, one security layer, zero copies.

‍

![Enter e6data’s “Unify, Don’t Migrate” Motto](/images/blog/Wiv0Rhqw0xPPDWsSivEPd49PWdA.png)

‍

Same table, same optimizer, same file reads—only now the planner injects a vector projection. The promise is seductive: no pipelines, zero duplication, federated permissions. Let’s answer the architectural “why”: pushing vector search *into* the query engine keeps compute close to data locality, sharing cache, scheduler, and security layers already battle-tested for petabyte SQL scans. It turns a maintenance nightmare into a compiler problem.

‍

## Semantic Search Up Close

![Medium product screenshot](/images/blog/psOxsg4t8yN2CdvGSt1sPhu1G8.png)

*Image credits: Medium*

‍

Semantic search works by transforming the unstructured data (images, documents, audio, video) into high-dimensional vector embeddings, i.e., numeric representations. But the way this works depends on the source data.

Take a look at this image:

![Semantic Search Up Close](/images/blog/wOLt0CbgMukyX2DjP9KaQ3Wc7FE.png)

‍

Translate “too expensive” into \[0.815, 0.642, -0.233, …\] and suddenly “price is a bit much,” “can’t afford this,” and “costs more than expected” land in the same neighborhood even though they share zero lexical tokens.

**Vectors turn free text into geometry; cosine similarity measures angular distance regardless of magnitude.**  

‍

### A Tiny Experiment

Open your favorite SQL prompt and try the most innocent query against an Amazon reviews export:

```sql
SELECT review_id, review_headline
FROM   reviews
WHERE  review_headline ILIKE '%too expensive%'
LIMIT  10;
```

It returns a handful of hits and misses tens of thousands of paraphrases hiding behind different wording.

With Vector Search, the above query can be rewritten as:

```sql
SELECT review_id, review_headline
FROM   reviews
WHERE  cosine_distance(review_headline, 'too expensive') <0.1
LIMIT  10;
```

‍

This represents a Nearest Neighbour search on review\_headline column.

This query can now be extended to have more SQL constructs and use the best of SQL and Vector operations.

‍

### How Vector, Semantic, & Similarity Search Work

We’ve seen why keyword search buckles under the weight of customer slang, emojis, and multilingual nuances, and we’ve explored how silo-heavy architectures force teams into cost-intensive copy-and-paste gymnastics.

The takeaway is clear - to mine real insight from the 80% of data that hides in free-form text, we need an engine that speaks both SQL and semantics without spawning yet another datastore. That’s where vector, semantic, and similarity searches step in. But *how* do they actually convert messy language into precise geometry that the optimizer can race through?

-   **Embedding** – Models such as SBERT or OpenAI’s text-embedding-3-large convert each document into a 1 k-dimensional vector.
-   **ANN Index** – There are several indices you can use based on your data set and Vector DBs. Structures like **HNSW**, **SCaNN**, and **DiskANN** accelerate Approximate Nearest Neighbor lookups.
-   **Query Flow for e6data bottom-up SQL engine architecture**
    
    -   Run the SQL filter
        
    -   Reduce search space
    -   Carry out vector search on the reduced search space

‍

## Conclusion

Unstructured data isn’t a side gig anymore; it’s the majority of your lakehouse. **Vector search, semantic search, and similarity search** turn those blobs into queryable geometry without abandoning SQL or spinning up another silo. In our next blog post, we’ll dive into **embedding best practices, cosine similarity math, and writing first-class SQL over vectors**.

‍

‍