1. Semantic search with Pinecone
Welcome back! In this video, we'll learn how to build a semantic search engine using Pinecone.
2. Semantic search engines
Semantic search engines use embeddings to return the most semantically similar results to a user input.
This involves embedding and ingesting a series of documents to search through into a Pinecone index, embedding a user query, and using the user query vector to query the index and return results.
We've seen how to ingest vectors and query indexes, so the final piece we need is to be able to embed the document and user query to create the vectors.
3. Setting up Pinecone and OpenAI for semantic search
For this, we'll use an OpenAI embedding model.
We'll initialize Pinecone and OpenAI clients; and create a new serverless index specifically for our semantic search application.
We give this index a dimensionality of 1536, which is the dimensionality of the vectors outputted by the OpenAI model we'll be using. If another embedding model is used, be sure to check the output dimensions and use that here.
Finally, we connect to the index.
4. Ingesting documents to Pinecone index
With our Pinecone index ready,
let's prepare our documents for indexing. We'll use embedding and ingesting SQuAD dataset, comprising of Wikipedia articles that we can semantically search through.
We start by importing pandas for reading the data with read_csv(), NumPy for a couple of data manipulation functions, and uuid, which we'll use to generate unique record IDs before upserting.
The dataset includes columns for article id, text, and title.
5. Ingesting documents to Pinecone index
To begin ingesting documents into our Pinecone index,
we set a batch_limit to process 100 documents per request for efficiency.
We'll iterate through the documents in batches, using NumPy's array_split() function and batch_limit to create batches from the DataFrame.
With each batch, we'll extract the metadata into a dictionary using a list comprehension. This creates a metadata dictionary for each row in the batch, assigning the row values to dictionary keys. We also extract the article text to embed and create unique IDs for each row using the uuid4() function.
Next, we create a request to OpenAI's API to embed these texts, extracting the raw embeddings from the response.
Finally, we zip together the ids, vectors, and metadatas and upsert them to a new namespace called squad_dataset.
6. Ingesting documents to Pinecone index
As before, we can verify the upsertion with .describe_index_stats()
7. Querying with Pinecone
Let's now perform semantic search with Pinecone!
Consider the following query.
To initiate semantic search, we encode this query using the same embedding model again. It's crucial to align the embedding model used for the query and documents, ensuring consistency in dimensions and processing methodology.
Once the query is encoded, we query the namespace with it. We added the include_metadatas argument here so we can access the article text, which is stored in the metadata, for the returned records.
8. Querying with Pinecone
We can loop through the returned records and extract the similarity scores and texts. Not bad!
9. Time to build!
Time to build your own semantic search engine!