Defining a function for chunking
To be able to batch upserts in a reproducible way, you'll need to define a function to split your list of vectors into chunks.
The built-in itertools module has already been imported for you.
Deze oefening maakt deel uit van de cursus
Vector Databases for Embeddings with Pinecone
Oefeninstructies
- Convert the
iterableinput into an iterator. - Slice
itinto chunks of sizebatch_sizeusing theitertoolsmodule. - Yield the current chunk.
Praktische interactieve oefening
Probeer deze oefening eens door deze voorbeeldcode in te vullen.
def chunks(iterable, batch_size=100):
"""A helper function to break an iterable into chunks of size batch_size."""
# Convert the iterable into an iterator
it = ____
# Slice the iterator into chunks of size batch_size
chunk = tuple(itertools.____(it, ____))
while chunk:
# Yield the chunk
____
chunk = tuple(itertools.islice(it, batch_size))