1. Învăţa
    2. /
  2. Courses
    3. /
  3. Retrieval Augmented Generation (RAG) with LangChain
Connected

exercise

Splitting by tokens

Splitting documents using RecursiveCharacterTextSplitter or CharacterTextSplitter is convenient, and can give you good performance in some cases, but it does have one drawback: they split using characters as base units, rather than tokens, which are processed by the model.

In this exercise, you'll split documents using a token text splitter, so you can verify the number of tokens in each chunk to ensure that they don't exceed the model's context window. A PDF document has been loaded as document.

tiktoken and all necessary classes have been imported for you.

Instrucțiuni

100 XP
  • Get the encoding for gpt-4o-mini from tiktoken so you can check the number of tokens in each chunk.
  • Create a text splitter to split based on the number of tokens using the GPT-4o-Mini encoding.
  • Split the PDF, stored in document, into chunks using token_splitter.