Auto Models and Tokenizers
1. Auto Models and Tokenizers
Welcome back!2. Pipelines: fast and simple
Pipelines are a fantastic way to quickly perform tasks like text classification or summarization, as we explored previously. But what if you need more control?3. Two ways to use Hugging Face models
If you explored a model card within the Hub, you likely noticed the “Use in Transformers” button.4. Two ways to use Hugging Face models
This pops up a chunk of code that can be used to quickly get started with a model. There are two main ways to use models with the Transformers package: The first, which we already covered, is the high-level helper framework called pipeline.5. Two ways to use Hugging Face models
The second is a more hands-on and flexible approach using Auto classes. Let’s see what makes Auto classes so powerful!6. Auto Classes: flexible and powerful
Auto classes are a flexible way to load models, tokenizers, and other components without manual setup. They offer more control compared to pipelines, making them ideal for advanced tasks. While pipelines are great for quick experimentation, Auto classes let us customize every step.7. AutoModels
Now that we’ve introduced Auto classes, let’s see how they simplify working with models. To directly download a model, we import the relevant AutoModel class for your task. For example, in text classification (also known as Sequence Classification), we use AutoModelForSequenceClassification. To load the model, use the .from_pretrained() method and specify the model name.8. AutoTokenizers
To prepare text input for use in a model, we use tokenizers. It’s recommended to use the tokenizer paired with the model to ensure the input is processed exactly as it was during training. With pipelines, this tokenizer and model pairing happens automatically. However, with Auto classes, you handle this step yourself. To retrieve the tokenizer for our model, first import AutoTokenizer from transformers. Then, call the .from_pretrained() method and pass in the model name.9. Tokenizing text with AutoTokenizer
Tokenizers work by first cleaning the input, such as lowercasing words or removing accents, and then dividing the text into smaller chunks called tokens. In this example, we load the tokenizer paired with our model using .from_pretrained(). Then, we call the .tokenize() method to split the sentence into tokens. The output shows how the text is processed and broken into smaller parts that the model can understand.10. Different models, different tokenizers
Different models may handle tokenization differently. While our example tokenizer processes text in a specific way, other models may produce different outputs for the same input.11. Building a Pipeline with Auto Classes
Now let’s create a custom pipeline. We start by importing the necessary modules, downloading a model and tokenizer, and combining them into a pipeline. By specifying the task, model, and tokenizer, we gain full control over the process for sentiment analysis.12. Use Cases for AutoModels and AutoTokenizers
In a nutshell, we use AutoModels and AutoTokenizers instead of simple pipelines when tasks demand more control and customization. For example, advanced text preprocessing and tokenization enable cleaning tailored to specific use cases. In classification tasks, custom thresholding allows setting weights and thresholds to prioritize categories, like assigning 'Support' more often in a customer support model. Lastly, for complex text analysis workflows involving multiple processing stages, they provide precise control to customize and integrate each step effectively.13. Let's practice!
We’ve unlocked the power of AutoModels—now, let’s dive into practice!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.