Multi-input models

1. Multi-input models

Let's learn to build multi-input models!

2. Why multi-input?

Multi-input models, or models that accept more than one source of data, have many applications. First, we might want the model to use multiple information sources, such as two images of the same car to predict its model. Second, multi-modal models can work on different input types such as image and text to answer a question about the image. Next, in metric learning, the model learns whether two inputs represent the same object. Think about an automated passport control where the system compares our passport photo with a picture it takes of us. Finally, in self-supervised learning, the model learns data representation by learning that two augmented versions of the same input represent the same object. Multi-input models are everywhere!

3. Omniglot dataset

Throughout the chapter, we will be using the Omniglot dataset, a collection of images of 964 different handwritten characters from 30 different alphabets.

4. Character classification

Let's use the Omniglot dataset to build a two-input model to classify handwritten characters. The first input will be the image of the character, such as this Latin letter "k".

5. Character classification

The second input will the the alphabet that it comes from expressed as a one-hot vector.

6. Character classification

Both inputs will be processed separately, then we concatenate their representations.

7. Character classification

Finally a classification layer predicts one of the 964 classes. We need two elements to build such a model: a custom Dataset and an appropriate model architecture.

8. Two-input Dataset

Let's start with the custom Omniglot dataset. We set it up as a class based on torch Dataset. In the init method, we store transform and samples provided when instantiating the dataset as class attributes. Samples are tuples of three: image file path, alphabet as a one-hot vector, and target label as the character class index. In the exercises, samples will be provided. For personal projects, we would need to create them from data file paths. Next, we need to implement the len method that returns the number of samples. Finally, the getitem method returns one sample based on the index it receives as input. For the given index, we retrieve the sample and load the image using Image.open from PIL. The convert method with the argument "L" makes sure that the image is read as grayscale. Then, we transform the image and return a triplet: the transformed image, the alphabet vector, and the target label.

9. Tensor concatenation

Before we proceed to building the model, we need to understand tensor concatenation. torch.cat concatenates tensors along a specified dimension. We pass it the tensors and the dimension: for 2D tensors, 0 stands for "horizontal" and 1 stands for "vertical" concatenation.

10. Two-input architecture

It's time to define our two-input model. We start with defining a sub-network or layer to process our first input, the image. It should look familiar: a convolution, max pool, elu activation, flattened to a linear layer of shape 128 in the end. Next, we define a layer to process our second input, the alphabet vector. Its input size is 30, the number of alphabets, and we map it to an arbitrarily chosen output size of 8. Then, a classifier would accept input of size 128 plus 8 (image and alphabet outputs concatenated) and produce the output of size 964, the number of classes.

11. Two-input architecture

In the forward method, we pass each input through its corresponding layer. Then, we concatenate the outputs with torch.cat. Finally, we pass the result through the classifier layer and return.

12. Training loop

The training loop looks just like all the ones we have seen so far. The only difference is that now the training data consists of three items: the image, the alphabet vector, and the labels, and we pass the images and alphabets to the model.

13. Let's practice!

Let's build a multi-input model!

This exercise is part of the course

Intermediate Deep Learning with PyTorch

IntermediateSkill Level

4.8+

Start Course for Free

Learn how to train neural networks in a robust way. In this chapter, you will use object-oriented programming to define PyTorch datasets and models and refresh your knowledge of training and evaluating neural networks. You will also get familiar with different optimizers and, finally, get to grips with various techniques that help mitigate the problems of unstable gradients so ubiquitous in neural nets training.

Exercise 1: PyTorch and object-oriented programming Exercise 2: PyTorch Dataset Exercise 3: PyTorch DataLoader Exercise 4: PyTorch Model Exercise 5: Optimizers, training, and evaluation Exercise 6: Training loop Exercise 7: Optimizers Exercise 8: Model evaluation Exercise 9: Vanishing and exploding gradients Exercise 10: Initialization and activation Exercise 11: Activations: ReLU vs. ELU Exercise 12: Batch Normalization

Train neural networks to solve image classification tasks. In this chapter, you will learn how to handle image data in PyTorch and get to grips with convolutional neural networks (CNNs). You will practice training and evaluating an image classifier while learning about how to improve the model performance with data augmentation.

Exercise 1: Handling images with PyTorch Exercise 2: Image dataset Exercise 3: Data augmentation Exercise 4: Data augmentation in PyTorch Exercise 5: Convolutional Neural Networks Exercise 6: The convolutional layer Exercise 7: Building convolutional networks Exercise 8: Training image classifiers Exercise 9: Choosing augmentations Exercise 10: Dataset with augmentations Exercise 11: Image classifier training loop Exercise 12: Evaluating image classifiers Exercise 13: Multi-class model evaluation Exercise 14: Analyzing metrics per class

Build and train recurrent neural networks (RNNs) for processing sequential data such as time series, text, or audio. You will learn about the two most popular recurrent architectures, Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks, as well as how to prepare sequential data for model training. You will practice your skills by training and evaluating a recurrent model for predicting electricity consumption.

Exercise 1: Handling sequences with PyTorch Exercise 2: Generating sequences Exercise 3: Sequential Dataset Exercise 4: Recurrent Neural Networks Exercise 5: Sequential architectures Exercise 6: Building a forecasting RNN Exercise 7: LSTM and GRU cells Exercise 8: RNN vs. LSTM vs. GRU Exercise 9: LSTM network Exercise 10: GRU network Exercise 11: Training and evaluating RNNs Exercise 12: RNN training loop Exercise 13: Evaluating forecasting models

Build multi-input and multi-output models, demonstrating how they can handle tasks requiring more than one input or generating multiple outputs. You will explore how to design and train these models using PyTorch and delve into the crucial topic of loss weighting in multi-output models. This involves understanding how to balance the importance of different tasks when training a model to perform multiple tasks simultaneously.

Exercise 1: Multi-input models

Current Exercise

Exercise 2: Two-input dataset Exercise 3: Two-input model Exercise 4: Training two-input model Exercise 5: Multi-output models Exercise 6: Two-output Dataset and DataLoader Exercise 7: Two-output model architecture Exercise 8: Training multi-output models Exercise 9: Evaluation of multi-output models and loss weighting Exercise 10: Multi-output model evaluation Exercise 11: Loss weighting Exercise 12: Wrap-up