Sinusoidal and rotary positional embeddings

View the content before continuing.

Den här övningen är en del av kursen

Google DeepMind: Discover The Transformer Architecture

Visa kurs

Interaktiv övning med praktiskt arbete

Gör teori till handling med en av våra interaktiva övningar

Starta övningen

Den här övningen är en del av kursen

Google DeepMind: Discover The Transformer Architecture

SkillTag.level.intermediateSkillTag.label

4.8+

19 reviews

Starta kursen gratis

In this module, you will reflect on which tokens in a prompt have the biggest impact on the prediction of the next token. You will also visualize the attention weights of the Gemma model to see which tokens the model relies on when making predictions. Finally, you will explore how community values and perspectives shape the meaning and impact of AI technologies.

Exercise 1: The architecture of modern LLMs Exercise 2: What drives predictions?Exercise 3: Lab: Visualizing Attention Weights Exercise 4: Learning objectives Exercise 5: How to get the most out of this course Exercise 6: Quiz 1 - Question 1 Exercise 7: Quiz 1 - Question 2

In this module, you will implement the attention mechanism. You will learn how this mechanism is used to combine the information from individual tokens to create embeddings that represent the information of an entire prompt. You will also reflect on how everyday human interactions create shared meaning and reinforce values, such as community, belonging, and respect. Further, you will consider what may be lost when these practices are replaced by automated systems.

Exercise 1: Transformer architecture Exercise 2: Computing attention weights Exercise 3: Lab: Implement the Attention Mechanism Exercise 4: Masked attention Exercise 5: Multi-head attention Exercise 6: Lab: Implement Masked Multi-Head Attention Exercise 7: The attention mechanism Exercise 8: Community values and meaning in an automated world Exercise 9: Quiz 2 - Question 1 Exercise 10: Quiz 2 - Question 2

In this module, you will learn about the other components that are required for building a transformer model. You will investigate the importance of adding positional information to tokens and you will see what components a transformer block consists of. You will also explore the role multi-layer perceptrons and normalization play in the transformer block. Finally, you will walk through a complete implementation of a transformer language model and investigate the parameters that are part of each component.

Exercise 1: Positional embeddings Exercise 2: Lab: Positional Embeddings Exercise 3: Sinusoidal and rotary positional embeddings

Aktuell övning

Exercise 4: Transformer blocks Exercise 5: Multi-layer perceptron Exercise 6: Layer normalization Exercise 7: Lab: Trainable Parameters in the Transformer Model Exercise 8: Quiz 3 - Question 1 Exercise 9: Quiz 3 - Question 2

In this module, you will learn about the advantages and disadvantages of using a transformer model and discover sophisticated methods for generating texts with language models. Additionally, you will consider how technologies like chatbots are understood differently by different groups, revealing why meaningful engagement is essential to avoid reinforcing stereotypes, deepening inequalities, or overlooking social values. You will see how, by recognising diverse perspectives, developers can design AI that is more inclusive, fair, and responsive to community needs.

Exercise 1: Pros and cons of transformers Exercise 2: Decoding and generation Exercise 3: Why engagement matters: Gendered chatbots in Nigerian banks Exercise 4: Quiz 4 - Question 1 Exercise 5: Quiz 4 - Question 2

In this module, the stakeholder mapping and social values activity will help you identify who is affected by your project, what values matter to them, and how their influence shapes outcomes. This will be followed by a mini-engagement design which will guide you to plan simple, practical ways of involving these groups so their perspectives meaningfully shape your AI project.

Exercise 1: Mapping stakeholders and social values Exercise 2: Design a mini-engagement plan Exercise 3: Quiz 5 - Question 1 Exercise 4: Quiz 5 - Question 2

In this module, you will have the opportunity to consult additional resources and further reading to investigate the topics you have covered in more detail. Finally, you will consider your next steps and how you can build on what you have learned in the course.

Exercise 1: Summary Exercise 2: Looking forward Exercise 3: Additional resources and further reading Exercise 4: Glossary Exercise 5: Feedback