Meerderheidsstemming over meerdere databronnen

Je team ontwikkelt een AI-model dat automatisch kwaliteitscontrole- (QC) rapporten voor smartphones genereert. Hiervoor heb je voorkeurdata verzameld uit drie verschillende QC-bronnen: een "Automated Vision System", een "Human Inspector" en "Customer Feedback". Ze hebben elk gepaarde tekstvoorbeelden gelabeld als 'chosen' en 'rejected'. Elk paar heeft een unieke 'id', en elke invoer laat een geprefereerde QC-beoordeling zien.

quality_df is een gecombineerde DataFrame die is ingeladen met pandas. Het bevat data uit de drie verschillende databronnen. Daarnaast is de klasse Counter vooraf geïmporteerd uit de module collections.

Deze oefening maakt deel uit van de cursus

Reinforcement Learning from Human Feedback (RLHF)

Cursus bekijken

Oefeninstructies

Tel hoe vaak elk (chosen, rejected)-paar voorkomt in de vote-functie.
Zoek het (chosen, rejected)-paar met het hoogste aantal stemmen.

Praktische interactieve oefening

Probeer deze oefening eens door deze voorbeeldcode in te vullen.

def majority_vote(df):
  	# Count occurrences of each (chosen, rejected) pair
    votes = ____
    # Find the (chosen, rejected) pair with the highest vote count
    winner = ____
    return winner

final_preferences = quality_df.groupby(['id']).apply(majority_vote)

print(final_preferences)

Code bewerken en uitvoeren

Deze oefening maakt deel uit van de cursus

Reinforcement Learning from Human Feedback (RLHF)

SkillTag.level.advancedSkillTag.label

4.8+

Begin de cursus gratis

This chapter introduces the basics of Reinforcement Learning with Human Feedback (RLHF), a technique that uses human input to help AI models learn more effectively. Get started with RLHF by understanding how it differs from traditional reinforcement learning and why human feedback can enhance AI performance in various domains.

Exercise 1: Introduction to RLHF Exercise 2: Text generation with RLHF Exercise 3: Classifying generated text for RLHF Exercise 4: RL vs. RLHF Exercise 5: Exploring pre-trained LLMs Exercise 6: Tokenize a text dataset Exercise 7: Fine-tuning for review classification Exercise 8: Preparing data for RLHF Exercise 9: Preparing the preference dataset Exercise 10: Extracting prompts

Discover how to set up systems for gathering human feedback in this Chapter. Learn best practices for collecting high-quality data, from pairwise comparisons to uncertainty sampling, and explore strategies for enhancing your data collection.

Exercise 1: Methods for high-quality feedback gathering Exercise 2: Understanding comparison and rating in RLHF Exercise 3: Comparing slogans for a gym campaign Exercise 4: Measuring feedback quality and relevance Exercise 5: Low confidence Exercise 6: K-means for feedback clustering Exercise 7: Active learning Exercise 8: Implementing an active learning pipeline Exercise 9: Active learning loop

In this Chapter, you'll get into the core of Reinforcement Learning from Human Feedback training. This includes exploring fine-tuning with PPO, techniques to train efficiently, and handling potential divergences from your metrics' objectives.

Exercise 1: Reward models explored Exercise 2: Initializing the reward Exercise 3: Setting up the reward trainer Exercise 4: Training with PPO Exercise 5: Initialize the PPO trainer Exercise 6: PPO fine-tuning Exercise 7: Efficient fine-tuning in RLHF Exercise 8: Prepare for 8-bit Training Exercise 9: Train with LoRA

Explore key techniques for assessing and improving model performance in this last Chapter of Reinforcement Learning from Human Feedback (RLHF): from fine-tuning metrics to incorporating diverse feedback sources, you'll be provided with a comprehensive toolkit to refine your models effectively.

Exercise 1: Modelmetingen en bijsturing Exercise 2: Negatieve KL-divergentie beperken Exercise 3: Het rewardmodel controleren Exercise 4: Diverse feedbackbronnen integreren Exercise 5: Meerderheidsstemming over meerdere databronnen

Huidige oefening

Exercise 6: Onbetrouwbare gegevensbron identificeren Exercise 7: RLHF-modellen evalueren Exercise 8: Curves interpreteren Exercise 9: RLHF evalueren met metriek Exercise 10: Afronding van je RLHF-reis