具優先級經驗回放的 DQN
在這個練習中，你會導入優先級經驗回放（Prioritized Experience Replay，PER）來改進 DQN 演算法。PER 旨在最佳化每一步用來更新網路的轉移批次。
供參考，你為 PrioritizedReplayBuffer 宣告的方法名稱有：
push()（將轉移推入緩衝區）sample()（從緩衝區取樣一個轉移批次）increase_beta()（提高重要性抽樣的影響）update_priorities()（更新已取樣的優先度）
describe_episode() 函式再次用來描述每個 episode。
本練習屬於課程
Python 的深度強化學習
練習說明
- 以容量 10000 筆轉移，建立一個優先級經驗回放緩衝區。
- 隨時間提高重要性抽樣的影響，方法是更新
beta參數。 - 根據最新的 TD 誤差，更新已取樣經驗的優先度。
動手互動練習
試著完成這個範例程式碼，體驗一下這個練習。
# Instantiate a Prioritized Replay Buffer with capacity 10000
replay_buffer = ____(____)
for episode in range(5):
state, info = env.reset()
done = False
step = 0
episode_reward = 0
# Increase the replay buffer's beta parameter
replay_buffer.____
while not done:
step += 1
total_steps += 1
q_values = online_network(state)
action = select_action(q_values, total_steps, start=.9, end=.05, decay=1000)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
replay_buffer.push(state, action, reward, next_state, done)
if len(replay_buffer) >= batch_size:
states, actions, rewards, next_states, dones, indices, weights = replay_buffer.sample(64)
q_values = online_network(states).gather(1, actions).squeeze(1)
with torch.no_grad():
next_q_values = target_network(next_states).amax(1)
target_q_values = rewards + gamma * next_q_values * (1-dones)
td_errors = target_q_values - q_values
# Update the replay buffer priorities for that batch
replay_buffer.____(____, ____)
loss = torch.sum(weights * (q_values - target_q_values) ** 2)
optimizer.zero_grad()
loss.backward()
optimizer.step()
update_target_network(target_network, online_network, tau=.005)
state = next_state
episode_reward += reward
describe_episode(episode, reward, episode_reward, step)