带优先经验回放的 DQN
在本练习中，您将引入优先经验回放（PER），以改进 DQN 算法。PER 的目标是在每一步更新网络时，优化用于训练的转移批次的选择。
供参考，您为 PrioritizedReplayBuffer 声明的方法如下：
push()（将转移推入缓冲区）sample()（从缓冲区采样一批转移）increase_beta()（提高重要性采样的权重）update_priorities()（更新已采样样本的优先级）
describe_episode() 函数再次用于描述每个 episode。
本练习是课程的一部分
Python 中的深度强化学习
练习说明
- 实例化一个容量为 10000 个转移的优先经验回放缓冲区。
- 随时间提升重要性采样的影响力，方法是更新
beta参数。 - 根据最新的 TD 误差更新已采样经验的优先级。
交互式实操练习
通过完成这段示例代码来试试这个练习。
# Instantiate a Prioritized Replay Buffer with capacity 10000
replay_buffer = ____(____)
for episode in range(5):
state, info = env.reset()
done = False
step = 0
episode_reward = 0
# Increase the replay buffer's beta parameter
replay_buffer.____
while not done:
step += 1
total_steps += 1
q_values = online_network(state)
action = select_action(q_values, total_steps, start=.9, end=.05, decay=1000)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
replay_buffer.push(state, action, reward, next_state, done)
if len(replay_buffer) >= batch_size:
states, actions, rewards, next_states, dones, indices, weights = replay_buffer.sample(64)
q_values = online_network(states).gather(1, actions).squeeze(1)
with torch.no_grad():
next_q_values = target_network(next_states).amax(1)
target_q_values = rewards + gamma * next_q_values * (1-dones)
td_errors = target_q_values - q_values
# Update the replay buffer priorities for that batch
replay_buffer.____(____, ____)
loss = torch.sum(weights * (q_values - target_q_values) ** 2)
optimizer.zero_grad()
loss.backward()
optimizer.step()
update_target_network(target_network, online_network, tau=.005)
state = next_state
episode_reward += reward
describe_episode(episode, reward, episode_reward, step)