训练 double DQN
您现在将基于已有的 DQN 代码进行修改，来实现 double DQN。
double DQN 只需对 DQN 算法做极少量调整，但能显著缓解 Q 值高估问题，且常常比 DQN 表现更好。
本练习是课程的一部分
Python 中的深度强化学习
练习说明
- 使用
online_network()计算用于 Q 目标的下一步动作，确保得到正确的动作与形状。 - 使用
target_network()估计这些动作对应的 Q 值，同样需要确保获取的数值与形状正确。
交互式实操练习
通过完成这段示例代码来试试这个练习。
for episode in range(10):
state, info = env.reset()
done = False
step = 0
episode_reward = 0
while not done:
step += 1
total_steps += 1
q_values = online_network(state)
action = select_action(q_values, total_steps, start=.9, end=.05, decay=1000)
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
replay_buffer.push(state, action, reward, next_state, done)
if len(replay_buffer) >= batch_size:
states, actions, rewards, next_states, dones = replay_buffer.sample(64)
q_values = online_network(states).gather(1, actions).squeeze(1)
with torch.no_grad():
# Obtain next actions for Q-target calculation
next_actions = ____.____.____
# Estimate next Q-values from these actions
next_q_values = ____.____.____
target_q_values = rewards + gamma * next_q_values * (1-dones)
loss = nn.MSELoss()(q_values, target_q_values)
optimizer.zero_grad()
loss.backward()
optimizer.step()
update_target_network(target_network, online_network, tau=.005)
state = next_state
episode_reward += reward
describe_episode(episode, reward, episode_reward, step)