训练 PPO 算法
现在，您将使用熟悉的 A2C 训练循环来训练 PPO 算法。
此训练循环并未充分利用裁剪后的代理目标函数，因此该算法的表现不应明显优于 A2C；它主要用于演示与裁剪代理目标和熵奖励相关的概念。
本练习是课程的一部分
Python 中的深度强化学习
练习说明
- 从 actor 损失中移除熵奖励，
c_{entropy}参数取 0.01。
交互式实操练习
通过完成这段示例代码来试试这个练习。
for episode in range(10):
state, info = env.reset()
done = False
episode_reward = 0
step = 0
while not done:
step += 1
action, action_log_prob, entropy = select_action(actor, state)
next_state, reward, terminated, truncated, _ = env.step(action)
episode_reward += reward
done = terminated or truncated
actor_loss, critic_loss = calculate_losses(critic, action_log_prob, action_log_prob,
reward, state, next_state, done)
# Remove the entropy bonus from the actor loss
actor_loss -= ____ * ____
actor_optimizer.zero_grad(); actor_loss.backward(); actor_optimizer.step()
critic_optimizer.zero_grad(); critic_loss.backward(); critic_optimizer.step()
state = next_state
describe_episode(episode, reward, episode_reward, step)