從 PER 緩衝區抽樣
在你能用 Prioritized Experience Buffer 類別來訓練代理之前，還需要實作 .sample() 方法。這個方法的參數是你想抽取的樣本大小，並會回傳取樣到的轉移（tensors），以及它們在記憶緩衝區裡的索引和對應的重要性權重。
環境中已預先載入一個容量為 10 的緩衝區，供你進行抽樣。
本練習屬於課程
Python 的深度強化學習
練習說明
- 計算每個轉移對應的抽樣機率。
- 抽取樣本中各轉移的索引；
np.random.choice(a, s, p=p)會依照機率陣列p，從陣列a以放回方式抽取大小為s的樣本。 - 計算每個轉移對應的重要性權重。
動手互動練習
試著完成這個範例程式碼，體驗一下這個練習。
def sample(self, batch_size):
priorities = np.array(self.priorities)
# Calculate the sampling probabilities
probabilities = ____ / np.sum(____)
# Draw the indices for the sample
indices = np.random.choice(____)
# Calculate the importance weights
weights = (1 / (len(self.memory) * ____)) ** ____
weights /= np.max(weights)
states, actions, rewards, next_states, dones = zip(*[self.memory[idx] for idx in indices])
weights = [weights[idx] for idx in indices]
states_tensor = torch.tensor(states, dtype=torch.float32)
rewards_tensor = torch.tensor(rewards, dtype=torch.float32)
next_states_tensor = torch.tensor(next_states, dtype=torch.float32)
dones_tensor = torch.tensor(dones, dtype=torch.float32)
weights_tensor = torch.tensor(weights, dtype=torch.float32)
actions_tensor = torch.tensor(actions, dtype=torch.long).unsqueeze(1)
return (states_tensor, actions_tensor, rewards_tensor, next_states_tensor,
dones_tensor, indices, weights_tensor)
PrioritizedReplayBuffer.sample = sample
print("Sampled transitions:\n", buffer.sample(3))