Twin Delayed DDPG (TD3)
Twin Delayed DDPG (TD3) é uma melhoria do Gradiente de Política Determinística Profunda (DDPG) algorithm, specifically designed for solving aprendizado por reforço problems in continuous action spaces. It addresses some of the key challenges faced by DDPG, such as o viés de superestimação e instabilidade durante o treinamento.
O TD3 melhora o DDPG por meio de três inovações principais:
- Redes Q gêmeas: Instead of using a single Q-network to estimate the value of actions, TD3 employs two separate Q-networks. This helps to mitigate the overestimation of action values, which is a common issue in Aprendizado Q algorithms. By taking the minimum value from the two Q-networks when updating the policy, TD3 achieves more reliable estimates.
- Atualizações de política atrasadas: In TD3, the policy and target networks are updated less frequently than the Q-networks. This means that the policy is updated only after a certain number of Q-network updates, allowing for more stable learning. This delay helps prevent the policy from changing too rapidly based on potentially noisy Q-value estimates.
- Suavização da política alvo: TD3 adds noise to the target policy during training, which encourages exploration and helps the algorithm to avoid overfitting to specific actions. This is done by applying a small amount of random noise to the target actions, leading to more robust learning.
Overall, TD3 has shown significant improvements in performance and stability over its predecessor, DDPG, making it a popular choice for various applications in robotics, gaming, and sistemas de controle onde espaços de ação contínuos de alta dimensão estão envolvidos.