Twin Delayed DDPG (TD3)
Twin Delayed DDPG (TD3) es una mejora de Gradiente de Políticas Determinísticas Profundas (DDPG) algorithm, specifically designed for solving aprendizaje por refuerzo problems in continuous action spaces. It addresses some of the key challenges faced by DDPG, such as sesgo de sobreestimación e inestabilidad durante el entrenamiento.
TD3 mejora el DDPG a través de tres innovaciones principales:
- Redes Q gemelas: Instead of using a single Q-network to estimate the value of actions, TD3 employs two separate Q-networks. This helps to mitigate the overestimation of action values, which is a common issue in Aprendizaje Q algorithms. By taking the minimum value from the two Q-networks when updating the policy, TD3 achieves more reliable estimates.
- Actualizaciones retrasadas de la política: In TD3, the policy and target networks are updated less frequently than the Q-networks. This means that the policy is updated only after a certain number of Q-network updates, allowing for more stable learning. This delay helps prevent the policy from changing too rapidly based on potentially noisy Q-value estimates.
- Suavizado de la política objetivo: TD3 adds noise to the target policy during training, which encourages exploration and helps the algorithm to avoid overfitting to specific actions. This is done by applying a small amount of random noise to the target actions, leading to more robust learning.
Overall, TD3 has shown significant improvements in performance and stability over its predecessor, DDPG, making it a popular choice for various applications in robotics, gaming, and sistemas de control donde se involucran espacios de acción continuos de alta dimensión.