Update python code

Author: wgyhhhh

.history/README_20251108180336.md

@@ -0,0 +1,43 @@
+<div align="center">
+  <p><a href="https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Preface1/">📚 在线阅读地址</a></p>
+  <h3>🤖 《强化学习中的数学原理》-个人笔记与思考总结</h3>
+  <p><em>理解强化学习的数学原理，并通过实例深入掌握核心算法</em></p>
+</div>
+
+### &#8627; Stargazers
+[![Stargazers repo roster for @wgyhhhh/Mathematical-Foundations-of-Reinforcement-Learning-Notes](https://reporoster.com/stars/wgyhhhh/Mathematical-Foundations-of-Reinforcement-Learning-Notes)](https://github.com/wgyhhhh/Mathematical-Foundations-of-Reinforcement-Learning-Notes/stargazers)
+
+### &#8627; Forkers
+[![Forkers repo roster for @wgyhhhh/Mathematical-Foundations-of-Reinforcement-Learning-Notes](https://reporoster.com/forks/wgyhhhh/Mathematical-Foundations-of-Reinforcement-Learning-Notes)](https://github.com/wgyhhhh/Mathematical-Foundations-of-Reinforcement-Learning-Notes/network/members)
+
+## 🎯 笔记介绍
+
+&emsp;&emsp;本笔记是对赵世钰老师所著《强化学习中的数学原理》的个人思考与总结，**笔者将其做成了网页模式，方便大家随时随地在掌上设备阅读**。在此基础上，我还补充了对书中核心算法的实现，以便读者能获得更直观的理解。书中首先从基础概念入手，讲解Bellman公式和Bellman最优公式，接着扩展到基于模型（model-based）和无模型（model-free）的强化学习算法，最终推广到基于函数逼近的强化学习算法。若读者在强化学习方面没有背景知识，只需具备一定的线性代数和概率论基础即可阅读本书。而对于已有一些强化学习知识的读者，本笔记则可以帮助他们深入理解相关问题。
+
+## 📖 内容导航
+
+| 章节 | 关键内容 | 状态 |
+| --- | --- | --- |
+| [前言](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Preface1/) | 本笔记的缘起、背景及阅读建议 | ✅ |
+| [第一章 基本概念](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-1/intro/) | 强化学习的基本概念 | ✅ |
+| [第二章 状态值与贝尔曼方程](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-2/intro/) | 回报、状态值、Bellman方程 | ✅ |
+| [第三章 最优状态值与贝尔曼最优方程](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-3/intro/) | 最优状态值、最优策略、Bellman最优方程 | ✅ |
+| [第四章 值迭代与策略迭代](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-4/intro/) | 值迭代算法、策略迭代算法、截断策略迭代算法 | ✅ |
+| [第五章 蒙特卡罗方法](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-5/intro/) | MC Basic、MC Exploring Starts、MC-Greedy | ✅ |
+| [第六章 随机近似算法](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-6/intro/) | Robbins-Monro算法、Dvoretzky定理、随机梯度下降 | ✅ |
+| [第七章 时序差分算法](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-7/intro/) | Sarsa、n步Sarsa、Q-learning、 Off-policy、On-policy| ✅ |
+| [第八章 值函数方法](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-8/intro/) | 基于值函数的TD算法、Sarsa、Q-learning | ✅(润色中) |
+| [第九章 策略梯度方法](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-9/intro/) | 策略梯度、REINFORCE | ✅(润色中) |
+| [第十章 演员-评论家算法](https://wgyhhhh.github.io/Mathematical-Foundations-of-Reinforcement-Learning-Notes/Chapter-10/intro/) | 优势演员-评论家、异策略演员-评论家、确定性演员-评论家 | ✅(润色中) |
+| 算法实现详解 | 核心算法Python实现 | 🚧 |
+
+### 🚧 算法实现详解
+
+笔者正在使用Python实现本书中的部分核心算法，读者可以通过结合阅读，获得更直观的理解。
+
+## 🤝 如何贡献
+
+如果你对强化学习感兴趣，可以参与到该笔记的完善中！❤️
+
+- 💡**内容完善** - 帮助改进笔记内容
+- 📝**报告问题** - 发现问题请提交 Issue

.history/docs/python/examples/arguments_20251108184311.py

@@ -0,0 +1,55 @@
+__credits__ = ["Intelligent Unmanned Systems Laboratory at Westlake University."]
+'''
+Specify parameters of the env
+'''
+from typing import Union
+import numpy as np
+import argparse
+
+parser = argparse.ArgumentParser("Grid World Environment")
+
+## ==================== User settings ===================='''
+# specify the number of columns and rows of the grid world
+parser.add_argument("--env-size", type=Union[list, tuple, np.ndarray], default=(5,5) )   
+
+# specify the start state
+parser.add_argument("--start-state", type=Union[list, tuple, np.ndarray], default=(2,2))
+
+# specify the target state
+parser.add_argument("--target-state", type=Union[list, tuple, np.ndarray], default=(4,4))
+
+# sepcify the forbidden states
+parser.add_argument("--forbidden-states", type=list, default=[ (2, 1), (3, 3), (1, 3)] )
+
+# sepcify the reward when reaching target
+parser.add_argument("--reward-target", type=float, default = 10)
+
+# sepcify the reward when entering into forbidden area
+parser.add_argument("--reward-forbidden", type=float, default = -5)
+
+# sepcify the reward for each step
+parser.add_argument("--reward-step", type=float, default = -1)
+## ==================== End of User settings ====================
+
+
+## ==================== Advanced Settings ====================
+parser.add_argument("--action-space", type=list, default=[(0, 1), (1, 0), (0, -1), (-1, 0), (0, 0)] )  # down, right, up, left, stay           
+parser.add_argument("--debug", type=bool, default=False)
+parser.add_argument("--animation-interval", type=float, default = 0.2)
+## ==================== End of Advanced settings ====================
+
+
+args = parser.parse_args()     
+def validate_environment_parameters(env_size, start_state, target_state, forbidden_states):
+    if not (isinstance(env_size, tuple) or isinstance(env_size, list) or isinstance(env_size, np.ndarray)) and len(env_size) != 2:
+        raise ValueError("Invalid environment size. Expected a tuple (rows, cols) with positive dimensions.")
+    
+    for i in range(2):
+        assert start_state[i] < env_size[i]
+        assert target_state[i] < env_size[i]
+        for j in range(len(forbidden_states)):
+            assert forbidden_states[j][i] < env_size[i]
+try:
+    validate_environment_parameters(args.env_size, args.start_state, args.target_state, args.forbidden_states)
+except ValueError as e:
+    print("Error:", e)

.history/docs/python/examples/arguments_20251108184323.py

@@ -0,0 +1,51 @@
+from typing import Union
+import numpy as np
+import argparse
+
+parser = argparse.ArgumentParser("Grid World Environment")
+
+## ==================== User settings ===================='''
+# specify the number of columns and rows of the grid world
+parser.add_argument("--env-size", type=Union[list, tuple, np.ndarray], default=(5,5) )   
+
+# specify the start state
+parser.add_argument("--start-state", type=Union[list, tuple, np.ndarray], default=(2,2))
+
+# specify the target state
+parser.add_argument("--target-state", type=Union[list, tuple, np.ndarray], default=(4,4))
+
+# sepcify the forbidden states
+parser.add_argument("--forbidden-states", type=list, default=[ (2, 1), (3, 3), (1, 3)] )
+
+# sepcify the reward when reaching target
+parser.add_argument("--reward-target", type=float, default = 10)
+
+# sepcify the reward when entering into forbidden area
+parser.add_argument("--reward-forbidden", type=float, default = -5)
+
+# sepcify the reward for each step
+parser.add_argument("--reward-step", type=float, default = -1)
+## ==================== End of User settings ====================
+
+
+## ==================== Advanced Settings ====================
+parser.add_argument("--action-space", type=list, default=[(0, 1), (1, 0), (0, -1), (-1, 0), (0, 0)] )  # down, right, up, left, stay           
+parser.add_argument("--debug", type=bool, default=False)
+parser.add_argument("--animation-interval", type=float, default = 0.2)
+## ==================== End of Advanced settings ====================
+
+
+args = parser.parse_args()     
+def validate_environment_parameters(env_size, start_state, target_state, forbidden_states):
+    if not (isinstance(env_size, tuple) or isinstance(env_size, list) or isinstance(env_size, np.ndarray)) and len(env_size) != 2:
+        raise ValueError("Invalid environment size. Expected a tuple (rows, cols) with positive dimensions.")
+    
+    for i in range(2):
+        assert start_state[i] < env_size[i]
+        assert target_state[i] < env_size[i]
+        for j in range(len(forbidden_states)):
+            assert forbidden_states[j][i] < env_size[i]
+try:
+    validate_environment_parameters(args.env_size, args.start_state, args.target_state, args.forbidden_states)
+except ValueError as e:
+    print("Error:", e)