RL表格型方法求解——Sarsa
参考来自百度强化学习课程
0. Sarsa 简介
Sarsa
全称是state-action-reward-state'-action'
,目的是学习特定的state
下,特定action
的价值Q
,最终建立和优化一个Q
表格,以state
为行,action
为列,根据与环境交互得到的reward
来更新Q
表格,更新公式为:
Sarsa
在训练中为了更好的探索环境,采用ε-greedy
方式来训练,有一定概率随机选择动作输出。
1. 安装环境
gym是一个openai的强化学习可视化环境库,拥有一些基本的场景,可调用交互。
1 | !pip install gym --user |
Requirement already satisfied: gym in /home/jupyter-sysustudent-1904/.local/lib/python3.6/site-packages (0.17.3)
Requirement already satisfied: numpy>=1.10.4 in /opt/tljh/user/lib/python3.6/site-packages (from gym) (1.16.2)
Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /opt/tljh/user/lib/python3.6/site-packages (from gym) (1.4.1)
Requirement already satisfied: scipy in /opt/tljh/user/lib/python3.6/site-packages (from gym) (1.3.2)
Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /home/jupyter-sysustudent-1904/.local/lib/python3.6/site-packages (from gym) (1.5.0)
Requirement already satisfied: future in /opt/tljh/user/lib/python3.6/site-packages (from pyglet<=1.5.0,>=1.4.0->gym) (0.18.2)
2. 导入函数包
1 | import gym |
3. 写Agent类
Agent
是智能体predict()
方法:对输入observation输出actionsample()
方法:在predict()
的基础上使用$\epsilon$-greedy算法增加探索learn()
方法:输入训练数据(sarsa),完成Q表格的更新
1 | # |
4. 训练和测试
run_episode()
:agent在一个episode中训练的过程,使用agent.sample()与环境交互,使用agent.learn()训练Q表格。
test_episode()
:agent在一个episode中测试效果,评估目前的agent能在一个episode中拿到多少总reward。
1 | def run_episode(env,agent,render=False): |
5. 创建环境和Agent,开始训练
1 | env = gym.make('CliffWalking-v0') #0 up, 1 right, 2 down, 3 left |
Episode 0: steps = 834, reward = -2319.0
Episode 1: steps = 400, reward = -598.0
Episode 2: steps = 270, reward = -369.0
Episode 3: steps = 369, reward = -567.0
Episode 4: steps = 259, reward = -358.0
Episode 5: steps = 114, reward = -114.0
Episode 6: steps = 109, reward = -109.0
Episode 7: steps = 142, reward = -142.0
Episode 8: steps = 148, reward = -148.0
Episode 9: steps = 77, reward = -77.0
Episode 10: steps = 110, reward = -209.0
Episode 11: steps = 53, reward = -53.0
Episode 12: steps = 105, reward = -204.0
Episode 13: steps = 107, reward = -107.0
Episode 14: steps = 88, reward = -88.0
Episode 15: steps = 204, reward = -402.0
Episode 16: steps = 109, reward = -208.0
Episode 17: steps = 64, reward = -64.0
Episode 18: steps = 105, reward = -105.0
Episode 19: steps = 104, reward = -104.0
Episode 20: steps = 71, reward = -170.0
Episode 21: steps = 133, reward = -133.0
Episode 22: steps = 75, reward = -75.0
Episode 23: steps = 92, reward = -92.0
Episode 24: steps = 138, reward = -138.0
Episode 25: steps = 49, reward = -49.0
Episode 26: steps = 119, reward = -218.0
Episode 27: steps = 139, reward = -238.0
Episode 28: steps = 80, reward = -179.0
Episode 29: steps = 91, reward = -91.0
Episode 30: steps = 49, reward = -49.0
Episode 31: steps = 33, reward = -33.0
Episode 32: steps = 120, reward = -120.0
Episode 33: steps = 100, reward = -199.0
Episode 34: steps = 50, reward = -50.0
Episode 35: steps = 84, reward = -84.0
Episode 36: steps = 112, reward = -211.0
Episode 37: steps = 57, reward = -57.0
Episode 38: steps = 119, reward = -218.0
Episode 39: steps = 91, reward = -91.0
Episode 40: steps = 97, reward = -97.0
Episode 41: steps = 53, reward = -53.0
Episode 42: steps = 70, reward = -70.0
Episode 43: steps = 41, reward = -41.0
Episode 44: steps = 116, reward = -215.0
Episode 45: steps = 74, reward = -173.0
Episode 46: steps = 69, reward = -69.0
Episode 47: steps = 73, reward = -172.0
Episode 48: steps = 81, reward = -81.0
Episode 49: steps = 58, reward = -58.0
Episode 50: steps = 67, reward = -67.0
Episode 51: steps = 37, reward = -37.0
Episode 52: steps = 83, reward = -83.0
Episode 53: steps = 31, reward = -31.0
Episode 54: steps = 54, reward = -54.0
Episode 55: steps = 50, reward = -50.0
Episode 56: steps = 86, reward = -185.0
Episode 57: steps = 59, reward = -59.0
Episode 58: steps = 37, reward = -136.0
Episode 59: steps = 46, reward = -46.0
Episode 60: steps = 78, reward = -78.0
Episode 61: steps = 50, reward = -50.0
Episode 62: steps = 58, reward = -58.0
Episode 63: steps = 33, reward = -33.0
Episode 64: steps = 48, reward = -48.0
Episode 65: steps = 85, reward = -85.0
Episode 66: steps = 146, reward = -542.0
Episode 67: steps = 35, reward = -35.0
Episode 68: steps = 29, reward = -29.0
Episode 69: steps = 53, reward = -53.0
Episode 70: steps = 43, reward = -43.0
Episode 71: steps = 48, reward = -48.0
Episode 72: steps = 57, reward = -156.0
Episode 73: steps = 52, reward = -52.0
Episode 74: steps = 25, reward = -25.0
Episode 75: steps = 88, reward = -187.0
Episode 76: steps = 53, reward = -53.0
Episode 77: steps = 33, reward = -33.0
Episode 78: steps = 67, reward = -67.0
Episode 79: steps = 52, reward = -52.0
Episode 80: steps = 53, reward = -53.0
Episode 81: steps = 23, reward = -23.0
Episode 82: steps = 37, reward = -37.0
Episode 83: steps = 41, reward = -41.0
Episode 84: steps = 47, reward = -47.0
Episode 85: steps = 37, reward = -37.0
Episode 86: steps = 23, reward = -23.0
Episode 87: steps = 43, reward = -43.0
Episode 88: steps = 42, reward = -42.0
Episode 89: steps = 27, reward = -27.0
Episode 90: steps = 47, reward = -47.0
Episode 91: steps = 41, reward = -41.0
Episode 92: steps = 47, reward = -47.0
Episode 93: steps = 56, reward = -56.0
Episode 94: steps = 30, reward = -30.0
Episode 95: steps = 47, reward = -47.0
Episode 96: steps = 21, reward = -21.0
Episode 97: steps = 95, reward = -293.0
Episode 98: steps = 51, reward = -150.0
Episode 99: steps = 31, reward = -31.0
Episode 100: steps = 37, reward = -37.0
Episode 101: steps = 49, reward = -49.0
Episode 102: steps = 56, reward = -155.0
Episode 103: steps = 41, reward = -41.0
Episode 104: steps = 41, reward = -41.0
Episode 105: steps = 47, reward = -47.0
Episode 106: steps = 35, reward = -35.0
Episode 107: steps = 40, reward = -40.0
Episode 108: steps = 47, reward = -47.0
Episode 109: steps = 118, reward = -118.0
Episode 110: steps = 23, reward = -23.0
Episode 111: steps = 21, reward = -21.0
Episode 112: steps = 37, reward = -136.0
Episode 113: steps = 47, reward = -47.0
Episode 114: steps = 51, reward = -51.0
Episode 115: steps = 43, reward = -43.0
Episode 116: steps = 41, reward = -41.0
Episode 117: steps = 46, reward = -46.0
Episode 118: steps = 36, reward = -36.0
Episode 119: steps = 23, reward = -23.0
Episode 120: steps = 27, reward = -27.0
Episode 121: steps = 27, reward = -27.0
Episode 122: steps = 51, reward = -51.0
Episode 123: steps = 32, reward = -32.0
Episode 124: steps = 42, reward = -42.0
Episode 125: steps = 27, reward = -27.0
Episode 126: steps = 23, reward = -23.0
Episode 127: steps = 43, reward = -43.0
Episode 128: steps = 25, reward = -25.0
Episode 129: steps = 49, reward = -49.0
Episode 130: steps = 29, reward = -29.0
Episode 131: steps = 44, reward = -143.0
Episode 132: steps = 39, reward = -39.0
Episode 133: steps = 35, reward = -35.0
Episode 134: steps = 21, reward = -21.0
Episode 135: steps = 34, reward = -34.0
Episode 136: steps = 31, reward = -31.0
Episode 137: steps = 61, reward = -160.0
Episode 138: steps = 42, reward = -240.0
Episode 139: steps = 35, reward = -35.0
Episode 140: steps = 36, reward = -36.0
Episode 141: steps = 27, reward = -27.0
Episode 142: steps = 50, reward = -149.0
Episode 143: steps = 23, reward = -23.0
Episode 144: steps = 41, reward = -41.0
Episode 145: steps = 44, reward = -44.0
Episode 146: steps = 26, reward = -26.0
Episode 147: steps = 31, reward = -31.0
Episode 148: steps = 28, reward = -28.0
Episode 149: steps = 27, reward = -27.0
Episode 150: steps = 47, reward = -47.0
Episode 151: steps = 27, reward = -27.0
Episode 152: steps = 35, reward = -35.0
Episode 153: steps = 23, reward = -23.0
Episode 154: steps = 49, reward = -49.0
Episode 155: steps = 26, reward = -26.0
Episode 156: steps = 37, reward = -37.0
Episode 157: steps = 39, reward = -39.0
Episode 158: steps = 27, reward = -27.0
Episode 159: steps = 49, reward = -49.0
Episode 160: steps = 29, reward = -29.0
Episode 161: steps = 28, reward = -28.0
Episode 162: steps = 27, reward = -27.0
Episode 163: steps = 49, reward = -49.0
Episode 164: steps = 23, reward = -23.0
Episode 165: steps = 20, reward = -20.0
Episode 166: steps = 28, reward = -28.0
Episode 167: steps = 36, reward = -36.0
Episode 168: steps = 19, reward = -19.0
Episode 169: steps = 27, reward = -27.0
Episode 170: steps = 24, reward = -24.0
Episode 171: steps = 33, reward = -33.0
Episode 172: steps = 19, reward = -19.0
Episode 173: steps = 19, reward = -19.0
Episode 174: steps = 29, reward = -29.0
Episode 175: steps = 29, reward = -29.0
Episode 176: steps = 26, reward = -26.0
Episode 177: steps = 40, reward = -40.0
Episode 178: steps = 31, reward = -31.0
Episode 179: steps = 25, reward = -25.0
Episode 180: steps = 19, reward = -19.0
Episode 181: steps = 36, reward = -135.0
Episode 182: steps = 24, reward = -24.0
Episode 183: steps = 15, reward = -15.0
Episode 184: steps = 22, reward = -22.0
Episode 185: steps = 29, reward = -128.0
Episode 186: steps = 29, reward = -29.0
Episode 187: steps = 17, reward = -17.0
Episode 188: steps = 23, reward = -23.0
Episode 189: steps = 23, reward = -122.0
Episode 190: steps = 17, reward = -17.0
Episode 191: steps = 16, reward = -16.0
Episode 192: steps = 29, reward = -29.0
Episode 193: steps = 25, reward = -25.0
Episode 194: steps = 21, reward = -21.0
Episode 195: steps = 17, reward = -17.0
Episode 196: steps = 19, reward = -19.0
Episode 197: steps = 23, reward = -23.0
Episode 198: steps = 38, reward = -38.0
Episode 199: steps = 24, reward = -24.0
Episode 200: steps = 20, reward = -20.0
Episode 201: steps = 25, reward = -25.0
Episode 202: steps = 25, reward = -25.0
Episode 203: steps = 25, reward = -25.0
Episode 204: steps = 21, reward = -21.0
Episode 205: steps = 34, reward = -34.0
Episode 206: steps = 17, reward = -17.0
Episode 207: steps = 23, reward = -23.0
Episode 208: steps = 32, reward = -32.0
Episode 209: steps = 21, reward = -21.0
Episode 210: steps = 27, reward = -27.0
Episode 211: steps = 23, reward = -23.0
Episode 212: steps = 39, reward = -39.0
Episode 213: steps = 21, reward = -21.0
Episode 214: steps = 27, reward = -27.0
Episode 215: steps = 24, reward = -123.0
Episode 216: steps = 31, reward = -31.0
Episode 217: steps = 31, reward = -31.0
Episode 218: steps = 27, reward = -27.0
Episode 219: steps = 21, reward = -21.0
Episode 220: steps = 20, reward = -119.0
Episode 221: steps = 24, reward = -24.0
Episode 222: steps = 20, reward = -20.0
Episode 223: steps = 23, reward = -23.0
Episode 224: steps = 23, reward = -23.0
Episode 225: steps = 21, reward = -21.0
Episode 226: steps = 34, reward = -34.0
Episode 227: steps = 19, reward = -19.0
Episode 228: steps = 23, reward = -23.0
Episode 229: steps = 33, reward = -33.0
Episode 230: steps = 19, reward = -19.0
Episode 231: steps = 32, reward = -32.0
Episode 232: steps = 19, reward = -19.0
Episode 233: steps = 23, reward = -23.0
Episode 234: steps = 19, reward = -19.0
Episode 235: steps = 21, reward = -21.0
Episode 236: steps = 21, reward = -21.0
Episode 237: steps = 20, reward = -20.0
Episode 238: steps = 19, reward = -19.0
Episode 239: steps = 17, reward = -17.0
Episode 240: steps = 17, reward = -17.0
Episode 241: steps = 22, reward = -22.0
Episode 242: steps = 17, reward = -17.0
Episode 243: steps = 32, reward = -131.0
Episode 244: steps = 17, reward = -17.0
Episode 245: steps = 21, reward = -21.0
Episode 246: steps = 31, reward = -31.0
Episode 247: steps = 21, reward = -21.0
Episode 248: steps = 25, reward = -25.0
Episode 249: steps = 17, reward = -17.0
Episode 250: steps = 26, reward = -125.0
Episode 251: steps = 17, reward = -17.0
Episode 252: steps = 32, reward = -32.0
Episode 253: steps = 18, reward = -18.0
Episode 254: steps = 19, reward = -19.0
Episode 255: steps = 20, reward = -20.0
Episode 256: steps = 20, reward = -20.0
Episode 257: steps = 19, reward = -19.0
Episode 258: steps = 17, reward = -17.0
Episode 259: steps = 32, reward = -32.0
Episode 260: steps = 19, reward = -19.0
Episode 261: steps = 19, reward = -19.0
Episode 262: steps = 38, reward = -38.0
Episode 263: steps = 19, reward = -19.0
Episode 264: steps = 17, reward = -17.0
Episode 265: steps = 21, reward = -21.0
Episode 266: steps = 26, reward = -26.0
Episode 267: steps = 17, reward = -17.0
Episode 268: steps = 31, reward = -31.0
Episode 269: steps = 17, reward = -17.0
Episode 270: steps = 22, reward = -22.0
Episode 271: steps = 27, reward = -27.0
Episode 272: steps = 25, reward = -25.0
Episode 273: steps = 18, reward = -18.0
Episode 274: steps = 29, reward = -29.0
Episode 275: steps = 27, reward = -27.0
Episode 276: steps = 31, reward = -31.0
Episode 277: steps = 20, reward = -20.0
Episode 278: steps = 31, reward = -31.0
Episode 279: steps = 38, reward = -38.0
Episode 280: steps = 27, reward = -27.0
Episode 281: steps = 35, reward = -35.0
Episode 282: steps = 42, reward = -240.0
Episode 283: steps = 18, reward = -18.0
Episode 284: steps = 25, reward = -25.0
Episode 285: steps = 17, reward = -17.0
Episode 286: steps = 29, reward = -29.0
Episode 287: steps = 19, reward = -19.0
Episode 288: steps = 21, reward = -21.0
Episode 289: steps = 19, reward = -19.0
Episode 290: steps = 23, reward = -23.0
Episode 291: steps = 19, reward = -19.0
Episode 292: steps = 18, reward = -18.0
Episode 293: steps = 19, reward = -19.0
Episode 294: steps = 30, reward = -30.0
Episode 295: steps = 20, reward = -20.0
Episode 296: steps = 17, reward = -17.0
Episode 297: steps = 19, reward = -19.0
Episode 298: steps = 22, reward = -22.0
Episode 299: steps = 23, reward = -23.0
Episode 300: steps = 15, reward = -15.0
Episode 301: steps = 23, reward = -23.0
Episode 302: steps = 19, reward = -19.0
Episode 303: steps = 19, reward = -19.0
Episode 304: steps = 21, reward = -21.0
Episode 305: steps = 23, reward = -23.0
Episode 306: steps = 21, reward = -21.0
Episode 307: steps = 27, reward = -126.0
Episode 308: steps = 20, reward = -20.0
Episode 309: steps = 17, reward = -17.0
Episode 310: steps = 22, reward = -22.0
Episode 311: steps = 19, reward = -19.0
Episode 312: steps = 17, reward = -17.0
Episode 313: steps = 23, reward = -23.0
Episode 314: steps = 20, reward = -20.0
Episode 315: steps = 22, reward = -22.0
Episode 316: steps = 24, reward = -24.0
Episode 317: steps = 23, reward = -23.0
Episode 318: steps = 19, reward = -19.0
Episode 319: steps = 23, reward = -23.0
Episode 320: steps = 17, reward = -17.0
Episode 321: steps = 26, reward = -26.0
Episode 322: steps = 30, reward = -30.0
Episode 323: steps = 21, reward = -21.0
Episode 324: steps = 21, reward = -21.0
Episode 325: steps = 19, reward = -19.0
Episode 326: steps = 19, reward = -19.0
Episode 327: steps = 17, reward = -17.0
Episode 328: steps = 17, reward = -17.0
Episode 329: steps = 21, reward = -21.0
Episode 330: steps = 28, reward = -28.0
Episode 331: steps = 23, reward = -23.0
Episode 332: steps = 19, reward = -19.0
Episode 333: steps = 23, reward = -23.0
Episode 334: steps = 17, reward = -17.0
Episode 335: steps = 20, reward = -20.0
Episode 336: steps = 41, reward = -140.0
Episode 337: steps = 17, reward = -17.0
Episode 338: steps = 22, reward = -22.0
Episode 339: steps = 18, reward = -18.0
Episode 340: steps = 23, reward = -23.0
Episode 341: steps = 18, reward = -18.0
Episode 342: steps = 23, reward = -23.0
Episode 343: steps = 23, reward = -23.0
Episode 344: steps = 20, reward = -20.0
Episode 345: steps = 15, reward = -15.0
Episode 346: steps = 15, reward = -15.0
Episode 347: steps = 17, reward = -17.0
Episode 348: steps = 23, reward = -23.0
Episode 349: steps = 21, reward = -21.0
Episode 350: steps = 16, reward = -115.0
Episode 351: steps = 20, reward = -20.0
Episode 352: steps = 17, reward = -17.0
Episode 353: steps = 15, reward = -15.0
Episode 354: steps = 15, reward = -15.0
Episode 355: steps = 17, reward = -17.0
Episode 356: steps = 17, reward = -17.0
Episode 357: steps = 15, reward = -15.0
Episode 358: steps = 17, reward = -17.0
Episode 359: steps = 19, reward = -19.0
Episode 360: steps = 19, reward = -19.0
Episode 361: steps = 15, reward = -15.0
Episode 362: steps = 17, reward = -17.0
Episode 363: steps = 17, reward = -17.0
Episode 364: steps = 17, reward = -17.0
Episode 365: steps = 25, reward = -25.0
Episode 366: steps = 19, reward = -19.0
Episode 367: steps = 20, reward = -20.0
Episode 368: steps = 19, reward = -19.0
Episode 369: steps = 19, reward = -19.0
Episode 370: steps = 25, reward = -25.0
Episode 371: steps = 19, reward = -19.0
Episode 372: steps = 20, reward = -20.0
Episode 373: steps = 22, reward = -22.0
Episode 374: steps = 17, reward = -17.0
Episode 375: steps = 23, reward = -23.0
Episode 376: steps = 17, reward = -17.0
Episode 377: steps = 17, reward = -17.0
Episode 378: steps = 19, reward = -19.0
Episode 379: steps = 21, reward = -21.0
Episode 380: steps = 19, reward = -118.0
Episode 381: steps = 17, reward = -17.0
Episode 382: steps = 19, reward = -19.0
Episode 383: steps = 21, reward = -21.0
Episode 384: steps = 17, reward = -17.0
Episode 385: steps = 19, reward = -19.0
Episode 386: steps = 21, reward = -21.0
Episode 387: steps = 21, reward = -120.0
Episode 388: steps = 19, reward = -19.0
Episode 389: steps = 17, reward = -17.0
Episode 390: steps = 19, reward = -118.0
Episode 391: steps = 19, reward = -19.0
Episode 392: steps = 20, reward = -20.0
Episode 393: steps = 19, reward = -19.0
Episode 394: steps = 17, reward = -17.0
Episode 395: steps = 17, reward = -17.0
Episode 396: steps = 19, reward = -19.0
Episode 397: steps = 22, reward = -22.0
Episode 398: steps = 17, reward = -17.0
Episode 399: steps = 18, reward = -18.0
Episode 400: steps = 20, reward = -20.0
Episode 401: steps = 19, reward = -19.0
Episode 402: steps = 21, reward = -21.0
Episode 403: steps = 20, reward = -20.0
Episode 404: steps = 19, reward = -19.0
Episode 405: steps = 19, reward = -19.0
Episode 406: steps = 41, reward = -140.0
Episode 407: steps = 37, reward = -37.0
Episode 408: steps = 19, reward = -19.0
Episode 409: steps = 23, reward = -23.0
Episode 410: steps = 15, reward = -15.0
Episode 411: steps = 17, reward = -17.0
Episode 412: steps = 23, reward = -23.0
Episode 413: steps = 15, reward = -15.0
Episode 414: steps = 15, reward = -15.0
Episode 415: steps = 15, reward = -15.0
Episode 416: steps = 18, reward = -117.0
Episode 417: steps = 15, reward = -15.0
Episode 418: steps = 19, reward = -19.0
Episode 419: steps = 17, reward = -17.0
Episode 420: steps = 17, reward = -17.0
Episode 421: steps = 17, reward = -17.0
Episode 422: steps = 15, reward = -15.0
Episode 423: steps = 19, reward = -19.0
Episode 424: steps = 17, reward = -17.0
Episode 425: steps = 29, reward = -128.0
Episode 426: steps = 17, reward = -17.0
Episode 427: steps = 17, reward = -17.0
Episode 428: steps = 17, reward = -17.0
Episode 429: steps = 19, reward = -19.0
Episode 430: steps = 17, reward = -17.0
Episode 431: steps = 22, reward = -22.0
Episode 432: steps = 20, reward = -20.0
Episode 433: steps = 17, reward = -17.0
Episode 434: steps = 19, reward = -19.0
Episode 435: steps = 17, reward = -17.0
Episode 436: steps = 17, reward = -17.0
Episode 437: steps = 17, reward = -17.0
Episode 438: steps = 19, reward = -19.0
Episode 439: steps = 26, reward = -26.0
Episode 440: steps = 20, reward = -20.0
Episode 441: steps = 18, reward = -18.0
Episode 442: steps = 22, reward = -22.0
Episode 443: steps = 20, reward = -20.0
Episode 444: steps = 19, reward = -19.0
Episode 445: steps = 26, reward = -26.0
Episode 446: steps = 20, reward = -20.0
Episode 447: steps = 21, reward = -21.0
Episode 448: steps = 18, reward = -18.0
Episode 449: steps = 19, reward = -19.0
Episode 450: steps = 17, reward = -17.0
Episode 451: steps = 21, reward = -21.0
Episode 452: steps = 17, reward = -17.0
Episode 453: steps = 19, reward = -19.0
Episode 454: steps = 19, reward = -19.0
Episode 455: steps = 20, reward = -20.0
Episode 456: steps = 17, reward = -17.0
Episode 457: steps = 16, reward = -16.0
Episode 458: steps = 19, reward = -19.0
Episode 459: steps = 17, reward = -17.0
Episode 460: steps = 18, reward = -18.0
Episode 461: steps = 15, reward = -15.0
Episode 462: steps = 15, reward = -15.0
Episode 463: steps = 15, reward = -15.0
Episode 464: steps = 15, reward = -15.0
Episode 465: steps = 19, reward = -19.0
Episode 466: steps = 19, reward = -19.0
Episode 467: steps = 20, reward = -20.0
Episode 468: steps = 17, reward = -17.0
Episode 469: steps = 15, reward = -15.0
Episode 470: steps = 19, reward = -19.0
Episode 471: steps = 19, reward = -19.0
Episode 472: steps = 17, reward = -17.0
Episode 473: steps = 17, reward = -17.0
Episode 474: steps = 24, reward = -24.0
Episode 475: steps = 20, reward = -20.0
Episode 476: steps = 25, reward = -25.0
Episode 477: steps = 19, reward = -19.0
Episode 478: steps = 17, reward = -17.0
Episode 479: steps = 17, reward = -17.0
Episode 480: steps = 19, reward = -19.0
Episode 481: steps = 21, reward = -21.0
Episode 482: steps = 17, reward = -17.0
Episode 483: steps = 18, reward = -18.0
Episode 484: steps = 18, reward = -18.0
Episode 485: steps = 24, reward = -24.0
Episode 486: steps = 18, reward = -18.0
Episode 487: steps = 17, reward = -17.0
Episode 488: steps = 21, reward = -21.0
Episode 489: steps = 18, reward = -18.0
Episode 490: steps = 18, reward = -18.0
Episode 491: steps = 17, reward = -17.0
Episode 492: steps = 19, reward = -19.0
Episode 493: steps = 17, reward = -17.0
Episode 494: steps = 24, reward = -24.0
Episode 495: steps = 17, reward = -17.0
Episode 496: steps = 21, reward = -21.0
Episode 497: steps = 17, reward = -17.0
Episode 498: steps = 19, reward = -19.0
Episode 499: steps = 26, reward = -26.0
test reward = -17.0
可以看到最终测试结果不是最优策略,实际上是走远离悬崖边的一条路径,故非最优。