RL-赵-(九)-Policy函数拟合算法-Policy Gradient算法02-2：目标函数/metrics的选取02【average one-step reward】

2、average one-step reward2种常见表达形式：rˉπ≐∑s∈Sdπ(s)rπ(s)≐lim⁡n→∞1nE[∑k=1nRt+k]\begin{aligned} \bar{r}_{\pi}\doteq\sum_{s\in\mathcal{S}}d_{\pi}\left(s\right)r_{\pi}\left(s\right) \\[4ex] \doteq\lim_{n\to\infty}\frac{1}{n}\mathbb{E}\left[\sum_{k=1}^{n}R_{t+k}\right] \\[4ex] \end{aligned}rˉπ​​≐s∈S∑​dπ​(s)rπ​(s)≐n→∞lim​n1​E[k=1∑n​Rt+k​]​第二大类是average one-step reward, 或者简称为 average reward.具体地′_{\prime}′​the metrics是rˉπ≐∑s∈Sdπ(s)rπ(s)=E[rπ(S)]\color{red}{ \bar{r}_{\pi}\doteq\sum_{s\in\mathcal{S}}d_{\pi}\left(s\right)r_{\pi}\left(s\right)=\mathbb{E}[r_{\pi}\left(S\right)]}rˉπ​≐