Value Functions and Bellman Equations

Value functions turn delayed reward into a prediction problem. They answer “how much future return should I expect if I start here?” or “how much future return should I expect if I take this action first?”

Defining Math

For a policy $\pi (a\mid s)$, the state-value function is

$$V^{\pi }(s)={\mathbb{E}}_{\pi }[G_t\mid S_t=s].$$

The action-value function is

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }[G_t\mid S_t=s,A_t=a].$$

The Bellman expectation equation rewrites value recursively:

$$V^{\pi }(s)=\sum _a\pi (a\mid s)\sum _{s^{\prime },r}P(s^{\prime },r\mid s,a)\left[r+\gamma V^{\pi }(s^{\prime })\right].$$

The value of a state is the expected immediate reward plus the discounted value of the next state.

Optimality

The optimal action-value function satisfies

$$Q^{\ast }(s,a)=\sum _{s^{\prime },r}P(s^{\prime },r\mid s,a)\left[r+\gamma \underset{a^{\prime }}{\max }{Q}^{\ast }(s^{\prime },a^{\prime })\right].$$

The ${\max }_{a^{\prime }}$ term says that after taking action $a$, the agent assumes optimal behavior from the next state onward. This is the target behind many value-based control algorithms.

Worked Calculation

Suppose a state has two actions. Action $a_1$ gives reward $1$ and moves to a next state with value $10$. Action $a_2$ gives reward $4$ and moves to a next state with value $3$. With $\gamma =0.8$:

$$Q(s,a_1)=1+0.8\cdot 10=9,Q(s,a_2)=4+0.8\cdot 3=\mathrm{6.4.}$$

Action $a_1$ has the smaller immediate reward but higher total value because it leads to a much better future state. That is the central point of value functions: they price delayed consequences.

Intuition

Bellman equations are consistency equations. A value estimate is good when the value assigned to the current state agrees with the observed reward and the value assigned to the next state. Temporal-difference learning uses this mismatch as a learning signal.

Caveats

Exact Bellman updates require known transition probabilities or enough samples to estimate them. Large state spaces replace tables with function approximators, which introduces approximation error, instability, and distribution-shift risk.

Connections

• Q-Learning and DQN use the Bellman optimality target for action values.
• Policy Gradients and Actor-Critic Methods use value functions as critics and baselines.
• Markov Decision Processes define the transition and reward process that Bellman equations summarize.

References

• Bellman, 1957, Dynamic Programming
• Sutton and Barto, 2018, Reinforcement Learning: An Introduction

Nav

Section — Reinforcement Learning

← Markov Decision Processes Temporal-Difference Learning →

Learning path — Reinforcement learning

← Markov Decision Processes Temporal-Difference Learning →