(Model) Value-guided task construals, after Ho et al. (2022)

Overview

@2022hoPeople introduce value-guided task construals to model the process of adaptively selecting simplified representations of cause-effect relationships during task planning. Intuitively, a construal “picks out” details in a task to consider.

The problem of selecting a task construal is formulated as an approximately optimal trade-off between cognitive cost and task performance, or behavioral utility. This gives a normative, resource-rational account of planning (@2023hoRational).

The key idea is to treat model and policy selection as a two-level optimization process: an outer loop selects a construal, or simplified model of cause-effect relationships, by optimizing the value of representation over task construals; this is then used by an inner loop planning algorithm to compute the optimal policy.

---

Preliminaries: MDP models of sequential decision-making

MDP model of sequential decision-making tasks

A task representation $\mathcal{T}$ consists of the following data: a state space $\mathcal{S}$ with initial state $s_0\in \mathcal{S}$; an action space $\mathcal{A}$; a transition function $P:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$; and a utility function $U:\mathcal{S}\to \mathbb{R}$. The value of a plan $\pi :S\times A\to [0,1]$ is defined for all states $s\in \mathcal{S}$ by the expected cumulative utility of using that plan:

$$V_{\pi }(s)=U(s)+\sum _a\pi (a|s)\sum _{s^{\prime }}P(s^{\prime }|s,a)V_{\pi }(s^{\prime }).$$

---

Value-guided task construals

Construal

Suppose an agent has $N$ primitive cause-effect relationships assigning probabilities to state, action, and next-state transitions

$${\varphi }_i:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1],i=1,\dots ,N,$$

where each ${\varphi }_i(s’|s,a)$ is a potential function representing the local effect of taking some action. A construal $c\subseteq \{{\varphi }_1,\dots ,{\varphi }_N\}$ is a subset of primitive cause-effect relationships that produces a task construal ${\mathcal{T}}_c$ that shares the same state space, action space, and utility function with $\mathcal{T}$, but has a construed transition function

$$P_c(s^{\prime }|s,a)\propto \prod _{{\varphi }_i\in c}{\varphi }_i(s^{\prime }|s,a).$$

Behavioral utility, value of representation

Given a decision-maker with task construal ${\mathcal{T}}_c$, the behavioral utility of the computed optimal plan ${\pi }_c$ when starting at state $s_0$ is defined as its performance when interacting with actual transition dynamics $P$:

$$U({\pi }_c)=U(s_0)+\sum _a{\pi }_c(a|s_0)\sum _{s^{\prime }}P(s^{\prime }|s_0,a)V_{{\pi }_c}(s^{\prime }).$$

The value of representation for the construal $c$ is

$$\text{VOR}(c)=U({\pi }_c)-|c|,$$

where $|c|$ is the cognitive cost defined as the cardinality of $c$.

---

Model implementation

• Given a value of representation function $\text{VOR}$ that assigns a value to each construal, decision-makers are modeled as selecting a construal according to a softmax decision rule: $P(c)\propto \exp ({\alpha }^{-1}\text{VOR}(c),$ where $\alpha >0$ is temperature parameter.
• The process of revisiting and modifying construals at each stage of planning is represented as a sequential decision-making problem, the construal modification Markov decision process.

Construal modification Markov decision process

Given a set of $N$ cause-effect relationships, let $\mathcal{C}=\mathcal{P}(\{{\varphi }_1,\dots ,{\varphi }_N\})$ be the set of all possible construals (i.e., the powerset of cause-effect relationships). The construal modification Markov decision process has state space $\mathcal{S}\times \mathcal{C}$ and an action space corresponding to possible next construals $c’\in \mathcal{C}$. After selecting a new construal $c’$, the probability of transitioning from task state $s$ to $s’$ is given by

$$P(s^{\prime }|s,c^{\prime })=\sum _a{\pi }_{c^{\prime }}(a|s)P(s^{\prime }|s,a);$$

that is, first calculating a joint distribution using the actual transition function $P(s’|s,a)$ and plan ${\pi }_{c’}(a|s)$, then marginalizing over task actions $a$.

The optimal construal modification value function is defined for all $s\in \mathcal{S}$ and $c\in \mathcal{C}$ by

$$V(s,c)=U(s)+\underset{c^{\prime }}{\max }\left\{\sum _{s^{\prime }}P(s^{\prime }|s,c^{\prime })V(s^{\prime },c^{\prime })-|c^{\prime }-c|\right\},$$

where $|c’-c|$ is the number of additional cause-effect relationships in $c’$ compared to $c$.