pomdp_problems.tiger package

pomdp_problems.tiger.tiger_problem module

The classic Tiger problem.

This is a POMDP problem; Namely, it specifies both the POMDP (i.e. state, action, observation space) and the T/O/R for the agent as well as the environment.

The description of the tiger problem is as follows: (Quote from POMDP: Introduction to Partially Observable Markov Decision Processes by Kamalzadeh and Hahsler )

A tiger is put with equal probability behind one of two doors, while treasure is put behind the other one. You are standing in front of the two closed doors and need to decide which one to open. If you open the door with the tiger, you will get hurt (negative reward). But if you open the door with treasure, you receive a positive reward. Instead of opening a door right away, you also have the option to wait and listen for tiger noises. But listening is neither free nor entirely accurate. You might hear the tiger behind the left door while it is actually behind the right door and vice versa.

States: tiger-left, tiger-right Actions: open-left, open-right, listen Rewards:

+10 for opening treasure door. -100 for opening tiger door. -1 for listening.

Observations: You can hear either “tiger-left”, or “tiger-right”.

Note that in this example, the TigerProblem is a POMDP that also contains the agent and the environment as its fields. In general this doesn’t need to be the case. (Refer to more complicated examples.)

pomdp_problems.tiger.tiger_problem.build_states(strings)[source]
pomdp_problems.tiger.tiger_problem.build_actions(strings)[source]
pomdp_problems.tiger.tiger_problem.build_observations(strings)[source]
class pomdp_problems.tiger.tiger_problem.State(name)[source]

Bases: pomdp_py.framework.basics.State

class pomdp_problems.tiger.tiger_problem.Action(name)[source]

Bases: pomdp_py.framework.basics.Action

class pomdp_problems.tiger.tiger_problem.Observation(name)[source]

Bases: pomdp_py.framework.basics.Observation

class pomdp_problems.tiger.tiger_problem.ObservationModel(probs)[source]

Bases: pomdp_py.framework.basics.ObservationModel

This problem is small enough for the probabilities to be directly given externally

probability(self, observation, next_state, action, **kwargs)[source]

Returns the probability of \(\Pr(o|s',a)\).

Parameters
  • observation (Observation) – the observation \(o\)

  • next_state (State) – the next state \(s'\)

  • action (Action) – the action \(a\)

Returns

the probability \(\Pr(o|s',a)\)

Return type

float

sample(self, next_state, action, **kwargs)[source]

Returns observation randomly sampled according to the distribution of this observation model.

Parameters
  • next_state (State) – the next state \(s'\)

  • action (Action) – the action \(a\)

Returns

the observation \(o\)

Return type

Observation

argmax(next_state, action, normalized=False, **kwargs)[source]

Returns the most likely observation

get_distribution(next_state, action, **kwargs)[source]

Returns the underlying distribution of the model; In this case, it’s just a histogram

get_all_observations(self)[source]

Returns a set of all possible observations, if feasible.

class pomdp_problems.tiger.tiger_problem.TransitionModel(probs)[source]

Bases: pomdp_py.framework.basics.TransitionModel

This problem is small enough for the probabilities to be directly given externally

probability(self, next_state, state, action, **kwargs)[source]

Returns the probability of \(\Pr(s'|s,a)\).

Parameters
  • state (State) – the state \(s\)

  • next_state (State) – the next state \(s'\)

  • action (Action) – the action \(a\)

Returns

the probability \(\Pr(s'|s,a)\)

Return type

float

sample(self, state, action, **kwargs)[source]

Returns next state randomly sampled according to the distribution of this transition model.

Parameters
  • state (State) – the next state \(s\)

  • action (Action) – the action \(a\)

Returns

the next state \(s'\)

Return type

State

argmax(state, action, normalized=False, **kwargs)[source]

Returns the most likely next state

get_distribution(state, action, **kwargs)[source]

Returns the underlying distribution of the model

get_all_states(self)[source]

Returns a set of all possible states, if feasible.

class pomdp_problems.tiger.tiger_problem.RewardModel(scale=1)[source]

Bases: pomdp_py.framework.basics.RewardModel

probability(self, reward, state, action, next_state, **kwargs)[source]

Returns the probability of \(\Pr(r|s,a,s')\).

Parameters
  • reward (float) – the reward \(r\)

  • state (State) – the state \(s\)

  • action (Action) – the action \(a\)

  • next_state (State) – the next state \(s'\)

Returns

the probability \(\Pr(r|s,a,s')\)

Return type

float

sample(self, state, action, next_state, **kwargs)[source]

Returns reward randomly sampled according to the distribution of this reward model.

Parameters
  • state (State) – the next state \(s\)

  • action (Action) – the action \(a\)

  • next_state (State) – the next state \(s'\)

Returns

the reward \(r\)

Return type

float

argmax(state, action, next_state, normalized=False, **kwargs)[source]

Returns the most likely reward

get_distribution(state, action, next_state, **kwargs)[source]

Returns the underlying distribution of the model

class pomdp_problems.tiger.tiger_problem.PolicyModel[source]

Bases: pomdp_py.algorithms.po_uct.RandomRollout

This is an extremely dumb policy model; To keep consistent with the framework.

probability(self, action, state, **kwargs)[source]

Returns the probability of \(\pi(a|s)\).

Parameters
  • action (Action) – the action \(a\)

  • state (State) – the state \(s\)

Returns

the probability \(\pi(a|s)\)

Return type

float

sample(self, state, **kwargs)[source]

Returns action randomly sampled according to the distribution of this policy model.

Parameters

state (State) – the next state \(s\)

Returns

the action \(a\)

Return type

Action

argmax(state, normalized=False, **kwargs)[source]

Returns the most likely reward

get_all_actions(self, *args, **kwargs)[source]

Returns a set of all possible actions, if feasible.

class pomdp_problems.tiger.tiger_problem.TigerProblem(obs_probs, trans_probs, init_true_state, init_belief)[source]

Bases: pomdp_py.framework.basics.POMDP

STATES = {State(tiger-right), State(tiger-left)}
ACTIONS = {Action(listen), Action(open-right), Action(open-left)}
OBSERVATIONS = {Observation(tiger-right), Observation(tiger-left)}
pomdp_problems.tiger.tiger_problem.test_planner(tiger_problem, planner, nsteps=3)[source]

Runs the action-feedback loop of Tiger problem POMDP

Parameters
  • tiger_problem (TigerProblem) – an instance of the tiger problem.

  • planner (Planner) – a planner

  • nsteps (int) – Maximum number of steps to run this loop.

pomdp_problems.tiger.tiger_problem.build_setting(setting)[source]
pomdp_problems.tiger.tiger_problem.main()[source]