We recommend using virtualenv and pip to manage package versions and installation.

Install Python and Cython

  1. Install Python 3.7+ from official website.

  2. Install Cython.:

    pip install Cython

Install pomdp_py

  1. First, download pomdp_py latest release on github, or clone the repository by:

    git clone
  2. Next, go to the package root directory (where is located). Run:

    pip install -e .

    This will build and install pomdp_py package. This will build .so files and copy them to the python source directory. When you make changes to .pyx or .pyd files, run the following to rebuild those .so libraries, so that the python imports can get those changes you made:

    make build
  3. Finally, verify that Tiger problem and RockSample problem work:

    python pomdp_problems/tiger/
    python pomdp_problems/rocksample/

    For the Tiger problem, you should see output like

    ** Testing value iteration **
    ==== Step 1 ====
    True state: tiger-left
    Belief: [(State(tiger-right), 0.5), (State(tiger-left), 0.5)]
    Action: listen
    Reward: -1
    >> Observation: tiger-left

    There will be plots that visualize the MCTS trees displayed,

    For the RockSample problem, you should see something like:

    *** Testing POMCP ***
    ==== Step 1 ====
    Particle reinvigoration for 66 particles
    True state: State((0, 4) | ('bad', 'good', 'bad', 'good', 'good') | False)
    Action: sample
    Observation: None
    Reward: 0.0
    Reward (Cumulative): 0.0
    Reward (Cumulative Discounted): 0.0
    __num_sims__: 1217