rollout

R2026b

Executes rollout for a vectorized environment with a given policy

Since R2026b

collapse all in page

    Syntax

    [NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps)
    [NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps,MaxStepsPerEpisode=Nmax)

    Description

    Use rollout to execute a specified number of simulation steps of a given policy on all instances of a vectorized environment.

    [NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps) executes NumSteps simulation steps using the policy policy on each instance of the vectorized environment venv. The function generates a total of NumSteps*venv.NumEnv samples.

    [NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps,MaxStepsPerEpisode=Nmax) also specifies the maximum steps for each environment instance. The function resets all environment instances that reach or exceed this value.

    example

    Examples

    collapse all

    Open Live Script

    This example shows how to use the rollout function to execute six simulation steps of a given policy on all instances of a vectorized environment.

    First, create a vectorized version of the cart-pole environment with continuous action space. This environment has four states.

    venv = rlVectorEnv(@(info) rlPredefinedEnv("CartPole-Continuous"))
    venv = 
  rlVectorEnv with properties:

    NumEnv: 2

    By default, the vectorized environment has two instances.

    Create a PPO agent for that environment.

    agent = rlPPOAgent(getObservationInfo(venv),getActionInfo(venv));

    Extract the exploration policy from the PPO agent.

    policy = getExplorationPolicy(agent);

    Simulate the policy on all instances of the vectorized environment for 6 steps.

    [NextObs,Obs,Act,Rwd,Isd,policy] = rollout(venv,policy,6);

    Display the sizes of the observation batch.

    Obs
    Obs = 1×1 cell array
    {4×1×12 double}

    The batch dimension is the number of environment instances multiplied by the number of steps.

    Display the is-done batch.

    Isd
    Isd = 1×12 uint8 row vector

   0   0   0   0   0   3   0   0   0   0   0   3

    Within a custom training loop, you can use the rollout function to simulate a policy against a vectorized environment.

    Input Arguments

    collapse all

    Vectorized environment, specified as an rlVectorEnv object.

    Example: venv = rlVectorEnv(@(info) rlPredefinedEnv("DoubleIntegrator-Discrete")) creates the vector environment venv from the predefined double-integrator environment with discrete action space.

    For more information on reinforcement learning policies, see Create Actors, Critics, and Policy Objects.

    Example: policy = getExplorationPolicy(rlPPOAgent(rlNumericSpec([2 1]),rlNumericSpec([1 1]))) extracts the object that implements the exploration policy from a default PPO agent and assigns it to the variable policy.

    Number of simulation steps to take for each environment instance, specified as a positive integer scalar.

    Example: 128

    Maximum number of simulation steps to take for each environment instance, specified as a positive integer scalar. If you do not specify this argument, the function uses the default value of 500.

    Example: 1000

    Output Arguments

    collapse all

    Batch of next observations, returned as an array whose first dimensions are specified in getObservationInfo(venv). The batch dimension is equal to venv.NumEnv multiplied by NumSteps.

    Batch of current observations, returned as an array whose first dimensions are specified in getObservationInfo(venv). The batch dimension is equal to venv.NumEnv multiplied by NumSteps.

    Batch of actions, returned as an array whose first dimensions are specified in getActionInfo(venv). The batch dimension is equal to venv.NumEnv multiplied by NumSteps.

    Batch of actions, returned as a row vector with length equal to venv.NumEnv multiplied by NumSteps.

    Is-done batch, returned as an uint8 row vector with length equal to venv.NumEnv multiplied by NumSteps.

    Updated policy, returned as the same policy object as the policy input argument. Typically, the policy updates its noise parameters during simulation.

    Version History

    Introduced in R2026b

    See Also

    Functions

    Objects

    Topics