Main Content

rlEvolutionStrategyTrainingOptions

Options for training off-policy reinforcement learning agents using an evolutionary strategy

Since R2023b

expand all in page

    Description

    Use an rlEvolutionStrategyTrainingOptions object to specify options to train an DDPG, TD3 or SAC agent within an environment. Evolution strategy training options include the population size and its update method, the number of training epochs, as well as criteria for stopping training and saving agents. After setting its options, use this object as an input argument for trainWithEvolutionStrategy.

    For more information on the training algorithm, see Train agent with evolution strategy. For more information on training agents, see Train Reinforcement Learning Agents.

    Creation

    Syntax

    trainOpts = rlEvolutionStrategyTrainingOptions
    trainOpts = rlEvolutionStrategyTrainingOptions(Name=Value)

    Description

    trainOpts = rlEvolutionStrategyTrainingOptions returns the default options for training a DDPG, TD3 or SAC agent using an evolutionary strategy.

    trainOpts = rlEvolutionStrategyTrainingOptions(Name=Value) creates the training option set trainOpts and sets its Properties using one or more name-value arguments.

    example

    Properties

    expand all

    Number of individuals in the population, specified as a positive integer. Every individual corresponds to an actor.

    Example: PopulationSize=50

    Percentage of individuals surviving to form the next population, specified as an integer between 1 and 100.

    Example: PercentageEliteSize=30

    Maximum number of episodes run per individual, specified as a positive integer. Here, episode run means all the environment steps performed by given individual from the beginning of a given generation until termination of the same generation.

    Example: EvaluationsPerIndividual=2

    Number of training epochs used to update the gradient-based agent. If you set TrainEpochs to 0, then the agents are updated without using any gradient-based agent (therefore using only an pure evolutionary search strategy). For more information on the training algorithm, see Train agent with evolution strategy.

    Example: TrainEpochs=5

    Population update options, specified as a GaussianUpdateOptions object. For more information on the training algorithm, see Train agent with evolution strategy.

    The properties of the GaussianUpdateOptions object, which determine how the evolution algorithm updates the distribution, and which you can modify using dot notation after creating the rlEvolutionStrategyTrainingOptions object, are as follows.

    Update method for the population distribution, specified as either:

    • "WeightedMixing" — When calculating the sum used to calculate the mean and standard deviation of the population distribution, weights each actor according to its fitness index (that is, better actors are weighted more).

    • "UniformMixing" — When calculating the sum used to calculate the mean and standard deviation of the population distribution, weights each actor equally.

    Example: Mode="async"

    Initial mean of the population distribution, specified as a scalar.

    Example: InitialMean=-0.5

    Initial standard deviation of the population distribution, specified as a scalar.

    Example: InitialStandardDeviation=0.5

    Initial bias of the standard deviation of the population distribution, specified as a scalar. A larger value promotes exploration.

    Example: InitialStandardDeviationBias=0.2

    Final bias of the standard deviation of the population distribution, specified as a nonnegative scalar.

    Example: FinalStandardDeviationBias=0.002

    Decay rate of the bias of the standard deviation of the population distribution, specified as a positive scalar.

    At the end of each training time step, if the bias of the population standard deviation StdBias is updated as follows.

    StdBias = (1-StandardDeviationBiasDecayRate)*StdBias + ...
          StandardDeviationBiasDecayRate*FinalStandardDeviationBias

    Note that StdBias is conserved between the end of an episode and the start of the next one. Therefore, it keeps on uniformly evolving over multiple episodes until it reaches FinalStandardDeviationBias.

    Example: StandardDeviationBiasDecayRate=0.99

    Type of the policy returned once training is terminated, specified as either "AveragedPolicy" or "BestPolicy".

    Example: ReturnedPolicy="BestPolicy"

    Maximum number of generations that the population is updated, specified as a positive integer.

    Example: MaxGenerations=1000

    This property is read-only.

    Maximum number of environment steps to run per episode, specified as a positive integer. In general, you define episode termination conditions in the environment. This value is the maximum number of steps to run in the episode if other termination conditions are not met.

    Example: MaxStepsPerEpisode=1000

    Window length for averaging the scores, rewards, and number of steps, specified as a scalar or vector.

    For options expressed in terms of averages, ScoreAveragingWindowLength is the number of episodes included in the average. For instance, if StopTrainingCriteria is "AverageReward", and StopTrainingValue is 500, training terminates when the average reward over the number of episodes specified in ScoreAveragingWindowLength equals or exceeds 500.

    Example: ScoreAveragingWindowLength=10

    Training termination condition, specified as one of the following strings:

    • "AverageReward" — Stop training when the running average reward equals or exceeds the critical value.

    • "EpisodeReward" — Stop training when the reward in the current episode equals or exceeds the critical value.

    Example: StopTrainingCriteria="AverageReward"

    Critical value of the training termination condition, specified as an scalar.

    Training ends when the termination condition specified by the StopTrainingCriteria option equals or exceeds this value.

    For instance, if StopTrainingCriteria is "AverageReward", and StopTrainingValue is 100, training terminates when the average reward over the number of episodes specified in ScoreAveragingWindowLength equals or exceeds 100.

    Example: StopTrainingValue=100

    Condition for saving agents during training, specified as one of the following strings:

    • "none" — Do not save any agents during training.

    • "AverageReward" — Save the agent when the running average reward over all episodes equals or exceeds the critical value.

    • "EpisodeReward" — Save the agent when the reward in the current episode equals or exceeds the critical value.

    Set this option to store candidate agents that perform well according to the criteria you specify. When you set this option to a value other than "none", the software sets the SaveAgentValue option to 500. You can change that value to specify the condition for saving the agent.

    For instance, suppose you want to store for further testing any agent that yields an episode reward that equals or exceeds 100. To do so, set SaveAgentCriteria to "EpisodeReward" and set the SaveAgentValue option to 100. When an episode reward equals or exceeds 100, train saves the current agent in a MAT-file in the folder specified by the SaveAgentDirectory option. The MAT-file is called AgentK.mat, where K is the number of the corresponding episode. The agent is stored within that MAT-file as saved_agent.

    Example: SaveAgentCriteria="EpisodeReward"

    Critical value of the condition for saving agents, specified as a scalar.

    When you specify a condition for saving candidate agents using SaveAgentCriteria, the software sets this value to 500. Change the value to specify the condition for saving the agents. See the SaveAgentCriteria option for more details.

    Example: SaveAgentValue=100

    Folder name for saved agents, specified as a string or character vector. The folder name can contain a full or relative path. When an episode occurs in which the conditions specified by the SaveAgentCriteria and SaveAgentValue options are satisfied, the software saves the current agent in a MAT-file in this folder. If the folder does not exist, the training function creates it. When SaveAgentCriteria is "none", this option is ignored and no folder is created.

    Example: SaveAgentDirectory = pwd + "\run1\Agents"

    Option to display training progress at the command line, specified as the logical values false (0) or true (1). Set to true to write information from each training episode to the MATLAB® command line during training.

    Example: Verbose=true

    Option to display training progress with Reinforcement Learning Training Monitor, specified as "training-progress" or "none". By default, calling train opens Reinforcement Learning Training Monitor, which graphically and numerically displays information about the training progress, such as the reward for each episode, average reward, number of episodes, and total number of steps. For more information, see train. To turn off this display, set this option to "none".

    Example: Plots="none"

    Option to use parallel training, specified as a logical. Setting this option to true configures training to use multiple processes (which can run on different cores, processors, computer clusters or cloud resources) to simulate the environment. This option scales up the number of simulations with the environment, and can speed up the generation of data for learning.

    To specify options for parallel training, use the ParallelizationOptions property.

    Note that if you want to speed up deep neural network calculations (such as gradient computation, parameter update and prediction) using a local GPU, you do not need to set UseParallel to true. Instead, when creating your actor or critic, set its UseDevice option to "gpu" instead of "cpu".

    Using parallel computing or the GPU requires Parallel Computing Toolbox™ software. Using computer clusters or cloud resources additionally requires MATLAB Parallel Server™. For more information about training using multicore processors and GPUs, see Train Agents Using Parallel Computing and GPUs.

    Example: UseParallel=true

    Options for parallel training, specified as a ParallelTraining object. For more information about training using parallel computing, see Train Agents Using Parallel Computing and GPUs.

    The ParallelTraining object has the following properties, which you can modify using dot notation after creating the rlTrainingOptions object.

    Parallel computing mode, specified as one of the following:

    • "sync" — Use parpool to run synchronous training on the available workers. In this case, each worker pauses execution until all workers are finished. The parallel pool client updates the actor and critic parameters based on the results from all the workers and sends the updated parameters to all workers. When training a PG agent using gradient-based parallelization Mode must be set to "sync".

    • "async" — Use parpool to run asynchronous training on the available workers. In this case, each worker sends its data back to the parallel pool client as soon as it finishes and then receives updated parameters from the client. The worker then continues with its task.

    Example: Mode="async"

    Randomizer initialization for workers, specified as one of the following:

    • –1 — Assign a unique random seed to each worker. The value of the seed is the worker ID.

    • –2 — Do not assign a random seed to the workers.

    • Vector — Manually specify the random seed for each worker. The number of elements in the vector must match the number of workers.

    Example: WorkerRandomSeeds=[1 2 3 4]

    Option to send model and workspace variables to parallel workers, specified as "on" or "off". When the option is "on", the client sends to the workers the variables defined in the base MATLAB workspace and used in the approximation models.

    Example: TransferBaseWorkspaceVariables="off"

    Additional files to attach to the parallel pool, specified as a string or string array.

    Example: AttachedFiles="myInitFile.m"

    Function to run before training starts, specified as a handle to a function having no input arguments. This function is run once per worker before training begins. Write this function to perform any processing that you need prior to training.

    Example: AttachedFiles=@mySetupFcn

    Function to run after training ends, specified as a handle to a function having no input arguments. You can write this function to clean up the workspace or perform other processing after training terminates.

    Example: AttachedFiles=@myCleanupFcn

    Option to stop training when an error occurs during an episode, specified as "on" or "off". When this option is "off", errors are captured and returned in the SimulationInfo output of train, and training continues to the next episode.

    Example: StopOnError="off"

    Storage type for environment data, specified as "memory", "file", or "none". This option specifies the type of storage used for data generated during training or simulation by a Simulink® environment. Specifically, the software saves anything that appears as the output of a sim (Simulink) command.

    Note that this option does not affect (and is not affected by) any option to save agents during training specified within a training option object, or any data logged by a FileLogger or MonitorLogger object.

    The default value is "memory", indicating that data is stored in an internal memory variable. When you set this option to "file", data is stored to disk, in MAT-files in the directory specified by the SaveSimulationDirectory property, and using the MAT-file version specified by the SaveFileVersion property. When you set this option to "none", simulation data is not stored.

    You can use this option to prevent out-of-memory issues during training or simulation.

    Example: "none"

    Folder used to save environment data, specified as a string or character vector. The folder name can contain a full or relative path. When you set the SimulationStorageType property to "file", the software saves data generated during training or simulation by a Simulink environment in MAT-files in this folder, using the MAT-file version specified by the SaveFileVersion property. If the folder does not exist, the software creates it.

    Example: "envSimData"

    MAT-file version used to save environment data, specified as a string or character vector. When you set the SimulationStorageType property to "file", the software saves data generated by a Simulink environment in MAT-files in the version specified by SaveFileVersion, in the folder specified by the SaveSimulationDirectory property. For more information, see MAT-File Versions.

    Example: Version="-v7.3"

    Object Functions

    trainWithEvolutionStrategyTrain DDPG, TD3 or SAC agent using an evolutionary strategy within a specified environment

    Examples

    collapse all

    Open Live Script

    Create an options set for training a DDPG, TD3 or SAC agent using an evolutionary strategy. Set the population size, the number of train epochs, and the maximum number of steps per episode. You can set the options using name-value pair arguments when you create the options set. Any options that you do not explicitly set have their default values.

    esOpts = rlEvolutionStrategyTrainingOptions(...
    PopulationSize=50, ...
    TrainEpoch=10, ...
    MaxStepsPerEpisode=500)
    esOpts = 
  rlEvolutionStrategyTrainingOptions with properties:

                PopulationSize: 50
           PercentageEliteSize: 50
      EvaluationsPerIndividual: 1
                   TrainEpochs: 10
       PopulationUpdateOptions: [1x1 rl.option.GaussianUpdateOptions]
                ReturnedPolicy: "AveragedPolicy"
                MaxGenerations: 500
            MaxStepsPerEpisode: 500
    ScoreAveragingWindowLength: 5
          StopTrainingCriteria: "AverageReward"
             StopTrainingValue: 500
             SaveAgentCriteria: "none"
                SaveAgentValue: 500
            SaveAgentDirectory: "savedAgents"
                       Verbose: 0
                         Plots: "training-progress"
                   UseParallel: 0
        ParallelizationOptions: [1x1 rl.option.ParallelSimulation]
                   StopOnError: "on"
         SimulationStorageType: "memory"
       SaveSimulationDirectory: "savedSims"
               SaveFileVersion: "-v7"

    Alternatively, create a default options set and use dot notation to change some of the values. 

    esOpts = rlEvolutionStrategyTrainingOptions;
esOpts.PopulationSize=30;
esOpts.TrainEpochs=15;
esOpts.MaxStepsPerEpisode=500;

    Set the population update method and the initial standard deviation in the PopulationUpdateOptions property.

    esOpts.PopulationUpdateOptions.UpdateMethod = "UniformMixing";
esOpts.PopulationUpdateOptions.InitialStandardDeviation  =  0.2;

    To train a supported off-policy agent with an evolutionary strategy, you can now use esOpts as an input argument to trainWithEvolutionStrategy.

    Algorithms

    expand all

    Version History

    Introduced in R2023b

    See Also

    Functions

    Objects

    Topics