Run Experiment and Analyze Results Programmatically

Run Experiment and Retrieve Result

Suppose that you have a MATLAB^® project, MyProject, that contains experiments you previously configured in the Experiment Manager app. To run an experiment programmatically, you need the name of the experiment. To list the names of experiments in the project, use the experimentNames function.

expNames = experimentNames("MyProject")

expNames =

  2×1 string array

    "Experiment1"
    "Experiment2"

To run one of the experiments in the project, use the runExperiment function. While the experiment runs, runExperiment displays its progress using timestamped messages. The runExperiment function returns an ExperimentResult object.

result = runExperiment("MyProject",expNames(1))

[16:22:16] [MyProject/Experiment1] Starting experiment.
[16:22:16] [MyProject/Experiment1] Initializing experiment and validating experiment details.
[16:22:17] [MyProject/Experiment1] Running 4 trials.
[16:22:17] [MyProject/Experiment1] Starting trial 1.
[16:22:18] [MyProject/Experiment1] Trial 1 Status : Complete.
[16:22:18] [MyProject/Experiment1] Starting trial 2.
...
[16:22:20] [MyProject/Experiment1] Finishing experiment.

result = 
matlab.experiment.ExperimentResult with properties:

     ProjectFolder: "C:\MyProjects\MyProject"
    ExperimentName: "Experiment1"
        ResultName: "Result1"
          Strategy: "exhaustive-sweep"
            Status: complete
             Error: [0x0 MException]
     ResultSummary: [4x3 table]
            Trials: [4x1 matlab.experiment.trial.ExperimentTrial]

View the Experiment Result

An ExperimentResult object contains the configuration details and outcomes of a single execution of an experiment. Each experiment can have multiple results. The experiment creates a new result each time you run it, whether you run it in the app or programmatically.

To get information about an experiment result, access the properties of an ExperimentResult object.

For example, to see how the experiment explored the parameter space, access the Strategy property. You choose the strategy in the Experiment Manager app when you configure the experiment.

strategy = result.Strategy

strategy = 

    "exhaustive-sweep"

To see if the experiment completed successfully, check the Status property.

status = result.Status

status = 

  ResultStatus enumeration

    complete

You can compare trials by viewing the result summary table, which is similar to the table that the app displays for a result. Each row in the table corresponds to one trial, and the columns include trial execution status, trial duration, parameters, and outputs. View the table by accessing the ResultSummary property.

result.ResultSummary

resultSummary =

  4×3 table

            Experiment Details             Parameters    Outputs
    ___________________________________    __________    _______

    Trial      Status      Elapsed Time      x    y         z   
    _____    __________    ____________      _    _         _   
                                                                
      1      "Complete"      00:00:02        1    4         5   
      2      "Complete"      00:00:01        2    4         6   
      3      "Complete"      00:00:01        1    5         6   
      4      "Complete"      00:00:01        2    5         7   

Analyze Individual Trials

After you review the experiment result at a high level, you might want additional information about a specific trial. The ExperimentTrial object stores this information.

If you already have an ExperimentResult object in your workspace, access its Trials property to get an array of ExperimentTrial objects. The number of trials depends on the experiment strategy and parameter configuration. For example, for an experiment using exhaustive sweep, the number of trials equals the number of parameter combinations.

trial = result.Trials

trial = 

  4x1 ExperimentTrial array with properties:

    ProjectFolder
    ExperimentName
    ResultName
    Number
    Status
    Error
    Parameter
    Information
    Metric
    Output
    Visualization

If you want to identify a trial of interest, you can operate on the ResultSummary table, which contains one row per trial. For example, find the trial that maximizes an output and return the ExperimentTrial object for that trial.

[~,bestIdx] = max(result.ResultSummary.Outputs.z);
bestTrial = result.Trials(bestIdx);

To get more information about the trial, access the properties of the ExperimentTrial object. For example, return the parameter values and the corresponding output for the best trial.

maxParams = bestTrial.Parameter
maxOutput = bestTrial.Output.z

maxParams = 

  Parameter with properties:

    y: 5
    x: 2

maxOutput =

     7

If your experiment is a built-in training experiment (requires Deep Learning Toolbox™), you can return the trained network for a trial.

net = bestTrial.Output.TrainedNetwork;

If your experiment creates visualizations, you can access them using the Visualization property of an ExperimentTrial object. Built-in training experiments create visualizations automatically. Custom training and general-purpose experiments include visualizations if the experiment function creates figures. For information about creating your own visualizations, see Visualize Experiment Results with Plots.

To see what visualizations are available from the trial, access the FigureNames property of the Visualization object from the trial. To open the visualizations, use the openfig function with the Visualization object and one or more figure names.

vis = bestTrial.Visualization;
figNames = vis.FigureNames;
figs = openfig(vis,figNames);

Access Existing Experiment Result and Trials

An experiment creates a result each time you run it in the Experiment Manager app or call runExperiment. If you already ran an experiment, you can retrieve the result without rerunning the experiment. List the names of the results for the experiment using the experimentResultNames function. Then, retrieve a result by using the experimentResult function.

resultNames = experimentResultNames("MyProject","Experiment1");
previousResult = experimentResult("MyProject","Experiment1",resultNames(1));

You can also retrieve a specific trial by specifying the trial number with the experimentTrial function. This function does not require having the ExperimentResult object in your workspace. For example, you can run an experiment in the Experiment Manager app and then programmatically access information about a trial.

trialNum = 3;
trial = experimentTrial("MyProject","Experiment1","Result1",trialNum);

Switch Between the App and Functions

You can use either the Experiment Manager app or the ExperimentResult object to interact with an experiment result and its trials. So, you can choose the approach that best suits your goal:

Before calling the runExperiment function, close the Experiment Manager app.