Deploy Trained Reinforcement Learning Policy as Microservice Docker Image

Required Products

Type ver at the MATLAB command prompt to verify whether the following products are installed:

Prerequisites

Create MATLAB Function to Evaluate the Policy

For this example, use a pretrained DQN agent to balance a cart-pole system. See Train DQN Agent to Balance Cart-Pole System (Reinforcement Learning Toolbox) for more details. Write a policy-evaluation function called evaluatePolicy.m using the following code.

function action = evaluatePolicy(observation)
% Generate action from policy

% Load the policy from a MAT file
% policy.mat might contain an agent or policy with the name "agent"
persistent policy
if isempty(policy)
    policy = load("policy.mat");
end

% Evaluate the policy
action = getAction(policy.agent,observation);
end

Test the function from the MATLAB command line:

load("policy.mat")
evaluatePolicy(rand(agent.ObservationInfo.Dimension))

ans =

1×1 cell array

{[10]}

Create Deployable Archive

Package the evaluatePolicy function into a deployable archive using the compiler.build.productionServerArchive function.

You can specify additional options in the compiler.build command by using name-value arguments.

buildResults = compiler.build.productionServerArchive('evaluatePolicy.m',...
    'ArchiveName','cartPoleDQN','Verbose',true);

buildResults =

Results with properties:

BuildType: 'productionServerArchive'

Files: {'\home\mluser\work\cartPoleDQNproductionServerArchive\cartPoleDQN.ctf'}

IncludedSupportPackages: {}

Options: [1×1 compiler.build.ProductionServerArchiveOptions]

The compiler.build.Results object buildResults contains information on the build type, generated files, included support packages, and build options.

Once the build is complete, the function creates a folder named cartPoleDQNproductionServerArchive in your current directory to store the deployable archive.

Package Archive into Microservice Docker Image

Build the microservice Docker image using the buildResults object that you created. You can specify additional options in the compiler.build command by using name-value arguments. For details, see compiler.package.microserviceDockerImage

compiler.package.microserviceDockerImage(buildResults, ...
    'ImageName','cartpoledqn-microservice', ...
    'DockerContext',fullfile(pwd,'microserviceDockerContext'));

The function generates the following files within a folder named microserviceDockerContext in your current working directory:

Test Docker Image

In a system command window, verify that your cartPoleDQN-microservice image is in your list of Docker images.

docker images

This command returns a list of Docker images, including your microservice.

Run the cartpoledqn-microservice microservice image from the system command prompt. Port 9910 is the default port exposed by the microservice within the Docker container. You can map it to any available port on your host machine. For this example, map it to 9900.

docker run --rm -p 9900:9910 cartpoledqn-microservice -l trace &

You can specify additional options in the Docker command. For a complete list of options, see Microservice Command Arguments.

Once the microservice container is running in Docker, you can check the status of the service by going to http://hostname:9900/api/health in a web browser.

Note: Use localhost as the hostname if Docker is running on the same machine as the browser.

If the service is ready to receive requests, you see the message "status: ok".

Test the running service. In the terminal, use the curl command to send a JSON query to the service through port 9900. For more information on constructing JSON requests, see JSON Representation of MATLAB Data Types.

curl --location 'http://localhost:9900/cartPoleDQN/evaluatePolicy' \
--header 'Content-Type: application/json' \
--data '{"nargout":1,"rhs":[{"mwdata":[0,0,0,0],"mwsize":[4,1],"mwtype":"double"}]}'

The output is:

{"lhs":[{"mwdata":[{"mwdata":[-10],"mwsize":[1,1],"mwtype":"double"}],"mwsize":[1,1],"mwtype":"cell"}]}

You can also test from the MATLAB desktop:

%% Import MATLAB HTTP interface packages
import matlab.net.*
import matlab.net.http.*
import matlab.net.http.fields.*

%% Set up message body
body = MessageBody;
body.Payload = ...
    '{"nargout":1,"rhs":[{"mwdata":[0,0,0,0],"mwsize":[4,1],"mwtype":"double"}]}';

%% Set up request
requestUri = URI('http://hostname:9900/cartPoleDQN/evaluatePolicy');
options = matlab.net.http.HTTPOptions('ConnectTimeout',20,...
    'ConvertResponse',false);
request = RequestMessage;
request.Header = HeaderField('Content-Type','application/json');
request.Method = 'POST';
request.Body = body;

%% Send request and view raw response
response = request.send(requestUri, options);
disp(response.Body.Data)

%% Decode JSON
lhs = mps.json.decoderesponse(response.Body.Data);

%% Display response as a table
T = cell2table(lhs{:},'VariableNames',{'Action'})

Replace hostname in the code above with the appropriate host. Since Docker is running locally, use localhost.

To stop the service, use the following command to display the container ID.

docker ps

Stop the service using the specified container id.

docker stop 0662c1e1fa85

Share Docker Image

You can share your Docker image in various ways.

For details about pushing your image to DockerHub or your private registry, consult the Docker documentation.

Save Docker Image as Tar Archive

To save your Docker image as a tar archive, open a system command window, navigate to the Docker context folder, and type the following.

docker save cartpoledqn-microservice -o cartpoledqn-microservice.tar

This command creates a file named cartpoledqn-microservice.tar in the current folder. Set the appropriate permissions (for example, using chmod) prior to sharing the tar archive with other users.

Load Docker Image from Tar Archive

Load the image contained in the tar archive on the end user machine.

docker load --input cartpoledqn-microservice.tar

Verify that the image is loaded.

docker images

Run Docker Image

docker run --rm -p 9900:9910 cartpoledqn-microservice