Main Content

Configure Metrics for MATLAB Job Scheduler

Since R2024b

Configure MATLAB® Job Scheduler to export cluster monitoring metrics such as cluster status, worker utilization, and licenses in use from the job manager. These metrics help you:

  • Monitor the health of the MATLAB Job Scheduler cluster

  • Diagnose cluster issues

  • Optimize cluster performance

You can gather the exported metrics with a cluster monitoring system such as Prometheus® and visualize the metrics in a preconfigured Grafana® dashboard with usage analysis and alerting tools.

Prerequisites

If this is the first time you are integrating MATLAB Job Scheduler with your cluster, see this page for the most common configuration options: Install for MATLAB Job Scheduler with Network License Manager.

In these instructions, matlabroot refers to the location of your installed MATLAB Parallel Server™ software. Where you see this term used in these instructions, substitute the path with the location of your installation.

Edit MATLAB Job Scheduler Parameter File

To set up your MATLAB Job Scheduler to export metrics, you must edit the mjs_def file on your headnode before installing the mjs service and starting MATLAB Job Scheduler. You can find this file in these locations:

  • matlabroot\toolbox\parallel\bin\mjs_def.bat on Windows® operating systems

  • matlabroot/toolbox/parallel/bin/mjs_def.sh on Linux® operating systems

To learn more about the parameters in the mjs_def file, see Define MATLAB Job Scheduler Startup Parameters.

Use these parameters to configure the MATLAB Job Scheduler job manager to export metrics. Edit the parameters in the mjs_def file with the required values.

Parameter

Description

Required Values

EXPORT_METRICS

Option to export cluster monitoring metrics from job manager.

true

METRICS_PORT

Port for exporting metrics.

The job manager uses an HTTP(S) server at the port you specify to export metrics.

By default, the job managers uses port number 8001.

USE_SECURE_METRICS

Option to use encrypted communication to export metrics.

By default, USE_SECURE_METRICS is set to true and the job manager exports metrics on an encrypted HTTPS server.

To disable encryption and export metrics on an HTTP server, set USE_SECURE_METRICS to false.

METRICS_CA_FILE

Job manager's trusted certificate authority (CA) for encrypted metrics.

If USE_SECURE_METRICS is true, the job manager only forms encrypted connections to clients that present certificates signed by this CA. You must set this parameter if USE_SECURE_METRICS is true.

Specify a path to a CA certificate, for example:

METRICS_CA_FILE=<~/mjs-metrics>/ca.crt
You can use the mjssetup tool to generate the job manager's CA certificate. To learn more, see Generate Certificate and Key Files.

METRICS_CERT_FILE

Job manager's certificate for encrypted metrics.

The job manager presents this certificate to clients that want to form encrypted connections for metrics. You must set this parameter if USE_SECURE_METRICS is true.

Specify a path to the job manager's certificate, for example:

METRICS_CERT_FILE=<~/mjs-metrics>/jobmanager.crt
You can use the mjssetup tool to generate the job manager's certificate file. To learn more, see Generate Certificate and Key Files.

METRICS_KEY_FILE

Job manager's private key for encrypted metrics.

You must set this parameter if USE_SECURE_METRICS is true.

Specify a path to the job manager's private key file. The private key must be associated with the certificate file you specify for METRICS_CERT_FILE. For example:

METRICS_KEY_FILE=<~/mjs-metrics>/jobmanager.key
You can use the mjssetup tool to generate the job manager's private key file. To learn more, see Generate Certificate and Key Files.

Generate Certificate and Key Files

You can use the mjssetup tool to generate the job manager's certificate and key files you need to export metrics using encrypted communication. The mjssetup tool ships with MATLAB. You can find the executable for the mjssetup tool in these folders:

  • matlabroot/toolbox/parallel/bin/glnxa64 on Linux operating systems.

  • matlabroot\toolbox\parallel\bin\win64 on Windows operating systems.

To generate the job manager's certificate and key files, specify the generate-metrics-certificates-and-keys command, the hostname of the job manager, and the output folder for the certificate and key files. For example, on a Linux operating system, generate certificate and key files for a job manager with the hostname MJSJobManager in a folder with the name mjs-metrics.

cd matlabroot/toolbox/parallel/bin/glnxa64
mjssetup generate-metrics-certificates-and-keys -jobmanagerhost <MJSJobManager> -outdir <mjs-metrics>
The command also generates the prometheus.crt public certificate and the prometheus.key private key that you use to set up a Prometheus data source. To learn more, see Monitor Cluster Metrics with Prometheus and Grafana.

Access Server Metrics

When you configure MATLAB Job Scheduler to export metrics, the job manager starts an HTTP(S) server on the headnode that opens up a port at the number specified using METRICS_PORT and exposes a page with the cluster metrics. The cluster metrics is in a human-readable Prometheus text-based format that you can easily integrate with a server performance monitoring platform.

You can use the nodestatus command to find the URL of the page with the cluster metrics. To learn how to use the metrics URL to set up a Prometheus data source, see Monitor Cluster Metrics with Prometheus and Grafana.

nodestatus
Job manager lookup process:
     Status                        Running
 
Job manager:
     Name                          MJSJobManager
     Running on host               host1.dhcp.mycompany.com
     Number of workers             4
     Status                        running
     Supported releases            R2024b
     Metrics URL                   https://host1.dhcp.mycompany.com:8001

Monitor Cluster Metrics with Prometheus and Grafana

After you have set up your MATLAB Job Scheduler cluster to export metrics, you can set up a Prometheus data source to gather the cluster metrics and a Grafana dashboard to visualize and analyze the metrics.

To configure the Prometheus data source and access the preconfigured Grafana dashboard, follow the instructions provided in this GitHub® repository:

See Also

Related Topics