Resolve License Service Issues

MATLAB^® Online Server™ requires you to have valid licenses for both MATLAB Online Server and MATLAB before you install either. After you have completed installation with valid licenses, your users can sign in successfully to MATLAB Online™.

License Service Pod State is Running but Users Cannot Connect to MATLAB Online

When the services are running successfully and can connect with the network license manager license server, then the status of pods is Running.

However, if the services are having issues connecting with network license manager, the status of the pods is still shown as Running. This is expected, and it means that the MATLAB Online Server is running successfully but is not authorized because it is unable to connect to the network license manager.

NAME                                                     READY   STATUS    RESTARTS   AGE
your-namespace-authnz-7994c9866d-675fb                   1/1     Running   0          10m
your-namespace-core-ui-cfdcccc4c-5bhrc                   1/1     Running   0          10m
your-namespace-gateway-88ffd446d-mbf2l                   1/1     Running   0          10m
your-namespace-gateway-proxy-6f85db9cbb-8ftbr            1/1     Running   0          10m
your-namespace-gateway-proxy-6f85db9cbb-mdhr7            1/1     Running   0          10m
your-namespace-ingress-nginx-controller-7fdcd49d74-bnbls 1/1     Running   0          10m
your-namespace-license-5cc85b97cd-zg4vd                  1/1     Running   0          10m
your-namespace-matlab-pool-9cc6b6465-9rdz8               2/2     Running   0          10m
your-namespace-matlab-pool-9cc6b6465-t7wp2               2/2     Running   0          10m
your-namespace-matlab-pool-helpsearch-8479fbdc88-4r6sd   1/1     Running   0          10m
your-namespace-matlab-pool-ui-8484bbbd4d-t6777           1/1     Running   0          10m
your-namespace-resource-78f9b97745-fzwlq                 1/1     Running   0          10m

If the license service is not operational, other services also do not work as expected. This is because the license service checks out the MATLAB Online Server license, which in turn allows all other services to run successfully.

Get More Information

Run the following command to get more information, replacing license-pod-name with the name of your license pod and your-namespace with the namespace you used for MATLAB Online Server:

kubectl logs license-pod-name --namespace your-namespace

If you get output similar to the following display, then it is likely that the license service is unable to check out the MATLAB Online Server license.

00000006 YYYY-Mmm-DD 21:10:55.104617 0x00007fa8d8fd5700 mos::license I 
Checking out license
00000007 YYYY-Mmm-DD 21:10:55.104626 0x00007fa8d8fd5700 mos::license I 
Server: 27000@flexlm
00000008 YYYY-Mmm-DD 21:11:15.237586 0x00007fa8d8fd5700 
startup::license_manager I enter checkout for feature: MATLAB_Online_Server
The server is running.
Error while trying to check out the license: , will retry in 30 seconds.
00000009 YYYY-Mmm-DD 21:11:28.314607 0x00007fa8d8fd5700 mos::license I 
Registered service
00000010 YYYY-Mmm-DD 21:11:56.040342 0x00007fa8d203c700 mos::license I Check 
license request mathworks-gateway-669dd7865f-t6l74
00000011 YYYY-Mmm-DD 21:11:58.430979 0x00007fa8c71fa700 mos::license I 
Checking out license
00000012 YYYY-Mmm-DD 21:11:58.431050 0x00007fa8c71fa700 mos::license I 
Server: 27000@flexlm
00000013 YYYY-Mmm-DD 21:12:03.866191 0x00007fa8d303e700 mos::license I Check 
license request from mathworks-authnz-authnz-7c7999589c-x74d9
00000014 YYYY-Mmm-DD 21:12:04.565262 0x00007fa8c71fa700 
startup::license_manager I enter checkout for feature: MATLAB_Online_Server
00000015 YYYY-Mmm-DD 21:12:04.869012 0x00007fa8d283d700 mos::license I Check 
license request from mathworks-resource-resource-894447df7-r9ngb
Error while trying to check out the license: ^^, will retry in 30 seconds.

Possible Issues

MATLAB Online Server is unable access license server when both servers are on the same machine
If you installed MATLAB Online Server and the network license manager server on the same machine, license checkouts from MATLAB or MATLAB Online Server cluster might be blocked because the firewall prevents access to the license server ports.
One workaround is to modify the firewall to enable access to the network license manager ports. By default, the network license manager listens on ports 27000 and 27001. For example, to enable these ports in Ubuntu^®, use this command:
```
sudo ufw allow 27000 && sudo ufw allow 27001
```
Network license manager server listening on ports other than 27000/27001
Network policy is enabled by default for the license service, which allows communicating with network license manager only on port 27000 and with the MLM daemon on 27001. The default policy assumes that the license file has the following SERVER lines:
SERVER <HOSTID> ID=0 27000 DAEMON MLM "<FLEXLM_INSTALL_LOCATION>/etc/MLM" port=27001
If your network license manager server is listening on ports other than 27000 and the MLM daemon on other than 27001, update the license service override file and then restart the pod:
networkPolicy: enabled: true egress: ports: - port: <flexlm_port_here> protocol: "TCP"
The network license manager server is using an incorrect or an expired license file or the network license manager server is misconfigured
Check the network license manager server logs to see details about the reason for failure. Try to connect to the network license manager server from a local machine using the following commands:
```
nc -vz <FelxLM_DNS_NAME> 27000
nc -vz <FelxLM_DNS_NAME> 27001
```
In cases where the network license manager server is using an expired license file:
- Update the license file. See Update Network License in the Help Center.
- Set up both a listener on the network license manager log file and an alert, so that you are notified when the license is about to expire.
- If the license service pod is not in the Running state, see Resolve License Service Issues.
- See Resolve MATLAB Pod Issues.

License Checkout Fails Due to Missed Heartbeats

If you are running MATLAB Online Server on a cloud-managed Kubernetes^® cluster, such as through Azure^® AKS, the license service might unexpectedly exceed the license checkout count. Exceeding the license count results in failed license checkouts.

When this issue occurs, the logs for the license service contain several "missed heartbeat" warnings. A heartbeat is when the license service periodically pings the Kubernetes cluster to confirm that the licenses for the MATLAB Online Server or MATLAB instances running inside it are still checked out. If the license service misses a heartbeat, meaning it is unable to connect, the service checks out a new license. After too many missed heartbeats and license checkouts, the service exceeds the checkout count.

To see if the failed checkouts are due to missed heartbeats, run this command to check the log for the license service:

kubectl logs license-pod-name --namespace namespace

where:

namespace is the MATLAB Online Server namespace as listed in the install.config file (default = "mathworks").
license-pod-name is the name of the pod running the license service. To get the license pod name, run kubectl get pods --namespace namespace.

If the log contains repeated missed heartbeat warnings, such as the ones shown here, then the issue is caused by Kubernetes closing its connection to the license service after a period of inactivity.

00000034 YYYY-Mmm-DD 09:26:32.863576 0x00007f2c08c8v400 mos::license I Missed heartbeat: 2
00000035 YYYY-Mmm-DD 09:26:32.863640 0x00007f2c08c8v400 mos::license I Current state is: 1
00000036 YYYY-Mmm-DD 09:27:32.643434 0x00007f2c08c8v400 mos::license I Missed heartbeat: 3
00000037 YYYY-Mmm-DD 09:27:32.643488 0x00007f2c08c8v400 mos::license I Current state is: 1
00000038 YYYY-Mmm-DD 09:28:32.700154 0x00007f2c08c8v400 mos::license I Missed heartbeat: 4
00000039 YYYY-Mmm-DD 09:28:32.700234 0x00007f2c08c8v400 mos::license I Current state is: 1
00000040 YYYY-Mmm-DD 09:29:32.739331 0x00007f2c08c8v400 mos::license I Missed heartbeat: 5
00000041 YYYY-Mmm-DD 09:29:32.739374 0x00007f2c08c8v400 mos::license I Current state is: 1
00000042 YYYY-Mmm-DD 09:30:32.702587 0x00007f2c08c8v400 mos::license I Missed heartbeat: 6
00000043 YYYY-Mmm-DD 09:30:32.702636 0x00007f2c08c8v400 mos::license I Current state is: 1
00000044 YYYY-Mmm-DD 09:31:32.670748 0x00007f2c08c8v400 mos::license I Missed heartbeat: 7
00000045 YYYY-Mmm-DD 09:31:32.670799 0x00007f2c08c8v400 mos::license I Current state is: 1
00000046 YYYY-Mmm-DD 09:32:32.709340 0x00007f2c08c8v400 mos::license I Missed heartbeat: 8
00000047 YYYY-Mmm-DD 09:32:32.709388 0x00007f2c08c8v400 mos::license I Current state is: 1
00000048 YYYY-Mmm-DD 09:33:32.776049 0x00007f2c08c8v400 mos::license I Missed heartbeat: 9
00000049 YYYY-Mmm-DD 09:33:32.776103 0x00007f2c08c8v400 mos::license I Current state is: 1
00000050 YYYY-Mmm-DD 09:34:32.676145 0x00007f2c08c8v400 mos::license I Missed heartbeat: 10
00000051 YYYY-Mmm-DD 09:34:32.676193 0x00007f2c08c8v400 mos::license I Current state is: 1
00000052 YYYY-Mmm-DD 09:35:32.897212 0x00007f2c08c8v400 mos::license I Missed heartbeat: 11
00000053 YYYY-Mmm-DD 09:35:32.897262 0x00007f2c08c8v400 mos::license I Current state is: 1

To resolve this issue, specify a shorter interval between heartbeat checks.

In the license.yaml and matlab-pool.yaml override files, set the heartbeatIntervalInSeconds field. If left unspecified, the default interval is 600 seconds (10 minutes). Setting the interval to 120 seconds is sufficient to resolve the issue. The minimum interval is 60 seconds, but more frequent interval consume additional server resources.

The tables show the location of the YAML override files and the values to set.

`matlab-online-server-root/overrides/cluster/namespace/license.yaml`
flexlm: servers: "27000@flexlm.company.com" heartbeatIntervalInSeconds: 120

`matlab-online-server-root/overrides/cluster/namespace/matlab-pool.yaml`
flexlm: servers: "27000@flexlm.company.com" heartbeatIntervalInSeconds: 120

After making these changes, you must redeploy the license and MATLAB pool services. For example:

./mosadm undeploy license
./mosadm deploy license

./mosadm undeploy matlab-pool
./mosadm deploy matlab-pool

Get Help

If you are unable to resolve the issue with these resources, do the following:

Have ready the output from the following kubectl commands:

kubectl describe pod <license-pod-name> --namespace <namespace>
kubectl logs <license-pod-name> --namespace <namespace>

Contact MathWorks Support.