My questions lie in, what is the exact sequence of when licenses are reserved and when the job gets scheduled and when the job runs?
- From my testing, it SEEMS like the PBS job won't be scheduled until there are enough MATLAB_Distrib_Comp_Engine available, is that corract?
We'd have to see your PBS job script, but likely, PBS doesn't know that you need any MATLAB_Distrib_Comp_Engine licenses and will run once there are enough resources (cores, memory, etc.).
- While my matlab command is waiting for MATLAB_Distrib_Comp_Engine seats to become available, is it using up one of the main MATLAB licenses?
MATLAB submits the job to PBS via parpool (synchronous) or batch (asynchronou). If your code calls parpool, then yes, you'll continue to checkout a MATLAB license. If you're using batch, MATLAB could possibly finishing and therefore return the MATLAB license before the whatever batch calls is finished.
- What about while the job is scheduled through PBS, but before the nodes get allocated?
It's unclear which job you mean here. The one you submitted, or the one MATLAB submitted. When you submit the job, MATLAB will checkout a license once PBS launches MATLAB. The MATLAB_Distrib_Comp_Engine license(s) will get checked out once the job that parpool/batch starts.
- What about after the nodes are allocated and the function Hello is running on them--the worker nodes "inherit" the client license when they're launched, does that mean that start taking a main license seat at that point?
The workers dynamically unlock toolboxes (slightly different than checking out in the licenses traditional sense) that the user is entitled to, based on the MATLAB license used to submit the job to begin with. The workers don't take any licenses (other than MATLAB_Distrib_Comp_Engine)
- If no seats are available when nodes are allocated, will the workers fail?
Yes.
