parallel.pool.DataQueue

parfor i = 1:3
    send(q, i); 
end;

     1

     2

     3

For more details on listening for data using a DataQueue, see afterEach.

When you create a parfor-loop, you offload each iteration to workers in a parallel pool. Information is only returned from the workers when the parfor-loop completes. You can use a DataQueue to update a wait bar at the end of each iteration.

When you update a wait bar with the progress of your parfor-loop, the client must record information about how many iterations remain.

The helper function parforWaitbar, defined at the end of this example, updates a wait bar. The function uses persistent to store information about the number of remaining iterations.

Use waitbar to create a wait bar, w.

w = waitbar(0,'Please wait ...');

Create a DataQueue, D. Then use afterEach to run parforWaitbar after messages are sent to the DataQueue.

% Create DataQueue and listener
D = parallel.pool.DataQueue;
afterEach(D,@parforWaitbar);

Set the number of iterations for your parfor-loop, N. Use the wait bar w and the number of iterations N to initialize the function parforWaitbar.

At the end of each iteration of the parfor-loop, the client runs parforWaitbar and incrementally updates the wait bar.

N = 100;
parforWaitbar(w,N)

The function parforWaitbar uses persistent variables to store the number of completed iterations on the client. No information is required from the workers.

Run a parfor-loop with N iterations. For this example, use pause and rand to simulate some work. After each iteration, use send to send a message to the DataQueue. When a message is sent to the DataQueue, the wait bar updates. Because no information is required from the workers, send an empty message to avoid unnecessary data transfer.

After the parfor-loop completes, use delete to close the wait bar.

parfor i = 1:N
    pause(rand)
    send(D,[]);
end

delete(w);

Define the helper function parforWaitbar. When you run parforWaitbar with two input arguments, the function initializes three persistent variables (count, h, and N). When you run parforWaitbar with one input argument, the wait bar updates.

function parforWaitbar(waitbarHandle,iterations)
    persistent count h N
    
    if nargin == 2
        % Initialize
        
        count = 0;
        h = waitbarHandle;
        N = iterations;
    else
        % Update the waitbar
        
        % Check whether the handle is a reference to a deleted object
        if isvalid(h)
            count = count + 1;
            waitbar(count / N,h);
        end
    end
end

Set Up Parallel Environment

Create a pool of parallel of thread workers by using the parpool function.

parpool("Threads");

Starting parallel pool (parpool) using the 'Threads' profile ...
Connected to parallel pool with 6 workers.

Create Parameter Grid

Define the range of parameters that you want to explore in the parameter sweep.

gridSize = 40;
sigma = linspace(5,45,gridSize);
rho = linspace(50,100,gridSize);
beta = 8/3;

Create a 2-D grid of parameters by using the meshgrid function.

[rho,sigma] = meshgrid(rho,sigma);

Perform Parallel Parameter Sweep

After you define the parameters, you can perform the parallel parameter sweep.

To visualize the interim results of the parameter sweep, create a surface plot. Note that initializing the Z component of the surface with NaN creates an empty plot.

figure;
surface = surf(rho,sigma,NaN(size(sigma)));
xlabel('\rho','Interpreter','Tex')
ylabel('\sigma','Interpreter','Tex')

To send interim data from the workers, create a DataQueue object. Set up a function that updates the surface plot each time a worker sends data by using the afterEach function. The updatePlot function is a supporting function defined at the end of the example.

Q = parallel.pool.DataQueue;
afterEach(Q,@(data) updatePlot(surface,data));

parfeval works more efficiently when you distribute the workload. To distribute the workload, group the parameters to explore into partitions. For this example, split into uniform partitions of size step by using the colon operator (:). The resulting array partitions contains the boundaries of the partitions. Note that you must add the end point of the last partition.

step = 100;
partitions = [1:step:numel(sigma),numel(sigma)+1]

partitions = 1×17

           1         101         201         301         401         501         601         701         801         901        1001        1101        1201        1301        1401        1501        1601

For best performance, try to split into partitions that are:

To represent function executions on parallel workers and hold their results, use future objects.

f(1:numel(partitions)-1) = parallel.FevalFuture;

Offload computations to parallel workers by using the parfeval function. parameterSweep is a helper function defined at the end of this script that solves the Lorenz system on a partition of the parameters to explore. It has one output argument, so you must specify 1 as the number of outputs in parfeval.

for ii = 1:numel(partitions)-1
    f(ii) = parfeval(@parameterSweep,1,partitions(ii),partitions(ii+1),sigma,rho,beta,Q);
end

parfeval does not block MATLAB, so you can continue working while computations take place. The workers compute in parallel and send intermediate results through the DataQueue as soon as they become available.

If you want to block MATLAB until parfeval completes, use the wait function on the future objects. Using the wait function is useful when subsequent code depends on the completion of parfeval.

wait(f);

After parfeval finishes the computations, wait finishes and you can execute more code. For example, plot a selection of the Lorenz system solutions. Use the fetchOutputs function to retrieve the results stored in the future objects.

results = fetchOutputs(f);
idxs = randperm(numel(results),4);
figure
for n = 1:numel(idxs)
    nexttile
    a = results{idxs(n)};
    plot3(a(:,1),a(:,2),a(:,3))
    grid on
    xlabel("x")
    ylabel("y")
    zlabel("z")
    title("Lorenz System Solution", ...
        "\rho = "+ num2str(rho(idxs(n)),'%5.2f') + " \sigma = "+ num2str(sigma(idxs(n)),'%5.2f'),Interpreter="tex")
end

If your parameter sweep needs more computational resources and you have access to a cluster, you can scale up your parfeval computations. For more information, see Scale Up from Desktop to Cluster.

Define Helper Functions

Define a helper function that solves the Lorenz system on a partition of the parameters to explore. Send intermediate results to the MATLAB client by using the send function on the DataQueue object.

function results = parameterSweep(first,last,sigma,rho,beta,Q)
    results = cell(last-first,1);
    for ii = first:last-1
        lorenzSystem = @(t,a) [sigma(ii)*(a(2) - a(1)); a(1)*(rho(ii) - a(3)) - a(2); a(1)*a(2) - beta*a(3)];
        [t,a] = ode45(lorenzSystem,[0 100],[1 1 1]);
        send(Q,[ii,a(end,3)]);
        results{ii-first+1} = a;
    end
end

Define another helper function that updates the surface plot when new data arrives.

function updatePlot(surface,data)
    surface.ZData(data(1)) = data(2);
    drawnow('limitrate');
end