In framing my question, I realised what I should have been looking for. By placing an ampersand after the command, the system command does not wait. I then studied the command format for Windows cmd, and discovered you can run consecutive commands. So the following code now runs the installer, followed by restarting the app (assumes it's installed in the same location as the original), while closing the original instance:
% find path of executable
if isdeployed % Stand-alone mode.
[~, result] = system('set PATH');
ExecutablePath = char(regexpi(result, 'Path=(.*?);', 'tokens', 'once'));
else % Running from MATLAB.
ExecutablePath=pwd;
end
% launch the installer, and then the newly installed GUI, without waiting
system(['GUIInstaller, ' && "', ExecutablePath,'GUIExecutable"',' &']);
closereq();
The code for finding the path of the executable so it can be relaunched came from: