Looks like you already found an apparent solution, but I'll go ahead and point out the things I noticed, since they may still be useful for future debugging.
First, this condition is always false:
% Check if the line needs to be extended
if any(j <= numel(app.LivePlotLine)) && isempty(app.LivePlotLine(j))
The any is unnecessary because you are comparing two scalars, but also isempty(app.LivePlotLine(j)) is always false for the following reason:
app.LivePlotLine is a vector of line objects, so if j <= numel(app.LivePlotLine) then app.LivePlotLine(j) is a scalar, (i.e., a single line object). Since a scalar is not empty by definition, that condition is always false and the if branch there (i.e., the "Use existing axes to plot" line) is never executed.
So, ignoring that line and focusing on the "First-Time Setup" line and the "Update existing line" line, the relation between app.LivePlotLine and app.SelectedChannels seems inconsistent between the two.
When the first lines are created, you create as many lines as there are selected channels:
% First-Time Setup
app.LivePlotLine = plot(app.LiveAxes, app.TimestampsFIFOBuffer, app.DataFIFOBufferch1(:,app.SelectedChannels));
so that app.LivePlotLine(1) corresponds to app.SelectedChannels(1), app.LivePlotLine(2) corresponds to app.SelectedChannels(2), and so on. For example, if app.SelectedChannels is [2,5,9], then LivePlotLine(1) has data from channel 2, LivePlotLine(2) is channel 5, and LivePlotLine(3) is channel 9. No problem (yet).
However, later, when updating those same lines (suppose app.SelectedChannels is still [2,5,9]) in the "Update existing plot" branch, j runs from 1 to numel(app.SelectedChannels), so that this:
% Update existing line
set(app.LivePlotLine(j), 'XData', app.TimestampsFIFOBuffer, 'YData', app.DataFIFOBufferch1(:, j));
would be updating app.LivePlotLine(1) from channel 1 (not channel 2), app.LivePlotLine(2) from channel 2 (not channel 5), and app.LivePlotLine(3) from channel 3 (not channel 9).
Seems like it should be the following, which would be consistent with how the lines were created in the First-Time Setup:
% Update existing line
set(app.LivePlotLine(j), 'XData', app.TimestampsFIFOBuffer, 'YData', app.DataFIFOBufferch1(:,app.SelectedChannels(j)));
More generally, I think it's worth considering a simpler code design if possible, where you create as many lines as you have channels (all channels, not selected channels) when the app starts, giving you a nice 1:1 correspondence between channels and lines. Initially the lines can be invisible and have no data. When a channel is selected, the line becomes visible; when a channel is deselected, the line becomes invisible. When you update the lines' data, you only need to update the selected channels' lines, whose index in app.LivePlotLine is the same as app.SelectedChannels, because they are 1:1 now.
