It can be done by manipulating undocumented properties, but there's a good chance it might cause problems, particularly when trying to save the figure. It's up to you to accept the risk of extra complications. That said, I was able to save the figure without it breaking (at least in R2019b).
% the example code
x = linspace(0, 10, 100);
y1 = sin(x);
y2 = cos(x);
plot(x, y1, 'red', x, y2, 'blue');
grid on
hold on;
h1 = plot(nan, nan, 'red', 'LineWidth', 2);
h2 = plot(nan, nan, 'blue', 'LineWidth', 2);
% choose the legend orientation as needed
hl = legend([h1, h2], 'sin(x)', 'cos(x)', 'Orientation', 'vertical');
% rotate the legend icon lines so that they're vertical
drawnow % make sure everything is up to date
nlegicons = numel(hl.EntryContainer.Children);
for k = 1:nlegicons
% this vertex data is of the form [x0 x1; y0 y1; z0 z1],
% and are in normalized coordinates with respect to the little icon box area.
hl.EntryContainer.Children(k).Icon.Transform.Children.Children.VertexData = single([0.5 0.5; 0 1; 0 0]);
end
% resize the icon column
% like Les pointed out, this is canonically accessible in newer versions,
% but i'm doing it this way since i'm using R2019b
oldtokensize = hl.ItemTokenSize;
hl.ItemTokenSize = oldtokensize.*[0.3; 1]; % reduce the width
Given that the tokensize is typically much wider than tall, this may become problematic when trying to also represent a line+marker style, especially with a heavy lineweight. I'm sure the vertical size of the legend entries could be adjusted, but it would be an extra complication. It's not directly adjustable via the tokensize parameter at least. It likely requires manipulation of the parent box and the positions of the contained objects, but I haven't dug that far yet.
