In a matrix, all columns are the same length. That's one of the defining characteristics of matrices. You're not going to be able to have a matrix whose columns are different lengths.
If you want to store column vectors of different lengths in a matrix, then you'll need to use NaNs or some other placeholder value to fill out the columns and make them all the same length in the matrix.
If you want to store column vectors of different lengths without any placeholder values, then store them in a cell array, exactly as you already have in your all_azimuths variable.