There is no maximum number. Or, there is, but it is purely dependent on how much RAM you have, and the time you are willing to spend.
Looking at your question, I see you have 5*n^2 equations. With n=61,
this will involve creating and solving a system of equations where the matrices will take 2.5 gigabytes of RAM. And no matter what, you always need to accept that at least double that memory will be used, sometimes a factor of 3 is safer. So I would expect that 7.5 GB of RAM will be necessary to solve the smaller problem. But MATLAB and your OS will get in the way too. They use RAM too. So 10-12 GB would have been sufficient with no worries for the smaller problem.
But when n = 91,
Now we see close to 13 gigabytes of RAM necessry, just to store the basic system of equations formed. Double or triple that, and now you run out of room when you have only 40GB. It may have been close. I'd bet you would have not seen a problem had you tried n = 81 with the memory you have. 91 was just a bit too much though.
If your system is sparse, then ABSOLUTELY you need to make the problem a sparse one. The problem internally involves solving linear systems of equations. But if the equations are full, the solve takes a huge amount of memory and time. This is where the problem arises. So forcing them to be sparse helps a great deal.
You can enforce that using the JPattern option. This allows you to specify the sparsity pattern of the Jacobian matrix. And if your problem is that large, then it must surely have a sparsity pattern.
Next, I checked the docs to see which ODE solvers allow you to specify the Jpattern option. They are ODE15s, ODE23s, ODE23t, ODE23tb, ODE15i.
ODE45 is NOT one of the solvers that uses the JPattern option.
Of course, the simple alternative is just more RAM, which is cheap these days.
