Have you tried using the performance analysis tool in the MATLAB Editor to identify where the most time is being spent and this can help identify areas for improvement? How large a value of N are you trying to reach?
If you have access for parfor in the Parallel Computing Toolbox and a multi core machine that may help a lot.