Your objective function may be wrong. I did the fit with this:
% MAPPING: B(1) = a, B(2) = b
diffrax = @(B,x) (780e-9^2 * cos( (pi * B(2) .* sin(x) )/780e-9 ).^2 .* sin( (pi * B(1) .* sin(x)) /780e-9 ).^2 )./(B(1).^2 * pi^2 * sin(x).^2);
y1 = [0.2685, 1, 0.2981]; % Data to fit
y2 = [0.3339, 1, 0.3290];
y3 = [0.3039, 1, 0.3012];
y4 = [0.4269, 1, 0.4354];
y5 = [0.3232, 1, 0.3608];
y6 = [0.3582, 1, 0.4291];
y7 = [0.3767, 1, 0.4491];
y8 = [0.3186, 1, 0.4245];
D = [y1; y2; y3; y4; y5; y6; y7; y8];
D = sortrows(D, 1);
SSECF = @(B) sum((D(:,3) - diffrax(B,D(:,1))).^2);
B0 = [2E-5; 2E-5]*0.001;
[EstB, SSE] = fminsearch(SSECF, B0);
figure(1)
plot(D(:,1), D(:,3), 'bp')
hold on
plot(D(:,1), diffrax(EstB,D(:,1)), '-r')
hold off
grid
and did not get an acceptable result.
I’m not submitting this as an Answer because it isn’t one.