This must of course fail as you are doing it, since out must generally exceed 2^53-1, the limits of flintmax for a double precision number. Remember that E1,E2,E3,x1,x2,x3 are all doubles. So when we compute
(E3 * x1) + (E1 * x2) + (E2 * x3)
It is also a double. Just because you then push it into a uint64 is not sufficient. You have already tossed too many bits into the bit bucket.
But, suppose that we force x1,x2,x3 to be uint64 in advance? That is, what if x1,x2,x3 all are created as uint64? Now products like E1*x1 will now be created directly as uint64 numbers, not temporarily as doubles. And the sum in out will now already be uint64.
x1 = uint64(1510769636);
x2 = uint64(757236075);
x3 = uint64(1466008126);
out = (E3 * x1) + (E1 * x2) + (E2 * x3);
y = mod(out,2^32)
y is the correct integer value we would expect it to have. There is no need to convert y into a uint32 though, as we effectively achieved that using the mod operation. As long as out will never overflow uint64, we have no problems.
As you can see, it does not. But can there ever be a problem? No. Since the sum E1+E2+E3 == 2^32, we need not worry about that being a problem as long as x1,x2,x3 all live inside the 32 bit limit themselves, out is just a convex linear combination of the x's, so no problem will exist.
Essentially, there is no problem, as long as you are careful in how you do the computation. Do NOT let MATLAB use doubles here though, as that will fail.