This is due to the asynchronous nature of gpuArray operations. Where possible, operations on the GPU are queued up and run in the background. At some point, we run out of queue, and the operations have to actually run - that's the point at which you see the change in time.
The best way to time operations on the GPU is to use gputimeit which takes into account this asynchronous behaviour. The other option is to call wait on the gpuDevice before calling toc, like this:
d = gpuDevice();
t = tic();
% do stuff on the GPU
wait(d);
time = toc(t)
