It depends what you're optimizing. Use of the GPU only really make sense if the objective function is a large enough operation to fully utilize the GPU (e.g. it is multiplying a very large matrix by a vector, such as in iterative solvers). Use of a parallel pool with GPU computation only gains you anything if you have multiple GPUs, but is perfectly possible if you are implementing your own objective function.
Alternatively, if you are on linux, you can try using MPS to allow overlapping use of the same GPU on multiple workers. This can potentially make it viable to use the GPU with smaller operations.
