Yes, there can be a fair bit of overhead in parallel workers.
What you would ideally prefer to do is transfer as little data as possible with the workers, and have them do as much work as possible: you do this so that you amortize the overhead.
What you really do not want to do is transfer a lot of data and have the worker do very little: that wastes all of the time on setting up the task.

