Too much padding can have negative impact on the network performance. To prevent the training process from adding too much padding, you can sort the training data by sequence length, and choose a mini-batch size such that the sequences in a mini-batch have similar length.
For more information refer to the following link:
