The following link will take you to a list of easily accessible audio and speech datasets with generous usage permissions and some helper MATLAB code: https://www.mathworks.com/help/deeplearning/ug/data-sets-for-deep-learning.html#mw_121629f8-95a4-4c9d-b100-4543b08d535a
Out of those, the Free Spoken Digit Dataset looks like a good fit for spoken digits. Nothinig equivalent comes to mind right now for isolated spoken English letters, assuming I understood correctly what you are after.
I hope this helps.
