Downloading a zipped file from S3 via AWS

Question

John Spicer 2019-7-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472346-downloading-a-zipped-file-from-s3-via-aws

回答： Jingyu Si 2020-12-4

We're trying to download files (one at a time) from AWS S3.

I've used the example code I found on the web to get the list of files (works great), and I am trying to use this code to fetch the file:

fp= ['s3://ourBucketGoesHere/‘ selectedFile];

ds=fileDatastore(fp,'ReadFcn',@AWSRead, 'FileExtensions', {'.zip','','.txt'});

mydata=ds.read;

saveName = [filename ext];

save (saveName, 'mydata');

(also use this function for actual fetching) ->

function data = AWSRead(fileName)

fid = fopen(fileName);

data= fread(fid,inf);

fclose(fid);

end

Which does fetch the file.

The problem is that I can't save it in a format recognized. It should be gzip format (as it is coming from the server) but I can't make it work.

It's probably a setting I have wrong, I expect. Searching the net didn't find anything.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jingyu Si 2020-12-4

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/472346-downloading-a-zipped-file-from-s3-via-aws#answer_564923

Try this funciton

ds=fileDatastore(fp,'ReadFcn',@S3_download);
dataFile=read(ds);

function dataFile=S3_download(readFileName)
writeFileName=regexp(readFileName,'\','split');
writeFileName=writeFileName{end}
readFileId = fopen(readFileName, 'r'); 
assert(readFileId > 0);
writeFileId = fopen(writeFileName, 'w');
assert(writeFileId > 0);
%%
buffersize = 1024;
while ~feof(readFileId)
    fileData = fread(readFileId, buffersize, '*uint8');
    writeCount = fwrite(writeFileId, fileData, 'uint8');
end
dataFile= writeFileName; 
end