How to load or display content of the file from AWS S3

Question

Mukesh Dangi 2018-6-29

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/408135-how-to-load-or-display-content-of-the-file-from-aws-s3

回答： Mukesh Dangi 2018-7-3

I have following code which load files from AWS s3 however i'm not sure how to display os access the these files one by one: -

    %Set S3 Credentials 
    setenv('AWS_ACCESS_KEY_ID', '{ID}');
    setenv('AWS_SECRET_ACCESS_KEY','{KEY}'); 
    setenv('AWS_REGION', 'us-east-1');
    %Load Data
    imds = fileDatastore('s3://{bucket}/',...
   'FileExtensions',{'.txt', '.pdf'},...
   'ReadFcn',@load,...
   'IncludeSubfolders',true);
    file=imds;
    disp(file);

output

FileDatastore with properties:
                       Files: {
                              's3://{bucket}/2017%20EBA%20-%20Relets%20(part%201)%20-%20signed.pdf'
    's3://{bucket}/AQS94%20-%20Official%20Website.pdf'
    's3://{bucket}/summary.txt'
                              }
                 UniformRead: 0
                     ReadFcn: @load
    AlternateFileSystemRoots: {}

I tried

file=imds.Files;
  disp(file);
However it's not displaying the content of the file.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Mukesh Dangi 2018-7-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/408135-how-to-load-or-display-content-of-the-file-from-aws-s3#answer_327358

在 MATLAB Online 中打开

Finally. Well following worked :

fileName = strrep(fileName,"s3://","https://s3.amazonaws.com/");
%fileName ='https://s3.amazonaws.com/{bucket}/summary.txt';
options = weboptions('ContentType','text');
data = webread(fileName, options);

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Hatem Helal 2018-7-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/408135-how-to-load-or-display-content-of-the-file-from-aws-s3#answer_327148

在 MATLAB Online 中打开

You need to call read on the datastore. Its hard to make a specific recommendation, but if your goal is to open the txt and pdf files in your S3 bucket locally: you could try using the open command instead of load as the ReadFcn of your fileDatastore. Something like the following

fds = fileDatastore('s3://{bucket}/',...
    'FileExtensions',{'.txt', '.pdf'},...
    'ReadFcn',@open,...
    'IncludeSubfolders',true);
while hasdata(fds)
    read(fds)
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Mukesh Dangi 2018-7-2

编辑：Mukesh Dangi 2018-7-2

在 MATLAB Online 中打开

Thank You for the suggestion. Unfortunately due to some reasons hasdata() is giving an error stating the provide valid input. I'm not sure whether i should use fileDataStore or not. My in objective is just to get the files from S3 and play{/r/w} with those files. The problem with my code is that it's giving me following output and i'm unable to get the file individually from it. I'm not sure whether it's list

Files: {
                              's3://{bucket}/2017%20EBA%20-%20Relets%20(part%201)%20-%20signed.pdf'
    's3://{bucket}/AQS94%20-%20Official%20Website.pdf'
    's3://{bucket}/summary.txt'
                              }
                 UniformRead: 0
                     ReadFcn: @load
    AlternateFileSystemRoots: {}

Moreover My code is able to fetch the files however unable to display desired results which is content of the file.

function handle()
  %Set S3 Credentials 
  setenv('AWS_ACCESS_KEY_ID', '{ID}');
  setenv('AWS_SECRET_ACCESS_KEY','{KEY}'); 
  setenv('AWS_REGION', 'us-east-1');
  %Load Data
  imds = fileDatastore('s3://{buket}/',...
 'FileExtensions',{'.txt'},...
 'ReadFcn',@load,...
 'IncludeSubfolders',true);
    data=imds.Files{1};
    AWSRead(data);
end
function AWSRead(fileName)
 fileID = fopen(fileName,'r');
 txt= textscan(fileID, '%s');
 fclose(fileID);
 whos txt
end
Error : Error using textscan
Invalid file identifier. Use fopen to generate a valid file identifier.