Script to access Amazon S3 files

22 次查看（过去 30 天）
André Bibaud
André Bibaud 2017-9-25
评论： Stephane 2022-12-23
I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?
Thanks

请先登录，再回答此问题。

回答（3 个）

Jhon Wine
Jhon Wine 2018-1-24
I found datastore to be very easy to use. Run the code below in each worker.
%Set S3 Credentials
setenv('AWS_ACCESS_KEY_ID', 'ID');
setenv('AWS_SECRET_ACCESS_KEY','Key');
setenv('AWS_REGION', 'us-west-2');
%Load Data
fp= 's3://mybucket/data/file.data';
ds=fileDatastore(spectralFilePath,'ReadFcn',@AWSRead);
data=ds.read;
Also define this function
function data= AWSRead(fileName)
fid = fopen(fileName);
data= fread(fid,inf,'short');
fclose(fid);
end
Make sure the EC2 cloud is on the same site as S3 to avoid data transfer bottlenecks
David Fink
David Fink 2017-9-28
Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.
webread: https://www.mathworks.com/help/matlab/ref/webread.html
webwrite: https://www.mathworks.com/help/matlab/ref/webwrite.html
  1 个评论
Jhon Wine
Jhon Wine 2018-11-17
Is there an example, how to use webread and webwrite with S3? There is no good tutorial..

请先登录，再进行评论。

Harish Sundaresh
Harish Sundaresh 2021-6-26
can MATLAB read .json files from aws s3? I get these errors
%% webread
webread(fname)
Error using webread (line 119)
The 's3' protocol specified in URL, 'url', is not supported. Specify the URL with the protocol 'http://' or 'https://'.
%% jsondecode
Error using jsondecode
JSON syntax error: expected value but found end of text.
  2 个评论
Dmitry Kaplan
Dmitry Kaplan 2022-2-8
I could not get webread to work. I am sure there is a way, but I could not figure it out.
Instead, I use something like this
fds = fileDatastore('s3://blah/blah/filename.json','FileExtensions',{'.json'},'readfcn',@fileread);
file_data = read(fds);
This defines the read function for the datastore to be fileread(). Invoking the read() actually runs fileread() and passes to it the filename -- because the fds.ReadMode is 'file'.
Stephane
Stephane 2022-12-23
Thanks for sharing this Dmitry

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Startup and Shutdown 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by