How to open a file in the remote server, read the contents, and save it to work space

Question

Ching An Yang 2023-7-10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1994198-how-to-open-a-file-in-the-remote-server-read-the-contents-and-save-it-to-work-space

回答： Aditya 2023-11-17

Hi,

I have some large size NetCDF files store on a remote server. Those files are at least 100GB, and that is just the data for 1 day. I would need data for multiple days for some analysis. I certainly do not want to download each file to the local.

Is there any way that I can connect to the remote server from local Matlab, open these NetCDF files, grab the variables I need, and load them to the local workspace?

I have tried using

unix('ssh xxxxx@xxxx.gov')

It does connect me to the server. However, I do not know how to open the NetCDF files and grab the variables to my local workspace. Is this even doable?

Thank you very much for your help!

5 个评论
显示 3更早的评论隐藏 3更早的评论

Walter Roberson 2023-7-10

When you are working with cdf files stored on remote systems, there are three possibilities:

That the remote system might only support downloading the file; if this is your situation then you would need to copy each of the 100 gigabyte files onto your system even though you only want to access a portion of the data;
That the remote system might support basic reading (and possibly seeking) within the file; if this is your situation then there would end up being a lot of back and forth traffic but it might not be necessary to download the entire file;
That the file might support OpenDap operations; if this is your situation then the cdf library would be able to tell the remote system exactly which subset of the data you wanted to be able to access, and it would access the data on your behalf and send you only what you need. This is the best case scenario

Are you able to tell us which server you are accessing from? If it is a well-known system then we might be able to investigate.

Ching An Yang 2023-7-10

Hi Walter, thank you! I learn a lot from your comments. I am not sure which case is the server, is there a way to tell? I asked some collegues and one told me that server that requires Duo when log in would not work.

Walter Roberson 2023-7-11

I am not sure what Duo is: is that the Single Sign-On security software?

Your reference to xxxx.gov suggests a US government site. US government sites tend to support OpenDap for science data. If, however, you happen to be working with data that is restricted access (possibly classified) then I do not have any information about what various US government sites support.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Aditya 2023-11-17

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1994198-how-to-open-a-file-in-the-remote-server-read-the-contents-and-save-it-to-work-space#answer_1354437

Hi Ching,

I understand that you want to retrieve some data from the NetCDF files present on the remote server.

For this you can follow these steps:

Connect to the remote server using ‘ssh’ command via the ‘system’ function.
Navigate to the directory where the NetCDF files are located using the ‘cd’ command via the ‘system’ function.
Launch MATLAB command line interface on the remote server using the ‘matlab -nodesktop’ command via the ‘system’ function.
Use NetCDF related commands like ‘ncread’ and ‘ncinfo’ to read and retrieve information from the files.
Save the extracted variables to a file using MATLAB’s ‘save’ function.
Exit the remote MATLAB session using ‘exit’ command.
Download the extracted data file to your local maching using the ‘scp’ command via the ‘system’ function.