Best way to process large data from Dropbox
4 次查看(过去 30 天)
显示 更早的评论
I have a very large permanent Dropbox link which has 6 folders in it. Each folder has the following file organization (I specified the organization for Folder 1 only where - represents a folder and *represents a file and 3 lines with dots means continue same pattern, but all of the main folders and subfolders are organized the same).
- Folder 1
- 12390r3398
- 20253023432
* A2308432.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
- 20253023789
* A2308449.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
.
.
.
- 202530243808
* A23086903.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
- 12390r4490
- 20258900023
* A23489001.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
.
.
.
- 20258978290
* A23489876.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
.
.
.
- Folder 2
.
.
.
- Folder 3
.
.
.
- Folder 4
.
.
.
- Folder 5
.
.
.
-Folder 6
.
.
.
Now, I wrote a function that I will call myfunction. The inputs of myfunction are a .edf file and a .xlsx file. The output of myfunction is a 6 column array, and the number of rows depends on the input files.
I want to apply myfunction to all of the file pairs in each of the smallest subfolders in this Dropbox link. For example, I want to have arr1 = myfunction('A2308432.edf', 'EventList.xlsx') for the file pair located in subfolder '20253023432' above and then I want arr2 = myfunction('A2308449.edf', 'EventList.xlsx') for the file pair located in subfolder '20253023789' and so on up until the last smallest subfolder in Folder 3 (please notice that even though each subfolder contains an 'EventList.xlsx' file, that each of these Excel files are actually different despite having the same name). I cannot directly download this dropbox link, or even any of the 3/6 main folders, or even a certain subfolder of a main folder as they are too large (the entire Dropbox has about 100 GB of data).
Does anyone know how I can do this in MATLAB by calling myfunction on all of these file pairs directly from the Dropbox link, or if I can iteratively download file pairs from Dropbox and then run my function on the file pair and then delete the file pair before moving on to the next one? If you can provide code to help, I would very much appreciate as I do not have a systems background. Thanks.
4 个评论
Walter Roberson
2023-3-27
I mean writing code in MATLAB that would do the downloading.
Do you have "Dropbox Plus" ? That would allow you to install dropbox desktop to get a mountpoint such as ~/Dropbox that can be used to refer to files stored on Dropbox.
If you do not have the "Plus" subscription then Dropbox Desktop would want to copy all of the files to your local system.
回答(1 个)
Walter Roberson
2023-3-27
移动:Walter Roberson
2023-3-27
I will use MacOS / Linux syntax for referring to files for this purpose
DBdir = "~/Dropbox/Fab403/Infrared/Tests";
EL = "EventList.xlsx";
Tdir = tempname() + ".cache";
mkdir(Tdir);
edfinfo = dir(DBdir, '**/*.edf');
edfnames = fullfile({edfinfo.folder}, {edfinfo.name});
numedf = numel(edfnames);
for K = 1 : numedf
thisedf = edfnames{K};
[thisfolder, thisbase, thisext] = fileparts(thisedf);
thisEL = fullfile(thisfolder, EL);
copyfile({thisedf, thisEL}, TDir);
Tedfname = fullfile(TDir, [thisbase thisext]);
TELname = fullfile(TDir, EL);
try
myfunction(Tedfname, TELname);
end
try delete(Tedfname); end
try delete(TELname); end
end
3 个评论
Walter Roberson
2023-3-30
Sorry, I do not have Dropbox Plus to test this with (it will not work regular Dropbox), and I am not willing to pay for Dropbox Plus.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Filename Construction 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!