import over 1000 files

Question

Jeong Dae Kim 2021-7-31

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/889737-import-over-1000-files

评论： Rik 2023-12-15

采纳的回答： Paul Kaufmann

i want to change many files including files in subfolders.

but if file outnumber 1001, my code stops.

i'd like to import over 1000 files to "files" struct.

2 个评论
显示无隐藏无

Rik 2021-7-31

What code are you using to select the files?

Jeong Dae Kim 2021-7-31

@Rik

i edited my Question. take a look

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Paul Kaufmann 2021-8-4

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/889737-import-over-1000-files#answer_760837

在 MATLAB Online 中打开

Don't know what the problem is with exceeding N=1000. Works for me:

path = "data/"
files = [ dir(fullfile(path,'**','*.txt')); dir(fullfile(path,'**','*.txt')) ]; % do it twice, to exceed 1000
N = numel(files)

Gives this result for my datafolder:

ans = 1890

As I assume you want to rename files or something, you can now just iteratively work on them.

However the following will get you into trouble:

>> numel(files.name)
Warning: Number of elements exceeds maximum flint 2^53-1.
The result may be inaccurate. 
ans =
   Inf

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Chunru 2021-7-31

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/889737-import-over-1000-files#answer_757927

You may hit the limit os the number of files that can be opened simutaneously. While it possible to increase the maximum number of opened files in OS, you may want to try open file one by one and then close the file once it is not used.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 3

Image Analyst 2021-7-31

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/889737-import-over-1000-files#answer_757937

在 MATLAB Online 中打开

Try code in the FAQ: https://matlab.fandom.com/wiki/FAQ#How_can_I_process_a_sequence_of_files.3F

It will work for more than 1000 files, if you have more than that.

% Specify the folder where the files live.
myFolder = 'G:\';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isfolder(myFolder)
	errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
	uiwait(warndlg(errorMessage));
	myFolder = uigetdir(); % Ask for a new one.
	if myFolder == 0
		% User clicked Cancel
		return;
	end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*.png'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
numFiles = length(theFiles)
Result = zeros(numFiles, 1);
for k = 1 : numFiles
	baseFileName = theFiles(k).name;
	fullFileName = fullfile(theFiles(k).folder, baseFileName);
	fprintf('Now reading %s, file #%d of %d.\n', fullFileName, k, numFiles);
	% Now do whatever you want with this file name,
	% such as reading it in as an image array with imread()
	imageArray = imread(fullFileName);
	imshow(imageArray);  % Display image.
	drawnow; % Force display to update immediately.
	Result(k) = mean(imageArray(:)); % Whatever you want.
end

14 个评论
显示 12更早的评论隐藏 12更早的评论

Stephen23 2021-7-31

编辑：Stephen23 2021-7-31

在 MATLAB Online 中打开

@Jeong Dae Kim: you request a huge array, much larger than any normal computer could hold in memory:

Result = zeros(1:numFiles);

This requests an array of size 1x2x3x4x5...xnumFiles. For 1000 files this array will contain exactly 402387260077093773543702433923003985719374864210714632543799910429938512398629020592044208486969404800479988610197196058631666872994808558901323829669944590997424504087073759918823627727188732519779505950995276120874975462497043601418278094646496291056393887437886487337119181045825783647849977012476632889835955735432513185323958463075557409114262417474349347553428646576611667797396668820291207379143853719588249808126867838374559731746136085379534524221586593201928090878297308431392844403281231558611036976801357304216168747609675871348312025478589320767169132448426236131412508780208000261683151027341827977704784635868170164365024153691398281264810213092761244896359928705114964975419909342221566832572080821333186116811553615836546984046708975602900950537616475847728421889679646244945160765353408198901385442487984959953319101723355556602139450399736280750137837615307127761926849034352625200015888535147331611702103968175921510907788019393178114194545257223865541461062892187960223838971476088506276862967146674697562911234082439208160153780889893964518263243671616762179168909779911903754031274622289988005195444414282012187361745992642956581746628302955570299024324153181617210465832036786906117260158783520751516284225540265170483304226143974286933061690897968482590125458327168226458066526769958652682272807075781391858178889652208164348344825993266043367660176999612831860788386150279465955131156552036093988180612138558600301435694527224206344631797460594682573103790084024432438465657245014402821885252470935190620929023136493273497565513958720559654228749774011413346962715422845862377387538230483865688976461927383814900140767310446640259899490222221765904339901886018566526485061799702356193897017860040811889729918311021171229845901641921068884387121855646124960798722908519296819372388642614839657382291123125024186649353143970137428531926649875337218940694281434118520158014123344828015051399694290153483077644569099073152433278288269864602789864321139083506217095002597389863554277196742822248757586765752344220207573630569498825087968928162753848863396909959826280956121450994871701244516461260379029309120889086942028510640182154399457156805941872748998094254742173582401063677404595741785160829230135358081840096996372524230560855903700624271243416909004153690105933983835777939410970027753472000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 elements (approx. 1e2567). Multiply that by 8 to get the required number of bytes.

Does your computer have that much memory? (hint: the observable universe has around 3.28e80 particles)

Image Analyst 2021-8-1

在 MATLAB Online 中打开

I believe this should work to find all .mp4 and .mpg files in the folder plus subfolders. Make sure to assign the desired starting folder, myFolder. If it doesn't work, attach your m-file.

% Specify the folder where the files live.
myFolder = pwd; %'G:\';
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isfolder(myFolder)
	errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
	uiwait(warndlg(errorMessage));
	myFolder = uigetdir(); % Ask for a new one.
	if myFolder == 0
		% User clicked Cancel
		return;
	end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '**/*.mp4') % Change to whatever pattern you need.
pngFiles = dir(filePattern);
filePattern = fullfile(myFolder, '**/*.mpg') % Change to whatever pattern you need.
bmpFiles = dir(filePattern);
theFiles = [pngFiles; bmpFiles];
numFiles = length(theFiles)
Result = zeros(numFiles, 1);
for k = 1 : numFiles
	baseFileName = theFiles(k).name;
	fullFileName = fullfile(theFiles(k).folder, baseFileName);
	fprintf('Now reading %s, file #%d of %d.\n', fullFileName, k, numFiles);
	% Now do whatever you want with this file name,
	% such as reading it in as an image array with imread()
	Result(k) = whatever; % Whatever you want.
end

Jeong Dae Kim 2021-8-2

not working. that is all of my code

Rik 2021-8-2

Please show all your code, exactly as you try to use it.

请先登录，再进行评论。

Answer 4

michael adams 2023-12-14

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/889737-import-over-1000-files#answer_1371803

I can confirm this is an issue and it is not consistent. If I query many directories (all with more than 1000 files), sometimes 'dir' returns the correct structure of all files, but sometimes it just stops at 1000.

I do not run into this issue on local drives. I see this issue on network drives only.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Rik 2023-12-15

How did you confirm this is an issue with Matlab and not with the network architecture and/or the OSes involved?

请先登录，再进行评论。

import over 1000 files

2 个评论
显示无隐藏无

采纳的回答

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

更多回答（3 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

14 个评论
显示 12更早的评论隐藏 12更早的评论

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

另请参阅

类别

标签

产品

Community Treasure Hunt

import over 1000 files

2 个评论 显示 无隐藏 无

采纳的回答

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

更多回答（3 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

14 个评论 显示 12更早的评论隐藏 12更早的评论

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

另请参阅

类别

标签

产品

Community Treasure Hunt

2 个评论
显示无隐藏无

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

14 个评论
显示 12更早的评论隐藏 12更早的评论

1 个评论
显示 -1更早的评论隐藏 -1更早的评论