Can we manipulate a file without opening it
7 次查看(过去 30 天)
显示 更早的评论
Hello,
I have a question which I explain in bellow. Consider the following loop:
for i=1:10^6
A = Read a csv file;
A = perform some operations on A;
A= save the performed operations;
end
Apparently, the most time conssumming part is reading the file. If I use A=csvread(); then this is very time consumming. If I use fopen stuff it is
computationally cheaper but still time conssuming.
Do you have an idea to rewduce the computational time for what I intend to do?
I hope there is a way to do the above operations without actually opening any file (updating an existing file and saving the updates to the same file without opening it).
Any idea?
Thanks in advance!
Babak
8 个评论
Stephen23
2022-11-25
Don't read and write the file on every iteration. Just use an array and indexing.
回答(2 个)
Matt J
2022-11-25
编辑:Matt J
2022-11-25
If you have one single file, the reading and saving of the file should probably happen outside the loop. Use the parfor loop to loop over sections of the data and keep them in Matlab memory until you are ready to save all of the results.
A = Read a csv file;
parfor i=1:10^6
A(i,:) = perform some operations on A(i,:);
end
A= save the performed operations;
3 个评论
Matt J
2022-11-25
编辑:Matt J
2022-11-25
I don't think you should be using files to store and retrieve optimization results. I would structure the loops like this,
I=1000;
J=300;
bestValue=inf;
bestSolution=[];
for i=1:I %Loop over batches
s(1:J).Value=nan;
s(1:J).Solution=nan;
parfor j=1:J %Do a batch of optimizations in parallel
[x,fval,exitflag]=Run the optimizer
if exiflag<0 %optimization failed
continue
end
s(j).Value=fval;
s(j).Solution=x;
end
[minf,k]=min([s.Value]);
if minf<bestVal
bestVal=minf;
bestSolution=s(k).Solution;
end
end
Walter Roberson
2022-11-25
I suggest that you switch to using parfeval() . Approximately
while you haven't gotten tired of it all
while number of active workers is less than number of cores
use parfeval() to create a new worker passing in a different initial condition
end
wait for a worker to finish, using a timeout
if any worker has been active longer than you want, cancel() the worker, end
if any workers have finished, fetch their results and update the notion of best, end
end
when you get tired of it all, cancel all remaining workers
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Distributed Arrays 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!