Running out of memory

3 次查看(过去 30 天)
Theo Score
Theo Score 2017-2-6
Hi All,
I have a couple of data text files which I am extracting data of interest from.
The total text data files size is about 549 MB, which I am writing into a single text file and it comes to about 500 MB since I am disregarding some information in the text files.
When I try to write to read and write the output text file into an excel file;
1. The process consumes all my RAM (8G) and computer freezes
2. I got this error at the end
Error using xlswrite (line 219)
The specified data range is invalid or too large to write to the specified file format. Try writing to an XLSX file and use Excel A1
notation for the range argument, for example, A1:D4.
Error in Data_Overall_Ext_Dir (line 29)
xlswrite (output, data);
Is there any way I can handle this?
With limited number of files, the script works well but as the number of files (data size) increases, I have this issue.
Kindly help.
  7 个评论
Theo Score
Theo Score 2017-2-8
编辑:Theo Score 2017-2-8
Hi Walter,
Thank you for your response. I thought of a work around the problem. I will use Linux sort/ split commands in Linux but I need to get an extra component to my data. This script:
workDir = 'C:\Users\Theo_Score\Desktop\COM_CFDEK';
data = fullfile(workDir, '*.data');
files = dir(data);
fileout = fullfile(workDir, 'Merged.txt');
fout = fopen (fileout, 'w');
for k = 1000:1000:100000
cntfiles = fullfile (workDir, ['dump' num2str(k) '.data']);
fin = fopen (cntfiles);
linecount =0;
while ~feof(fin)
linecount = linecount + 1;
tline = fgetl(fin);
if linecount >= 10 % Number of header lines being skipped
fprintf (fout, '%s\r\n', tline);
end
end
fclose(fin);
end
Was skipping 9 lines of the .data files. I need some help to modify this script so that it reads line 2 (which is time step, say 1000) as shown in the first 9 lines I was previously skipping.
ITEM: TIMESTEP
1000
ITEM: NUMBER OF ATOMS
26
ITEM: BOX BOUNDS ff ff ff
-0.1 0.1
0 0.4
-0.1 0.1
id type x y z ix iy iz vx vy vz fx fy fz omegax omegay omegaz radius
I want to multiply 1000*0.00001 and return the value in the first column and the previous first column becomes the second column.
The first 5 lines from the previous script would be like
1 1 -0.0506691 0.301248 -0.0540098 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
7 1 0.0119942 0.300662 -0.0584242 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
6 1 0.0589997 0.30511 -0.0540171 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
3 1 -0.0512266 0.330591 -0.0441473 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
16 2 -0.0118166 0.320646 -0.046286 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
After the modification, I would want an output like
0.01 1 1 -0.0506691 0.301248 -0.0540098 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 7 1 0.0119942 0.300662 -0.0584242 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 6 1 0.0589997 0.30511 -0.0540171 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 3 1 -0.0512266 0.330591 -0.0441473 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
0.01 16 2 -0.0118166 0.320646 -0.046286 0 0 0 0 -1 -0 0 0 0 0 0 0 0.015 0
I would appreciate help on this.
Walter Roberson
Walter Roberson 2017-2-8
I would probably toss it all into perl...

请先登录,再进行评论。

回答(1 个)

cr
cr 2017-2-6
If xls format is not critical requirement, you may want to use fileIO operations and produce a csv (or some text delimited) format. Once a file text file is successfully generated it can be converted to xls using MS Excel.
  1 个评论
Theo Score
Theo Score 2017-2-7
编辑:Theo Score 2017-2-7
Hi cr,
I have embedded the script on this thread on a response to Walter.
Essentially, I need to plot graphs with the information I get so I need the data to be in .xlsx. Further comment is also given in that response.
I appreciate your further help.
Best regards.

请先登录,再进行评论。

标签

尚未输入任何标签。

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by