Cannot import large .dat files
显示 更早的评论
Dear All,
I have some huge data files, containing around 15000000x2 matrices in seperate files. That is, one of my files contains a 15000000x2 matrix. When I import it into Matlab using importdata, only a fraction of it is imported. It seems there is a problem with the memory, but could anyone can help me to solve this?
I am using Dell latitude with 8GB memory, core i5.
Cheers
2 个评论
per isakson
2014-2-19
>> 15000000*2*8/1e6
ans =
240
That is 0.24GB, which shouldn't be a problem. At least not with a 64bit Matlab.
"only a fraction of it is imported" make me believe there is a problem with the file.
You must provide more detail. Error message. The ten first lines of the file.
Batuhan
2014-3-4
采纳的回答
per isakson
2014-3-5
编辑：per isakson
2014-3-5
0 个投票
Comments:
- "[...]can open it with vim." That is not a sufficient requirement! There could be a special string indicating a missing numerical value in a line in the middle of the file. There could be a spurious non-printing character somewhere. Possibly, vim understands more text encodings than does importdata of Matlab.
- AFAIK: importdata analyzes (auto-magically) the beginning of the file to find out the format and delegates to an appropriate function. There might be a line which don't honor the format of the lines in the beginning.
- If you know the format it's better to use textscan.
- Wasn't there an error message?
- You must provide more detail. Error message. The ten first lines of the file.
3 个评论
Batuhan
2014-3-5
per isakson
2014-3-5
编辑：per isakson
2014-3-5
Strange! I don't know. Guessing:
- Something happened when you copied the file to the laptop. Did you check the size of the file on the different computers?
- AFAIK: there is no default value or preference, which could cause this.
- With importdata one may set "Range:". Did you by mistake use an old value?
However, there are many Matlab function, which should read that file, e.g. textscan and load. (The flexibility of textscan is not needed.)
>> fid = fopen( 'cssm.txt', 'r' ); % see doc for different encodings
>> buf = textscan( fid, '%f%f', inf, 'CollectOutput', true );
>> fclose(fid);
>> M = load( 'cssm.txt', '-ascii' );
where cssm.txt contains the data from your comment (copy&paste).
You might test your file for spurious characters. Read as characters and search for characters other than, digit, dot, "white space", plus and minus. Empty cac indicates that none was found.
>> str = fileread( 'cssm.txt' );
>> cac = regexp( str, '[^\d\.\s\+\-]', 'match' );
>> whos
Name Size Bytes Class Attributes
M 30x2 480 double
ans 1x1 8 double
buf 1x1 592 cell
cac 0x0 0 cell
fid 1x1 8 double
str 1x807 1614 char
Batuhan
2014-3-7
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Large Files and Big Data 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
