Reading data from file with breaks

4 次查看（过去 30 天）
Timothy Devenport
Timothy Devenport 2015-5-21
评论： Star Strider 2015-5-22
I'm trying to read values from some very large files and store them as a matrix. The issues is as well as having headers at the top, there are also headers repeated every 10k lines or so. For example
header 1, header 2, header 3;
1 2 3;
%(9998 rows of numbers)
4 5 6;
header 1, header 2, header 3;
7 8 9
dlmread doesn't work, and I've tried textscan and fread, which both sort of do what I want but not quite. Thanks in advance.
  2 个评论
Thorsten
Thorsten 2015-5-21
It is easier to answer such questions if you provide a sample file.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Star Strider
Star Strider 2015-5-21
The textscan function will work, but you have to use fseek to restart it each time it hits the interim text line every 10K or so lines.
This is an archived code snippet to illustrate the technique:
fidi = fopen('S21.txt');
D1 = textscan(fidi, '%f %f', 'HeaderLines',2, 'Delimiter','\n', 'CollectOutput',1);
D1{:}([1:5 end-4:end],:) % Diagnostic Look
fseek(fidi,0,0); % Position Start Of Second Part Of File
D2 = textscan(fidi, '%f %f', 'HeaderLines',2, 'Delimiter','\n', 'CollectOutput',1);
D2{:}([1:5 end-4:end],:) % Diagnostic Look
The ‘Diagnostic Look’ lines verify that the file is importing correctly. They are not important for the code and you can delete them.
  6 个评论
显示 4更早的评论
Timothy Devenport
Timothy Devenport 2015-5-22
This appears to be on the right lines, but I'm not sure it's working quite right. It seems to cut down the data files to just 30 cells in D1 and D2, and that's clearly incorrect, as there should be hundreds of cells.
I'm wondering if it's skipping the header lines, reading a line, then skipping the next lines (as if they're headers) and reading a line and so on...
Star Strider
Star Strider 2015-5-22
The ‘D1’ and ‘D3’ lines are there only to verify that the code is reading the file correctly. They display the first 5 and last 5 lines in each segment to be sure they were imported correctly. They are otherwise not part of the actual code, so you can delete them.
When I run the code I posted, then add these two lines:
SizeD1 = size(D{1}{1})
SizeD2 = size(D{2}{1})
I get:
SizeD1 =
200 3
SizeD2 =
335 3
I didn’t count the lines manually, but it seems to be working as it should, at least with the file you provided.
While I’m thinking about it, this is the way to concatenate them if you want to:
Dcat = [D{1}{1}; D{2}{1}];
producing a (535x3) double array with the file you provided.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Large Files and Big Data 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by