Hello!
I have a (tab-delimited) textfile with genetic data. The file is about 8 Gb in total and has 963 rows and about 1.5 million columns. (I have another 17Gb file that I have to tackle later...)
The format is: format = '%s %s repmat('%f ', 1, 1.5*10^6)'.
Now, I have to extract all 'float'-columns, transpose them and concatenate them with some (3) additional (string)values, and write them to a file as rows.
So, in effect, my problem is how to transpose a large text-file dataset.
Obviously I dont want to import the whole file into a cell-array, so I have been trying to do it column by column (or row by row in the outfile).
More specifically, I have been trying (for the first float-column):
col = textscan(fid, '%*s %*s %f %*[^\n]', 'bufsize', large_value);
and then repeat this in a for-loop, with different FORMAT for each pass.
This works, of course, but textscan still has to read the whole file every round in the loop (i think, hence the required BUFSIZE). So it takes a VERY long time.
One textscan takes approximately 100 seconds, so 100s * 1.5 million = 4.5 years.
PLEASE HELP! Is there any way to make this fast?
Do I HAVE to load full file into a cell-array?
Or is there maybe some way to WRITE COLUMNS to a textfile(?). That would do the trick, I think.
Maybe someone have a cool Pearl-script that take less time to extract the column?
Cheers! / Erik
