Unable to read a huge XML or text file

Question

JFz 2017-7-24

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/350006-unable-to-read-a-huge-xml-or-text-file

评论： Santa Raghavan 2017-7-27

Hi,

I have a XML of 2GB in size. I keep getting java heap memory error when loading it. So I am thinking of reading it in as a text file and remove many useless rows in that file before saving it into a new and smaller file.

How to do that? I cannot even read it with textpad. Thanks!

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Stephen23 2017-7-24

Try Notepad++.

Or process any large text file in blocks.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Santa Raghavan 2017-7-26

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/350006-unable-to-read-a-huge-xml-or-text-file#answer_275685

编辑：Santa Raghavan 2017-7-26

在 MATLAB Online 中打开

The amount of Java Heap memory available to MATLAB can be increased and this can be done in the following way:

In the MATLAB Desktop Window:

For versions of MATLAB R2010a and above, use - File -> Preferences -> General -> Java Heap Memory. Move the slider to adjust the allocated heap memory.

For versions of MATLAB prior to R2010a, refer to the link below-

Increase head space for 10a and earlier

If that does not work, you can read it in as a text file using the textscan function by specifying the block size you wish to read at a time.

fileID = fopen('bigfile.txt');
formatSpec = '%s %f %*f %*f %s';

Read a block of data in the file. Use the HeaderLines name-value pair argument to instruct textscan to skip two lines before reading data.

D = textscan(fileID,formatSpec,'HeaderLines',2,'Delimiter','\t')

Refer for more info: Import large text files

2 个评论
显示无隐藏无

JFz 2017-7-27

Thank! I will try it. I have increased the java heap memory to the maximum but still got the same error.

Santa Raghavan 2017-7-27

在 MATLAB Online 中打开

You can also try the datastore function that lets you read files that dont fit into the memory.

ds = datastore('Myfile.xml', ...
                   'TreatAsMissing','NA')
ds.ReadSize = 100; % Specifies the number of lines 
                   % you want to read at a time.
read(ds) % Reads first 100 lines in file
read(ds) % Reads next 100 lines in file