slicing a char array seems to internally allocate far more memory than expected
1 次查看(过去 30 天)
显示 更早的评论
My laptop has 64G ram.
I am reading a large text file (> 8G) with s=fileread(filename).
The resulting char array is
K>> whos('s')
Name Size Bytes Class Attributes
s 1x8656516488 17313032976 char
Grabbing everything but the first, e.g. 200 characters results in this:
K>> v = s(201:end);
Requested 8656516288x1 (64.5GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive.
See array size limit or preference panel for more information.
It seems that slicing a char array allocates 8x(number of chars). Is this expected?
[[Yes, I know of plenty of workarounds -- datastores, reading it pieces, fscanfs, etc., but this code is already a workaround to deal with the glacial speed of readtable(), so, I don't have to have it working, it's just somewhat surprising]]
0 个评论
采纳的回答
Walter Roberson
2022-2-16
That indexing 201:end is not internally optimized as passing subsref 'type', '()', 'subs', {{201, ':', 8656516488}}) and expecting the internal implementation of subsref to handle the range.
What is done instead is that the colon operator 201:8656516488 is executed, producing a double precision array of length 8656516288 to hold the indices, and that array is passed to subsref. But that array of indices is 64 gigabytes...
Copy the first 200 elements out of the array, and then
s(1:200) = [] ;
which only needs creating a vector of length 200 of double precision indices.
3 个评论
Walter Roberson
2022-2-16
See https://www.mathworks.com/matlabcentral/answers/1645930-what-is-the-fastest-and-smartest-way-to-import-and-manage-plot-many-text-files-in-matlab#answer_891795 for potential speed improvements compared to readtable()
更多回答(1 个)
Sourav Karmakar
2022-2-16
Hey Dmitri,
I've tried to create a char array of size 1x8656516488, but as it exceeds the maximum array size, it throws an error message. For example,
>> s = char(1:8656516488);
Requested 8656516488x1 (64.5GB) array exceeds maximum array size preference (48.0GB). This might cause
MATLAB to become unresponsive.
As you have 64G RAM , it should produce the same error. Try creating the 's' char array in your matlab workspace and check whether you are getting the same error message or not. Because, slicing only 200 chars from the array, does not change significantly in the memory( see difference between sizes of 's' and 'v' ).
You can refer to the following document for more reference:
Hope this helps!
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Logical 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!