Handling memory issues by connecting to database and retrieving only required data
2 次查看(过去 30 天)
显示 更早的评论
I have an excel file as large as 20MB with roughly abt 1.5 million rows. I run out of memory while I perform any sort of operation on the data though I am able to read data completely from this huge file. As an alternative would it be a good idea to connect to a database and fetch only the data required for each of my operation (Although this could mean connecting to the database several number of times.) ?
0 个评论
采纳的回答
Walter Roberson
2011-3-7
It seems odd that you would run out of memory on basic operations on such a small file. You are only averaging about 12 characters per row, which would tend to suggest one double precision number per row, which would add up to only 12 megabytes of storage. That's relatively small.
Please check that you are not accidentally using "*" (matrix multiplication) instead of ".*" (element-by-element multiplication) in your routines.
Does even something as simple as
B = A + 0;
cause you to run out of memory? If not then just read the file once and access the sub-ranges you need. If it does run out of memory with something that simple, please use "whos" to figure out what is taking up so much memory. If you are working on the 32 bit Windows version, the command "memstats" can tell you about available memory.
0 个评论
更多回答(1 个)
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!