Running a large array

2 次查看(过去 30 天)
Sophie
Sophie 2024-7-22
Hi,
I am trying to load in an array (the array is 362x332x75x1032 (34.7GB)), which exceeds maximum array size preference (16.0GB). How do i go about loading it in.
It is a variable which is part of a .nc dataset

回答(2 个)

John D'Errico
John D'Errico 2024-7-22
编辑:John D'Errico 2024-7-22
Memory is cheap. Get more memory.
I'm sorry, but if you want to work with big data, you will often need sufficient capabilities to handle that data. No matter what, working with huge arrays, lacking sufficient will be slow. So your next question will be, how can I make my code run faster. Again ... get sufficient memory. Or, solve smaller problems.
  2 个评论
Sophie
Sophie 2024-7-22
This is for my masters dissertation, so unfortunantly upgrading my memory isn't an option
Walter Roberson
Walter Roberson 2024-7-22
Did the professor assign the hardware and say "you must run it on this hardware" ?
If not, then:
  • if it is your own hardware, then there is always the option of upgrading it
  • if it is university hardware, then there is always negotiating with the university to obtain an upgrade

请先登录,再进行评论。


Voss
Voss 2024-7-22
You can read and process the file in sections, one section at a time, in a loop, by specifying the start and count input arguments to ncread.
And/or specify the stride argument to read only every so many values instead of all of them.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by