kmeans for dataset that's too big for memory

3 次查看(过去 30 天)
Hi,
I'd like to do kmeans for a dataset that is too big to be loaded in memory. What options do I have? I can load a part of the dataset to workspace by using MATFILE but general kmeans function requires loading entire dataset, which is not possible for my computer. Please help.

回答(1 个)

Jan
Jan 2013-10-7
编辑:Jan 2013-10-7
The obvious strategy is to install more RAM.
If you start with the claim, that the function you want to use requires the complete data set to be loaded at once, there is no way to escape from the need to load the complete data set at once. If this is not possible for your computer, upgrade it or run the code on a more powerful computer.
Sorry, I know that this answer is trivial. But it is the only possible answer matching your question exactly. Of course you could apply a data compression in the RAM or condense the data at first. But this is not exactly what you are asking for, and in addition, it would be much more expensive most likely.
  3 个评论
Jan
Jan 2013-10-7
And does "Big data" mean that it is more than matches into the RAM? How big are your data?
Jeff
Jeff 2013-10-8
Yes, big data here refers to dataset that's too large for the memory. The dataset I'm using is about 90GB in size.

请先登录,再进行评论。

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by