What to do when you really ARE out of memory?

Question

0 个投票

What is the solution for optimizing code when you really are just working with too large of a dataset?

Currently I need to perform triscatteredinterp using 3 vectors all (100,000,000 x 1).

scatteredInterpolant does not work any better in this instance.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

Answer 1

the cyclist 2015-8-4

编辑：the cyclist 2015-8-4

2 个投票

For very large datasets, processing a random sample of the data will often give satisfactory results.

Answer 2

Walter Roberson 2015-8-4

1 个投票

Store the data in hierarchies such as octrees that allow you to extract a subset that fits within working memory to do the fine-grained work on.

Answer 3

Robert Jenkins 2015-8-7

1 个投票

The solution I used in the end was to break up each of my vectors into manageable chunks and execute TriScatteredInterp on each of them.

The interpolant is valid on the new grid but only gives solutions inside of the block from which it came. I then simply stitched these back together.

Thank you both for your input!