working with 1 TB or greater data files

Question

James Buxton 2014-11-7

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/161787-working-with-1-tb-or-greater-data-files

评论： James Buxton 2014-12-5

I would like to know what is the best way to work with very large data files that range from 1 TB to 30 TB. The files include in-phase and quadrature (IQ) data captured from a real-time signal analyzer. I am running Matlab 2014b on a Windows 8 64-bit computer with 64 GB of RAM.

I would like to be able to read in the data files to conduct basic signal processing and analysis such as FFTs and advanced routines more specific to RF analysis such as error vector magnitude, adjacent channel power ration, etc.

I am not familiar with Matlab's parallel computing capabilities or other 'bid data' capabilities such as mapreduce, memmapfile, or datastore.

Any information, feedback or suggestions as to recommended practices would be most welcome.

thanks, JimB

3 个评论
显示 1更早的评论隐藏 1更早的评论

yashwanth annapureddy 2014-11-19

Yes, it would be good to know what type of files you are dealing with. datastore and mapreduce work with tabular text and mat files of a specific format.

Please do refer to the documentation of datastore and mapreduce and let us know for any questions using them.

James Buxton 2014-12-5