big dataset: how to find specific data that would require taking the mean?

Question

Maitiumc 2017-4-21

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/336581-big-dataset-how-to-find-specific-data-that-would-require-taking-the-mean

评论： Nikhil Sreekumar 2017-4-24

I have a csv file, millions of rows. It consists of time series data of a few thousand individual households. I need to find various contrasting household consumption profiles. Is there a way I can somehow find the type of profiles I am after? Trying to simply import the dataset is not an option, it is too large.

To look at the profiles I would probably have to put them into an hourly average for the year (ie. average for all the 0000, 0001, ..., 2300hrs) of the year and see how the consumption profiles looks. Though this isn't necessarily essential so long as I can see there is a general trend in the profile for a particular house.

2 个评论
显示无隐藏无

dpb 2017-4-21

Have you actually tried to import the data? What is the file organization; doesn't seem to match up unless this is something like several years' electric consumption on a minute-by-minute or even finer resolution basis, maybe? There are facilities in Matlab for "tall" arrays beginning in R2016b but there are other facilities prior to that which can be used effectively as well.

Certainly agglomeration is helpful but need a little more detail on what really intend to try to compute.

Nikhil Sreekumar 2017-4-24

Hi Maitiumc,

There are certain functionalities available in MATLAB for processing very large data. But certain of those functionalities have been introduced recently. If you want to have a quick look on various ways in MATLAB to handle very large data, please have a look into the below link:

Large Files and Big Data

Also regarding the computation of hourly average a year, you may have to group them and do a grpstats on the data. An example of Statistical and Machine Learning is provided in the following link: