How to organize a datastore with multiple devices for multiple days?

Question

Magsud Hasanov 2022-5-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1724795-how-to-organize-a-datastore-with-multiple-devices-for-multiple-days

回答： Magsud Hasanov 2022-6-14

So, I have a data from 75 smart meters for range of 17 days.

Each day all 75 smart meters did some measurements.

How it is better to organize the data? I really like the idea of datastore() in Matlab. But don't know how to do it automatically for so many variables(tables)

If I create a timetable for each day of each smart meter, it would require me to create 75*17 =1275 separate tables.

And also I would like to analyze each day (with all smart meter measurements combined) so it will make the total number 1275 +17 = 1292.

Can you suggest a good option of organizing such big amount of data?

I am analyzing it for load forecasting using Machine Learning.

Thank you very much in advance!

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

dpb 2022-5-22

That's really not much data at all in terms of observations -- unless there are gigabytes of data which it wouldn't seem there could be but a few measurements from each device.

I'd just collect the whole dataset in memory in a timetable and use retime and similar functions to do operations over the data. memapfile is just going to get in the way here.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Magsud Hasanov 2022-6-14

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1724795-how-to-organize-a-datastore-with-multiple-devices-for-multiple-days#answer_985530

I just used array with all paths to all files and made datastore(), then did readall()

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Seth Furman 2022-5-30

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1724795-how-to-organize-a-datastore-with-multiple-devices-for-multiple-days#answer_974570

Provided your data are already stored in files on disk in a common format, you can create a tall timetable and perform operations on it as if it were a large timetable.

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Magsud Hasanov 2022-6-13

Hello, after making path into all files, I just used datastore() and then readall(). It created the big dataset. The amount of data was enough small to use readall(). Thank you very much for your time!

请先登录，再进行评论。