Reading variables from a big matlab Table
5 次查看(过去 30 天)
显示 更早的评论
Hello There we have about 45 big matlab Tables (average size 12 GByte each). Please note that thee are .mat files. We will be running a script with a for loop that will read only a couple of rows from each of theses tables and do some easy computation on the obtained data. Our problem is that on matlab r2017a takes about 25 minutes average to load each table to the matlab environment, a lot of time considering that we are just interested to extract the data of a couple of rows of those tables. As far as we understand the command matfile is not working with tables (is this correct?) to be able to extract variables from the file without loading. What are our options here? If any! Thank you in advance
1 个评论
João Nunes
2018-3-28
Hi there, I am also interested in knowing the answer to this question if anybody can help. In my project I need to handle very large tables as well but it takes too much time and I only need to acess two or three variables of the entire table. I also tried to use matfile('file_name.mat') but without sucess... Is there any other solution that I do not know about?
采纳的回答
Guillaume
2018-3-28
Yes, matfile won't help since it can't partially load a table.
Unfortunately, as you are now there is no workaround I'm aware of. You could always contact Mathworks support directly to see if they have a solution.
If the data is to be accessed more than once, then what I'd do is convert all these tables to Tall Tables, then save them with the write function. As far as I know there is no way to convert a regular mat file to one compatible with tall arrays so You'd have to take the performance hit once when you load the tables for the first time as regular table, but subsequent loads would allow you efficiently load only the data your require as you'd be using tall arrays.
2 个评论
Guillaume
2018-3-28
Thanks for the replies. I have looked on the tall tables, using the datastore command, but if I am not wrong datastrore does not deal with mat files. Only csv, image , txt, etc seems to be supported. Not mat files. Is this correct? regards
Guillaume
2018-3-28
As said, you would have to
- load your tables once, as standard tables. As far as I know there's no way around that. So this is going to take a long time.
- convert the standard table into a tall table with the tall function
- save the tall table with the write. This will save the tall array in a different kind of mat file (not suitable for load)
At this point, you can reopen the new mat files as a TallDatastore which will allow you faster access to parts of the tables. But you'll have to go through these 3 initial steps first.
更多回答(1 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Large Files and Big Data 的更多信息
产品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!