Storing 3 large data sets in one workspace

Question

Mark Angelo 2024-10-10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2158820-storing-3-large-data-sets-in-one-workspace

回答： Aastha 2024-10-11

Hello i have 3 data sets from NESO for System Inertia, System Frequency and Historic generation mix and carbon intensity.

They are quite large in size, just wondering how to put into one workspace for data analysis.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Aquatris 2024-10-10

what is the problem you encounter when you try to import all 3 datasets to your workspace?

Unless you have memory limitations, this should not be a problem

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Aastha 2024-10-11

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2158820-storing-3-large-data-sets-in-one-workspace#answer_1530615

在 MATLAB Online 中打开

Hi Mark,

I understand that you are working with three large datasets, and you are looking for a way to load them into the workspace while managing memory constraints.

You can use "datastore" function in MATLAB, which is specifically designed for handling collections of data that are too large to fit into memory.

Kindly refer to the example mentioned below illustrating the use of "datastore" function for large datasets:

1. Configure the datastore function using the MATLAB code given below:

ds = tabularTextDatastore("large_file.csv"); % Create a datastore for tabular text data 
ds.SelectedVariableNames = {'Column_Name_1', 'Column_Name_2'}; % Specify columns to be read from the file 
ds.TreatAsMissing = 'NA'; % Replace missing data with NA 
ds.ReadSize = 15000; % Define the number of rows to be read at once based on memory constraints 

2. Use a while loop along with a "hasdata" function to determine if there is data available to read the data from the "datastore" function as shown in the MATLAB code below:

reset(ds); % reset the datastore to the initial state 
while hasdata(ds) 
    T = read(ds); % Read data from the datastore 
    % Process the intermediate data 
    % Insert your processing code here 
end 

This method allows you to handle datasets too large to fit into memory efficiently. Additionally, you can explore other methods for managing large datasets by referring to the link given below:

https://www.mathworks.com/help/matlab/large-files-and-big-data.html?s_tid=CRUX_lftnav

For more information on the "datastore" and "hasdata" function, kindly refer to the link of MathWorks documentation mentioned below:

-datastore: 

https://www.mathworks.com/help/matlab/datastore.html

-hasdata: 

https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.hasdata.html

I hope this is helpful!