Out of memory error for big data

2 次查看（过去 30 天）

Sharif Khalil 2020-12-29

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/704492-out-of-memory-error-for-big-data

编辑： Sharif Khalil 2020-12-31

The masterDataMatrxOffset, contains 464780 rows and 256 columns of complex numbers. I use meshgrid to get 256X256 matrix of the real and imaginary and store it in a cell, but I get this error: Is there a way to overcome the out of memory?

Out of memory.

Error in meshgrid (line 62)

yy = repmat(ycol,size(xrow));

Error in auto_encoder (line 34)

[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));

More information

tic;
clear; clc; close all;
%% Load Files
load masterDataMatrxOffset;
%% Extract Parameters
iAntenna   = size(masterData,2) - 16;
data       = masterData(:,1:iAntenna);
power      = normalize(real(data),'range');
phase      = normalize(imag(data),'range');
Region     = masterData(:,261);
iReceiver  = length(data);
iRegion    = max(Region);
%% Initializations
xTrn      = cell(1,max(Region));
xTst      = cell(1,max(Region));
datamat   = cell(1,size(data,1));
%% Split into training and testing
for idx    = 1:iRegion
    IDX    = find(Region==idx);
    train  = sort(randperm(size(IDX,1),ceil(0.8*size(IDX,1))));
    trnidx = IDX(train);
    tstidx = setdiff(IDX,trnidx);
    xTrn{idx} = trnidx;
    xTst{idx} = tstidx;
end
xTrnidx = cell2mat(xTrn');
xTstidx = cell2mat(xTst');
Trainreg = zeros(numel(Region), max(Region));
Trainreg(sub2ind(size(Trainreg), 1:numel(Region), Region')) = 1;
tTrain = Trainreg';
for obsrv = 1:size(power,1)
    sprintf('observation = %d',obsrv);
    [Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
    datamat{obsrv}    = Po.^2 + Ph.^2;  
end
toc;

4 个评论
显示 2更早的评论隐藏 2更早的评论

Ive J 2020-12-29

编辑：Ive J 2020-12-29

在 MATLAB Online 中打开

You still can store them in a cell array

myCell = {};
myCell{1} = tall(1:100); myCell{2} = tall(101:200);
myCell{3} = myCell{1}.^myCell{2};
myCell =
  1×3 cell array
    {1×100 tall}    {1×100 tall}    {1×100 tall}

Sharif Khalil 2020-12-31

编辑：Sharif Khalil 2020-12-31

在 MATLAB Online 中打开

It still shows out of memory. The cell is 1X464780, but the matrix inside each cell is 256X256.

for obsrv = 1:size(power,1)
    [Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
    Talldata          = tall(Po.^2 + Ph.^2);
    datamat{obsrv}    = Talldata;  
end

请先登录，再进行评论。

请先登录，再回答此问题。