Profile and debug load function

Question

Jan Kappen 2024-2-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2083448-profile-and-debug-load-function

评论： Jan Kappen 2024-2-27

I have a .mat file containing an object of a custom Matlab class that mainly consists of a big data table of type table, stored in one of the class' properties. The table contains ~20k rows and contains datetime, double, categorical arrays, and objects of class idnlarx, i.e. estimated nonlinear nlarx models.

The mat file is 520 MB in size (mat file v7 with enabled compression, v7.3 makes the size 10x) and takes between 240 and 290 seconds to load.

I'm trying to speed up the loading process which is clearly CPU bound. I think the problem is the idnlarx model which utilizes most of the space (> 1 GB when loaded to RAM and instantiated).

And I'd like to understand what takes so much time to load, but the profiler only shows this:

...

That means, 94% are self-time of the load function.

How do I find out what takes so long?

Thanks,

Jan

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Matt J 2024-2-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2083448-profile-and-debug-load-function#answer_1411083

编辑：Matt J 2024-2-18

You could pursue your hypothesis by extracting the table from an object, then saving the idnlarx column to its own .mat file. Then, you could check how much time it takes to load back in. I suspect your hypothesis is correct, however, because I routinely load .mat files that are ~500 MB and, while it takes some time, it is nowhere near a minute long, let alone 4 minutes.

8 个评论
显示 6更早的评论隐藏 6更早的评论

Jan Kappen 2024-2-18

移动：Matt J 2024-2-18

在 MATLAB Online 中打开

I made some tests, and both, idnlarx (which does not support putting it into an array, just into a cell), and some of my custom classes (which support collecting them in an array), have significant impacts on load and save times.

I understand Matlab is not that good at looping and instantiating lots objects, I might have to live with some overhead, and the idnlarx class is quite old and complex.

However, even this simple example here causes issues and shows a big gap in the load profile:

testClassBenchmark.m

%% benchmark test class
N = 100000;
tsc = testScalarClass.empty(N,0);
for k = 1:N
    tsc(k) = testScalarClass(k, "myName" + k);
end
tac = testArrayClass(1:N, "myName" + string(1:N));
%% save
tic
save("tsc.mat", "tsc")
save("tac.mat", "tac")
toc
%% load benchmark
useProfiler = true;
val = loadData("tsc");
assert(isequal(val,tsc));
val = loadData("tac");
assert(isequal(val,tac));
if useProfiler
    profile on
end
meadianTimeTsc = timeit(@()loadData("tsc"),1)
meadianTimeTsa = timeit(@()loadData("tac"),1)
if useProfiler
    profview
end
%% load fcn
function val = loadData(fn)
    val = load(fn + ".mat").(fn);
end

testScalarClass.m

classdef testScalarClass
    properties
        InstanceNum (1,1) double = nan
        InstanceName (1,1) string = "default"
    end
    methods
        function obj = testScalarClass(num, name)
            arguments
                num double = nan
                name string = "default"
            end
            obj.InstanceNum = num;
            obj.InstanceName = name;
        end
    end
    methods (Static)
        function obj = loadobj(s)
            if isstruct(s)
                disp("compatibility issues!")
            else
                obj = s;
            end
        end
    end % static methods
end

testArrayClass.m

classdef testArrayClass
    properties
        InstanceNum (:,1) double = nan
        InstanceName (:,1) string = "default"
    end
    methods
        function obj = testArrayClass(num, name)
            arguments
                num double = nan
                name string = "default"
            end
            obj.InstanceNum = num;
            obj.InstanceName = name;
        end
    end
    methods (Static)
        function obj = loadobj(s)
            if isstruct(s)
                disp("compatibility issues!")
            else
                obj = s;
            end
        end
    end % static methods
end

Are there any patterns how to improve save/load speed with many class instances, probably with additional class objects nested in their properties? I could implement lazy load and split the built-in datatypes in my table from the custom ones, but the table container is not the bottleneck. I might be able to transform some of my classes from an array of objects construct to object of arrays construct, but that only works for very simple classes with scalar or simple properties.

Profiling both classes lead to:

The array class obviously is way faster, but I can't refactor everything to array classes.

Key question: What causes the overhead? Is it the class instantiation? Is it the properties assignment? What is causing the slowdown? The loadobj method seems not to be the issue. What is happening in the time not detailed profiled during load? Can I profile it?

Thanks!

Matt J 2024-2-26

编辑：Matt J 2024-2-26

According to your original post, the idnlarx objects are members of your custom class. It is that class whose saveobj/loadobj method I am suggesting you customize, not the internal loadobj method of idnlarx. Transform the idnlarx members into something simpler and more compact, something which can be used to rebuild the original idnlarx memebrs later.

Jan Kappen 2024-2-27

Understood. And that's where I struggle since I can't access the internals of idnlarx to rebuild the original idnlarx objects later. At least its constructor does not let me set all class properties to rebuild and I can't set them from "outside", because they are read only.

But I'll investigate, maybe there are other ways to serialize/deserialize.

请先登录，再进行评论。

Profile and debug load function

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

8 个评论
显示 6更早的评论隐藏 6更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Profile and debug load function

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

8 个评论 显示 6更早的评论隐藏 6更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

8 个评论
显示 6更早的评论隐藏 6更早的评论