Need help with "Out of memory" error

1 次查看(过去 30 天)
Hi everyone, I'm working with Matlab and InfluxDB in need to make them communicate and work with files containing millions of rows.
When I have to import the csv file with readtable I get no memory problem, but when I get to work with influxdb through matlab I get that a lot. I managed to resolve the problem when writing the data in the database just segmenting the timetable, but when I use the commands that helps me extract the data from the database (around 5 millions of rows) i get that error and I can't find a way around it.
Can someone please help me? Thanks in advance.

回答(1 个)

Sam Chak
Sam Chak 2022-7-4
5 millions of rows...
Even Excel cannot handle more than 1,048,576 rows. What is the size of your data array?
N = 5e6;
n = round(sqrt(N));
A = rand(n)
A = 2236×2236
0.2871 0.2892 0.2362 0.8154 0.6719 0.9486 0.0286 0.0167 0.0037 0.9809 0.2334 0.3200 0.6854 0.6888 0.1695 0.9808 0.2442 0.7251 0.9363 0.5723 0.9718 0.5492 0.7839 0.9613 0.7380 0.8300 0.1917 0.6323 0.4454 0.1851 0.3742 0.5974 0.1385 0.3654 0.7016 0.4632 0.2757 0.1444 0.1932 0.9558 0.1683 0.4811 0.3795 0.9344 0.0477 0.0531 0.9365 0.8929 0.8917 0.5186 0.2335 0.4736 0.3354 0.0314 0.2693 0.9257 0.4876 0.6192 0.8157 0.0835 0.9680 0.8536 0.5761 0.4397 0.7299 0.4963 0.4466 0.7446 0.1443 0.3364 0.2759 0.5051 0.8596 0.1811 0.9695 0.0002 0.8675 0.0702 0.0458 0.7376 0.0446 0.7774 0.4039 0.0414 0.3167 0.7822 0.3230 0.2219 0.2134 0.8739 0.0221 0.9181 0.7088 0.0396 0.4419 0.8038 0.9004 0.5114 0.8154 0.0024 0.5422 0.6169 0.3261 0.2753 0.1199 0.0807 0.5046 0.4376 0.1845 0.8173 0.7529 0.4920 0.7203 0.8673 0.4909 0.5040 0.0431 0.2915 0.2074 0.6698 0.2228 0.5988 0.0644 0.4645 0.8457 0.3008 0.7301 0.7456 0.3475 0.8645 0.0832 0.5660 0.6291 0.3286 0.5295 0.7372 0.3966 0.8460 0.6487 0.0219 0.9833 0.0456 0.9380 0.3492 0.3861 0.3639 0.9372 0.4164 0.7274 0.6656 0.3047 0.7029 0.6693 0.6465 0.2201 0.7117 0.0093 0.2530 0.9470 0.2857 0.5328 0.3144 0.2076 0.8893 0.1145 0.7776 0.9828 0.7616 0.6983 0.9311 0.2531 0.9205 0.4533 0.5781 0.7889 0.1680 0.8151 0.9566 0.5768 0.7956 0.1690 0.4041 0.2299 0.0682 0.1962 0.4454 0.7184 0.8617 0.5937 0.0149 0.5761 0.2546 0.8043 0.2645 0.7105 0.1647 0.2057 0.4319 0.6486 0.3131 0.0807 0.6536 0.2650 0.6225 0.6905 0.0867 0.1694 0.8908 0.2788 0.9686 0.5583 0.2647 0.6359 0.0701 0.1135 0.1764 0.6828 0.2149 0.9057 0.2459 0.4584 0.2944 0.1062 0.8637 0.2237 0.4451 0.7978 0.6307 0.7105 0.6784 0.0101 0.0640 0.7792 0.0225 0.7677 0.6073 0.2113 0.8333 0.0855 0.1213 0.5186 0.2191 0.3643 0.3197 0.7782 0.6828 0.2533 0.6939 0.0983 0.1220 0.0086 0.6968 0.3239 0.0517 0.4423 0.6635 0.5764 0.5497 0.8355 0.9435 0.8975 0.6853 0.8019 0.0649 0.0176 0.7949 0.8224 0.2203 0.2157 0.3511 0.9187 0.2796 0.7165 0.5693 0.8563 0.6711 0.7393 0.8317 0.8806 0.9828 0.9553 0.1042 0.5307 0.3902 0.9207 0.3834 0.4921 0.5256 0.6935 0.4608 0.2148 0.2966 0.0602 0.5499 0.5002 0.3270 0.8936 0.9960 0.0837 0.0307
B = ones(N, 1)
B = 5000000×1
1 1 1 1 1 1 1 1 1 1
whos
Name Size Bytes Class Attributes A 2236x2236 39997568 double B 5000000x1 40000000 double N 1x1 8 double cmdout 1x33 66 char n 1x1 8 double
  3 个评论
Sam Chak
Sam Chak 2022-7-4
Hmm... 🤨 That's about 456 MB.
Please check if you can increase the array size limit to 100% of RAM.
What does InfluxDB do through MATLAB?
Sara Bruschi
Sara Bruschi 2022-7-4
I alredy checked that and it was already set as in your photo.
For let matlab and influxdb communicate I'm using a user code found on github: https://github.com/wardgssens/influxdb-matlab/tree/v2-compatibility.
After writing my timetable to influx via the write command I'm trying to call them back with the query command but I get this error if I don't limit the rows.
It's strange also because if I limit the query to 999'999 rows it works fine, but limiting it to 1'000'000 rows I get the same error..

请先登录,再进行评论。

类别

Help CenterFile Exchange 中查找有关 Tables 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by