Inverse huge matrix with tall array

Question

chen zhe 2024-6-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2130626-inverse-huge-matrix-with-tall-array

评论： halleyhit 2024-6-23

Hi,

I am trying to find x in my system Ax=b. A is a huge matrix, it is a size of 10 billion *9, and out of my memory, b is 10 billion *1 vector.

Therefore, I am using tall array, while I don know how to compute the inverse of A? (pseudo inverse)

One answer is here, while it is incorrect: https://uk.mathworks.com/matlabcentral/answers/530823-taking-the-inverse-of-a-tall-array-to-solve-a-linear-system-of-equations

A = tall(ds);
b_t = tall(ones(1,1));
SOL = gather(A\b);

This code works with SOL is 9*1 vector, however, the vector b_t has all elements equal to 1.

If I use my verctor b, it shows an error with incompatible tall array argument in tall/qrLeftSolve.

Any one can help this case?

Thanks in advance.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

halleyhit 2024-6-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2130626-inverse-huge-matrix-with-tall-array#answer_1475161

在 MATLAB Online 中打开

Hi chen, I am writing some lines code and find they are working fine. You can directly solve equeation AX=B or get x_inv by tall array. Would you please share more information, like MATLAB version and full error message?

A=tall(rand(100,100));
B1=tall(rand(100,1));
X=A\B1; % solve X in equation A*X=B1
B2=tall(eye(100));
X_inv=A\B2; % solve inv(X)
X=gather(X)
Evaluating tall expression using the Local MATLAB Session:
- Pass 1 of 1: 0% complete
- Pass 1 of 1: Completed in 0.45 sec
Evaluation completed in 0.79 sec
X = 100x1
    0.3758
    0.0945
    1.6328
    0.5906
   -0.6462
    0.2529
    1.8940
    1.8789
   -0.1144
    0.0478
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
X_inv=gather(X_inv)
Evaluating tall expression using the Local MATLAB Session:
- Pass 1 of 1: 0% complete
- Pass 1 of 1: Completed in 0.12 sec
Evaluation completed in 0.28 sec
X_inv = 100x100
    0.7743   -0.0693    0.8170    0.8577   -0.3118    0.3562   -0.9650   -0.7043    1.7218   -1.4824   -0.3475   -0.1169    0.8369    0.6821    0.7398    0.2457   -0.4201    0.1088    0.1201   -0.2784   -0.0559   -1.2019   -0.5790   -0.3116    0.0530   -0.0006    0.3367    0.9886   -1.0026   -0.1638
   -0.0558    0.4031    0.0743   -0.3376    0.2502   -0.0059   -0.4564    0.2059   -0.6678   -0.0947   -0.0655    0.4893   -0.3262   -0.0874   -0.4319   -0.5594    0.1062    0.0800   -0.5376    0.1450   -0.1239    0.2719    0.1130   -0.3080   -0.2385    0.8645   -0.9948   -0.2167    0.1881    0.0734
   -0.0784    0.1249    0.2214   -0.3601    0.2519   -0.1688   -0.2442   -0.2114   -0.6418    0.4962    0.1870   -0.0858   -0.1424   -0.2332   -0.6112   -0.1980   -0.3017    0.5306   -0.4729   -0.0702   -0.1993   -0.3099   -0.0528   -0.1944   -0.1935    0.5906   -0.3473   -0.4316    0.2758    0.0678
    0.1676   -0.4320    0.4946    0.1798   -0.4843   -0.1374    0.0295   -0.1248    0.8595   -0.2044   -0.2073   -0.3594    0.5226   -0.0816    0.6570    0.5772   -0.1445    0.1138    0.3007   -0.0707   -0.0532   -0.2033   -0.3197   -0.0642    0.2278   -0.7270    0.5173    0.4822   -0.1869   -0.0182
   -1.0476    0.2718   -1.6123   -0.8449    0.4432   -0.3391    2.0997    1.0481   -1.5932    1.9536    0.6653    0.2580   -0.8852   -0.7632   -1.1088   -0.2306    0.3132   -0.4577   -0.2821    0.2983    0.3981    1.8093    0.9012    1.0211    0.2953   -0.9724   -0.1735   -1.0113    0.8892    0.0465
   -0.1037    0.5654   -0.1123   -0.3934    0.4696    0.1766   -0.2697    0.2524   -0.6914    0.1512    0.0448    0.3207   -0.4046   -0.0659   -0.6561   -0.4781   -0.1712   -0.1590   -0.2617    0.0846    0.1668    0.3875    0.2468   -0.0771   -0.4338    0.5762   -0.6102   -0.3239    0.3225    0.1269
   -0.0470    0.2204   -0.9893   -0.5586    0.7420   -0.8195    1.3611    0.7751   -1.1705    1.5683    0.4142    0.4074   -0.6538   -0.4146   -0.7123   -0.2845   -0.1614    0.2445   -0.9458    0.3083    0.2387    1.3137    0.4728    0.6883    0.0753    0.0331   -0.7331   -0.5261   -0.0115    0.4966
    0.6588   -0.6303    1.2606    0.5455   -0.4444    0.0983   -1.5520   -1.2083    1.4617   -1.1599   -0.4837   -0.3701    0.6419    0.6479    0.8795    0.4380   -0.6280    0.5090    0.2999   -0.4201   -0.3082   -1.9395   -0.8663   -0.8497   -0.1915    0.5109    0.5848    0.8014   -0.6520   -0.0784
   -1.1458    0.2776   -0.8118   -0.8092    0.5263   -0.2181    0.5056    0.2499   -1.7848    1.4072    0.7064    0.0356   -0.8161   -0.5013   -1.4115   -0.3823   -0.0601    0.0124   -0.2746    0.0059   -0.2659    0.3024    0.4954    0.3484   -0.1463    0.3013   -0.2698   -1.1395    0.9805   -0.3267
   -0.5385    0.7469   -0.4072   -0.7453    0.7073    0.1720   -0.2972    0.1257   -1.7257    0.7864    0.7099    0.4504   -0.7276   -0.0966   -1.5364   -0.8592   -0.3177    0.2224   -0.6054   -0.1317   -0.1317    0.0955    0.4519   -0.2018   -0.4185    0.9981   -0.6300   -0.9723    0.7656   -0.1510
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>

2 个评论
显示无隐藏无

chen zhe 2024-6-21

Hi halleyhit,

Thanks for your answer. It works well for me.

But I have another issue in pinv, if I need a Pseudo inverse.

X=pinv(A)*B1; % solve X in equation A*X=B1

X=gather(X);

The error shows incorrect number or type for function diag().

Do we have any good solution for this pinv()?

Thanks in advance.

halleyhit 2024-6-23

as dos shows, pinv does not support tall array

请先登录，再进行评论。

Answer 2

Ayush Modi 2024-6-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2130626-inverse-huge-matrix-with-tall-array#answer_1475166

编辑：Ayush Modi 2024-6-21

在 MATLAB Online 中打开

Hi Chen,

"If I use my verctor b, it shows an error with incompatible tall array argument in tall/qrLeftSolve."

Operator '\' requires both the inputs to be of same type. If either one is tall array and the other one is not, this error will be thrown. To resolve the error, you would need to convert b vector into a tall array.

Here is the sample code for your reference:

% Defining two random matrix
A = rand(100,100)
A = 100x100
    0.9947    0.4626    0.6908    0.9572    0.3930    0.2770    0.4175    0.1024    0.9190    0.9325    0.8319    0.8676    0.5394    0.6506    0.8565    0.8859    0.9501    0.9520    0.2930    0.4328    0.5271    0.1691    0.2264    0.2029    0.5648    0.0718    0.9155    0.5690    0.0737    0.3573
    0.0923    0.2488    0.7940    0.2614    0.8866    0.7586    0.4054    0.6688    0.5740    0.3765    0.7618    0.2080    0.1918    0.5318    0.7129    0.9550    0.4707    0.9276    0.0998    0.5540    0.7819    0.4010    0.8166    0.4177    0.1151    0.4744    0.3886    0.6050    0.6691    0.9335
    0.0192    0.5947    0.6057    0.9769    0.7346    0.8659    0.5971    0.8688    0.4415    0.4081    0.3207    0.3461    0.0195    0.1080    0.9258    0.0734    0.2192    0.3056    0.8795    0.5699    0.1733    0.7057    0.6939    0.8609    0.3452    0.2789    0.9620    0.2709    0.1687    0.0543
    0.0302    0.7422    0.7112    0.3936    0.1013    0.8690    0.9021    0.8979    0.7237    0.5793    0.3438    0.3597    0.6744    0.1376    0.2294    0.5965    0.6801    0.2358    0.0972    0.6335    0.5269    0.8279    0.6645    0.2513    0.0440    0.1707    0.9959    0.6991    0.8024    0.3426
    0.5075    0.7772    0.1627    0.6762    0.6031    0.6545    0.7705    0.3404    0.4068    0.9728    0.8714    0.3318    0.0450    0.7205    0.8281    0.0297    0.5191    0.5438    0.6343    0.3165    0.5878    0.9132    0.9126    0.8596    0.8247    0.0339    0.4097    0.7741    0.0685    0.4046
    0.0702    0.9067    0.2362    0.9370    0.2169    0.0860    0.8906    0.7138    0.4679    0.5370    0.5455    0.1653    0.3492    0.2454    0.8324    0.1050    0.2765    0.9222    0.0056    0.3227    0.7864    0.9552    0.3336    0.6613    0.0659    0.7436    0.2887    0.8740    0.7749    0.9455
    0.5397    0.2529    0.7490    0.6075    0.7590    0.4994    0.8931    0.9234    0.2500    0.1606    0.8784    0.5149    0.1548    0.2430    0.5835    0.8846    0.0499    0.2962    0.8501    0.6615    0.2866    0.0072    0.9357    0.1078    0.1873    0.4558    0.5208    0.8972    0.9301    0.1669
    0.5393    0.8417    0.4028    0.8963    0.9119    0.8722    0.5371    0.8855    0.2501    0.0766    0.1744    0.5684    0.1131    0.2407    0.8461    0.0656    0.9170    0.9014    0.1651    0.3962    0.1488    0.3718    0.7011    0.1383    0.8359    0.8441    0.6202    0.0033    0.6855    0.6126
    0.1507    0.4941    0.0794    0.4621    0.2253    0.4462    0.3381    0.9387    0.4232    0.8856    0.9501    0.0953    0.7915    0.1792    0.6617    0.6468    0.9575    0.4384    0.1039    0.6810    0.3056    0.1640    0.5208    0.2375    0.3266    0.8771    0.6450    0.7747    0.9686    0.1680
    0.7484    0.2843    0.4546    0.6945    0.6782    0.4360    0.6349    0.4398    0.5671    0.7586    0.3504    0.3579    0.2416    0.3331    0.8744    0.4183    0.3727    0.0479    0.9523    0.9842    0.7074    0.0672    0.3116    0.6295    0.6847    0.3250    0.4385    0.5122    0.3505    0.7533
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
b = rand(100,1)
b = 100x1
    0.5972
    0.5849
    0.6938
    0.5226
    0.1276
    0.1076
    0.8850
    0.6889
    0.6440
    0.6508
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
% Converting them to tall arrays
tallA = tall(A)
tallA =

  100x100 tall double matrix

  Columns 1 through 18

    0.9947    0.4626    0.6908    0.9572    0.3930    0.2770    0.4175    0.1024    0.9190    0.9325    0.8319    0.8676    0.5394    0.6506    0.8565    0.8859    0.9501    0.9520
    0.0923    0.2488    0.7940    0.2614    0.8866    0.7586    0.4054    0.6688    0.5740    0.3765    0.7618    0.2080    0.1918    0.5318    0.7129    0.9550    0.4707    0.9276
    0.0192    0.5947    0.6057    0.9769    0.7346    0.8659    0.5971    0.8688    0.4415    0.4081    0.3207    0.3461    0.0195    0.1080    0.9258    0.0734    0.2192    0.3056
    0.0302    0.7422    0.7112    0.3936    0.1013    0.8690    0.9021    0.8979    0.7237    0.5793    0.3438    0.3597    0.6744    0.1376    0.2294    0.5965    0.6801    0.2358
    0.5075    0.7772    0.1627    0.6762    0.6031    0.6545    0.7705    0.3404    0.4068    0.9728    0.8714    0.3318    0.0450    0.7205    0.8281    0.0297    0.5191    0.5438
    0.0702    0.9067    0.2362    0.9370    0.2169    0.0860    0.8906    0.7138    0.4679    0.5370    0.5455    0.1653    0.3492    0.2454    0.8324    0.1050    0.2765    0.9222
    0.5397    0.2529    0.7490    0.6075    0.7590    0.4994    0.8931    0.9234    0.2500    0.1606    0.8784    0.5149    0.1548    0.2430    0.5835    0.8846    0.0499    0.2962
    0.5393    0.8417    0.4028    0.8963    0.9119    0.8722    0.5371    0.8855    0.2501    0.0766    0.1744    0.5684    0.1131    0.2407    0.8461    0.0656    0.9170    0.9014
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :

  Columns 19 through 36

    0.2930    0.4328    0.5271    0.1691    0.2264    0.2029    0.5648    0.0718    0.9155    0.5690    0.0737    0.3573    0.6437    0.8474    0.7992    0.8280    0.0741    0.1939
    0.0998    0.5540    0.7819    0.4010    0.8166    0.4177    0.1151    0.4744    0.3886    0.6050    0.6691    0.9335    0.4819    0.5598    0.4953    0.4115    0.4782    0.3217
    0.8795    0.5699    0.1733    0.7057    0.6939    0.8609    0.3452    0.2789    0.9620    0.2709    0.1687    0.0543    0.0242    0.3062    0.0516    0.6704    0.0306    0.7953
    0.0972    0.6335    0.5269    0.8279    0.6645    0.2513    0.0440    0.1707    0.9959    0.6991    0.8024    0.3426    0.4167    0.6541    0.2446    0.0995    0.2396    0.4477
    0.6343    0.3165    0.5878    0.9132    0.9126    0.8596    0.8247    0.0339    0.4097    0.7741    0.0685    0.4046    0.8334    0.2130    0.6875    0.6769    0.3877    0.4635
    0.0056    0.3227    0.7864    0.9552    0.3336    0.6613    0.0659    0.7436    0.2887    0.8740    0.7749    0.9455    0.5906    0.3272    0.5108    0.3575    0.0480    0.4427
    0.8501    0.6615    0.2866    0.0072    0.9357    0.1078    0.1873    0.4558    0.5208    0.8972    0.9301    0.1669    0.3334    0.5827    0.3716    0.8463    0.3661    0.2470
    0.1651    0.3962    0.1488    0.3718    0.7011    0.1383    0.8359    0.8441    0.6202    0.0033    0.6855    0.6126    0.3333    0.6425    0.1980    0.4854    0.8663    0.5186
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :

  Columns 37 through 54

    0.5355    0.0628    0.5854    0.7285    0.3470    0.4697    0.0952    0.2652    0.5367    0.3317    0.5564    0.9495    0.7961    0.8552    0.7955    0.6598    0.0790    0.2758
    0.1317    0.1247    0.5925    0.3688    0.3156    0.2509    0.2378    0.4523    0.1257    0.3662    0.2784    0.7689    0.7046    0.9002    0.3912    0.0351    0.6458    0.8515
    0.0865    0.6577    0.8280    0.9986    0.2409    0.1537    0.6579    0.1930    0.0249    0.4879    0.7781    0.6314    0.9368    0.0240    0.0143    0.2097    0.7062    0.7341
    0.9884    0.2811    0.9029    0.6874    0.0348    0.4320    0.6434    0.8410    0.5180    0.4068    0.9110    0.1273    0.4360    0.5249    0.4062    0.9801    0.6685    0.9761
    0.0288    0.8181    0.5334    0.4100    0.8383    0.0385    0.5114    0.2457    0.0255    0.4325    0.5701    0.6968    0.2131    0.8715    0.9579    0.5394    0.1259    0.3427
    0.6658    0.1941    0.3923    0.7473    0.8542    0.1991    0.5978    0.0734    0.1934    0.6157    0.8895    0.7805    0.9647    0.7126    0.5998    0.1779    0.3588    0.6053
    0.4122    0.1348    0.8091    0.4853    0.9496    0.1586    0.8800    0.0572    0.1071    0.4716    0.7755    0.0318    0.7086    0.1122    0.3572    0.6548    0.3940    0.6840
    0.7034    0.2817    0.6467    0.2609    0.2060    0.7199    0.8099    0.2952    0.0169    0.8177    0.3036    0.2856    0.1215    0.2466    0.9243    0.0432    0.3923    0.5641
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :

  Columns 55 through 72

    0.1232    0.8125    0.3342    0.3624    0.6900    0.7983    0.9980    0.8447    0.7832    0.7575    0.0067    0.1952    0.2662    0.5321    0.0664    0.9825    0.9733    0.8893
    0.0374    0.8396    0.6496    0.5587    0.9625    0.6297    0.3435    0.6432    0.2192    0.5261    0.8477    0.8670    0.6865    0.4448    0.0893    0.0777    0.3255    0.2446
    0.7908    0.6795    0.1366    0.4217    0.7251    0.2381    0.5144    0.5793    0.6458    0.7338    0.7580    0.5616    0.4369    0.6664    0.6267    0.3804    0.3380    0.2379
    0.5076    0.2598    0.8806    0.6885    0.9630    0.9466    0.9165    0.0427    0.4639    0.2303    0.8713    0.6334    0.5789    0.5165    0.1985    0.1758    0.5908    0.5495
    0.9266    0.6116    0.9526    0.6330    0.8698    0.4270    0.3496    0.9524    0.4305    0.7161    0.9226    0.2925    0.0856    0.7509    0.1776    0.9825    0.3673    0.4231
    0.3078    0.9878    0.5069    0.3448    0.8062    0.8439    0.0511    0.2587    0.2708    0.8813    0.3075    0.0586    0.4917    0.8142    0.1890    0.0054    0.9303    0.4693
    0.9174    0.2580    0.3145    0.4633    0.1351    0.6166    0.6503    0.9601    0.2325    0.5454    0.5390    0.4688    0.0713    0.5822    0.1256    0.0251    0.5389    0.3943
    0.8867    0.6729    0.7401    0.9365    0.5946    0.3014    0.0888    0.9111    0.3976    0.8435    0.8404    0.6172    0.3288    0.5484    0.8531    0.1393    0.3151    0.2739
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :

  Columns 73 through 90

    0.1418    0.4728    0.5474    0.4756    0.7492    0.7575    0.9610    0.9560    0.3065    0.2035    0.2640    0.2454    0.1366    0.5136    0.2233    0.1296    0.6005    0.8043
    0.1893    0.4862    0.4133    0.2853    0.9044    0.8362    0.7122    0.3697    0.4223    0.9724    0.4888    0.3134    0.2218    0.4102    0.4233    0.1391    0.1185    0.7081
    0.0672    0.4929    0.4596    0.5012    0.8577    0.7600    0.2484    0.3413    0.3446    0.0896    0.4449    0.8577    0.8031    0.8912    0.0473    0.3659    0.5192    0.9020
    0.8129    0.0201    0.7949    0.7919    0.3383    0.9086    0.1702    0.7679    0.5813    0.3368    0.8705    0.3670    0.9006    0.9876    0.4270    0.3445    0.5984    0.8060
    0.5353    0.4185    0.3806    0.4322    0.0480    0.7821    0.2039    0.3469    0.5749    0.2550    0.5728    0.0580    0.8129    0.2231    0.5661    0.7490    0.9359    0.3340
    0.4876    0.9417    0.9716    0.1828    0.2480    0.4886    0.1627    0.5514    0.1248    0.7397    0.3727    0.1695    0.9214    0.4645    0.9173    0.5430    0.6221    0.0176
    0.5162    0.1587    0.7328    0.5016    0.3008    0.5404    0.9609    0.4512    0.3910    0.8881    0.9625    0.0213    0.6435    0.8747    0.2997    0.8007    0.4106    0.3149
    0.8028    0.6989    0.2706    0.2003    0.8490    0.4253    0.8486    0.8889    0.2723    0.4114    0.3909    0.1060    0.2967    0.8794    0.5920    0.0155    0.5504    0.6084
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :
      :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :         :

  Columns 91 through 100

    0.3253    0.8026    0.1349    0.1779    0.5445    0.7229    0.8003    0.1008    0.3925    0.6121
    0.3294    0.3681    0.5884    0.7623    0.0899    0.5957    0.0224    0.3019    0.7777    0.6986
    0.3077    0.1976    0.9783    0.5364    0.0945    0.2875    0.4918    0.1006    0.7761    0.0023
    0.6713    0.1236    0.1948    0.5170    0.1872    0.9505    0.3202    0.6634    0.0846    0.9351
    0.6131    0.7419    0.9973    0.6152    0.3016    0.2948    0.8884    0.6059    0.3350    0.1469
    0.5383    0.3988    0.6944    0.4074    0.4004    0.8078    0.5827    0.2220    0.3726    0.4620
    0.8897    0.4469    0.8567    0.9446    0.6511    0.4413    0.7349    0.8421    0.3107    0.8229
    0.5451    0.0544    0.0301    0.8424    0.8300    0.1674    0.7776    0.5627    0.4716    0.4457
      :         :         :         :         :         :         :         :         :         :
      :         :         :         :         :         :         :         :         :         :
tallb = tall(b)
tallb =

  100x1 tall double column vector

    0.5972
    0.5849
    0.6938
    0.5226
    0.1276
    0.1076
    0.8850
    0.6889
      :
      :
% We can do operation A\b
A\b
ans = 100x1
   -2.1302
   -0.6963
    2.2138
   -4.1181
    3.3345
   -0.1259
    2.6496
   -4.5379
    4.1526
   -0.1002
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
% We can also do tallA\tallb
gather(tallA\tallb)
Evaluating tall expression using the Local MATLAB Session:
- Pass 1 of 1: 0% complete
- Pass 1 of 1: Completed in 0.16 sec
Evaluation completed in 0.3 sec
ans = 100x1
   -2.1302
   -0.6963
    2.2138
   -4.1181
    3.3345
   -0.1259
    2.6496
   -4.5379
    4.1526
   -0.1002
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
% ERROR - if they are different
gather(tallA\b)  % Note b is matrix, not tall array 
Error using tall/qrLeftSolve (line 16)
Incompatible non-scalar tall array arguments. Each of the tall arrays must be the same size in the first dimension, must be derived from a single tall array, and must not have been indexed differently in the first dimension (indexing operations include functions such as VERTCAT, SPLITAPPLY, SORT, CELL2MAT, SYNCHRONIZE, RETIME and so on).

Error in \ (line 29)
    [~,Z] = qrLeftSolve(X,Y);