How do I compute and visualize complex gradients? (ReLU example)

Question

Julian Knaup 2022-2-15

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1650420-how-do-i-compute-and-visualize-complex-gradients-relu-example

回答： Aditya 2024-1-24

How do I compute and visualize complex gradients?

In the real case I use:

x = -5:0.01:5; 
ReLU = max(0,x);
dReLU = gradient(ReLU)./gradient(x);
% visualize:
a = figure;
plot(x,ReLU);
hold on;
plot(x,dReLU);
axis([-1 5 -1 5]);
xlabel('x');
ylabel('y');
legend({'y=ReLU(x)','y=ReLU''(x)'},'Location','northwest');
title('Real-Valued ReLU');

and for the complex case I want to use:

x = -5:0.2:5; 
y = x;
[X,Y]=meshgrid(x,y);
z = X+1i*Y;
% Complex ReLU
f1 = figure;
complexReLU = max(0,real(z))+1i*(max(0,imag(z)));
complexReLU_abs = min(6.2, abs(complexReLU)); % 6.2 as upper limit just for nicer colors 
sc = surfc(x,y,complexReLU_abs);
sc(2).LineStyle = 'none';
axis([-5 5 -5 5 0 6]);
xlabel('Real(z)');
ylabel('Imag(z)');
zlabel('|ReLU(z)|');
title('Magnitude of ReLU(z)');
colormap jet;
% Derivative of Complex ReLU
f3 = figure;
dcomplexReLU = gradient(complexReLU)./gradient(z);
dcomplexReLU_abs = abs(dcomplexReLU);
sc = surfc(x,y,dcomplexReLU_abs);
sc(2).LineStyle = 'none';
axis([-5 5 -5 5 0 6]);
xlabel('Real(z)');
ylabel('Imag(z)');
zlabel('|dReLU(z)|');
title('Magnitude of dReLU(z)');
colormap jet;

I receive a plot for the derivative of the complex ReLU, which is zero for real(z)<0 and imag(z)>0. But as you can see in the plot of the complex ReLU, this is cannot be true. How do I calculate complex gradients or is there a function that works in the complex as opposed to the gradient() function?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Aditya 2024-1-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1650420-how-do-i-compute-and-visualize-complex-gradients-relu-example#answer_1396211

在 MATLAB Online 中打开

Hi Julian,

I understand that you're looking to calculate the gradient of a complex function. This process is quite different from that for real-valued functions because standard gradient calculations don't apply to complex numbers. They overlook the Cauchy-Riemann equations, which are necessary to determine if a complex function is differentiable.

In the complex domain, a function is differentiable (holomorphic) at a point if it satisfies the Cauchy-Riemann equations there.

For the ReLU function, which is piecewise linear, the complex derivative does not exist everywhere because it is not holomorphic; it is not differentiable at the points where the piecewise linear sections meet (i.e., the nonlinearity at zero).

However, you can compute a sort of "generalized gradient" by considering the “Wirtinger derivatives”, which are used in complex optimization problems.

Here is how you can modify your code:

% Compute the Wirtinger derivatives 
dcomplexReLU_dz = 0.5 * (X > 0) + 0.5 * (-1i) * (Y > 0); 
dcomplexReLU_dz_star = 0.5 * (X > 0) + 0.5 * (1i) * (Y > 0); 
% Absolute values for visualization 
dcomplexReLU_dz_abs = abs(dcomplexReLU_dz); 
dcomplexReLU_dz_star_abs = abs(dcomplexReLU_dz_star); 
% Visualize the magnitude of the Wirtinger derivative with respect to z 
f2 = figure; 
sc = surfc(x, y, dcomplexReLU_dz_abs); 
sc(2).LineStyle = 'none'; 
axis([-5 5 -5 5 0 1]); 
xlabel('Real(z)'); 
ylabel('Imag(z)'); 
zlabel('|dReLU(z)/dz|'); 
title('Magnitude of dReLU(z)/dz'); 
colormap jet; 
% Visualize the magnitude of the Wirtinger derivative with respect to z* 
f3 = figure; 
sc = surfc(x, y, dcomplexReLU_dz_star_abs); 
sc(2).LineStyle = 'none'; 
axis([-5 5 -5 5 0 1]); 
xlabel('Real(z)'); 
ylabel('Imag(z)'); 
zlabel('|dReLU(z)/dz*|'); 
title('Magnitude of dReLU(z)/dz*'); 
colormap jet;

Since MATLAB does not provide a built-in method to compute the “Wirtinger derivatives”, the manual implementation is necessary. However, for real-valued functions or functions that can be separated into real and imaginary parts treated as independent variables, you can use “dlgradient” to compute the gradients of each part separately.