I found a solution.
clear; clc;
J = 152e-6; % moment of inertia
W = 200*pi; % desired crossover freq
K = 0.1; % torque constant
syms Ki Kp real
assumeAlso(Ki>0);
assumeAlso(Kp>0);
G = Ki/(i*W)*(1 + i*W/(Ki/Kp))*K/(i*W*J); % transfer-function
F = [abs(G), angle(G)]; % symbolic output
H = [rewrite(F(1),'sqrt') == 1, simplify(F(2)) == -120*pi/180]; % 2 equations
[Ki,Kp] = solve(H,Ki,Kp);
So 'rewrite' and 'simplify' made the equations more managable for 'solve'