The error you are receiving is due to the fact that "fu" and "fx" need to be functions of the same independent variable, otherwise there is nothing to link the two functions and, hence, compute the functional derivative.
For example, assume the second argument passed to "functionalDerivative" is "f(x)" as you have it. Then, a valid first argument would be an expression of "x", "f(x)", "diff(f(x),x)", "diff(f(x),x,x)", etc. Here is an example:
>> syms f(x)
>> g(x) = f(x)^3 + diff(f(x),x,x,x)*diff(f(x), x);
>> functionalDerivative(g(x), f(x))
Hope this helps!
