fsolve(@NL_Syst_HW4, [0,0,0])
Remember that matlab processes arguments before making the calls. The argument NL_Syst_HW4 has to be processed, and at that point, matlab has no idea that a function handle is expected there. MATLAB does not look and see "oh, fsolve expects a function handle so treat NL_SYST_HW4 as a function handle": matlab processes NL_Syst_HW4 first and passes the result to fsolve.
But what is the result of processing NL_SYST_HW4? Well, that is not a variable so matlab checks to see if it is a function. Having found it to be a function, matlab calls the function with all the provided parameters.. which is no parameters at all. And then the function complained because you did not pass enough parameters.