The bottom line is that both of these plots show the expected numerical behavior of the differential equations as you implemented them. My recommendation is to replace the [A:B] term with max(0,[A:B]).
In your case, the reactions for the two cases proceed at vastly different rates. In Case 1, EC50 = 10 nanomole. In Case 2, EC50*comp1 = 10 nanomole/liter*microliter = 1e-5 nanomole.This leads to much faster dynamics for Case 2, and eventually the solver takes a large enough time step that the concentration goes negative. And it turns out that the solver tolerances are still satisifed under these conditions, due to the form of your reaction rate. The problem here is that a "negative amount" of [A:B] of -10 nanomole results in the exact same reaction rate as +10 nanomole of [A:B]. So a slightly negative amount leads to even more negative amounts of [A:B]. I typically say that negative concentrations that are within the solver's absolute tolerance of zero should be considered as equivalent to zero. And you can actually enforce such a constraint on your equations by replacing terms involving [A:B] with max(0,[A:B]).
I made such a change in your model, and this eliminated the negative concentrations. Note however that if you make such a change and still see concentrations with large magnitude negative values (say, -10 or -100 nanomole, for this model), then you may have a modeling error that leads to non-physical behavior.
