I have not used the MuPAD Symbolic Toolbox, and have not seen much written about using syms to define a function, so I am going to have to give an interpretation here.
It looks to me that when you use syms to define a function, then when you reference the function name without indicating any parameters, that the parameter names you gave in the syms command are always added on automatically (except perhaps on the left side of an assignment, and I would never expect them to be added inside a string.)
When you reference the function name and give something in () afterwards, then you are doing parameter substitution, so u(t) is the function "u" with the actual argument "t" substituted for its first parameter (which happens to be "t"). For example,
syms p; u(p) + u
I suspect would be would be a(p) * b(p) + u(t)
Now, add to this that
v = b(t) * c(t)
is defining the symbol v to be b(t) * c(t), with the fixed parameter "t", rather than defining a function. v(r) would then be (b(t) * c(t))(r)
and everything you show in your test follows.
The error message you are getting is saying that if you want u(t) to mean indexing, then you need to use formula(u)(t) where "formula" is a fairly new MuPAD function. It might be, though, that you have to assign the result of formula(u) to a variable and index that, as I do not know if MATLAB has special-cased formula() to allow indexing immediately afterwards.
