Sorry, but no, there is no simple mathematical formula to perform maximal circle packing of a known radius circle into a given domain. Since your cylinders will be the height of the domain, the height is known. And that means this is just a circle packing problem, in a rectangular domain, so just a 2-d problem. A hexagonal close packed solution is probably at least close to optimal, but along the edges, you may be able to do just a bit better, depending on the dimensions of the rect and the circle radius.
If you then say you don't know the orientation of the cylinders, so you don't know the height, then it is just THREE independent circle packing problems. Solve it three times, once along each axis of the domain. In each case, it becomes a circle packing problem in a rect.
Once you know how many circles can fit into the rect, computing the volume is trivial. Just multiply the area of one circle with the height, then multiply by the number of circles you were able to fit inside. But there is no formula to give you the circle locations or the number of circles.
