"Is there a simpler and faster way to do it?"
Calculating the matrix inverse is rarely required or nor efficient when doing numeric computing. Read this to know why:
If you are really interested in efficiency, then use better tools than inv, just as the documentation recommends.
The determinant also tends to be overused by beginners who do not realize how useless it is in numeric computing. Search this forum to know why the determinant is essentially useless in numeric computing.
