Any reasonable GA algorithm permits you to use a function handle to be evaluated. The function handle can be to a function you wrote that sets up libsvm calls and executes them and calculates a scalar "score" based upon the results.
I wonder, though, why you are using libsvm instead of using the Statistics Toolbox SVM routines, either the older svmtrain (now no longer supported) or the newer fitcsvm and predict functions? https://www.mathworks.com/help/stats/support-vector-machines-for-binary-classification.html#bsr5o09
