Interactively Build Experiments to Fine-Tune and Compare Deep Learning Networks

Hi. My name's Joe Hicklin. I'm a senior developer at the MathWorks. I'm going to show you how to use Experiment Manager to make your deep learning projects produce better results, go faster, and take less work.

When I'm trying to produce the very best deep learning solution that I can, I'll go through a lot of experimentation to see which strategies work best. I'll try different sets of training data or different treatments of my training data, I'll try different network architectures or maybe different parameters within my network, and I'll try different training options to see which strategy produces the best result.

All this experimentation is the most time-consuming part of my project, and before Experiment Manager came along, I did a pretty disorganized job of it. I'd write a script like this one that would set up my training data, set up my network, set my training options, and finally train the network.

I'd run that, see how the network performed, and then to try out alternatives. I'd edit the script, and run it again. I'd say, what if I wanted to try more channels in this layer, or what if I wanted a different learning rate?

And I would go through a great many different combinations of all these sorts of options to figure out which network performs the best. When I was all finished, I'd think that I'd covered most of the areas, but I wasn't really sure. And I'd wind up rerunning some experiments several times to validate or recover previous results.

The Experiment Manager addresses all of those problems. Let's look at an Experiment Manager project that I've been working with. I'll go to Apps Gallery, choose the Experiment Manager. When the Experiment Manager comes up, I'll open a pre-existing project. There it is.

And here's my project. My project has three experiments in it, one called data sets, one called networks, one called training options. We're going to look at the results for the first experiment, data sets.

In this experiment, I was trying to figure out if I was using enough data or not. So I ran four different trials. And each one using a different amount of data or with or without augmentation. And these trials have already been run, and they're finished.

For each one of the trials, I can look and see how long they took, what the training accuracy was, the validation accuracy, and a few other things. For any trial I want, I can look at the training plot. We will recreate that, and there's the training plot for that trial. There's the training plot for this trial.

I can also look at the confusion matrix for each trial. And so that was a set of-- That was an experiment, a set of trials for an experiment that I did. I did another experiment. We'll look at those results. And this is trying different networks.

I tried a basic network. I tried MobileNet, two-layer network, and a three-layer network. And, again, for each one of these, I can look at the training network, I can look at the training plot and things like that.

One of these networks didn't work. I tried MobileNet, and I don't have MobileNet on my machine. But that doesn't stop the Experiment Manager. It noted the error and kept on going and ran all four of these experiments.

In my third experiment, you can see that I don't have any results. I haven't run that one yet. Let's do that now. If I double-click on Training Options, I'll see the experiment description that I wrote earlier.

In this experiment, I'm going to do an exhaustive sweep over four parameters. I've named them here. And each parameter has a set of values it's going to cover. So what this means is it's going to run a trial for every combination of values for these four parameters.

That's going to result in, I think, 48 different trials. That's a lot, so I'm going to use parallel and run those things in parallel. And I'll go ahead and start that now.

So what we see here is this table has all 48 trials it's going to run in it. And it started the first 12, and the rest are waiting. If we look up here, we can see that it has finished 0. 12 are running right now. 42 are queued up, and nothing has crashing yet. Oh, there's going to be a total of 54 trials.

These 12 trials have started. We can watch their progress and watch them go. When any of these trials finish, it'll just start the next trial. And it'll slowly work its way through all of the different trials we need to get done.

You can see for each trial what particular combination of parameters that's being used. The first trial used that solver for two epics, that learn rate and that mini batch size. And by the time it's finished, it will have run through all 54 different permutations of the things that I asked for.

This will take quite a while. So we're not going to sit here and wait for it. But when it's finished, we'll be able to look through all those. So let's let the clock go for a minute or two.

[CLOCK TICKING]

OK. It's finished. The Experiment Manager has trained up 54 neural nets for us. In real life, we probably would have had to go out to lunch or maybe gone home for the evening and come back the next day. But it's done all 54 of these.

And these results are reproducible because the Experiment Manager has saved the script that ran everything. It saved the parameters that it used for each one of these trials. It's even saved the trained network that you could export back to the workspace to work with these further.

If we wanted to see what worked out the best, we could sort on validation accuracy and see the top one here used this solver, this many epics, this learn rate, that sort of thing. And you can compare and see which strategies work the best.

So that's an example of the kind of experimentation that Experiment Manager can automate for you. In the next video, I'll show you what it took to get Experiment Manager to do this for me.