trainAutoencoder

An autoencoder is a neural network which is trained to replicate its input at its output. Training an autoencoder is unsupervised in the sense that no labeled data is needed. The training process is still based on the optimization of a cost function. The cost function measures the error between the input x and its reconstruction at the output $$\widehat{x}$$.

An autoencoder is composed of an encoder and a decoder. The encoder and decoder can have multiple layers, but for simplicity consider that each of them has only one layer.

If the input to an autoencoder is a vector $$x\in {ℝ}^{D_x}$$, then the encoder maps the vector x to another vector $$z\in {ℝ}^{D^{\left(1\right)}}$$ as follows:

where the superscript (1) indicates the first layer. $$h^{\left(1\right)}:{ℝ}^{D^{\left(1\right)}}\to {ℝ}^{D^{\left(1\right)}}$$ is a transfer function for the encoder, $$W^{\left(1\right)}\in {ℝ}^{D^{\left(1\right)}\times D_{{}^x}}$$ is a weight matrix, and $$b^{\left(1\right)}\in {ℝ}^{D^{\left(1\right)}}$$ is a bias vector. Then, the decoder maps the encoded representation z back into an estimate of the original input vector, x, as follows:

where the superscript (2) represents the second layer. $$h^{\left(2\right)}:{ℝ}^{D_x}\to {ℝ}^{D_x}$$ is the transfer function for the decoder,$$W^{\left(1\right)}\in {ℝ}^{D_{{}^x}\times D^{\left(1\right)}}$$ is a weight matrix, and $$b^{\left(2\right)}\in {ℝ}^{D_x}$$ is a bias vector.

Encouraging sparsity of an autoencoder is possible by adding a regularizer to the cost function [2]. This regularizer is a function of the average output activation value of a neuron. The average output activation measure of a neuron i is defined as:

where n is the total number of training examples. x_j is the jth training example, $$w_i^{\left(1\right)T}$$ is the ith row of the weight matrix $$W^{\left(1\right)}$$, and $$b_i^{\left(1\right)}$$ is the ith entry of the bias vector, $$b^{\left(1\right)}$$. A neuron is considered to be ‘firing’, if its output activation value is high. A low output activation value means that the neuron in the hidden layer fires in response to a small number of the training examples. Adding a term to the cost function that constrains the values of $${\widehat{\rho }}_i$$ to be low encourages the autoencoder to learn a representation, where each neuron in the hidden layer fires to a small number of training examples. That is, each neuron specializes by responding to some feature that is only present in a small subset of the training examples.

Sparsity regularizer attempts to enforce a constraint on the sparsity of the output from the hidden layer. Sparsity can be encouraged by adding a regularization term that takes a large value when the average activation value, $${\widehat{\rho }}_i$$, of a neuron i and its desired value, $$\rho$$, are not close in value [2]. One such sparsity regularization term can be the Kullback-Leibler divergence.

Kullback-Leibler divergence is a function for measuring how different two distributions are. In this case, it takes the value zero when $$\rho$$ and $${\widehat{\rho }}_i$$ are equal to each other, and becomes larger as they diverge from each other. Minimizing the cost function forces this term to be small, hence $$\rho$$ and $${\widehat{\rho }}_i$$ to be close to each other. You can define the desired value of the average activation value using the SparsityProportion name-value pair argument while training an autoencoder.

When training a sparse autoencoder, it is possible to make the sparsity regulariser small by increasing the values of the weights w^(l) and decreasing the values of z⁽¹⁾ [2]. Adding a regularization term on the weights to the cost function prevents it from happening. This term is called the L₂ regularization term and is defined by:

where L is the number of hidden layers, n_l is the output size of layer l, and k_l is the input size of layer l. The L₂ regularization term is the sum of the squared elements of the weight matrices for each layer.

The cost function for training a sparse autoencoder is an adjusted mean squared error function as follows:

where λ is the coefficient for the L₂ regularization term and β is the coefficient for the sparsity regularization term. You can specify the values of λ and β by using the L2WeightRegularization and SparsityRegularization name-value pair arguments, respectively, while training an autoencoder.