Normalizing Neural Networks

Lets look at various different strategies for normalizing neural networks:

Batch Normalization: Batch normalization is a type of layer that can adaptively normalize the data. Before we go deep in that lets first examine the phenomenon of Internal covariate shift. The change in the distribution of the internal nodes of a deep network in the course of training is called as Internal Covariate Shift. This is disadvantageous because the layers need to continuously adapt to the change in the distribution. Batch Normalization transform takes care of the problem as follows:
  1. For a layer, normalize each feature dimension as: 
    x'(k) = x(k) - E[x(k)]/√ Var[x(k)]
  2. Simply normalizing the input can change what the layer can represent. For example, normalizing the inputs of sigmoid would constrain them to linear regime. To understand why, note that 95% of the values of a Gaussian distribution lie within the range μ ± 2σ. Hence we need to make sure that the transformation inserted in the network can represent the identity transform. To do this 2 parameters are introduced which scale and shift the normalized value as: 
    y(k) = γ(k)x'(k) + β(k)
    Note that these parameters are learned along with the original model parameters. Further, the BN transform is a differentiable transformation which is important as during training we need to backpropagate the gradient of the loss through this transformation as well.
  3. The BN layer is neither desirable nor necessary during test time. At the training time, from each mini batches compute the E[μb] and unbiased sample variance to rescale the values.
Problems: BN requires large batches to compute accurate statistics, has problems when the batches are not iid and is hard to adapt to RNNs.

Layer Normalization: Layer normalization attempts to address some of the short comings of batch normalization. It solves the problem of covariate shift by normalizing using the statistics collected from all units within a layer for an input example instead of computing batch statistics per dimension for a mini batch. It results in more stable hidden to hidden dynamics.

Weight Normalization: Weight normalization normalizes the weights of a layer directly using the L2 norm. As an example consider the input layer to be of dimension 10 and the hidden layer to be of dimension 50. The weight matrix is thus of size 10 x 50. In order to normalize the weight matrix, consider the 10 inputs going to every of the 50 nodes. Take the L2 norm of all the 10 weights and then normalize the weights.

Comments

Post a Comment

Popular posts from this blog

MinHash, Bloom Filters and LSH

Lets talk about some large scale algorithms widely used in the document world. MinHash: Minhash is a really cute algorithm to determine how does a document compare to another. One simple way to compute document similarity is to compute the jaccard similarity between them. The jaccard similarity between the documents is computed as follows: def jaccard_similarity(set1: set, set2: set): if len(set1) == 0 or len(set2) == 0: return 0.0 common = len(set1.intersection(set2)) if common == 0: return 0.0 union = len(set1.union(set2)) return common / union One problem with Jaccard similarity is that it can take a lot of time to compute especially for large scale documents. The idea behind minhashing is to first operate on the space of document shingles. Shingles are basically any combination of k consecutive words of the document. For example, for the sentence "hi i am jaya" the possible 3 shingles are "hi i am" and "i am jaya...
Read more

Perceptron Algorithm

Image
The perceptron algorithm is a linear classifier using a linear function (w'x + b) to classify the points into two classes. The loss function for the perceptron algorithm just counts the points misclassified and then incrementally adjusts the weight vector and thus the decision boundary to appropriately classify the two classes. Perhaps this is the simplest algorithm one can think of to update the weight vector and is shown as follows: w(k+1) = w(k) + yt*xt if yt ≠ f(xt; w) i.e. we cycle through the training points and adjust the weight vector only if the points are misclassified. Note that our class labels are -1 and 1 and hence if the points are misclassified then y_t* f(xt; w) will be negative. In order to understand why the seemingly simple update rule works and leads to a convergence, lets closely examine it. One of the underlying assumption in the perceptron model is that the points are linearly separated hence there exists an w* such that y_t * f(xt; w*) ≥ γ for all...
Read more

Logistic Regression

The logistic regression method even though called regression has a unique ability to give posterior probability values for a data point to belong to a particular class and hence serve as a discriminative function useful for classification. The idea is simple, to use the logistic sigmoid function on the linear function of feature vector. p(y = 1| x) = σ(w'x +b) The likelihood function can be written as p(t|w) = Π y_i^t_i*(1 - y_i)^(1 - t_i) ln p(t|w) = Σ t_i*ln(y_i) + (1 - t_i)*ln(1 - y_i) The error function is computed by taking the negative of the log likelihood function and is computed as follows: E(w) = -ln p(t|w) Now, ∇E(w) is computed as follows: y = 1 / (1 + exp(-w'x) Taking derivative of both sides, ∇y/∇w = exp(-w'x)(x)/(1 + exp(-w'x))^2 ∇y/∇w = xy(1-y) Substituting the value in ∇E(w), we get ∇ E(w) = Σ (y_i - t_i)*x_i The derivative of the error function is similar to the least squares problem, although w...
Read more