Universal sentence encoder

Ok so I might be a bit late to join the NLP bus but am so glad I boarded! Lets start with the universal sentence encoder. The universal sentence encoder was released a couple of years ago by Google and is widely appreciated by the NLP community as a quick way to generate a sentence embedding before any further processing can be done on it. One reason to not to use a naive encoding scheme based upon term frequency is that it ignores word ordering and can have high similarity even when the meaning of the sentence is not the same. Some examples mentioned in the blog below shows that the sentence it is cool and it and is it cool have a high similarity.

The original paper mentions two ways to encode the natural language sentence - a) Transformer encoder - This consists of 6 stacked transformer layers (each has a self-attention module followed by a feed-forward network).The self attention takes care of the nearby context to generate the word embeddings. b) Deep averaging network - begins by averaging the embeddings for the word and bi-grams and then puts it through a 4-layer feed-forward deep DNN as output.

A question then arises is when to use USE vs BERT. [3] is an interesting reference showing the comparion numbers using BERT vs others. Without tuning BERT is not meant to find similar sentences. BERT is trained on two main tasks - masked language model where words are hidden and the model is trained to predict the missing words and next sentence prediction to identify whether sentence B follows sentence A. USE is better suited for the similarity of sentence task.

References
[1] A very nice blog.
[2] Original paper of Universal sentence encoder
[3] Nice blog on BERT vs USE and others

Comments

Post a Comment

Popular posts from this blog

MinHash, Bloom Filters and LSH

Lets talk about some large scale algorithms widely used in the document world. MinHash: Minhash is a really cute algorithm to determine how does a document compare to another. One simple way to compute document similarity is to compute the jaccard similarity between them. The jaccard similarity between the documents is computed as follows: def jaccard_similarity(set1: set, set2: set): if len(set1) == 0 or len(set2) == 0: return 0.0 common = len(set1.intersection(set2)) if common == 0: return 0.0 union = len(set1.union(set2)) return common / union One problem with Jaccard similarity is that it can take a lot of time to compute especially for large scale documents. The idea behind minhashing is to first operate on the space of document shingles. Shingles are basically any combination of k consecutive words of the document. For example, for the sentence "hi i am jaya" the possible 3 shingles are "hi i am" and "i am jaya&quo
Read more

My learnings of Karpathy 1-3

Andrej Karpathy has put together one of the most awesome video blogs on NNs and I finished watching the first three of them. Wanted to put out some of my ramblings on the same. These are my notes so that I internalize them well and also so that I do not forget :) Have become a bit forgetful of late (blame it on the forties! :| ). Lecture 1: Bacpropagation is the core of any modern NN . Derivative of function: If you slightly bump up f(x) by h how does it respond. f(x + h) - f(h) / h. Lecture 2: Bigram Model . Lecture 3: Basic Language model
Read more

Perceptron Algorithm

Image
The perceptron algorithm is a linear classifier using a linear function (w'x + b) to classify the points into two classes. The loss function for the perceptron algorithm just counts the points misclassified and then incrementally adjusts the weight vector and thus the decision boundary to appropriately classify the two classes. Perhaps this is the simplest algorithm one can think of to update the weight vector and is shown as follows: w(k+1) = w(k) + yt*xt if yt ≠ f(xt; w) i.e. we cycle through the training points and adjust the weight vector only if the points are misclassified. Note that our class labels are -1 and 1 and hence if the points are misclassified then y_t* f(xt; w) will be negative. In order to understand why the seemingly simple update rule works and leads to a convergence, lets closely examine it. One of the underlying assumption in the perceptron model is that the points are linearly separated hence there exists an w* such that y_t * f(xt; w*) ≥ γ for all t=1, 2
Read more