Two Tower

The two tower model is a popularly used model for retrieval mainly because it can be set up efficiently for a large scale. At a very high level, the model consists of two towers - a user tower and a query tower. Here are some practical tips and tricks on training two tower model -
  1. Candidate Sampling: To solve a multi-class multi-label problem with a large number of classes, one of the tricks often applied is that of candidate sampling where you do not need to compute F(x,y) for every class y for every training example x. Candidate sampling involves constructing a training task such that you only update a subset of the classes. Some examples of candidate sampling algorithms are Noise Contrastive Estimation (NCE), Negative Sampling, Sampled Logistic, etc.
  2. Popularity correction : A typical two tower setup involves

References
  1. Candidate Sampling Tutorial by TensorFlow

Comments

Post a Comment

Popular posts from this blog

MinHash, Bloom Filters and LSH

Lets talk about some large scale algorithms widely used in the document world. MinHash: Minhash is a really cute algorithm to determine how does a document compare to another. One simple way to compute document similarity is to compute the jaccard similarity between them. The jaccard similarity between the documents is computed as follows: def jaccard_similarity(set1: set, set2: set): if len(set1) == 0 or len(set2) == 0: return 0.0 common = len(set1.intersection(set2)) if common == 0: return 0.0 union = len(set1.union(set2)) return common / union One problem with Jaccard similarity is that it can take a lot of time to compute especially for large scale documents. The idea behind minhashing is to first operate on the space of document shingles. Shingles are basically any combination of k consecutive words of the document. For example, for the sentence "hi i am jaya" the possible 3 shingles are "hi i am" and "i am jaya&quo
Read more

My learnings of Karpathy 1-3

Andrej Karpathy has put together one of the most awesome video blogs on NNs and I finished watching the first three of them. Wanted to put out some of my ramblings on the same. These are my notes so that I internalize them well and also so that I do not forget :) Have become a bit forgetful of late (blame it on the forties! :| ). Lecture 1: Bacpropagation is the core of any modern NN . Derivative of function: If you slightly bump up f(x) by h how does it respond. f(x + h) - f(h) / h. Lecture 2: Bigram Model . Lecture 3: Basic Language model
Read more

Perceptron Algorithm

Image
The perceptron algorithm is a linear classifier using a linear function (w'x + b) to classify the points into two classes. The loss function for the perceptron algorithm just counts the points misclassified and then incrementally adjusts the weight vector and thus the decision boundary to appropriately classify the two classes. Perhaps this is the simplest algorithm one can think of to update the weight vector and is shown as follows: w(k+1) = w(k) + yt*xt if yt ≠ f(xt; w) i.e. we cycle through the training points and adjust the weight vector only if the points are misclassified. Note that our class labels are -1 and 1 and hence if the points are misclassified then y_t* f(xt; w) will be negative. In order to understand why the seemingly simple update rule works and leads to a convergence, lets closely examine it. One of the underlying assumption in the perceptron model is that the points are linearly separated hence there exists an w* such that y_t * f(xt; w*) ≥ γ for all t=1, 2
Read more