Assignment #1 Invariant to constant offsets

Assignment #1

1 Softmax (10 points)
(a) (5 points) Prove that softmax is invariant to constant offsets in the input, that is, for any input vector
x and any constant c,
softmax(x) = softmax(x + c)
where x + c means adding the constant c to every dimension of x. Remember that
sof tmax(x)i =
e
xi
P
j
e
xj
(1)
Note: In practice, we make use of this property and choose c = − maxi xi when computing softmax
probabilities for numerical stability (i.e. subtracting its maximum element from all elements of x).
(b) (5 points) Given an input matrix of N rows and d columns, compute the softmax prediction for each row.
Write your implementation in q1 softmax.py. You may test by executing python q1 softmax.py.
Note: The provided tests are not exhaustive. Later parts of the assignment will reference this code so it is
important to have a correct implementation. Your implementation should also be efficient and vectorized
whenever possible. A non-vectorized implementation will not receive full credit!
2 Neural Network Basics (30 points)
(a) (3 points) Derive the gradients of the sigmoid function and show that it can be rewritten as a function
of the function value (i.e. in some expression where only σ(x), but not x, is present). Assume that the
input x is a scalar for this question. Recall, the sigmoid function is
σ(x) = 1
1 + e−x
(2)
(b) (3 points) Derive the gradient with regard to the inputs of a softmax function when cross entropy loss
is used for evaluation, i.e. find the gradients with respect to the softmax input vector θ, when the
prediction is made by yˆ = softmax(θ). Remember the cross entropy function is
CE(y, yˆ) = −
X
i
yi
log(ˆyi) (3)
1
CS 224d: Assignment #1
where y is the one-hot label vector, and yˆ is the predicted probability vector for all classes. (Hint: you
might want to consider the fact many elements of y are zeros, and assume that only the k-th dimension
of y is one.)
(c) (6 points) Derive the gradients with respect to the inputs x to an one-hidden-layer neural network (that
is, find ∂J
∂x where J is the cost function for the neural network). The neural network employs sigmoid
activation function for the hidden layer, and softmax for the output layer. Assume the one-hot label
vector is y, and cross entropy cost is used. (feel free to use σ
0
(x) as the shorthand for sigmoid gradient,
and feel free to define any variables whenever you see fit)
x
h
yˆ
Recall that the forward propagation is as follows
h = sigmoid(xW1 + b1) yˆ = softmax(hW2 + b2)
Note that here we’re assuming that the input vector (thus the hidden variables and output probabilities)
is a row vector to be consistent with the programming assignment. When we apply the sigmoid function
to a vector, we are applying it to each of the elements of that vector. Wi and bi (i = 1, 2) are the weights
and biases, respectively, of the two layers.
(d) (2 points) How many parameters are there in this neural network, assuming the input is Dx-dimensional,
the output is Dy-dimensional, and there are H hidden units?
(e) (4 points) Fill in the implementation for the sigmoid activation function and its gradient in q2 sigmoid.py.
Test your implementation using python q2 sigmoid.py. Again, thoroughly test your code as the provided tests may not be exhaustive.
(f) (4 points) To make debugging easier, we will now implement a gradient checker. Fill in the implementation for gradcheck naive in q2 gradcheck.py. Test your code using python q2 gradcheck.py.
(g) (8 points) Now, implement the forward and backward passes for a neural network with one sigmoid
hidden layer. Fill in your implementation in q2 neural.py. Sanity check your implementation with
q2 neural.py.
3 word2vec (40 points + 5 bonus)
(a) (3 points) Assume you are given a predicted word vector vc corresponding to the center word c for
skipgram, and word prediction is made with the softmax function found in word2vec models
yˆo = p(o | c) = exp(u

o vc)
PW
w=1 exp(u
w vc)
(4)
where w denotes the w-th word and uw (w = 1, . . . , W) are the “output” word vectors for all words in
the vocabulary. Assume cross entropy cost is applied to this prediction and word o is the expected word
(the o-th element of the one-hot label vector is one), derive the gradients with respect to vc.
Page 2 of 4
CS 224d: Assignment #1
Hint: It will be helpful to use notation from question 2. For instance, letting yˆ be the vector of softmax
predictions for every word, y as the expected word vector, and the loss function
Jsof tmax−CE(o, vc, U) = CE(y, yˆ) (5)
where U = [u1,u2, · · · ,uW ] is the matrix of all the output vectors. Make sure you state the orientation
of your vectors and matrices.
(b) (3 points) As in the previous part, derive gradients for the “output” word vectors uw’s (including uo).
(c) (6 points) Repeat part (a) and (b) assuming we are using the negative sampling loss for the predicted
vector vc, and the expected output word is o. Assume that K negative samples (words) are drawn, and
they are 1, · · · , K, respectively for simplicity of notation (o /∈ {1, . . . , K}). Again, for a given word, o,
denote its output vector as uo. The negative sampling loss function in this case is
Jneg−sample(o, vc, U) = − log(σ(u

o vc)) −
X
K
k=1
log(σ(−u

k vc)) (6)
where σ(·) is the sigmoid function.
After you’ve done this, describe with one sentence why this cost function is much more efficient to
compute than the softmax-CE loss (you could provide a speed-up ratio, i.e. the runtime of the softmaxCE loss divided by the runtime of the negative sampling loss).
Note: the cost function here is the negative of what Mikolov et al had in their original paper, because we
are doing a minimization instead of maximization in our code.
(d) (8 points) Derive gradients for all of the word vectors for skip-gram and CBOW given the previous parts
and given a set of context words [wordc−m, . . . , wordc−1, wordc, wordc+1, . . . , wordc+m], where m is the
context size. Denote the “input” and “output” word vectors for wordk as vk and uk respectively.
Hint: feel free to use F(o, vc) (where o is the expected word) as a placeholder for the Jsof tmax−CE(o, vc, …)
or Jneg−sample(o, vc, …) cost functions in this part — you’ll see that this is a useful abstraction for the
coding part. That is, your solution may contain terms of the form ∂F (o,vc)
∂… .
Recall that for skip-gram, the cost for a context centered around c is
Jskip-gram(wordc−m…c+m) = X
−m≤j≤m,j6=0
F(wc+j , vc) (7)
where wc+j refers to the word at the j-th index from the center.
CBOW is slightly different. Instead of using vc as the predicted vector, we use vˆ defined below. For (a
simpler variant of) CBOW, we sum up the input word vectors in the context
vˆ =
X
−m≤j≤m,j6=0
vc+j (8)
then the CBOW cost is
JCBOW(wordc−m…c+m) = F(wc, vˆ) (9)
Note: To be consistent with the vˆ notation such as for the code portion, for skip-gram vˆ = vc.
(e) (12 points) In this part you will implement the word2vec models and train your own word vectors
with stochastic gradient descent (SGD). First, write a helper function to normalize rows of a matrix in
q3 word2vec.py. In the same file, fill in the implementation for the softmax and negative sampling cost
and gradient functions. Then, fill in the implementation of the cost and gradient functions for the skipgram model. When you are done, test your implementation by running python q3 word2vec.py.
Note: If you choose not to implement CBOW (part h), simply remove the NotImplementedError so that
your tests will complete.
Page 3 of 4
CS 224d: Assignment #1
(f) (4 points) Complete the implementation for your SGD optimizer in q3 sgd.py. Test your implementation by running python q3 sgd.py.
(g) (4 points) Show time! Now we are going to load some real data and train word vectors with everything
you just implemented! We are going to use the Stanford Sentiment Treebank (SST) dataset to train
word vectors, and later apply them to a simple sentiment analysis task. There is no additional code to
write for this part; just run python q3 run.py.
Note: The training process may take a long time depending on the efficiency of your implementation
(an efficient implementation takes approximately an hour). Plan accordingly!
When the script finishes, a visualization for your word vectors will appear. It will also be saved as
q3 word vectors.png in your project directory. Include the plot in your homework write up.
Briefly explain in at most three sentences what you see in the plot.
(h) Extra credit (5 points) Implement the CBOW model in q3 word2vec.py. Note: This part is optional
but the gradient derivations for CBOW in part (d) are not!.
4 Sentiment Analysis (20 points)
Now, with the word vectors you trained, we are going to perform a simple sentiment analysis. For each
sentence in the Stanford Sentiment Treebank dataset, we are going to use the average of all the word vectors
in that sentence as its feature, and try to predict the sentiment level of the said sentence. The sentiment
level of the phrases are represented as real values in the original dataset, here we’ll just use five classes:
“very negative”, “negative”, “neutral”, “positive”, “very positive”
which are represented by 0 to 4 in the code, respectively. For this part, you will learn to train a softmax
regressor with SGD, and perform train/dev validation to improve generalization of your regressor.
(a) (10 points) Implement a sentence featurizer and softmax regression. Fill in the implementation in
q4 softmaxreg.py. Sanity check your implementation with python q4 softmaxreg.py.
(b) (2 points) Explain in fewer than three sentences why we want to introduce regularization when doing
classification (in fact, most machine learning tasks).
(c) (4 points) Fill in the hyperparameter selection code in q4 sentiment.py to search for the “optimal”
regularization parameter. What value did you select? Report your train, dev, and test accuracies. Justify your hyperparameter search methodology in at most one sentence. Note:
you should be able to attain at least 30% accuracy on dev.
(d) (4 points) Plot the classification accuracy on the train and dev set with respect to the regularization
value, using a logarithmic scale on the x-axis. This should have been done automatically. Include
q4 reg acc.png in your homework write up. Briefly explain in at most three sentences what you
see in the plot.
Page 4 of 4