Homework 3
1. Implement the Logistic Regression learning by gradient ascent as described in class.
Before using logistic regression, be sure to normalize the variables of the training set
to have zero mean and standard deviation 1, and to do the exact same transformation to
the test set, using the mean and standard deviation of the training set.
a) Using the Gisette data, train a logistic regressor on the training set, starting
with w(0) = 0, with 300 gradient ascent iterations and shrinkage λ = 0.0001 in
the update equation:
w(t+1) ← w(t) − ηλw(t) −
η
N
∂
∂w
L(w(t)
)
where L(w(t)
) is the negative log likelihood. Observe that there is an extra factor
of 1/N in the loss term compared to the class notes.
Find a good learning rate η such that the negative log-likelihood converges in
at most 300 iterations and is monotonically decreasing. Plot the negative loglikelihood vs iteration number. Report in a table the misclassification error on
the training and test set. (4 points)
b) Repeat point a) on the hill-valley dataset. (2 points)
c) Repeat point a) on the dexter dataset. (2 points)
1
