Homework 3
1. Implement the Logistic Regression learning by gradient ascent as described in class.
Before using logistic regression, be sure to normalize the variables of the training set
to have zero mean and standard deviation 1, and to do the exact same transformation to
the test set, using the mean and standard deviation of the training set.
a) Using the Gisette data, train a logistic regressor on the training set, starting
with w(0) = 0, with 300-1000 gradient ascent iterations and shrinkage λ =
0.0001 in the update equation:
w(t+1) ← w(t) − ηλw(t) +
η
N
∂
∂w
L(w(t)
)
Observe that there is an extra factor of 1/N in the loss term compared to the
class notes.
Find a good learning rate η such that the log-likelihood converges in 300-1000
iterations and is monotonically increasing. Plot the log-likelihood vs iteration
number. Report in a table the misclassification error on the training and test set.
(4 points)
b) Repeat point a) on the madelon dataset. (2 points)
c) Repeat point a) on the dexter dataset. (2 points)
1
