Homework 3  Activation Function in Neural Networks

Homework 3 

Instructions for homework submission
Please submit on eCampus a single pdf file containing your solutions.
a) Please typewrite in Latex the answers to the math problems. If this is not possible, please
handwrite your solution very clearly, scan it and merge it to the final pdf file. Make sure that
your solution is visible after scanning. Non-visible solutions will not be graded: we wouldn’t
like our TA to have to guess what you are writing :)
b) Please write a brief report for the experimental problems. At the end of the pdf file, please
include your code. The code has to be directly converted instead of scanned (i.e. the text in
the code must be selectable).
c) Please start early :)
Question 1: Activation Function in Neural Networks
Consider a two-layer fully-connected network with D input features, one hidden layer with M
nodes, and one output layer. The activation function of each node is a sigmoid function of the
form
sigmoid(α) = 1
1 + exp(−α)
(a) Provide a schematic representation of the network. Define the variables for the input, the
output, and the weights.
(b) Express the output as a function of the input.
(c) Calculate the number of parameters that need to be inferred.
(d) Show that there exists an equivalent network with hidden unit activation functions given
by the hyperbolic tangent, which computes exactly the same function, where the hyperbolic
tangent is given by
tanh(α) = exp(α) − exp(−α)
exp(α) + exp(−α)
Hint: First find the relation between sigmoid(α) and tanh(α), then show that the parameters
of the two networks differ by linear transformations.
Question 2: Image processing for human faces
In this problem, we will process face images coming from the Yale Face Dataset: http:
//vision.ucsd.edu/content/yale-face-database. This dataset contains images of the faces
of 15 individuals. For each individual there are 11 images taken under a variety of conditions
e.g., the person makes a happy expression, wears glasses etc.
1
(a) Download the dataset from the above URL. Implement Principal Component Analysis
(PCA) on the input images. Assume that the input vector of PCA contains all rows of an
image stacked one on top of the other. You can use available libraries that calculate eigenvalues
and eigenvectors of a matrix. Hint: Don’t forget to normalize the data.
(b) Plot a curve displaying the first k eigenvalues λ1, . . . , λK, i.e. the energy of the first K
principal components. How many components do we need to capture 50% of the energy?
(c) Plot the top 10 eigenfaces, i.e. the eigenvectors uk, k = 1, . . . , 10 obtained by PCA.
(d) Select a couple of images from the data. Use the first k eigenfaces as a basis to reconstruct
the images. Visualize the reconstructed images using 1, 10, 20, 30, 40, 50 components. How
many components do we need to achieve a visually good result?
Hint: Reconstruction of an input vector x based on the eigenvectors u1, . . . , uK is given by the
following expression x ≈ x0 +
PK
k=1 ckuk, where ck = u
T
k
x is the projection of the input image
to the k
th eigenvector.
(e) Perform face recognition: Split the input data into training and testing making sure that
every person is included in each set. Use any of the classification methods that we have learnt
so far to recognize a person’s identity. Use as input features the transformed feature space that
resulted from PCA. Experiment with different number of PCA components through a 5-fold
cross-validation. Report the recognition accuracy on the test set.
(f) Bonus: Data augmentation: Data augmentation is a way to increase the size of our dataset
and reduce overfitting, especially when we use complicated models with many parameters to
learn. Using any available toolbox or your own code, implement some of these techniques and
augment the original Yale Face Dataset.
Hint: You can find more information in hw3 DataAugmentationUsingDeepLearning.pdf from
Homework 3 folder on Piazza and in the following link:
https://machinelearningmastery.com/image-augmentation-deep-learning-keras/
(g) Bonus: Face recognition with CNNs: Use CNNs to perform face recognition in the augmented Yale Face Dataset. Use the same split for the train and test set as in Question 2e.
Experiment with different CNN parameters, e.g. filter size, stride size, activation function,
dropout, pool size, etc.
2