COMS 4771 HW0

COMS 4771 HW0

This is a calibration assignment (HW0). The goal of this assignment is for you to recall basic concepts, and get familiarized with the homework submission system (Gradescope). Everyone
enrolled must submit this assignment by the due date. The score received on this assignment will
not count towards your final grade in this course. You must show your work to receive full credit.
You should cite all resources (including online material, books, articles, help taken from specific
individuals, etc.) you used to complete your work.
This homework assignment is to be done individually. All homeworks (including this one)
should be typesetted properly in pdf format. Handwritten solutions will not be accepted. You must
include your name and UNI in your homework submission.
1.1 [Probability and Statistics] Let X and Y be discrete random variables, and consider the joint
distribution (X, Y ) given by
Y=1 Y=2 Y=3
X=1 0.1 0.2 0.3
X=2 0.2 0.1 0.1
(i) What is the marginal distribution of X?
(ii) What is Pr[Y = 1|X = 2]?
(iii) Let f : x 7→ x
2
. What is E[f(X)|Y = 1]?
1.2 Fix some θ 0, and consider the function gθ : [0, ∞) → R defined as x 7→ 1
θ
e
− x
θ .
(i) Verify that gθ is a probability distribution.
(ii) Let X be a random variable distributed as gθ. What is E[X]?
(iii) What is Variance(X)?
1.3 You live with your cat in an apartment in a relatively safe neighborhood with a crime rate of
5%. Being a cautious person, you have invested in a good quality burglar alarm that is rated
to be 99% effective at the time of a break-in. Over time you have observed that your cat likes
to play with the alarm causing it to trip 10% of the time. One day while you are at work your
neighbor calls you telling that the alarm is ringing. What is the probability that there was a
break-in?
1
2.1 [Linear Algebra] Consider the subspace S spanned by vectors 1
2
3
4
!
,
2
8
3
2
!
,
3
10
6
6
!
in R
4
.
(i) What is the dimension of the subspace S.
(ii) Compute the orthogonal linear projection of the point 6
5
9
2
!
, onto the subspace S.
2.2 Prove that for any m × n real matrix A and any ρ 0, the matrix A
TA + ρI is invertible
(where I is the n × n identity matrix). (Hint: show that all eigenvalues of A
TA + ρI are real
and analyze the smallest eigenvalue.)
3.1 [Calculus and optimization] Let A ∈ R
d×D be a real matrix, and b ∈ R
d
be a real vector.
Define the function
f : R
D → R
x 7→ kAx − bk
2 + kxk
2
.
(i) What is ∇f(x)?
(ii) What value of x minimizes f, that is, find arg minx f(x)? (Hint: compute the stationary
points of f.)
4.1 [Programming practice] Download the Matlab data file hw0data.mat (instructions on Piazza on where to download the file). Write a script that does the following.
Special note for those who are not using Matlab: Python users can use scipy to read in
the mat file, R users can use R.matlab package to read in the mat file, Julia users can use
JuliaIO/MAT.jl. Octave users should be able to load the file directly.
(i) Load the data in hw0data.mat. It contains one matrix variable is called M.
(ii) Print the dimensions of M.
(iii) Print the 4th row and 5th column entry of M.
(iv) Print the mean value of the 5th column of M.
(v) Compute the histogram of the 4th row of M and show the figure.
(vi) Compute and print the top three eigenvalues of the matrix MTM.
4.2 We will try to understand the geometry of eigenvectors and eigenvalues of a matrix via experimentation. Let L =
"
5/4 −3/2
−3/2 5 #
be a 2 × 2 matrix. To understand eigenvectors and
eigenvalues, we will study the action of L on random vectors and relate it to eigenvectors and
eigenvalues. Write a script that does the following.
(i) Create the 2 × 2 matrix L (as defined above).
2
(ii) Create 500 random, unit length, two-dimensional vectors. (Hint: to generate a random
d-dimensional unit length vector, draw d independent samples from the Gaussian distribution N(0, 1) and assign each sample as one component of the vector. Now, normalize
the vector to have length one.) Let R be the set of these 500 random 2-dimensional unit
vectors.
(iii) For each vector r ∈ R, compute how the matrix L “distorts” r, that is, compute ˜r := Lr.
(iv) Compute the eigenvalues of L. Let λmax and λmin denote the maximum and the minimum eigenvalue respectively.
(v) For each distorted vector ˜r, compute the length k˜rk.
(vi) Create a histogram of values of k˜rk (use 50 bins) and compare it to λmax and λmin.
(vii) What relationship can you infer between k˜rk, λmax and λmin?
(viii) Now, compute the eigenvectors of L. Let vmax denote the eigenvector corresponding to
the maximum eigenvalue λmax.
(ix) Make a two-dimensional plot of all the distorted vectors ˜r (in blue color) and the eigenvector Lvmax (in red color). (make sure that the x- and the y-axis are displayed at the
same scale).
(x) What can you infer about the vmax from studying this plot?
3