EECE5644 - Homework 1

EECE5644 - Homework 1
Please submit your solutions to Blackboard in a single PDF file that includes all math and
numerical results in the main body (please start a new page for each question). Make sure that
you cite all resources you benefit from (books, papers, software packages you use) as appropriate
in your submissions. Also include your code that is used to generate numerical results for your
answers in one of the following ways: (Undesirable) copy/paste into an appendix section of the
PDF document, (Acceptable) upload a ZIP file containing all code files, (Preferred) keep your code
in an online version control repository (such as GitHub, BitBucket, etc) and as part of your answer
to each question that requires code, provide a link to the relevant online repository.
Note that this is a graded assignment and the entirety of your submission must contain only
your own work. You may use written or other forms of literature and software packages available
to you, as long as these sources are properly acknowledged in your submission as cited references.
Question 1 (10%)
1. Let x ∈ R be a random variable with mean E[x] = µ. Show that the variance of x, E[(x−µ)
2
]
is E[x
2
]− µ
2
.
2. Let x ∈ R
n be an n-dimensional random vector with mean E[x] = µ. Show that the covariance
matrix of x, E[(x− µ)(x− µ)
T
] is E[xxT
]− µµT
.
Question 2 (20%)
Suppose that a real-valued measurement x is generated by one of two equally probable probability density functions (pdfs), both of the parametric form p(x|L = l) ∝ e
−|x−al
|/bl for l ∈ 1,2 and
bl > 0 for both pdfs.
1. Find the normalization factor for this parametric pdf family as a function of the respective a
and b parameters, such that these pdfs integrate to 1 along the real axis.
2. Determine the simplest analytical expression for the log-likelihood ratio between class labels
1 and 2 evaluated at a given x, `(x) = lnp(x|L = 1)−lnp(x|L = 2).
3. Generate a plot of this log-likelihood-ratio function for the case a1 = 0,b1 = 1 and a2 =
1,b2 = 2 using a suitable programming language. Label the axes of the plot properly as `(x)
and x using legible font sizes, and include an intelligible title and caption for the plot. In
all future assignments, this will be expected without explicit instruction. All figures must be
properly labeled, titled, and captioned. If there are multiple curves or data visualizations in
the plot also include a legend to help identify each component.
Question 3 (20%)
Consider a two-class setting where class prior probabilities are P(L = 1) = P(L = 2) = 1/2. We
measure a one dimensional real-valued feature variable x, which takes values from Uni f orm[a,b]
for class 1, and Uni f orm[r,t] for class 2. Here a < r < b < t, so the support of these two uniform
class-conditioned feature pdfs p(x—L=1) and p(x—L=2) overlap. Derive a minimum probability
of error classification rule.
1
Question 4 (30%)
Consider a two-class setting with equal class priors and a real-valued feature with Gaussian
class-conditioned pdfs N (0,1) and N (µ,σ
2
).
1. Derive an expression for the classification/decision rule that achieves minimum probability
of error. This will be a function of x that depends on parameters µ and σ. Please simplify
the expression to a nice form.
2. For the case µ = 1 and σ
2 = 2 generate plots that visualize the class-conditional pdfs p(x|L =
l) for l ∈ 1,2, as well as class posterior probabilities p(L = l|x) for l ∈ 1,2. Demonstrate the
decision boundary you found earlier in these visualizations.
3. Numerically estimate the achieved minimum probability of error for the specific case you
visualized above. Describe your procedure and the numerical result intelligibly.
4. Consider the case where µ = 0 and σ  1 for the second class conditional pdf. Discuss
what happens to the decision boundary and regions. Imagine and briefly describe a practical
situation when this kind of data distribution might arise.
Question 5 (20%)
The pdf of an n-dimensional random vector drawn from N (µ,Σ) is
p(ζ ) = (2π)
−n/2
|Σ|
−1/2
e
−
1
2
(ζ −µ)
TΣ
−1
(ζ −µ)
(1)
1. Determine the pdf of x: Let x = Az + b be an n-dimensional random vector where A ∈
R
n×n
and b ∈ R
n
are fixed and known, z ∼ N (0,I). Assuming that A is full rank, and by
using/citing an appropriate theorem from probability theory, show that x ∼ N (b,AAT
).
2. Determine Linear Transformation: We want to generate a random vector drawn from
N (µ,Σ) using the linear transformation technique introduced and studies above. Determine
A and b to achieve the desired mean vector and covariance matrix. Specifically, determine
A and b in terms of µ and Σ.
3. Implement Code: Write code that takes in N, n, µ, and Σ and produces N samples of independent and identically distributed (iid) n-dimensional random vectors {x1,...,xN} drawn
from N (µ,Σ) using the linear transformation technique applied to samples of z ∼ N (0,I).
2