EECS 491 Assignment 1

EECS 491 Assignment 1
 140 points total.
Submitting assignments to Canvas
For jupyter notebooks, submit the .ipynb file and a pdf export of the notebook.
Make sure you check that the pdf export represents the latest state of your notebook and that the equations
and figures are properly rendered.
If your are not using notebooks, writeup your assignment using latex and submit a pdf with your code. The
writeup should include relevant code with description if it can fit on a page.
Use the following format for filenames:
EECS491-A1-yourcaseid.ipynb
EECS491-A1-yourcaseid.pdf
If you have more than these two files, put any additional files in a directory named EECS491-A1-
yourcaseid . Do not include binaries or large data files. Then zip this directory and submit it with the name
EECS491-A1-yourcaseid.zip . Do not use other compression formats. The .ipynb file can be included in
the zipped directory, but make sure you submit the .pdf file along with the .zip file. This is so it appears at
the top level on canvas, which allows for easier grading.
Some of questions below aren't specified in great detail and you may need to spend sometime making sense of the
questions themselves, which you can do from the reads and other sources. You also might need to fill in some
blanks or make some assumptions. The spirit behind this approach is explained in The Problem with Problems by
Eric Mazur, which I encourage everyone to read.
Put your name and Case ID here
Q1. Basic probability (10 pts)
In the proofs below you should use general probability distributions (as opposed to specific examples) and the
basic laws of probability. Be concise and clear. The proof should be in terms of mathematical facts of probability
theory.
1.1. Prove (5 pts)
1.2. Prove (5 pts)
Q2. Independence (10 pts)
Again these proofs should use general probability distributions and the basic laws of probability. Note that the proof
should be in terms of mathematical facts. It should not be an argument that depends on real-world knowledge. The
example should use common real-world knowledge and interpretation should convey the ideas of the proof.
2.1 Prove that independence is not transitive, i.e. . Define a joint probability distribution
 for which the previous expression holds and provide an example with an interpretation. (5 pts)
2.2 Prove that conditional independence does not imply marginal independence, i.e. . Again
provide an example that illustrates the statement. (5 pts)
Q3. Inspector Clouseau re-revisited (20 pts)
3.1 Write a program to evaluate in Example 1.3 in Barber. Write your code and choose your data
representations so that it is easy to use it to solve the remaining questions. Show that it correctly computes the
value in the example. (5 pts)
3.2 Define a different distribution for . Your new distribution should result in the outcome that
is either or , i.e. reasonably strong evidence. Use the original values of and from the
example. Provide (invent) a reasonble justification for the value of each entry in . (5 pts)
3.3 Derive the equation for . (5 pts)
3.4 Calculate it's value for both the original and the one you defined yourself. Is it possible to provide a
summary of the main factors that contributed to the value? Why/Why not? Explain. (5 pts)
Q4. Biased views (20 pts)
4.1 Write a program that calculates the posterior distribution of the (probability of heads) from the Binomial
distribution given heads out of trials. Feel to use a package where the necessary distributions are defined as
primitives. (5 pts)
4.2 Imagine three different views on the coin bias:
"I believe strongly that the coin is biased to either mostly heads or mostly tails."
"I believe strongly that the coin is unbiased".
"I don't know anything about the bias of the coin."
Define and plot prior distributions that expresses each of these beliefs. Provide a brief explanation. (5 pts)
4.3 Perform Bernoulli trials where one of these views is correct. Show how the posterior distribution of changes
for each view for =0, 1, 2, 5, 10, and 100. Each view should have its own plot, but with the curves of the posterior
after different numbers of trials overlayed. (5 pts)
4.4 Is it possible that each view will always arrive at an accurate estimate of ? How might you determine which
view is most consistent with the data after trials? (5 pts)
Q5. Inference using the Poisson distribution (20 pts)
Suppose you observe for 3 seconds and detect a series of events that occur at the following times (in seconds):
0.53, 0.65, 0.91, 1.19, 1.30, 1.33, 1.90, 2.01, 2.48.
5.1 Model the rate at which the events are produced using a Poisson distribution where is the number of events
observed per unit time (1 second). Show the likelihood equation and plot it for three different values of : less,
about equal, and greater than what you estimate (intuitively) from the data. (5 pts)
5.2 Derive the posterior distribution of assuming a Gamma prior (usually defined with parameters and ). The
posterior should have the form where is the total duration of the observation period and is the
number of events observed within that period. (5 pts)
5.3 Show that the Gamma distribution is a conjugate prior for the Poisson distribution, i.e. it is also a Gamma
distribution, but defined by parameters and that are functions of the prior and likelihood parameters. (5 pts)
5.4 Plot the posterior distribution for the data above at times = 0, 0.5, and 1.5. Overlay the curves on a single plot.
Comment how it is possible for your beliefs to change even though no new events have been observed. (5 pts)
Q6. Probability Distribution Example (20 pts)
In this problem you will illustrate a probability distribution in a settings of your choosing. It can be discrete or
continuous. This is meant to be a simpler version of the letter seqeunce example shown in class (so don't use that).
Your example should use two random variables that each have at least three distinct values (if it is discrete), i.e.
don't use binary variables. The variables should not be independent, in other words, the setting you are modeling
should have structure, and ideally structure that is interesting and interpretable in some way. Your example should
include the following:
a decription of the setting
an illustration of the joint probability and how it captures the structure
an illustration of a conditional probability
an illustration of marginal probability
Note that "illustration" here means to explain with tables or figures that convey the ideas of the mathematical
operations. The motivation behind this exercise is to help you develop a better understanding of how joint
probability distributions model probabilistic structure in a simplified setting, so try to choose something you are
very familiar with. If find this is getting too long, you can continue it as part of the exploration, but there you will
also need to add and inference problem.
Exploration (40 pts)
In these problems, you are meant to do creative exploration. Define and explore:
E.1 A discrete inference problem (20 pts)
E.2 A continuous inference problem (20 pts)
This is meant to be open-ended; you should not feel the need to write a book chapter; but neither should you just
change the numbers in one of the problems above. After doing the readings and problems above, you should pick a
concept you want to understand better or an simple modeling idea you want to try out. You can also start to explore
ideas for your project. The general idea is for you to teach yourself (and potentially a classate) about a concept
from the assignments and readings or solidify your understanding of required technical background. For additional
guidance, see the grading rubric below.
You can use the readings and other sources for inspiration, but here are a few ideas:
An inference problem using categorical data
A disease for which there are two different tests
A two-dimensional continuous inference problem
The idea of a conjugate prior
Exploration Grading Rubric
Exploration problems will be graded according the elements in the table below. The scores in the column headers
indicate the number of points possible for each rubric element (given in the rows). A score of zero for an element is
possible if it is missing entirely.
Substandard (+1) Basic (+2) Good (+3) Excellent (+5)
Pedagogical
Value
No clear statement of
idea or concept being
explored or explained;
lack of motivating
questions.
Simple problem with
adequate motivation;
still could be a useful
addition to an
assignment.
Good choice of problem with
effective illustrations of
concept(s). Demonstrates a
deeper level of understanding.
Problem also illustrates or
clarifies common conceptual
difficulties or
misconceptions.
Novelty of
Ideas
Copies existing
problem or makes only
a trivial modification;
lack of citation(s) for
source of inspiration.
Concepts are similar to
those covered in the
assignment but with
some modifications of
an existing exericse.
Ideas have clear pedagogical
motivation; creates different
type of problem or exercise to
explore related or foundational
concepts more deeply.
Applies a technique or
explores concept not
covered in the assignment or
not discussed at length in
lecture.
Clarity of
Explanation
Little or confusing
explanation; figures
lack labels or useful
captions; no
explanation of
motivations.
Explanations are
present, but unclear,
unfocused, wordy or
contain too much
technical detail.
Clear and concise explanations
of key ideas and motivations.
Also clear and concise, but
includes illustrative figures;
could be read and
understood by students from
a variety of backgrounds.
Depth of
Exploration
Content is obvious or
closely imitates
assignment problems.
Uses existing problem
for different data.
Applies a variation of a
technique to solve a problem
with an interesting motivation;
explores a concept in a series of
related problems.
Applies several concepts or
techniques; has clear focus
of inquiry that is approached
from multiple directions.
p(x, y|z) = p(x|z)p(y|x, z)
p(x|y, z) = p(y|x, z)p(x|z)
p(y|z)
a ⊥ b ∧ b ⊥ c ⇏ a ⊥ c
p(a, b, c)
a ⊥ b|c ⇏ a ⊥ b
p(B|K)
p(K|M, B) p(B|K)
< 0.1 > 0.9 p(B) p(M)
p(K|M, B)
p(M|K)
p(K|M, B)
θ
y n
θ
n
θ
n
λ n
λ
λ α β
p(λ|n, T, α, β) T n
α′ β′
T
In [ ]: