Assignment #1 MLE & MAP

Assignment #1
Machine Learning
Posterior Examples
Problem 1. In this problem we will find the maximum likelihood estimator (MLE) and maximum a posteriori (MAP) estimator for the mean of a univariate normal distribution. Specifically, we assume that we have N samples, x1,...,xN independently drawn from a normal distribution with known variance σ2 and unknown mean µ.
a) Derive the MLE estimator for the mean µ. Make sure to show all of your work.
b) Now derive the MAP estimator for the mean µ. Assume that the prior distribution for the mean is itself a normal distribution with mean ν and variance β2. Please show all of your work. HINT: You may want to make use of the fact that: β2 N X i=1 (xi −µ)2!+ σ2(µ−ν)2 = µpNβ2 + σ2 − σ2ν + β2PN i=1 xi pNβ2 + σ2 !2 − (σ2ν + β2PN i=1 xi)2 Nβ2 + σ2 + β2 N X i=1 x2 i! c) Please comment on what happens to the MLE and MAP estimators as the number of samples N goes to infinity. Problem 2. Suppose that x,w be independent random variables where w ∼N(0,σ2) and p(x = 0) = q,p(x = 1) = 1 − q. Find the function f : R → {0,1} that minimizes p(x 6= f(x + w)). Problem 3. Given b, y ∼ N(0.1 + b,0.1 + b) for b ∈ {0,1}. Assume that p(b = 0) = 0.2,p(b = 1) = 0.8. Given y = a, find the maxiumum a posteriori(MAP) rule: ˆ b(a).
Problem 4. Recall the Monty Hall problem that we discussed in class. We discussed the situation where there are 3 doors. Now, we further extend the problem with n doors. In other words, the host opens one door, and you are given a chance to change the door to any of the other n−2 doors. In this situation, what is the probability of winning a prize when you a) change your first choice and b) do not?
1
Problem 5. Suppose that there are two species of panda bears. Both are equally common in the wild and live in the same place. They look exactly alike and eat the same food, and there is yet no genetic assay capable of telling them apart. They differ, however, in their family sizes. Species A gives birth to twins 10% of the time, otherwise birthing a single infant. Species B births twins 20% of the time, otherwise birthing a single infant. Assume that these numbers are known with certainty, from many years of field research.
a) Suppose you have a new female panda of unknown species, and she has just given birth to twins. What is the probability that her next birth will also be twins?
b) Suppose that the same panda mother has a second birth and that it is not twins, but a single infant. What is the probability that this panda is species A?
Problem 6. Show that the beta distribution is a conjugate prior of the binomial distribution. Interpret the effect of the beta distribution on the posterior.
Problem 7. (Simple Programming) Let X1,X2,··· ,X10 | θ are i.i.d. Binary(θ). Given observed data (X1,X2,··· ,X10) = (x1,x2,··· ,x10) withP10 j=1 xj = 5 and a) uniform prior distribution over θ, i.e. p(θ) = 1 for all θ ∈ [0,1], b) beta prior distribution over θ ∼ beta(3,2), plot the posterior distributions over θ for range θ ∈ {0.4,0.41,··· ,0.6} for each case. (If needed, you should use normalization constants a) 2772, b) 24024.)
Monte Carlo Examples
From here, we introduce the concept of generating observations from a specified distribution or sample. This is often called Monte Carlo generation or Monte Carlo method. For the most part, a generator of random uniform observations is all that we need.
Simple Example: Consider an experiment where a fair six-sided die is rolled and the random variable X is 1 if the upface is in {1,2}, 0 otherwise. Then the mean of the random variable X is 1 3. (Since, E[X] = 1 3 ·1 + 2 3 ·0.) If the random variable U ∼U(0,1), we canrecognize the random variable X by X =(1 for 0 < U ≤ 1 3 0 for 1 3 < U < 1 Using this fact, we can draw random samples x1,x2,...,xn with the uniform distribution U(0,1), and 1 nPn j=1 xj will be the unbiased estimate of E[X]. (in this case, 1 3) Problem 1. (Programming, Estimation of π) Similarly to the simple example above, consider the experiment where a pair of numbers (U1,U2) is chosen at random in the unit square [0,1]×[0,1]; that is, U1 and U2 are i.i.d. U(0,1) random variables. Since the points will be chosen at random, the probability of (U1,U2) lying within the unit circle is π 4. Let X be the random variable, X =(1 if U2 1 + U2 2 < 1 0 otherwise.
2
With this random variable and the uniform distribution U(0,1), write your program estimating the value of π. Your program should return the list estimating results for sample size n = 100,500,1000,10000,100000.
Problem 2. (Hand, Monte Carlo Integration) For a continuous function f over the closed and bounded interval [a,b], we want to compute the integralRb a f(x)dx. If the antiderivative of f does not exist (ex. f(x) = sin(x2),exp(−x2), etc.), numerical integration using the method of Monte Carlo can be made. Derive the unbiased estimator of this integral in closed form using the uniform distribution. (Hint: Rb a f(x)dx = (b−a)Rb a f(x) 1 b−a dx = (b−a)E[f(X)] where X ∼U(a,b)). Problem 3. (Programming, Estimation of π with Monte Carlo Integration) Using your unbiased estimator in problem 2, estimate the π with a suitable choice of function f. Your program should return the list of estimating results for sample size n = 100,500,1000,5000,10000.
Instructions and Some Hints • You should write a program in Python (Version: 2.7 or 3.5) • We highly recommend that you implement using NumPy. (Hint: np.random.rand will give you a random number generator) • Since we deal with random numbers, you should set fixed random seed number for reproducibility! (This is quite important. In NumPy, it can be done by the function np.random.seed) • For plotting the distribution, Matplotlib library will help you. • Please submit your homework in [StudentID Homework1].zip on KLMS. Your .zip should contain 6 files, which are solutions for problems (in .pdf or .docx, whatever), three .py files for each programming problem, and pictures of plotted distributions for Problem 7 (.png, or .jpg whatever). • Feel free to ask questions on KLMS Q&A board!
3