Homework 4: Least squares

CS/ECE/ISyE 524 Introduction to Optimization 
Homework 4: Least squares

See the course website for instructions and submission details.
1. [10 pts] Moving averages. There are many ways to model the relationship between an input sequence
{u1, u2, . . . } and an output sequence {y1, y2, . . . }. In class, we saw the moving average (MA) model,
where each output is approximated by a linear combination of the k most recent inputs:
MA: yt ≈ b1ut + b2ut−1 + · · · + bkut−k+1
We then used least-squares to find the coefficients b1, . . . , bk. What if we didn’t have access to the
inputs at all, and we were asked to predict future y values based only on the previous y values? One
way to do this is by using an autoregressive (AR) model, where each output is approximated by a
linear combination of the ` most recent outputs (excluding the present one):
AR: yt ≈ a1yt−1 + a2yt−2 + · · · + a`yt−`
Of course, if the inputs contain pertinent information, we shouldn’t expect the AR method to outperform the MA method!
a) Using the same dataset from class uy_data.csv, plot the true y, and on the same axes, also plot
the estimated ˆy using the MA model and the estimated ˆy using the AR model. Use k = 5 for
both models. To quantify the difference between estimates, also compute ky − yˆk for both cases.
b) Yet another possible modeling choice is to combine both AR and MA. Unsurprisingly, this is
called the autoregressive moving average (ARMA) model:
ARMA: yt ≈ a1yt−1 + a2yt−2 + · · · + a`yt−` + b1ut + b2ut−1 + · · · + bkut−k+1
Solve the problem once more, this time using an ARMA model with k = ` = 1. Plot y and ˆy as
before, and also compute the error ky − yˆk.
2. [10 pts] Voltage smoothing. We would like to send a sequence of voltage inputs to the manipulator
arm of a robot. The desired signal is shown in the plot below (also available in voltages.csv)
0 50 100 150 200
time
−2
−1
0
1
2
voltage
Unfortunately, abrupt changes in voltage cause undue wear and tear on the motors over time, so we
would like to modify the signal so that the transitions are smoother. If the voltages above are given
by v1, v2, . . . , v200, one way to characterize smoothness is via the sum of squared differences:
R(v) = (v2 − v1)
2 + (v3 − v2)
2 + · · · + (v200 − v199)
2
When R(v) is smaller, the voltage is smoother. Solve a regularized least squares problem that explores
the tradeoff between matching the desired signal above and making the signal smooth. Explain your
reasoning, and include a plot comparing the desired voltages with your smoothed voltages.
1 of 2
CS/ECE/ISyE 524 Introduction to Optimization L. Lessard, Spring 2017
3. [10 pts] Spline fitting. We are running a series of experiments to evaluate the properties of a
new fluorescent material. As we vary the intensity of the incident light, the material should fluoresce
different amounts. Unfortunately, the material isn’t perfectly uniform and our method for measuring
fluorescence is not very accurate. After testing 200 different intensities, we obtained the result below
(also available in xy_data.csv). The intensities xi and fluorescences yi are recorded in the first and
second columns of the data matrix, respectively.
0 1 2 3 4 5 6 7 8 9 10
light intensity
0.0
0.2
0.4
0.6
0.8
1.0
fluorescence
The material has interesting nonlinear properties, and we would like to characterize the relationship
between intensity and fluorescence by using an approximate model that agrees well with the trend of
our experimental data. Although there is noise in the data, we know from physics that the fluorescence
must be zero when the intensity is zero. This fact must be reflected in all of our models!
a) Polynomial fit. Find the best cubic polynomial fit to the data. In other words, look for a
function of the form y = a1x
3 + a2x
2 + a3x + a4 that has the best possible agreement with the
data. Remember that the model should have zero fluorescence when the intensity is zero! Include
a plot of the data along with your best-fit cubic on the same axes.
b) Spline fit. Instead of using a single cubic polynomial, we will look for a fit to the data using two
quadratic polynomials. Specifically, we want to find coefficients pi and qi so that our data is well
modeled by the piecewise quadratic function:
y =
(
p1x
2 + p2x + p3 if 0 ≤ x < 4
q1x
2 + q2x + q3 if 4 ≤ x < 10
These quadratic functions must be designed so that:
• as in the cubic model, there is zero fluorescence when the intensity is zero.
• both quadratic pieces have the same value at x = 4.
• both quadratic pieces have the same slope at x = 4.
In other words, we are looking for a smooth piecewise quadratic. This is also known as a spline
(this is just one type of spline, there are many other types!). Include a plot of the data along with
your best-fit model.
2 of 2