Neural Networks, SGD, and Back Propagation

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Homework 4: Neural Networks, SGD, and Back Propagation \n",
"***\n",
"\n",
"**Name**: \n",
"\n",
"***\n",
"\n",
"This assignment is due on Moodle by **5pm on Friday March 23rd**. Your solutions to theoretical questions should be done in Markdown/MathJax directly below the associated question. Your solutions to computational questions should include any specified Python code and results as well as written commentary on your conclusions. Remember that you are encouraged to discuss the problems with your instructors and classmates, but **you must write all code and solutions on your own**. For a refresher on the course **Collaboration Policy** click [here](https://github.com/chrisketelsen/CSCI-4622-Machine-Learning/blob/master/resources/syllabus.md#collaboration-policy).\n",
"\n",
"**NOTES**: \n",
"\n",
"- Do **NOT** load or use any Python packages that are not available in Anaconda 3.6. \n",
"- Some problems with code may be autograded. If we provide a function API **do not** change it. If we do not provide a function API then you're free to structure your code however you like. \n",
"- Submit only this Jupyter notebook to Moodle. Do not compress it using tar, rar, zip, etc. "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:09.992631Z",
"start_time": "2018-03-13T09:10:08.906587Z"
},
"scrolled": false
},
"outputs": [],
"source": [
"import pickle, gzip\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pylab as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [15 points] Problem 1 - Single-Layer and Multilayer Perceptron Learning \n",
"***\n",
"\n",
"**Part A**: Consider learning the following concepts with either a single-layer or multilayer perceptron where all hidden and output neurons utilize indicator activation functions. For each of the following concepts, state whether the concept can be learned by a single-layer perceptron. **Briefly** justify your response: \n",
"\n",
"i. $~ \\texttt{ NOT } x_1$ \n",
"\n",
"ii. $~~x_1 \\texttt{ NOR } x_2$ \n",
"\n",
"iii. $~~x_1 \\texttt{ XNOR } x_2$ \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"scrolled": true
},
"source": [
"#### I. NOT $x_{1}$ can be learned by a single-layer perceptron. This is because the concept can be either 0 or 1 and if represented on a graph, these data points are linearly seperable. Meaning that it can be classifed by the perceptron.\n",
"\n",
"#### II. $x_{1}$ NOR $x_{2}$ can be learned by a single-layer perceptron. As data points, these values could be represented as:\n",
"|$x_{1}$| $x_{2}$|$~~x_1$ NOR $x_2$|\n",
"|-------|--------|------------------|\n",
"| 0 | 0 | 1 |\n",
"| 0 | 1 | 0 |\n",
"| 1 | 0 | 0 |\n",
"| 1 | 1 | 0 |\n",
"##### Which when plotted, it is clear that this data is linearly seperable as well.\n",
"\n",
"#### III. $x_{1}$ XNOR $x_{2}$ can NOT be learned by a single-layer perceptron because when these data points are plotted, they are not lineraly seperable.\n",
"\n",
"|$x_{1}$| $x_{2}$|$~~x_1$ XNOR $x_2$|\n",
"|-------|--------|------------------|\n",
"| 0 | 0 | 1 |\n",
"| 0 | 1 | 0 |\n",
"| 1 | 0 | 0 |\n",
"| 1 | 1 | 1 |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part B**: Determine an architecture and specific values of the weights and biases in a single-layer or multilayer perceptron with indicator activation functions that can learn $x_1 \\texttt{ XNOR } x_2$. Describe your architecture and state your weight matrices and bias vectors in Markdown below. Then demonstrate that your solution is correct by implementing forward propagation for your network in Python and showing that it correctly produces the correct boolean output values for each of the four possible combinations of $x_1$ and $x_2$. "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"scrolled": true
},
"source": [
"##### Truth table for $x_1 \\texttt{ XNOR } x_2$ taken from part III gives us the same logic as doing $(x_1 \\texttt{ OR } x_2) \\texttt{ NAND } (x_1 \\texttt{ NAND } x_2)$\n",
"\n",
"##### From lecture, we know the weights and beta for $OR$ is $w = [1,1]$ & $Beta = 0$. \n",
"##### From lecture we the weights and beta for $NAND$ is $w = [-1,-1]$ & $Beta = 1.5$"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1.]\n",
"[0.]\n",
"[0.]\n",
"[1.]\n"
]
}
],
"source": [
"def act(z):\n",
" for ii, ee in enumerate(z):\n",
" if ee 0:\n",
" z[ii] = 1\n",
" else:\n",
" z[ii] = 0\n",
" return z\n",
"\n",
"def xnor(x1, x2):\n",
" w1 = np.array([[1,1], [-1,-1]])\n",
" b1 = np.array([0,1.5])\n",
" w2 = np.array([-1, -1])\n",
" b2 = np.array([1.5])\n",
" x1 = np.array([x1, x2])\n",
" \n",
" z2 = np.dot(w1, x1) + b1\n",
" a2 = act(z2)\n",
" z3 = np.dot(w2, a2) + b2\n",
" \n",
" return act(z3)\n",
"\n",
"for (x1,x2) in [(0,0), (0,1), (1,0), (1,1)]:\n",
" print(xnor(x1,x2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [20 points] Problem 2 - Back Propagation and Deep Networks\n",
"***\n",
"\n",
"In this problem you'll gain some intuition about why training deep neural networks can be very time consuming. Consider training the chain-like neural network seen below: \n",
"\n",
"\n",
"\n",
"Note that this network has three weights $W^1, W^2, W^3$ and three biases $b^1, b^2,$ and $b^3$ (for this problem you can think of each parameter as a single value or as a $1 \\times 1$ matrix). Suppose that each hidden and output neuron is equipped with a sigmoid activation function and the loss function is given by \n",
"\n",
"$$\n",
"C(y, a^4) = \\frac{1}{2}(y - a^4)^2 \n",
"$$\n",
"\n",
"where $a^4$ is the value of the activation at the output neuron and $y \\in \\{0,1\\}$ is the true label associated with the training example. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part A**: Suppose each of the weights is initialized to $W^k = 1.0$ and each bias is initialized to $b^k = -0.5$. Use forward propagation to find the activities and activations associated with each hidden and output neuron for the training example $(x, y) = (0.5,0)$. Show your work. "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"##### With training example $X = 0.5$ and $y=0$ , our weight is initialized to $W^1 = 1.0$ and bias $b^{1} = -0.5$\n",
"\n",
"#### <center Layer 1\n",
"$$Z^{2} = W^{1} \\cdot X + b^{1}$$\n",
"\n",
"$$Z^{2} = 1 \\cdot 0.5 - 0.5$$\n",
"\n",
"$$\\boxed{Z^{2} = 0}$$\n",
"\n",
"$$a^{2} = g(Z^{2})$$\n",
"\n",
"###### Where the function g is defined as our activation function. $g(Z^{l+1}) = \\frac{1}{1 + e^{-Z^{l+1}}}$\n",
"\n",
"$$a^{2} = \\frac{1}{1 + e^{0}}$$\n",
"\n",
"$$\\boxed{a^{2} = 0.5}$$\n",
"\n",
"#### <center Layer 2\n",
"$$Z^{3} = W^{2} \\cdot a^{2} + b^{2}$$\n",
"\n",
"$$Z^{3} = 1 \\cdot 0.5 - 0.5$$\n",
"\n",
"$$\\boxed{Z^{3} =0}$$\n",
"\n",
"$$a^{3} = g(Z^{3})$$\n",
"\n",
"$$a^{3} = \\frac{1}{1 + e^{0}}$$\n",
"\n",
"$$\\boxed{a^{3} = 0.5}$$\n",
"\n",
"\n",
"#### <center Layer 3\n",
"$$Z^{4} = W^{3} \\cdot a^{3} + b^{3}$$\n",
"\n",
"$$Z^{4} = 1 \\cdot 0.5 - 0.5$$\n",
"\n",
"$$\\boxed{Z^{4} =0}$$\n",
"\n",
"$$a^{4} = g(Z^{4})$$\n",
"\n",
"$$a^{4} = \\frac{1}{1 + e^{0}}$$\n",
"\n",
"$$\\boxed{a^{4} = 0.5}$$\n",
"\n",
"##### In Summary, \n",
"\n",
"||$Z^{Layer + 1}$ | $a^{Layer+1}$ |\n",
"|------|:------:|-------|\n",
"|Layer1| 0 | 0.5 | \n",
"|Layer2| 0 | 0.5 | \n",
"|Layer3| 0 | 0.5 | "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part B**: Compute the value of $\\delta^4$ associated with the given training example. Show all work. "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"##### The general formula for finding this value will be $\\delta^{4} = (a^4 - y) \\odot g(z^4) \\odot (1-g(z^4))$\n",
"\n",
"$$\\delta^{4} = (0.5 - 0) \\odot g(0) \\odot (1-g(0))$$\n",
"\n",
"$$\\delta^{4} = \\frac{1}{2} \\odot \\frac{1}{2} \\odot \\frac{1}{2}$$\n",
"\n",
"$$\\boxed{\\delta^{4} = \\frac{1}{8}}$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part C**: Use Back-Propagation to compute the weight and bias derivatives $\\partial C / \\partial W^k$ and $\\partial C / \\partial b^k$ for $k=1, 2, 3$. Show all work. "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"#### In general, $\\frac{\\partial C}{\\partial W^{k}}= \\delta^{k+1}(a^{k})^T$ and $\\frac{\\partial C}{\\partial b^{k}} = \\delta^{k+1}$\n",
"\n",
"### <center For k=3\n",
"$$\\frac{\\partial C}{\\partial W^{3}}= \\delta^{4}(a^{3})^T$$\n",
"\n",
"$$\\frac{\\partial C}{\\partial W^{3}}= \\frac{1}{8}\\cdot \\frac{1}{2}$$\n",
"\n",
"$$\\boxed{\\frac{\\partial C}{\\partial W^{3}}= \\frac{1}{16}}$$\n",
"\n",
"$$\\frac{\\partial C}{\\partial b^{3}} = \\delta^{4}$$\n",
"\n",
"$$\\boxed{\\frac{\\partial C}{\\partial b^{3}} = \\frac{1}{8}}$$\n",
"\n",
"### <center For k=2\n",
"<center First we need $\\delta^3$\n",
" \n",
"$$\\delta^3 = (W^{3})^T \\delta^4 \\odot g\\prime(z^3)$$\n",
"$$\\delta^3 = (1)^T \\frac{1}{8} \\odot \\frac{1}{2} \\cdot \\frac{1}{2}$$\n",
"$$\\delta^3 = \\frac{1}{32}$$\n",
"\n",
"$$\\frac{\\partial C}{\\partial W^{2}}= \\delta^{3}(a^{2})^T$$\n",
"\n",
"$$\\frac{\\partial C}{\\partial W^{2}}= \\frac{1}{32} \\cdot \\frac{1}{2}$$\n",
"\n",
"$$\\boxed{\\frac{\\partial C}{\\partial W^{2}}= \\frac{1}{64}}$$\n",
"\n",
"$$\\boxed{\\frac{\\partial C}{\\partial b^{2}} = \\frac{1}{32}}$$\n",
"\n",
"\n",
"\n",
"### <center For k=1\n",
"\n",
"<center First we need $\\delta^2$\n",
" \n",
"$$\\delta^2 = (W^{2})^T \\delta^3 \\odot g\\prime(z^2)$$\n",
"\n",
"$$\\delta^2 = (1)^T \\frac{1}{32} \\odot \\frac{1}{2} \\cdot \\frac{1}{2}$$\n",
"\n",
"$$\\delta^2 = \\frac{1}{128}$$\n",
"\n",
"$$\\frac{\\partial C}{\\partial W^{1}}= \\delta^{2}(a^{1})^T$$\n",
"\n",
"$$\\frac{\\partial C}{\\partial W^{1}}= \\frac{1}{128} \\cdot \\frac{1}{2}$$\n",
"\n",
"$$\\boxed{\\frac{\\partial C}{\\partial W^{1}}= \\frac{1}{256}}$$\n",
"\n",
"$$\\boxed{\\frac{\\partial C}{\\partial b^{1}} = \\frac{1}{128}}$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part D**: Comment on your observations in **Part C**. In particular, compare the rate at which weights will be learned in the earlier layers to the later layers. What would happen if we had an even deeper network? "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"scrolled": true
},
"source": [
"##### Initially our learning rate is very small on the first layer and as we progress through the layers the derivatives increase which in turn increases our learning rates for the later layers. If we have a deeper network , the first layer weights would not be nearly as good compared to the weights farther back in the network but as the network gets deeper, the more time consuming it becomes to calculate the weights."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [40 points] Problem 3: Building and Training a Feed-Forward Neural Network \n",
"***\n",
"\n",
"In this problem you'll implement a general feed-forward neural network class that utilizes sigmoid activation functions. Your tasks will be to implement `forward propagation`, `prediction`, `back propagation`, and a general `train` routine to learn the weights in your network via Stochastic Gradient Descent. \n",
"\n",
"The skeleton for the `Network` class is below. Note that this class is almost identical to the one you worked with in the **Hands-On Neural Network** in-class notebook, so you should look there to remind yourself of the details. Scroll down to find more information about your tasks as well as unit tests. \n",
"\n",
"**Important Note**: In **Problem 4** we'll be using the `Network` class to train a network to do handwritten digit recognition. Please make sure to utilize vectorized Numpy routines as much as possible, as writing inefficient code here will cause very slow training times in **Problem 4**. "
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:15.724906Z",
"start_time": "2018-03-13T09:10:15.345793Z"
},
"scrolled": true
},
"outputs": [],
"source": [
"class Network:\n",
" def __init__(self, sizes):\n",
" \"\"\"\n",
" Initialize the neural network \n",
" \n",
" :param sizes: a list of the number of neurons in each layer \n",
" \"\"\"\n",
" # save the number of layers in the network \n",
" self.L = len(sizes) \n",
" \n",
" # store the list of layer sizes \n",
" self.sizes = sizes \n",
" \n",
" # initialize the bias vectors for each hidden and output layer \n",
" self.b = [np.random.randn(n) for n in self.sizes[1:]]\n",
" \n",
" # initialize the matrices of weights for each hidden and output layer \n",
" self.W = [np.random.randn(n, m) for (m,n) in zip(self.sizes[:-1], self.sizes[1:])]\n",
" \n",
" # initialize the derivatives of biases for backprop \n",
" self.db = [np.zeros(n) for n in self.sizes[1:]]\n",
" \n",
" # initialize the derivatives of weights for backprop \n",
" self.dW = [np.zeros((n, m)) for (m,n) in zip(self.sizes[:-1], self.sizes[1:])]\n",
" \n",
" # initialize the activities on each hidden and output layer \n",
" self.z = [np.zeros(n) for n in self.sizes]\n",
" \n",
" # initialize the activations on each hidden and output layer \n",
" self.a = [np.zeros(n) for n in self.sizes]\n",
" \n",
" # initialize the deltas on each hidden and output layer \n",
" self.delta = [np.zeros(n) for n in self.sizes]\n",
" \n",
" #initialize empty arrays for accuracies on train and valid data sets\n",
" self.train_accuracy = []\n",
" self.valid_accuracy = []\n",
" \n",
" def g(self, z):\n",
" \"\"\"\n",
" sigmoid activation function \n",
" \n",
" :param z: vector of activities to apply activation to \n",
" \"\"\"\n",
" z = np.clip(z, -20, 20)\n",
" return 1.0/(1.0 + np.exp(-z))\n",
" \n",
" def g_prime(self, z):\n",
" \"\"\"\n",
" derivative of sigmoid activation function \n",
" \n",
" :param z: vector of activities to apply derivative of activation to \n",
" \"\"\"\n",
" return self.g(z) * (1.0 - self.g(z))\n",
" \n",
" def gradC(self, a, y):\n",
" \"\"\"\n",
" evaluate gradient of cost function for squared-loss C(a,y) = (a-y)^2/2 \n",
" \n",
" :param a: activations on output layer \n",
" :param y: vector-encoded label \n",
" \"\"\"\n",
" return (a - y)\n",
" \n",
" def forward_prop(self, x):\n",
" \"\"\"\n",
" take an feature vector and propagate through network \n",
" \n",
" :param x: input feature vector \n",
" \"\"\"\n",
" \n",
" # TODO: Initialize activation on initial layer to x \n",
" self.a[0] = x\n",
" # TODO: Loop over layers and compute activities and activations\n",
" for ii in range(0,self.L-1): #stop one before end \n",
" self.z[ii+1] = np.dot(self.W[ii],self.a[ii]) + self.b[ii]\n",
" self.a[ii+1] = self.g(self.z[ii+1])\n",
" \n",
" def predict(self, X):\n",
" \"\"\"\n",
" Predicts on the the data in X. Assume at least two output neurons so predictions\n",
" are one-hot encoded vectorized labels. \n",
" \n",
" :param X: a matrix of data to make predictions on \n",
" :return y: a matrix of vectorized labels \n",
" \"\"\"\n",
" \n",
" yhat = np.zeros((X.shape[0], self.sizes[-1]), dtype=int)\n",
" \n",
" # TODO: Populate yhat with one-hot-coded predictions \n",
" for row in range(X.shape[0]):\n",
" self.forward_prop(X[row])\n",
" indx = np.argmax(self.a[-1]) #get index of predic with highest prob\n",
" yhat[row][indx] = 1\n",
"\n",
" return yhat \n",
" \n",
" def accuracy(self, X, y):\n",
" \"\"\"\n",
" compute accuracy on labeled training set \n",
"\n",
" :param X: matrix of features \n",
" :param y: matrix of vectorized true labels \n",
" \"\"\"\n",
" yhat = self.predict(X)\n",
" return np.sum(np.all(np.equal(yhat, y), axis=1)) / X.shape[0]\n",
" \n",
" \n",
" def back_prop(self, x, y):\n",
" \"\"\"\n",
" Back propagation to get derivatives of C wrt weights and biases for given training example\n",
" \n",
" :param x: training features \n",
" :param y: vector-encoded label \n",
" \"\"\"\n",
" \n",
" # TODO: forward prop training example to fill in activities and activations \n",
" self.forward_prop(x)\n",
" # TODO: compute deltas on output layer \n",
" self.delta[-1] = self.gradC(self.a[-1], y) * self.g_prime(self.z[-1])\n",
" # TODO: loop backward through layers, backprop deltas, compute dWs and dbs\n",
" for ll in range(self.L-2, -1, -1):\n",
" self.dW[ll] = np.outer(self.delta[ll+1], self.a[ll])\n",
" self.db[ll] = self.delta[ll+1]\n",
" self.delta[ll] = np.dot(self.W[ll].transpose(), self.delta[ll+1]) * self.g_prime(self.z[ll])\n",
" \n",
" \n",
" def train(self, X_train, y_train, X_valid=None, y_valid=None, eta=0.25, lam=0.0, num_epochs=10, isPrint=True):\n",
" \"\"\"\n",
" Train the network with SGD \n",
" \n",
" :param X_train: matrix of training features \n",
" :param y_train: matrix of vector-encoded training labels \n",
" :param X_train: optional matrix of validation features \n",
" :param y_train: optional matrix of vector-encoded validation labels \n",
" :param eta: learning rate \n",
" :param lam: regularization strength \n",
" :param num_epochs: number of epochs to run \n",
" :param isPrint: flag indicating to print training progress or not \n",
" \"\"\"\n",
" \n",
" # initialize shuffled indices \n",
" shuffled_inds = list(range(X_train.shape[0]))\n",
" \n",
" # loop over training epochs \n",
" for ep in range(num_epochs):\n",
" if ep % 5 == 0 and X_valid.all() != None and y_valid.all() != None: # get training and validation accuracy every 5 epochs\n",
" self.train_accuracy.append(self.accuracy(X_train, y_train))\n",
" self.valid_accuracy.append(self.accuracy(X_valid, y_valid))\n",
" \n",
" # shuffle indices \n",
" np.random.shuffle(shuffled_inds)\n",
" \n",
" # loop over training examples \n",
" for ind in shuffled_inds:\n",
" \n",
" # TODO: back prop to get derivatives \n",
" self.back_prop(X_train[ind,:], y_train[ind,:])\n",
" # TODO: update weights and biases \n",
" self.W = [wll - eta*dwll - (eta *lam * wll) for wll, dwll in zip(self.W, self.dW)]\n",
" self.b = [bll - eta*dbll for bll, dbll in zip(self.b, self.db)]\n",
" \n",
" # occasionally print accuracy\n",
" if isPrint and ((ep+1)%10)==1:\n",
" self.epoch_report(ep, num_epochs, X_train, y_train, X_valid, y_valid)\n",
" \n",
" # print final accuracy\n",
" if isPrint:\n",
" self.epoch_report(ep, num_epochs, X_train, y_train, X_valid, y_valid)\n",
" \n",
" \n",
" def epoch_report(self, ep, num_epochs, X_train, y_train, X_valid, y_valid):\n",
" \"\"\"\n",
" Print the accuracy for the given epoch on training and validation data \n",
" \n",
" :param ep: the current epoch \n",
" :param num_epochs: the total number of epochs\n",
" :param X_train: matrix of training features \n",
" :param y_train: matrix of vector-encoded training labels \n",
" :param X_train: optional matrix of validation features \n",
" :param y_train: optional matrix of vector-encoded validation labels \n",
" \"\"\"\n",
" \n",
" print(\"epoch {:3d}/{:3d}: \".format(ep+1, num_epochs), end=\"\")\n",
" print(\" train acc: {:8.3f}\".format(self.accuracy(X_train, y_train)), end=\"\")\n",
" if X_valid is not None: print(\" valid acc: {:8.3f}\".format(self.accuracy(X_valid, y_valid)))\n",
" else: print(\"\") \n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part A**: Complete the `forward_prop` function in the `Network` class to implement forward propagation. Your function should take in a single training example `x` and propagate it forward in the network, setting the activations and activities on the hidden and output layers. When you think you're done, execute the following unit test. "
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:17.610306Z",
"start_time": "2018-03-13T09:10:17.602365Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"testForwardProp (__main__.TestNN) ... ok\n",
"\n",
"----------------------------------------------------------------------\n",
"Ran 1 test in 0.001s\n",
"\n",
"OK\n"
]
}
],
"source": [
"%run -i tests/tests.py \"prob 3A\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part B**: Complete the `predict` function in the `Network` class to take in a matrix of features and return a matrix of one-hot-encoded label predictions. Your predictions should correspond to the output neuron with the largest activation. \n",
"\n",
"Recall that our convention for encoding, e.g. the label $y=2$ in a classification problem with possible labels $y \\in \\{0,1,2,3\\}$ is \n",
"\n",
"$$\n",
"y=2 \\quad \\Leftrightarrow \\quad y=\\left[0, 0, 1, 0\\right]\n",
"$$\n",
"\n",
"So the equivalent matrix associated with the labels $y_1=3, y_2=2, y_3=0$ is \n",
"\n",
"$$\n",
"y = \\begin{bmatrix}{3\\\\2\\\\0}\\end{bmatrix} \\quad \\Leftrightarrow \\quad \n",
"y = \\begin{bmatrix}\n",
"0 & 0 & 0 & 1 \\\\\n",
"0 & 0 & 1 & 0 \\\\\n",
"1 & 0 & 0 & 0 \n",
"\\end{bmatrix}\n",
"$$\n",
"\n",
"When you think your `predict` function is working well, execute the following unit test. \n"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:19.290278Z",
"start_time": "2018-03-13T09:10:19.282762Z"
},
"scrolled": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"testPredict (__main__.TestNN) ... ok\n",
"\n",
"----------------------------------------------------------------------\n",
"Ran 1 test in 0.001s\n",
"\n",
"OK\n"
]
}
],
"source": [
"%run -i tests/tests.py \"prob 3B\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part C**: OK, now it's time to implement back propagation. Complete the function ``back_prop`` in the ``Network`` class to use a single training example to compute the derivatives of the loss function with respect to the weights and the biases. As in the **Hands-On** notebook, you may assume that the loss function for a single training example is given by \n",
"\n",
"$$\n",
"C(y, {\\bf a}^L) = \\frac{1}{2}\\|y - {\\bf a}^L\\|^2 \n",
"$$\n",
"\n",
"When you think you're done, execute the following unit test. "
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:21.407947Z",
"start_time": "2018-03-13T09:10:21.399745Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"testBackProp (__main__.TestNN) ... ok\n",
"\n",
"----------------------------------------------------------------------\n",
"Ran 1 test in 0.001s\n",
"\n",
"OK\n"
]
}
],
"source": [
"%run -i tests/tests.py \"prob 3C\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part D**: OK, now let's actually train a neural net! Complete the missing code in ``train`` to loop over the training data in random order, call `back_prop` to get the derivatives, and then update the weights and the biases via SGD. When you think you're done, execute the following code"
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:23.048222Z",
"start_time": "2018-03-13T09:10:23.039936Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"testSGD (__main__.TestNN) ... ok\n",
"\n",
"----------------------------------------------------------------------\n",
"Ran 1 test in 0.001s\n",
"\n",
"OK\n"
]
}
],
"source": [
"%run -i tests/tests.py \"prob 3D\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part E**: Last but not least, we should implement $\\ell$-$2$ regularization. Modify your `train` function to incorporate regularization of the weights (but **not** the biases) in your SGD update. As in the Lecture 18 slides, you should assume that the cost function with regularization takes the form \n",
"\n",
"$$\n",
"C_\\lambda = C + \\frac{\\lambda}{2} \\displaystyle\\sum_{w} w^2\n",
"$$\n",
"\n",
"where $\\sum_{w}$ sums over each weight in all layers of the network. Think carefully before you go making large changes to your code. This modification is much simpler than you think. When you think you're done, execute the following unit test. (Then go back and execute the test in **Part C** to make sure you didn't break anything.) "
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:25.040053Z",
"start_time": "2018-03-13T09:10:25.031009Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"testRegularizedSGD (__main__.TestNN) ... ok\n",
"\n",
"----------------------------------------------------------------------\n",
"Ran 1 test in 0.001s\n",
"\n",
"OK\n"
]
}
],
"source": [
"%run -i tests/tests.py \"prob 3E\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [25 points] Problem 4: A Neural Network Classifier for Handwritten Digit Recognition \n",
"***\n",
"\n",
"In this problem you'll use the Feed-Forward Neural Network framework you wrote in **Problem 3** to take an image of a handwritten digit and predict which digit it corresponds to. \n",
"\n",
"![Samples of Handwritten Digits](figs/mnist.png \"MNIST Digits\")\n",
"\n",
"To keep run times down we'll again only consider the subset of the MNIST data set consisting of the digits $3, 7, 8$ and $9$. \n",
"\n",
"**Part A**: Executing the following cells will load training and validation data and plot an example handwritten digit. Explore the training and validation sets and answer the following questions: \n",
"\n",
"- How many pixels are in each image in the data set? \n",
"- How do the true labels correspond to the associated one-hot-encoded label vectors? \n",
"- Give an example of a network architecture with a single hidden layer that is compatible with this data. \n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:27.264422Z",
"start_time": "2018-03-13T09:10:27.153070Z"
},
"scrolled": true
},
"outputs": [],
"source": [
"X_train, y_train, X_valid, y_valid = pickle.load(gzip.open(\"data/mnist21x21_3789_one_hot.pklz\", \"rb\"))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"ExecuteTime": {
"end_time": "2018-03-13T09:10:28.454747Z",
"start_time": "2018-03-13T09:10:28.350656Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"There are 441 pixels in each image in the data set.\n",
"The true value to the example image above is [1 0 0 0] where similiar to problem 3, the first index represents the value 3, the second represents 7, the third 8, and the last 9. \n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAALgAAADHCAYAAACqR5nTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAACG9JREFUeJzt3WtoVOkdx/HfXyW1aazGrUllqy1EqDbdgsKCxEu9VFdKqbUURELRF7ZYoRSWFdMU1kJL2ReFvhCaYlt7oe9qaVcCdotKlnq/m4KY7q7RoqxZV1TwstGYpy/mZA3B859JoiTzn+8HgmZ+55mcxJ/PzDxzco6llARENWGsdwB4nig4QqPgCI2CIzQKjtAoOEKj4AiNgiM0Co7QJg1nYzPjbU+MGyklK7YNMzhCo+AIjYIjNAqO0Cg4QqPgCI2CIzQKjtAoOEKj4AiNgiM0Co7QKDhCo+AIjYIjNAqO0Cg4QqPgCI2CIzQKjtAoOEIb1m/VY2SqqqrcvLa21s0bGhrcvKenJze7cuWKO7avr8/Nyx0zOEKj4AiNgiM0Co7QKDhCo+AIjYIjtIpZB6+pqXHzxsbG3Ky+vt4dO2vWLDevq6tz87lz57r5ihUr3Pz8+fO5WUtLizv2zJkzbt7f3+/m4x0zOEKj4AiNgiM0Co7QKDhCo+AIjYIjtIpZBy+21rxr167cbOrUqe7Y6upqNzfzLwY2efJkNy+2hr98+fLcbMOGDe7YCxcuuPn9+/fdfLxjBkdoFByhUXCERsERGgVHaBQcoVFwhFYx6+C3bt1y887Oztys2LlDzp496+bFzk2ydOlSN9+6daubP3r0KDc7deqUO/bhw4duXu6YwREaBUdoFByhUXCERsERGgVHaBQcoVXMOvilS5fcfMuWLblZsbVibx1aKn48+dq1a928mIMHD+Zmhw8fdsdyfnCgjFFwhEbBERoFR2gUHKFRcIRWMcuEKSU3v3fv3ojve8IEf55YvXq1m69atcrNr1+/7ua7d+/Oza5du+aOjY4ZHKFRcIRGwREaBUdoFByhUXCERsERWsWsgz9PM2fOdPPm5mY3nzZtmpu3tbW5+YEDB3Kzx48fu2OjYwZHaBQcoVFwhEbBERoFR2gUHKFRcITGOngJih3v3djY6ObLli1z8xMnTri5d4lDaXTHskfHDI7QKDhCo+AIjYIjNAqO0Cg4QqPgCI118BJUV1e7+Zo1a9x8ypQpbl7sePKWlhY337dvX27W3t7uju3t7XXzcscMjtAoOEKj4AiNgiM0Co7QKDhCo+AIjXXwzKRJ+T+KRYsWuWPXr1/v5sUuM1hMsfOLL168ODe7fPmyO/b06dMj2aWywQyO0Cg4QqPgCI2CIzQKjtAoOEKj4AiNdfBMXV1dbrZt2zZ3bLHjuY8dO+bmra2tbj5//nw337FjR27W0NDgjj137pybl/v5xZnBERoFR2gUHKFRcIRGwREaBUdoLBNmHjx4kJtNnDhxxGMlae/evW5+8uRJN589e7abe0t5d+7cccf29/e7ebljBkdoFByhUXCERsERGgVHaBQcoVFwhFY26+Bm5uYppVHdv7devGfPHnfsjRs33PzIkSNu3tTU5OYbN2508+7u7tzs6tWr7tjR/tzGO2ZwhEbBERoFR2gUHKFRcIRGwREaBUdoNpx1UDMbs0XThQsXunlVVZWbHz161M29UxzX1NS4YxcsWODmS5YscfNNmza5ebHjzbdv356b7d+/3x072lM7j6WUkv/miJjBERwFR2gUHKFRcIRGwREaBUdoFByhlc3x4PX19W6+efNmN+/q6nLzmzdv5mbz5s1zx86ZM8fNp0+f7ubt7e1uXux49OPHj+dmfX197tjomMERGgVHaBQcoVFwhEbBERoFR2gUHKGVzfHgxY7JXrlypZuvW7fOzWtra3OzYudkOXTokJt3dHS4+cWLF9387t27bh79HN95OB4cFY+CIzQKjtAoOEKj4AiNgiO0slkmBIZimRAVj4IjNAqO0Cg4QqPgCI2CIzQKjtAoOEKj4AiNgiM0Co7QKDhCo+AIjYIjNAqO0IZ7+uQPJV15HjsCDNPnS9loWL/wAJQbnqIgNAqO0Cg4QqPgGTP7lpm9Otb7USoze8PMOs3stpk9MLOLZva6mVWP9b6NJ7zIzJjZHyV9LaX0ubHel1KY2a8lvSOpS1KvpCZJP5H0Vkpp7Vju23hSNldZG0/M7BMppd6x3IeU0tYhNx3IZu8WM/tMSunDsdiv8YanKPp49t4o6UUzS9nH5Sxbln3+bTP7rZndkNQzMG5guyH312FmHUNum2FmvzGza2bWmz2l+P4z/lYGroVY2dcOHIQZvOBnkmZIelnSN7Pbhs7QOyXtk/RdSZOHc+dm9mlJhyR9UtJPJXVLekVSW/ZosHPQtknSn1JKm0q870nZ/iyU9Kqk3Sml28PZv8gouKSU0nvZzPwwpXQsZ7MTKSX/arP5fqTCO28vpZTeyW7bb2bTJO0ws7aU0sCs+zj7KMrMvizpP4Nu+rOkZ/2oUNYoeOn+PoqxayQdl9SdzbgD3pK0WdKXJHVKUkppOP8m76rwqPMpFV5k/liFf9PmUexrKBS8dO+PYmydpDmSHuXkL4zkTlNKH0k6lX36tpm9L+kPZrbTeSSqKBS8dE9bT/1IUtVTbn9BT17wKfv7Byo8VXmartHt2scGyj5HEgUXBR+sV4UXgcNxRVK9mc1IKd2QJDNrkPRFSUcGbfdPST+U9L+U0gfPYmdzfDX7873n+DXKCsuET1yQNN3MfmBmL5vZSyWM+asKM/tfzOwVM2uW9KYKhxUP9isVZvB/m9kWM1tuZt8ws9fM7M3BG5pZn5n93vuiZvYVM/uXmX3PzFaa2dfN7A1Jv5S0L6V0tMTvOTxm8Cd+p8JS2y8kTVNhdv6CNyCl9K6ZfUfSzyX9Q9J/VViqax2y3R0za5L0uqTtkl6UdFuFpyZ/G3K3E7MPT48K/4laJX1W0n1JlyS9ln0fyPBWPULjKQpCo+AIjYIjNAqO0Cg4QqPgCI2CIzQKjtD+D7QQyEKBqOIUAAAAAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x113427128"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def view_digit(x, label=None):\n",
" fig = plt.figure(figsize=(3,3))\n",
" plt.imshow(x.reshape(21,21), cmap='gray');\n",
" plt.xticks([]); plt.yticks([]);\n",
" if label: plt.xlabel(\"true: {}\".format(label), fontsize=16)\n",
" \n",
"training_index = 2\n",
"label_dict = dict({0:3, 1:7, 2:8, 3:9})\n",
"view_digit(X_train[training_index], label_dict[np.argmax(y_train[training_index])])\n",
"print('There are',X_train[training_index].shape[0], 'pixels in each image in the data set.')\n",
"print('The true value to the example image above is', y_train[training_index], 'where similiar to problem 3, the first index represents the value 3, the second represents 7, the third 8, and the last 9. ')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part B**: Train a network with a single hidden layer containing $30$ neurons on the first $500$ training examples in the training set using a learning rate of $\\eta = 0.01$ for at least $50$ epochs. What accuracy does your network achieve on the validation set? Do you see any clear signs of overfitting? "
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 1/ 50: train acc: 0.326\n",
"epoch 11/ 50: train acc: 0.612\n",
"epoch 21/ 50: train acc: 0.714\n",
"epoch 31/ 50: train acc: 0.834\n",
"epoch 41/ 50: train acc: 0.872\n",
"epoch 50/ 50: train acc: 0.880\n"
]
},
{
"data": {
"text/plain": [
"0.851"
]
},
"execution_count": 84,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"single_L_Network = Network([441, 30, 4])\n",
"single_L_Network.train(X_train[:500], y_train[:500], eta=0.01, num_epochs=50)\n",
"single_L_Network.accuracy(X_valid, y_valid)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### After 50 epochs, the training accuracy was roughly 88% and the accuracy on the validation set is 85%. This does not show signs of overfitting yet..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part C**: Modify the `Network` class so that it stores the accuracies on the training and validation data every $5$ epochs during the training process. Now increase the number of neurons in the hidden layer to $100$. On a single set of axes, plot the **validation accuracy** vs epoch for networks trained on the full training set for at least 50 epochs using the learning rates $\\eta = 0.01$, $\\eta = 0.25$ and $\\eta = 1.5$. Which learning rate seems to perform the best? What is the best accuracy achieved on the validation set? "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 1/ 50: train acc: 0.635 valid acc: 0.634\n",
"epoch 11/ 50: train acc: 0.914 valid acc: 0.889\n",
"epoch 21/ 50: train acc: 0.934 valid acc: 0.908\n",
"epoch 31/ 50: train acc: 0.948 valid acc: 0.912\n",
"epoch 41/ 50: train acc: 0.956 valid acc: 0.919\n",
"epoch 50/ 50: train acc: 0.959 valid acc: 0.921\n",
"epoch 1/ 50: train acc: 0.931 valid acc: 0.912\n",
"epoch 11/ 50: train acc: 0.987 valid acc: 0.945\n",
"epoch 21/ 50: train acc: 0.991 valid acc: 0.950\n",
"epoch 31/ 50: train acc: 0.993 valid acc: 0.949\n",
"epoch 41/ 50: train acc: 0.994 valid acc: 0.952\n",
"epoch 50/ 50: train acc: 0.994 valid acc: 0.951\n",
"epoch 1/ 50: train acc: 0.896 valid acc: 0.889\n",
"epoch 11/ 50: train acc: 0.970 valid acc: 0.952\n",
"epoch 21/ 50: train acc: 0.986 valid acc: 0.960\n",
"epoch 31/ 50: train acc: 0.988 valid acc: 0.960\n",
"epoch 41/ 50: train acc: 0.991 valid acc: 0.961\n",
"epoch 50/ 50: train acc: 0.987 valid acc: 0.964\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8HNWV6PHf6VZrtXZ5lSxvksA2OIYYDCRvhhAWAwkQzLBlD4GXDCQMkIXMg4QhmWwvK4Nn8khCAgQCPCYkJjGQECDkYRNswupNsoyN5N3aLFtSr+f9UdXtlqylLavUkvp8P5/6VNWt6upTWu6pqlt1S1QVY4wxBsCX7gCMMcaMHZYUjDHGJFhSMMYYk2BJwRhjTIIlBWOMMQmWFIwxxiRYUjAThoioiNSkO46RkLwvIvITEbk9lXWH8T0fFpE/DjdOM/FYUjCeEJFtItItIgeThrvTHddocCvx+/spf5eIBEWk7Gi2p6qfUdWvj0Bcs90EkpW07QdV9dxj3baZOCwpGC99UFUnJQ03pDugUXIfcKmIFPQp/yjwe1VtTUNMxqTEkoIZdSLyCRF5UUTuFpEOEdkkIu9PWj5DRFaKSKuIbBGRa5OW+UXkX0WkUUQ6ReQVEZmZtPmzRaRBRNpFZIWIiPu5GhH5i/t9+0XkkQFie1JEbuhT9rqIXCqOH4rIXhE5ICJvisgJfbehqmuAHcDy5LiBq4H73flTRWSNG+cu92eRPUBMvxSRbyTNf9H9zE4R+VSfdS8UkVfd+JpE5I6kxS+443b3zO1093fx/5I+f4aIrHV/TmtF5IykZc+LyNfd312niPxRRCr6i9mMX5YUTLosBRqBCuBrwG+SLqs8DDQDM4DLgG+KyFnuspuBq4ALgCLgU0BX0nY/AJwCLAIuB85zy78O/BEoBaqA/xggrl+72wdARBYAs4A/AOcC/wDUAcXu9lsG2M79wMeS5s8GAsAqdz4K3OTu/+nA+4F/HmBbCSKyDPgCcA5Q62432SH3e0uAC4HPisgl7rJ/cMcl7pnbmj7bLnP38y6gHPgB8AcRKU9a7Wrgk8AUINuNxUwglhSMl37rHgnHh2uTlu0FfqSqYVV9BNgMXOge9b8H+LKq9qjqa8DPOFzBfhq4TVU3q+N1VU2umL+tqu2q+g7wHLDYLQ/jVO4z3O3+P/r3OLBYRGa58x8GfqOqQXcbhcDxgKjqRlXdNcB2HgD+UUSq3PmPAQ+pahhAVV9R1ZdUNaKq24D/A/zjgD/Jwy4HfqGqb6nqIeCO5IWq+ryqvqmqMVV9AyfJpbJdcJJIg6o+4Mb1a2AT8MGkdX6hqvWq2g08yuGfr5kgLCkYL12iqiVJw0+Tlu3Q3r0xbsc5M5gBtKpqZ59lle70TJwzjIHsTpruAia5018CBHhZRNb3vewS537vH4Ar3aKrgAfdZc8CdwMrgL0ico+IFA2wnXdwLtd8REQmAZfgXjoCEJE6Efm9iOwWkQPAN3HOGoYyA2hKmt+evFBElorIcyKyT0Q6gM+kuN34trf3KUv+2cPAP18zQVhSMOlSGb/e76oGdrpDmYgU9lm2w51uAuYd7Zep6m5VvVZVZwD/E/jPQW7j/DVwlYicDuTinHHEt3OXqr4bWIBzGemLg3ztfTiNy8uBt1X1laRl/4VzFF6rqkXAv+IkraHswkmMcdV9lj8ErARmqmox8JOk7Q7VJfJOnLOpZMk/e5MBLCmYdJkCfF5EAiLyT8B8YJWqNgGrgW+JSK6ILAKuAX7lfu5nwNdFpNZt+F3U55p3v0Tkn5Iu5bThVJCxAVZfhVM53gk8oqoxdxunuEfiAZxr9z2DbAPgv3Eq1X/DSRDJCoEDwEEROR747FD74HoU+ISILBCRfJz2mL7bbVXVHhE5FacNIG6fG+/cAba9CqgTkatFJEtErsBJfr9PMTYzAVhSMF56Qno/p/B40rK/4TSU7gf+HbgsqW3gKmA2zpHr48DXVPUZd9kPcCrGP+JUqj8H8lKI5RTgbyJyEOdI+kZV3drfim77wW9wGnEfSlpUBPwUJ6lsx2lk/t8DfaF7zf+/cRq2H+yz+As4FXanu81+74bqZ5tPAj8CngW2uONk/wzcKSKdwFdxflbxz3bh/KxfdNt4Tuuz7Rachvpb3H37EvABVd2fSmxmYhB7yY4ZbSLyCeDTqvredMdijOnNzhSMMcYkWFIwxhiT4FlSEJF73Sc/3xpguYjIXeI8sfqGiJzsVSxmbFHVX9qlI2PGJi/PFH4JLBtk+fk4DY21wHU4t+gZY4xJo6yhVxkeVX1BRGYPssrFwP3uA0wviUiJiEwf5AlRACoqKnT27ME2a4wxpq9XXnllv6pOHmo9z5JCCirp/WRms1t2RFIQketwziaorq5m3bp1oxKgMcZMFCLS92n1fo2LhmZVvUdVl6jqksmTh0x0xhhjhimdSWEHvR/Xr8IepzfGmLRKZ1JYCXzMvQvpNKBjqPYEY4wx3vKsTUFEfg2cCVSISDNOHy0BAFX9CU4/KxfgPKrfhdNHuzHGmDTy8u6jq4ZYrsD1Xn2/McaYozcuGpqNMcaMDksKxhhjEtL5nIIxGSsYDdLW00ZrT2uv8aHIIYqyiyjOKaYkp4SSnBKKs4spzi2mMFBI7/cSGTPyLCkYc4xUla5IV6JiT1TywbbEfHw6vk5XpOuov8cvfopzihMJozj78HRJbomzLNtdllSe48/xYK8ntnAsTCgaoifSQygaAsDv85Ply8Ivh8d+n58syZpQydqSghkVqkp3pJuuSBdd4S4OhQ8lpoHEP5ff58cvfgK+QGI6vqzvP2L8nzQ+7ZORuRoa0xidoc7elXywlfae9iMq+/g6oVio321l+7IpzS2lLLeM0txSqouqKc05PJ9YluNMFwQK6Ax10h5spyPYQUewg/Zge2I+uXzXoV1sbN1IR7CDnmjPgPuTl5VHUXbR4TOPnMOJo9cZSVL5pMCk1F4O6rFILNKrcg5Gg/0OoWiInmhPSusGI0d+pu+2oho9qjj90s/fqvv32+vvNv736s4P+rfe5/NZviyWzV7GyVO97TvUkoLpVyQWSVTayZV4cmXeFe7iUOTQ4eXhrkHX0SFfEXxsBMHvc//Jkv7pBvsni/8TAhwIHaCtp432YPuAlUJBoICSnBLKcsuYnD+ZutK6ASv4stwy8rLyjvooMr6to9ET6emVNJITSd/y+rZ6DoQO0BHsOOrKbzwJ+ALk+HMSQ7Y/m9ysXLL92eT4cygJlBwu9+f2Guf4c3qtKwjhWJioRonGokQ1SiQWIRKLJKbjyyIa6TUf1ajzWXc6vk58HI6F6Y5297vdvtubXzbfkoIZmqoSioUSFfOh8CG6I929KvJeZX2O1JPXi68TjAZT/v68rDzys/IpCBSQH8gnPyuf8txyZhbOdMqy8hPl8XUKsgrICzifE5Fe/xCJcdI/T+Kfpc984h8neVksQiTSQzTSQyTSTTQSJBINEo2GiLrjcDRINHaQaDRM1P1nr5Ys3iXZlGXPoNSXTakvhzJfDqX+HEp9uZT6c8jxBUB8zhDzQY8PgofgQDfI7sPLRJyxz59U5jtyea/BP8RyX9I2ey/PdYep8bKsMghUQOEA2xAfMZSD0R46Il20hw/SHjpIR/ggHeFODkV6nO/x+d240nPa4Bd/r8q5VwXvzx2w0s/2ZeP3+dMS83hnSSENYhpzLqX0PQJPPtJ2j7C7w04lHT/a7ns0fijsrBPRSErf7Rd/7wraHZfmliYq6/xAfr/rxMvj6xQECsj15x77P180DKGDEDwIoUPOdPggBLud6b7L+s4HO5OWHXKGVM9KsnIhexIE8p2KT2NDDOqMY9Ejl3l8JjTSfDgvnS6id38z/a+cBf4c8AfAn+0MWe44UTbY8pze62b1WdeftG5W0rqI87OORSAUgVg3xDqd+V6Du0403Ht+wGGw9cNHfn6svLb4f9wCCy/x9CssKYyyn7z+E/7ztf9M+VJKjj+HgkABeVl5iQq6KLuIaQXTeh9591mnIFDQu8xdJ9uX3f/lDFXnHyTSDZEgRHog3OOM40NPO4R3HV6ePPRdNxKEcHc/6waT1g063xft/3r8kcSpwLMLIMcdZxdC4XR33h0Sy5LnB1jmH8F/AdXDSSMx9E0efZf3l2D6WafXtgZaHt/eUIkteRv9JLd4TNGQO4Sd31V8OlHuDpHk6SAED/ReLxLq89mgm0RHgS/gJDRflvO79iUPfncc6DPvDoE8Zxw/gxsLsgs8/wpLCqPsheYXmFU0i+W1y4c8Is/PyifLdwy/onA3HNgJ+xqhoxkO7HDG8emejt4V+jEd6YrzT5SV6w457nzO4aPx/Io+5e44UUnHK/vCpOlJh5cF8sE3Rv45+yPiXmYZwzGOFfGkEwn2n2jiSURjbqXtP7JC9weGqOTt9zAclhRGUUxjbGnfwqW1l/KJEz5xbBuLRuDg7t6VfEczdOyAA+64a/+Rn8uvgOIqKK+BvNJ+KvCkijy5Uo8PgaRKP3ldfyBt153NOOTzgy/P+RszY4olhVHU3NlMd6SbutK6wVdUhUP7D1fuB3ZAR1PSdDN07jryFDynCIoqnUp/xklQVAXF7nxRpTMEcr3bQWPMuGdJYRTVt9UDUJs/A/ZudCr5jia3ok+aPrDTvZyTxJ/jVPBFlTDnHw5X9IlxJeQWp2GvjDETiSWFUdTQ1oAA8362rPfdDOJzGkuLq2D6Yjj+Qvco3z3SL6qCggq7PGOM8ZwlhVFU31ZPtfrJnzzfubWs2K34J00b2btgjDFmmKwmGkUNbfXUdnfB3AvgxMvSHY4xGUVVicSUaMwZR6KxXvPRqBKJxdxl8fIYUXedmDrbiCnEVImqOvMxZ77v8sTgLlc9vF4s/lnF3fbAy51tONNnL5jK4pklnv6cLCmMkq5wF+90NnFhsBumLkx3OMaMuEg0RjASoycc7TXuNR2O0tNnHEwe9/pclFAkqeJOqqj7q7jD0aQKPtb7c5FYjNgYef5suPw+YXpJriWFiaKxvRFFqQ2FLSlksGhM6egO09YVor0rTHtXiLbEOMTBnsiYeC46GlN6wk7F3LeCD/at9N0KPnqMtW5Olo/cgJ+cLB85AR+5WX4Cfh8Bv+D3CVk+H36fkBPISsxn+QS/X5yxzxln+X295v3ueln+3vP+RFnv+cR2fD78fsEvzryI4BPw+ZyxM+8sFwGfCD6fO+5vufs5X/L6vbY58PLR7IXVksIoaWhvAKAuHIXJx6c5GnOsVJWuUDRRufet5AcqP9ATHrDHBL9PmJSThW8M3E/gE0mqoP1uhe2jJC9ATmHOEZV3TsBHTpaf3D7jvhV9r2VJ89l+34Tqfno88zQpiMgy4MeAH/iZqn67z/JZwL3AZKAV+IiqNnsZU7rUt9WTh1BVMseeFRgl6l6njeoA12xjh6/9Hj6Cj1fk8Ur98FF8fNqp5MOEogN31TApJ4uS/ACl+dmU5AeYWZZPaX6AkvxsSpPK4/Ml+dkU5mThGwsZwWQ0z5KCiPiBFcA5QDOwVkRWquqGpNW+B9yvqveJyFnAt4CPehVTOjW0NVATUXxTT0h3KGkRjsbY1d7DO61dNLV1OePWLprbuukJR3tV1vFGtd4VeT8NfbHBl49EH2YBv/SquOdUFFCan02xW7Efruid6eL8ACV52WRnWRcLZnzy8kzhVGCLqm4FEJGHgYuB5KSwALjZnX4O+K2H8aSNqlLfupn3dx+EuonZnqCq7D8YoqnNqeydoTuRBHa2d/dq6MvyCVWleVSV5jO1KCdx/fTwtdT+rrMenu51zdY3+PLBr9keXlaUFzjiKL4g22+XNUxG8TIpVAJNSfPNwNI+67wOXIpzielDQKGIlKtqS/JKInIdcB1AdXW1ZwF7ZX/3ftpDHdSGQjD1xHSHM2xdocjhir7VOdpvThz1d9Md7v3ClsmFOVSX5bNkVikzT6pkZlk+M0vzqS7PZ1pRLn67VGLMmJPuhuYvAHeLyCeAF4AdwBGvglLVe4B7AJYsWTIWbs44KvHuLerG+J1HkWiMXR09zlF+UmUfr/z3H+zdxXVBtp+ZZfnMKi/gf9ROZmZpHtXlTsVfVZpPXra95MSY8cbLpLCD3u/uqHLLElR1J86ZAiIyCViuqu0expQWiT6PJB+KZqQ5GghGoqxubGHDzgO9EsDO9p5etxVm+YQZJXnMLMvjnAVTqSrNp7osn5llzrg0P2CXVoyZYLxMCmuBWhGZg5MMrgSuTl5BRCqAVlWNAV/BuRNpwmloa2CK+iiZsjBt/Rf1hKO8UL+PJ9/azTMb9tAZdN7UVjEpm5ll+Zw0s5SL35XPzLK8xGWe6cW5ZPmtwdSYTOJZUlDViIjcADyNc0vqvaq6XkTuBNap6krgTOBbIqI4l4+u9yqedKpvq6c22APVo3vnUVcowvOb97HqzV08u2kvXaEoJfkBzj9xGuefMJ1T55RRkJPuK4jGmLHE0xpBVVcBq/qUfTVp+jHgMS9jSLdwLMzWjkbO6OkZlfaEzp4wz27ay5Nv7ub5+r30hGOUF2RzyUmVnH/CNE6bW07Ajv6NMQOww0SPbe/YTjgWce888iYpdHSHeWbDHp58axcvNOwnFIkxpTCHK5bMZJl7RmB3+hhjUmFJwWO9u7eYP2LbbT0U4k8bdvPkW7t5cct+wlFlRnEuH1k6iwtOnMbJ1aX2dKwx5qhZUvBYfVs9WcCcwpmQnX9M29rXGeTp9bt56q3drNnaQjSmzCzL41PvmcP5J07nXVXFdjeQMeaYWFLwWH1bPbOjkD3M7i12d/Tw1Fu7ePKt3by8rRVVmFtRwGf+cS7nnzCdhTOKLBEYY0aMJQWPNbTWs7j7EMxLPSnsaO/myTedRPDK9jYA6qZO4vNn1XLBidOpmzrJEoExxhOWFDx0IHSAXV27uTyFJ5m3txziybd28+Sbu3i9uQOABdOL+MK5dSw7YTo1UyaNRsjGmAxnScFDW9q2AFAXCsG0I88UGvcd5Mk3d7Hqzd1s2HUAgHdVFXPr+cdz/gnTmFVeMKrxGmOMJQUPJfo8IgeKnR4/Wg4GuX/Ndp56azeb93QC8O5Zpdx24XyWnTCNqtJja4w2xphjYUnBQ/Vt9RSqMHXy/ET3Fl/93XpWvbWLU2eXcccHF7DshOlMK7aX7hhjxgZLCh5qaGugNhRGKp1LR9GY8teGfVz+7pl857JFaY7OGGOOZP0deERVaWjbTF2wG9zbUTfsPMCBnghn1JSnOTpjjOmfJQWP7Dy0k0ORbmpD4URSWN24H4DT51lSMMaMTZYUPFLfGn+xTgimON1brG5soXbKJKYUWhuCMWZssqTgkXifR7X5MyBnEqFIjLXbWjnDzhKMMWOYJQWP1LfVUxmDgmnOO5nfaG6nKxTl9HkVaY7MGGMGZknBI/Wtm6nr7kpqT2hBBE6bW5bmyIwxZmCWFDwQjAbZfmC728jsdG+xprGFhTOKKMnPTnN0xhgzMEsKHmhsbySGOo3MUxfSE47yyjttnGGXjowxY5ynSUFElonIZhHZIiK39rO8WkSeE5FXReQNEbnAy3hGS0Ob28hMAEpm8/ftbYQiMU6fa43MxpixzbOkICJ+YAVwPrAAuEpEFvRZ7TbgUVU9CbgS+E+v4hlN9W315KhQXXYc+HysbmzB7xNOmWPtCcaYsW3IpCAinxOR0mFs+1Rgi6puVdUQ8DBwcZ91FChyp4uBncP4njGnvq2euZEIWUkPrb2rqphJOdariDFmbEvlTGEqsFZEHnUvB6X6dpdKoClpvtktS3YH8BERaQZWAZ/rb0Micp2IrBORdfv27Uvx69OnoXUzdT3dMHUhB4MRXm/usPYEY8y4MGRSUNXbgFrg58AngAYR+aaIzBuB778K+KWqVgEXAA+IyBExqeo9qrpEVZdMnjx5BL7WOy3dLbQE26gLhWHaiax9u5VoTO2hNWPMuJBSm4KqKrDbHSJAKfCYiHx3kI/tAGYmzVe5ZcmuAR51v2MNkAuM60PqxJPMbvcWqxv3k53l4+RZw7kCZ4wxoyuVNoUbReQV4LvAi8CJqvpZ4N3A8kE+uhaoFZE5IpKN05C8ss867wDvd79nPk5SGPvXhwaR6PMobyrkFrO6sYV3V5eSG/CnOTJjjBlaKi2fZcClqro9uVBVYyLygYE+pKoREbkBeBrwA/eq6noRuRNYp6orgVuAn4rITTiNzp9wz0rGrYb2BspiUD7lRNoOhdiw6wA3n12X7rCMMSYlqSSFJ4HW+IyIFAHzVfVvqrpxsA+q6iqcBuTksq8mTW8A3nNUEY9x9a2bqQv2wKyF/O3tFlSx9ycYY8aNVNoU/gs4mDR/0C0zfURjURrbtySeZF7d2EJ+tp9FVSXpDs0YY1KSSlKQ5Es6qhrDXuPZr3c63yEYCyderLO6sYVT55QR8FtvIsaY8SGV2mqriHxeRALucCOw1evAxqP6NreROeZjb9Z0tuw9aLeiGmPGlVSSwmeAM3BuJ20GlgLXeRnUeNXQ1oAPmFtay5pt7QD20JoxZlwZ8jKQqu7FuZ3UDKG+rZ5ZkRi5U09gTWMLxXkB5k8vGvqDxhgzRgyZFEQkF+chs4U4zxEAoKqf8jCucam+ZSMn9HQ77Ql/aWHpnDL8vlR7BTHGmPRL5fLRA8A04DzgLzhPJnd6GdR4dCh8iB1du6kNh9mbX8M7rV3WnmCMGXdSSQo1qno7cEhV7wMuxGlXMEni71CoC4V5sXMqAGfUWHuCMWZ8SSUphN1xu4icgNPF9RTvQhqfEn0eZZfzQlOEiknZ1E6ZlOaojDHm6KTyvME97vsUbsPpu2gScLunUY1D9a31FChMr1jA6sb9nD6vgtR7GTfGmLFh0KTgdmN9QFXbgBeAuaMS1TjU0FZPTTDEgfLj2HMgaO0JxphxadDLR+7Ty18apVjGLVWlvnUTdaEgb0ac3sItKRhjxqNU2hSeEZEviMhMESmLD55HNo7s6dpDZ6SLulCY59snU1mSR3VZfrrDMsaYo5ZKm8IV7vj6pDLFLiUlxLu3qInC997J48wF5daeYIwZl1J5onnOaAQynsWTQnX+bPa3xuzSkTFm3ErlieaP9VeuqvePfDjjU0NbA9OiSlvAeW316ZYUjDHjVCqXj05Jms7FeX3m3wFLCq76lo3UBXv4e2gGcysKmF6cl+6QjDFmWFK5fPS55HkRKQEe9iyicSYcDbOtcztnhkL8qXUypy+2swRjzPg1nLe/HAKsncG1tWMrEY1RGwrzanCGXToyxoxrqbQpPIFztxE4SWQB8GgqGxeRZcCPAT/wM1X9dp/lPwTe587mA1NUdVy9uzLeyFylk2ijiNPmWlIwxoxfqbQpfC9pOgJsV9XmoT4kIn5gBXAOzst51orISlXdEF9HVW9KWv9zwEmpBj5WNLQ3kKUQjs3k+GmFVEzKSXdIxhgzbKkkhXeAXaraAyAieSIyW1W3DfG5U4EtqrrV/dzDwMXAhgHWvwr4WkpRjyH1rZuZFw7z964ZnL7QzhKMMeNbKm0K/xeIJc1H3bKhVAJNSfPNbtkRRGQWTjvFswMsv05E1onIun379qXw1aOnoWUTtaEQb0aq7dWbxphxL5WkkKWqofiMO509wnFcCTymqtH+FqrqPaq6RFWXTJ48eYS/evjae9rZG2ylLhSinmpOnWO9fxhjxrdUksI+EbkoPiMiFwP7U/jcDmBm0nyVW9afK4Ffp7DNMSX+DoV5oRj504+nOC+Q5oiMMebYpNKm8BngQRG5251vBvp9yrmPtUCtiMzBSQZXAlf3XUlEjgdKgTUpRTyGxO888gfLOXXRtDRHY4wxxy6Vh9cagdNEZJI7fzCVDatqRERuAJ7GuSX1XlVdLyJ3AutUdaW76pXAw6qqA21rrGpoa6A4puwKz7L+jowxE0Iqzyl8E/iuqra786XALap621CfVdVVwKo+ZV/tM3/H0QQ8ljS0bKAuGKSBWVwwuzTd4RhjzDFLpU3h/HhCAHDfwnaBdyGNDzGN0dDeSF0oTKRiIfnZqVyJM8aYsS2VpOAXkcQTWSKSB2T8E1o7OnfQHQtRFwoxpebkdIdjjDEjIpXD2weBP4vIL9z5T2I9pCYamStCOVTOr0tzNMYYMzJSaWj+joi8DpztFn1dVZ/2Nqyxr76tHlGIhGawtHpcdddkjDEDSulCuKo+BTwFICLvFZEVqnr9EB+b0BraNlMVidA96XhysvzpDscYY0ZESklBRE7C6ZvocuBt4DdeBjUebNy3nuNDIXIqT0x3KMYYM2IGTAoiUoeTCK7CeYL5EUBU9X0DfSZTdEe62dG9mw+GwlQdf8rQHzDGmHFisDOFTcBfgQ+o6hYAEblpkPUzRmN7IwrMDUaoWWB3HhljJo7Bbkm9FNgFPCciPxWR9wMyOmGNbfE7j8q0gqwcex+zMWbiGDApqOpvVfVK4HjgOeBfgCki8l8icu5oBTgWvbZ7IzkxpaR4frpDMcaYETXkw2uqekhVH1LVD+L0dPoq8GXPIxvDXt/9JnWhEMWz7NKRMWZiSeWJ5gRVbXPfbfB+rwIa61SVnYe2UhsOM9meZDbGTDBHlRQM7OvaR490UxcK4Zt2QrrDMcaYEWVJ4Si92PQWALNjOVBo71AwxkwslhSO0vNvvw7A3OJaELsZyxgzsQyZFETkUhFpEJEOETkgIp0icmA0ghuL1u/bxORIlGlVJ6U7FGOMGXGpdHPxXeCDqrrR62DGOlWlLbiFUyIhxNoTjDETUCqXj/ZYQnBs3N1OOGs/daEwTF2Y7nCMMWbEpXKmsE5EHgF+CwTjhaqacZ3iPbnpDdQXozYcgcn24JoxZuJJ5UyhCOgCzgU+6A4fSGXjIrJMRDaLyBYRuXWAdS4XkQ0isl5EHko18HR48R3nzqO6/OkQyE1zNMYYM/JSecnOJ4ezYRHxAyuAc4BmYK2IrFTVDUnr1AJfAd6jqm0iMmU43zUaojGlob0Bf7Eyp8IuHRljJqZU7j6qEpHHRWSvO/y3iFSlsO1TgS2qulVVQ8DDwMV91rkWWKGqbQCquvdod2C0bNx1APW/w5xwmOxp9g4FY8zElMrlo18AK4EZ7vCEWzaUSqApab7VVWrqAAAbKUlEQVTZLUtWB9SJyIsi8pKILOtvQyJynYisE5F1+/btS+GrR97qxv0EcndSGwqDJQVjzASVSlKYrKq/UNWIO/wSmDxC358F1AJn4rzM56cicsQLj93+lpao6pLJk0fqq4/OXxubiAQO2Z1HxpgJLZWk0CIiHxERvzt8BGhJ4XM7gJlJ81VuWbJmYKWqhlX1baAeJ0mMKeFojFd2OXfl1hGAor4nPMYYMzGkkhQ+hfNu5t04L925DEil8XktUCsic0QkG7gS5zJUst/inCUgIhU4l5O2phT5KHqjuZ2wbycAdSU11r2FMWbCSuXuo+3ARUe7YVWNiMgNwNOAH7hXVdeLyJ3AOlVd6S47V0Q2AFHgi6qaylnIqFq9pQV/7k4KYzGmTnlXusMxxhjPDJgURORLqvpdEfkPQPsuV9XPD7VxVV0FrOpT9tWkaQVudocxa3VjC6WTdlIbsu4tjDET22BnCvGuLdaNRiBjVU84yivvtFJYu4fa7jBMtaRgjJm4BkwKqvqEO9mlqv83eZmI/JOnUY0hf9/eRlhaCBF27jyacny6QzLGGM+k0tD8lRTLJqTVjS0EcvcAUJc3BbIL0hyRMcZ4Z7A2hfOBC4BKEbkraVEREPE6sLFideN+pk9poxWoLV+Q7nCMMcZTg50p7MRpT+gBXkkaVgLneR9a+h0MRni9uYPiwj1UhiMUTLM7j4wxE9tgbQqvA6+LyEOqGh7FmMaMtW+3Eo0pQd1GXShkTzIbYya8VNoUZovIY2731lvjg+eRjQFrtraQnRVlT2iv272F3XlkjJnYUu0Q779w2hHeB9wP/MrLoMaK1Y37mT+rmyhKrfqhpDrdIRljjKdSSQp5qvpnQFR1u6reAVzobVjp194VYv3OA8yc2gFAXdEc697CGDPhpfI6zqCI+IAGt9uKHcAkb8NKv5e2tqIKuQV7yGlVqqcsSndIxhjjuVTOFG4E8oHPA+8GPgp83MugxoI1jfvJz/bT3lPP3FCYrOmWFIwxE18qHeKtdScPklrvqBPC6sYWTpldxpaOLbw3FLJGZmNMRhjs4bUn6KcjvDhVPeqeU8eLvZ09NOw9yAWLJ/H3poPO29amzE93WMYY47nBzhS+544vBaZx+I6jq4A9XgaVbmsand67p1W0QxPU5ZRBTmGaozLGGO8N9vDaXwBE5PuquiRp0RMiMqF7Tl3T2EJRbhYhv/OiuLoyO0swxmSGVBqaC0RkbnxGROYAE7pXuNWNLZw2t5wtLRspi0Ypn7Y43SEZY8yoSOWW1JuA592nmAWYBfxPT6NKo6bWLt5p7eKT75nN03vXW/cWxpiMksrdR0+JSC0Qf5HAJlUNehtW+qzZ6rQnLJ1byorGJi4PhcHetmaMyRADXj4SkbPc8aU4TzDPc4cL3bIhicgyEdksIltE5NZ+ln9CRPaJyGvu8Onh7cbIWdPYQnlBNrl5rQQ1Ql3UByWz0x2WMcaMisHOFP4ReBb4YD/LFPjNYBsWET+wAjgHaAbWishKVd3QZ9VHVPWG1EP2jqqyunE/p88rZ0v7FgDqCqvBl0rTizHGjH+D3X30NXc83AfWTgW2qOpWABF5GLgY6JsUxoyt+w+x50CQM+ZVUN+2Gp8qc6ecmO6wjDFm1Az28NrNg31QVX8wxLYrgaak+WZgaT/rLReRfwDqgZtUtanvCiJyHXAdQHW1dz2Vxp9POGNeOT/8+5vMCkfItRfrGGMyyGDXRQqHGEbCE8BsVV0E/Am4r7+VVPUeVV2iqksmT548Ql99pDWNLcwozmVWeT71rZupte4tjDEZZrDLR/92jNveAcxMmq9yy5K/oyVp9mfAd4/xO4ctFlPWbG3hfcdNoTvSTXOwhUvCYZhq72U2xmSOIW9JFZFc4BpgIZAbL1fVTw3x0bVArfuw2w7gSuDqPtuerqq73NmLgI2phz6yNu/ppPVQiDPmldPQ3gBAbVYJ5BanKyRjjBl1qdxW8wBO30fnAX/BOeLvHOpDqhoBbgCexqnsH1XV9SJyp4jEO9P7vIisF5HXcbrm/sTR78LIWO22J5w+r5z6tnoA6srq0hWOMcakRSpPNNeo6j+JyMWqep+IPAT8NZWNq+oqYFWfsq8mTX8F+MrRBOyVNY37mVNRwIySPBo2bqIgFmPGVOvewhiTWVI5Uwi743YROQEoBqZ4F9Loi0Rj/G1rK6fPKwegft8b1ITC+OxJZmNMhkklKdwjIqXAbcBKnOcMvuNpVKPsrZ0H6AxGOGNeOapKw4Ftbp9HlhSMMZllsOcUpqnqblX9mVv0AjB3oPXHs9WN+wE4bW45e7r2cCDaQ20UKJuQu2uMMQMa7EzhNRF5RkSuEZGSUYsoDdY0tnDc1EIqJuUcbmQuqAKfP82RGWPM6BosKVQC/xt4L7BZRH4nIleKSN7ohDY6gpEoa7cdbk9oaHNvR62wS0fGmMwzYFJQ1aiqPu32fTQTuBen76K3ReTB0QrQa6+9005POMYZSY3M0yIRiqbbnUfGmMyTUvefqhrCaWDeCBwAJsz7KVc3tuATWDrXPVNo2UidvUPBGJOhBk0KIjJTRL4oIn8Hfu+uf5Gqnjwq0Y2CNY0tnFBZTHFegHA0zNtdu50+j6ZY9xbGmMwz2N1Hq3HaFR4FrlXVV0YtqlHSHYryalMbn3rvHADePvA2EZQ6fyHkl6U5OmOMGX2DPdF8K/BXVdXRCma0rdveSjiqnDGvAiBx51Ftybx0hmWMMWkzWC+pL4xmIOmwurGFLJ9wyuxSAOpbNpGlyuypJ6U5MmOMSY+Mfs/k6sYWTqouIT/byY0Ne19nXihMYJq9bc0Yk5kyNikc6AnzZnM7p7uXjgDqOxqpDYetewtjTMZKOSmIyGki8pSIPC8il3gZ1Gh4eWsrMSXxfEJHsIO94U7qwjEor0lzdMYYkx5D9n2UVHQz8CFAgL8Bv/U4Nk+tbmwhJ8vHSdVODx6JRuaC6eBPpUdxY4yZeAar/X7iPp/wXVXtAdqBy4AYzgNs49rqxv2cMruMnCynf6NEn0fl9nyCMSZzDdbNxSXAq8DvReRjwL8AOUA5MK4vH7UcDLJpd2eivyOAhr1vUhKNMnma3XlkjMlcg7YpqOoTOK/hLAYeB+pV9S5V3TcawXnlpa2tAL2TQstb1IbCiHVvYYzJYAMmBRG5SESeA54C3gKuAC4WkYdFZFw/3bW6cT+TcrJYVFkMQExjNBzc4fR5ZHceGWMy2GBnCt8AzgcuB76jqu2qegtwO/DvqWxcRJaJyGYR2SIitw6y3nIRURFZcjTBD9eaxhZOnVNGlt/Z/R2dO+jWCLW+PCioGOLTxhgzcQ3W0NwBXArkA3vjharaAFw51IZFxA+sAM4BmoG1IrJSVTf0Wa8QuBHnjibP7eroZuv+Q1y9tDpRVt/uNjIXzRmNEIwx/QiHwzQ3N9PT05PuUMa13NxcqqqqCAQCw/r8YEnhQ8BVQBi4ehjbPhXYoqpbAUTkYZz3MWzos97Xcd75/MVhfMdRW9PYAvRuT6hv2YSoMm/qu0YjBGNMP5qbmyksLGT27NmISLrDGZdUlZaWFpqbm5kzZ3gHuYPdfbRfVf9DVX+iqsO5BbUSaEqab3bLEkTkZGCmqv5hsA2JyHUisk5E1u3bd2xt3KsbWyjJDzB/WlGirGHPq8yMRMifYXceGZMuPT09lJeXW0I4BiJCeXn5MZ1tpa2bCxHxAT8AbhlqXVW9R1WXqOqSyZMnD/s7VZU1jS2cPrccn+/wH15D+xa3kXnhsLdtjDl2lhCO3bH+DL1MCjtwXuMZV+WWxRUCJwDPi8g24DRgpZeNzU2t3exo7050bQHQHelme7CF2nAUymu9+mpjjBkXvEwKa4FaEZkjItk4jdMr4wtVtUNVK1R1tqrOBl7CeavbOq8CWt24H6BXJ3hb27eiQF3eFMjK9uqrjTHj2De/+c0R2U4wGOSKK66gpqaGpUuXsm3btn7Xe+qppzjuuOOoqanh29/+dqL87rvvpqamBhFh//79IxJTX54lBVWNADcAT+O82/lRVV0vIneKyEVefe9gVje2MKUwh3mTCxJliT6Pyo5PR0jGmHFgpJLCz3/+c0pLS9myZQs33XQTX/7yl49YJxqNcv311/Pkk0+yYcMGfv3rX7Nhg3N/znve8x6eeeYZZs2aNSLx9MfTnt9UdRWwqk/ZVwdY90yPY2F1YwvvrendkFW/7w3yYjGqpk2Y104bM+792xPr2bBzZLtYWzCjiK99cOh2w1/96lfcddddhEIhli5dSlFREd3d3SxevJiFCxfy4IMPcskll9DU1ERPTw833ngj1113XUox/O53v+OOO+4A4LLLLuOGG25AVXvVSS+//DI1NTXMnTsXgCuvvJLf/e53LFiwgJNO8v5mmIzpDnTL3oPsPxhMvHozrmHfG9SEwvjtxTrGZLyNGzfyyCOP8OKLLxIIBPjnf/5nTjzxRPLy8njttdcS6917772UlZXR3d3NKaecwvLlyykvL+eKK65g8+bNR2z35ptv5mMf+xg7duxg5kynqTUrK4vi4mJaWlqoqDhcLyWvA1BVVcXf/jYqj3E5cY3aN6XZ6n6eT1BV6jvf4X32Yh1jxpRUjui98Oc//5lXXnmFU045BYDu7m6mTJlyxHp33XUXjz/+OABNTU00NDRQXl7OI488MqrxeiFjksKiqmI+d1YNM8vyE2UtPS20RXuoIxsKp6YxOmPMWKCqfPzjH+db3/pWr/Lvfe97iennn3+eZ555hjVr1pCfn8+ZZ56ZeC5gqDOFyspKmpqaqKqqIhKJ0NHRQXl5ea914+vENTc3U1lZ2XeTnsmYpHBSdSknVZf2KqtvdRuZC71rtDHGjB/vf//7ufjii7npppuYMmUKra2tdHZ2EggECIfDBAIBOjo6KC0tJT8/n02bNvHSSy8lPj/UmcJFF13Efffdx+mnn85jjz3GWWeddcRzBaeccgoNDQ28/fbbVFZW8vDDD/PQQw95sr/9ydh3NAM0tG4CoHbKojRHYowZCxYsWMA3vvENzj33XBYtWsQ555zDrl27uO6661i0aBEf/vCHWbZsGZFIhPnz53Prrbdy2mmnpbz9a665hpaWFmpqavjBD36QuN10586dXHDBBYDT1nD33Xdz3nnnMX/+fC6//HIWLnQup911111UVVXR3NzMokWL+PSnPz3iPwNR1RHfqJeWLFmi69aNzKMM/+uZz/HS9mf487tvh8XD6d7JGDNSNm7cyPz589MdxoTQ389SRF5R1SEfDs7oM4X61k3U2jsUjDEmIWOTQiQWobF7L3XhCEw+Lt3hGGPMmJCxSWH7ge2EiVGbUw5ZOekOxxhjxoSMTQoNbQ0A1JXUpTkSY4wZOzI2KdTvewO/KnNmvDvdoRhjzJiRsUmhYe/rzAmHyZ5mb1szxpi4jE0K9R1vu3ce2Yt1jDGDG82us5uamnjf+97HggULWLhwIT/+8Y8Ty+644w4qKytZvHgxixcvZtWqVUd8/lhlZFLoDHWyM9JJnWZB4fR0h2OMGeNGs+vsrKwsvv/977NhwwZeeuklVqxYkeg6G+Cmm27itdde47XXXks88DaSMqabi2Rb2rcAUDepCuz1f8aMPU/eCrvfHNltTjsRzv/2kKulu+vs6dOnM326c7BaWFjI/Pnz2bFjBwsWLDj6fR6GjEwK9S1u9xYV1l22MeawsdB1drJt27bx6quvsnTp0kTZ3Xffzf3338+SJUv4/ve/T2lpab+fHa6MTAoNe/5OYTTGtOl255ExY1IKR/ReGEtdZx88eJDly5fzox/9iKKiIgA++9nPcvvttyMi3H777dxyyy3ce++9I/adkKFJob5lA7XhEDLNurcwxhw2FrrOBgiHwyxfvpwPf/jDXHrppYnyqVMPd/F/7bXX8oEPfOCY97kvT5OCiCwDfgz4gZ+p6rf7LP8McD0QBQ4C16nqhiM2NIJUlYZDu7gwFIEp1vmWMeawsdB1tqpyzTXXMH/+fG6++eZey3bt2pVob3j88cc54YSRP7D17O4jEfEDK4DzgQXAVSLSt6XkIVU9UVUXA98FfuBVPHG7Du3ioIapyy6BQJ7XX2eMGUfGQtfZL774Ig888ADPPvvsEbeefulLX+LEE09k0aJFPPfcc/zwhz8c8Z+BZ11ni8jpwB2qep47/xUAVf3WAOtfBXxMVc8fbLvH2nX2803P87lnP8cDOcex+MrHhr0dY8zIsq6zR86xdJ3t5eWjSqApab4ZWNp3JRG5HrgZyAbO8jAeABr2Obe51Uw7yeuvMsaYcSftD6+p6gpVnQd8Gbitv3VE5DoRWSci6/bt23dM31e/51UqwxEm2Z1HxhhzBC+Twg5gZtJ8lVs2kIeBS/pboKr3qOoSVV0yefLkYwqqoaOR2lDIurcwxph+eJkU1gK1IjJHRLKBK4GVySuISG3S7IVAg4fxEIwG2RZsoy7mg+IqL7/KGGPGJc/aFFQ1IiI3AE/j3JJ6r6quF5E7gXWquhK4QUTOBsJAG/Bxr+IB2Nq+lShKbcEM697CGGP64elzCqq6CljVp+yrSdM3evn9fTW01gNQVz46fYgYY8x4k/aG5tFUv3sd2TGlesYp6Q7FGDOOjFQvqS+88AInn3wyWVlZPPbYwLfEn3nmmRx33HGJ5xT27t0LpNb19rHKqKTQsH8988JhsqYtSncoxphxZKSSQnV1Nb/85S+5+uqrh1z3wQcfTHSRHe9/KZWut49VRvV9VH+oifeEwta9hTFj3Hde/g6bWjeN6DaPLzueL586dCXqZdfZs2fPBsDnG97xeCpdbx+rjEkKrT2t7I/2UJtVCNkF6Q7HGDMGed119tH45Cc/id/vZ/ny5dx2222IyFF3vT0cGZMUGtqcu13riuakORJjzFBSOaL3wljpOvvBBx+ksrKSzs5Oli9fzgMPPHDUSWW4MiYp1LvdW9ROXZzmSIwxY5XXXWenqrKyEnDevHb11Vfz8ssvH1XX28ciY5LCyVnFfL61nYpTh+wPyhiTobzuOjsVkUiE9vZ2KioqCIfD/P73v+fss88GUut6+1hlTFJY2H2IhR0HwF6sY4wZQHLX2bFYjEAgwIoVKxJdZ5988snce++9/OQnP2H+/Pkcd9xxR9V19tq1a/nQhz5EW1sbTzzxBF/72tdYv349AIsXL+a1114jGAxy3nnnEQ6HiUajnH322Vx77bWA0/X2Rz/6UWpqaigrK+Phhx8e8Z+BZ11ne2XYXWdv+gO8+iBc8SsYZsu/McY71nX2yBmrXWePLcdf6AzGGGMGZIfMxhhjEiwpGGPGjPF2OXssOtafoSUFY8yYkJubS0tLiyWGY6CqtLS0kJubO+xtZE6bgjFmTKuqqqK5uZljfbtipsvNzaWqavjvi7GkYIwZEwKBAHPmWI8D6WaXj4wxxiRYUjDGGJNgScEYY0zCuHuiWUT2AduH+fEKYP8IhjMe2D5nBtvnzHAs+zxLVScPtdK4SwrHQkTWpfKY90Ri+5wZbJ8zw2jss10+MsYYk2BJwRhjTEKmJYV70h1AGtg+Zwbb58zg+T5nVJuCMcaYwWXamYIxxphBWFIwxhiTkDFJQUSWichmEdkiIremOx4viMi9IrJXRN5KKisTkT+JSIM7Lk1njCNJRGaKyHMiskFE1ovIjW75RN7nXBF5WURed/f539zyOSLyN/fv+xERyU53rCNNRPwi8qqI/N6dn9D7LCLbRORNEXlNRNa5ZZ7/bWdEUhARP7ACOB9YAFwlIgvSG5Unfgks61N2K/BnVa0F/uzOTxQR4BZVXQCcBlzv/l4n8j4HgbNU9V3AYmCZiJwGfAf4oarWAG3ANWmM0Ss3AhuT5jNhn9+nqouTnk3w/G87I5ICcCqwRVW3qmoIeBi4OM0xjThVfQFo7VN8MXCfO30fcMmoBuUhVd2lqn93pztxKoxKJvY+q6oedGcD7qDAWcBjbvmE2mcAEakCLgR+5s4LE3yfB+D533amJIVKoClpvtktywRTVXWXO70bmJrOYLwiIrOBk4C/McH32b2M8hqwF/gT0Ai0q2rEXWUi/n3/CPgSEHPny5n4+6zAH0XkFRG5zi3z/G/b3qeQQVRVRWTC3YMsIpOA/wb+RVUPOAeRjom4z6oaBRaLSAnwOHB8mkPylIh8ANirqq+IyJnpjmcUvVdVd4jIFOBPIrIpeaFXf9uZcqawA5iZNF/llmWCPSIyHcAd701zPCNKRAI4CeFBVf2NWzyh9zlOVduB54DTgRIRiR/kTbS/7/cAF4nINpxLv2cBP2Zi7zOqusMd78VJ/qcyCn/bmZIU1gK17t0K2cCVwMo0xzRaVgIfd6c/DvwujbGMKPe68s+Bjar6g6RFE3mfJ7tnCIhIHnAOTlvKc8Bl7moTap9V9SuqWqWqs3H+d59V1Q8zgfdZRApEpDA+DZwLvMUo/G1nzBPNInIBznVJP3Cvqv57mkMacSLya+BMnO519wBfA34LPApU43Q5frmq9m2MHpdE5L3AX4E3OXyt+V9x2hUm6j4vwmlg9OMc1D2qqneKyFyco+gy4FXgI6oaTF+k3nAvH31BVT8wkffZ3bfH3dks4CFV/XcRKcfjv+2MSQrGGGOGlimXj4wxxqTAkoIxxpgESwrGGGMSLCkYY4xJsKRgjDEmwZKCMS4Ribo9UsaHEetsTERmJ/dea8xYZd1cGHNYt6ouTncQxqSTnSkYMwS3X/vvun3bvywiNW75bBF5VkTeEJE/i0i1Wz5VRB5333nwuoic4W7KLyI/dd+D8Ef3iWRE5PPuOyHeEJGH07SbxgCWFIxJltfn8tEVScs6VPVE4G6cJ+MB/gO4T1UXAQ8Cd7nldwF/cd95cDKw3i2vBVao6kKgHVjult8KnORu5zNe7ZwxqbAnmo1xichBVZ3UT/k2nBfbbHU74NutquUish+Yrqpht3yXqlaIyD6gKrnLBbdr7z+5L0dBRL4MBFT1GyLyFHAQp0uS3ya9L8GYUWdnCsakRgeYPhrJ/fJEOdymdyHOmwFPBtYm9fxpzKizpGBMaq5IGq9xp1fj9NoJ8GGczvnAeU3iZyHxQpzigTYqIj5gpqo+B3wZKAaOOFsxZrTYEYkxh+W5bzSLe0pV47ellorIGzhH+1e5ZZ8DfiEiXwT2AZ90y28E7hGRa3DOCD4L7KJ/fuBXbuIQ4C73PQnGpIW1KRgzBLdNYYmq7k93LMZ4zS4fGWOMSbAzBWOMMQl2pmCMMSbBkoIxxpgESwrGGGMSLCkYY4xJsKRgjDEm4f8DhqkEOLrutuIAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x113445860"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"for ee in [0.01, 0.25, 1.5]:\n",
" bigger_network = Network([441, 100, 4])\n",
" bigger_network.train(X_train, y_train, X_valid, y_valid, eta=ee, num_epochs=50) \n",
" plt.plot(np.linspace(0,50, 10), bigger_network.valid_accuracy, label='eta=%.2f' %ee)\n",
"\n",
"plt.xlabel('Epochs')\n",
"plt.ylabel('% Validation Accuracy')\n",
"plt.title('Epochs vs Validation')\n",
"plt.legend() \n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part D**: Now let's see if we can get better results with regularization. Using the best learning rate you found in **Part C**, on a single set of axes, plot the **validation accuracy** vs epoch for networks trained on the full training set for at least 50 epochs using the regularization strengths $\\lambda = 10^{-6}$, $\\lambda = 10^{-4}$ and $\\lambda = 10^{-2}$. Which regularization strength seems to perform the best? What is the best accuracy achieved on the validation set? "
]
},
{
"cell_type": "code",
"execution_count": 95,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"On Lambda 1e-05\n",
"On Lambda 0.001\n",
"On Lambda 0.1\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8FPX9+PHXO3dCwhXCISFcCZeCihHxqgelgq1YrVXUelRardVq7aH28MB+W4/218NKD1oPbKvU2qqoCFpFqQpyKB5AkHCFYMKRgxyQc9+/Pz6TsAk5lpDdTbLv5+Oxj92ZnZ15z2Yz75nPNaKqGGOMMQBR4Q7AGGNM12FJwRhjTCNLCsYYYxpZUjDGGNPIkoIxxphGlhSMMcY0sqRgwkZEVEQyO/jZDBGpEJHoTo7pTBHZ1Jnr7Er8v3MR+ZOI3BXIsh3YzpUi8mpH4zThI9ZPIbKJyHZgEFAPVABLgJtVtSIE21YgS1Vzg72trhxDoETkT0CSql7dbP7xwCpgiKoWt7OOgPc30GVFZASwDYhV1br21mu6NrtSMAAXqGoycAJwIvCjMMfTJhGJCXcMYbIAuFhEejWbfxXwUnsJwZhAWFIwjVS1EFiKSw4AiEi8iPxKRPJEZLdX5JDo9/7tIlIgIp+JyDeaFU+8KSLf8Fv2WhF5u6Vti8gXReQDESkTkZ0icq/feyO89c4RkTzgDb95MSJyqleU1PCo8q6AEJEpIrJCREq9OB8RkTjvveXeJj70PneZiJwtIvl+2x7v7UepiKwXkVl+7z0hIvNE5GURKReR90RkdCv794qI3Nxs3ocicrE4vxGRPd7+fywix7Xw91kB7AK+4reOaOAK4Mn29reFmJ4Qkf/zm/6h39/yukD/PkDD91jqfY+nNv9bi8hpIrJaRPZ7z6f5vfemiPxMRN7xvsdXRWRASzGb4LOkYBqJSDowE/AvLngAGINLFJnAUOBub/kZwPeAz3vvnX0Um68Ergb6Al8EbhSRLzdb5ixgPHCe/0xVXaGqyd7VTj/gPeBp7+164DZgAHAqMA34tve5z3nLHO99/p/+6xWRWOBF4FVgIPAd4B8iMtZvsdnAXG+7ucDPW9m/p4HL/dY9ARgOvAx8Afgc7nvuA1wKFLWynidx31ODzwOxwOL29rct3t/yB8B0IMtbr7+2/j4N32Nf73tc0Wzd/b39fBhIBX4NvCwiqX6LXQF8Hfc9x3mxmDCwpGAAnheRcmAnsAe4B0BEBLgeuE1Vi1W1HPgF7kAI7uD1uKquV9UDwL0dDUBV31TVj1XVp6of4Q6iZzVb7F5VrVTVg22s6mGgHPiJt961qrpSVetUdTvw5xbW25qpQDLwgKrWqOobwEv4HdyB51R1lVeW/g/8rrKaeQ44QUSGe9NXAv9R1WqgFkgBxuHq+TaqakEr6/kbcJaXwMEdqJ9S1dqj3N+Gv+UnqlpJs79lgH+f1nwR2Kyqf/PiehrIAS7wW+ZxVf3U+9s+Q+vfowkySwoG4MuqmoI70x+HO8sESAOSgLVecUQpriI6zXv/GFwiaeD/+oiIyCkiskxE9orIfuBbfnEEtH4RucHbhytU1efNGyMiL4lIoYiU4ZJaoEUTxwA7G9bl2YG7WmpQ6Pf6AC6JHMZLqC9zKKFejksieMnmEWAesEdE5otI71bWk4crrvmaiCQDX8YrOjrK/W3+t9zh/2aAf5+21r2j2bwOfY8m+CwpmEaq+hbwBPArb9Y+4CBwrKr29R59vGIagAIg3W8Vw5qtshKXVBoMbmPzTwGLgGGq2gf4EyDNQ2ztwyJyJvAz4EJVLfN764+4s9IsVe0N/LiF9bbmM2CYiPj/n2TgyvU74mngchE5FUgAljW8oaoPq+pJwARcMdIP21jPAlzl8leAbaq61u+9ju5vAU3/fhnN3m/r79NeE8bPcEVl/o7mezRBZEnBNPdbYLqIHO+dIf8F+I2IDAQQkaEi0lCm/wzwda8yNglo3uZ9Ha61TJK4yuc5bWw3BShW1SoRmYIrYw6IiAzzYrlaVT9tYb1lQIWIjANubPb+bmBUK6t+D3fWeruIxIrI2bgij4WBxtbMYtzB8T7gn35XMyd7Z+KxuERaBfhaXw3/xh1U5+IShL/29rc1zwDXisgE7295Twvrbe3vs9eLt7XvcTEwRkSuENcw4DJc8nspwNhMCFlSME2o6l5cccTd3qw7cBWoK73iiP8CY71lX8GV4S9rWMb7TLX3/BugBnfgXYBXXNKKbwP3eXUbd+MOUoGahutr8awcaoG03nvvB7gDWDkuwf2z2WfvBRZ4xWOX+r+hqjW4JDATd9X0B1ziyTmC2PzXVw38B1eJ+5TfW7292EpwxSpFwC/bWE8lLjGkc/h32t7+trbOV3AnBG/g/pZvNFuk1b+PV5/0c+Ad73uc2mzdRcCXgO97+3Y78CVV3RdIbCa0rPOa6TQiMh74BIi3TkzGdE92pWCOiohcJK4vQz/gQeBFSwjGdF+WFMzRugHXjHULro18oGXYxpguyIqPjDHGNAralYKIPOZ12/+klfdFRB4WkVwR+UhEJgcrFmOMMYEJ5sBiT+A65DzZyvszcd3ps4BTcO2rT2lvpQMGDNARI0Z0ToTGGBMh1q5du09V09pbLmhJQVWXixtStzUXAk+qK79aKSJ9RWRIG937ARgxYgRr1qzpxEiNMabnE5HmvcpbFM6K5qE07VafT9Nu741E5HoRWSMia/bu3RuS4IwxJhJ1i9ZHqjpfVbNVNTstrd2rH2OMMR0UzqSwi6ZjraRjY6EYY0xYhTMpLAKu9lohTQX2t1efYIwxJriCVtEsIk/jhjEeIO5OVvfgbgaCqv4JN0jW+bhxVg7gbrBhjDEmjILZ+ujydt5X4KZgbd8YY8yR6xYVzcYYY0IjmJ3XjDFd3IGaOooqaiiudI99FdUUV9ZQXeejT2Js46O399w3yT3HRtv5ZE9lScGYHkJVKa+uo7iihqLKhgN9tXvtHfgPza+hqLKaqtq27uXTuqS46CZJo/mjb9KhRNL8ERMhCcXnU6rrfFTV1nPQe1Q1PnwcrKmnqq7ee/ZRVXNomYPeMlVNpuuZc8Yopk8YFNS4LSkY00X5fMr+g7UtHuCbHtzdeyWVtdTUt3yQT4yNpn+vOFKT3SNrUDKpveLo3yvee46jf3Jc4+uE2GjKDtay/2Atpd5zw/T+A4fmNTzyig+4ZQ/UcrC2vs39So6P8bv6iKFvYpxLGEmHrkr6egkkITYanyo+VVTxXuNNKz7foXna8Iz/9KFlWvy8Nn2vtWVq6nxNDtjVtW7a/8BeXec7bLojYqKExNho4mOjSYyLIiEmmsS4aBJiognFAKaWFEzQ+HzKgdp6KqvrKK+qo7K6jgrvUdn4XE9cTBQpCTH0Toild0IMKQmx9E50zykJMd2+qKLep5RXNT2Ilh449LrMb17pwUMH+5IDtdT7Wj4IpMTH0D/ZHcCH9k1g4tDerR7gU3vFkxgXfcRxpybHk5ocf8Sfq6nzNdnX/QdrGpPJ/oN1XqKpadzvrfsqGve/owfSYIsSl1gT46KJbzhIx0aRGBtNcnwMA5LjSYiNJjE2ynuOJsF7NM7z/2xMlLeOaC8BRDV+Jty/d0sKpol6n1JZ4w7ahw7m9VRU11JRXd/kwF7RyoG+4WBfWVNHZ5zYJMRG0dtLEC5hxHpJJKbZ/BhS4r33Eg/NT4mPISoqkHvXt87nc0UzZa0c2A87APqdVZdXt/09xMVENZ4Z90mMZURqL04a3s8d3P0P9N6Zfv9eccTHHPlBPlTiYqJIS4knLeXIE0pVbX2T77i6zocIRIl4DxDvuWFe4/tRILSxTNSheQ2fEfyWiaJxG82XiY4SRI7uN9RdWFKIAFW19eSXHCS/5AA7vef84oMUV9YcdjA/UNP2pX+DmCghOSGG5Hj36BUfQ9+kONL7JTVOJ8dHk5zQ8DrGb773SIihV1wMNXU+yqpqKauqpbzKHXjLq+oor6qlzHt203WUeWfc+SUHKDvo3gvk7DIlPqZp8kiIbbw6SfFirKyua+Egf+hsvpWTdgBio4U+iXH0SXRFI2nJ8WSmJdM3Ke6wsvWGyto+fkUkxmk4ux7YOyHcoUQsSwo9QE2dj89KD7Kz5AD5JQfZWXygSRLYW17dZPm46CiG9ktkgFe+PDw16fCDdoLfgT0+ll7x0aR4z73iY4iPieq0M6fEuGj6JMV2+PPVdfWNSaO8qrYxWTQkEf/E0pBw9pRXkbvn0Pw6nxIdJY1n7L0TY+mXFMeI1F5NDuKNrXD8ysD7JMaSGBsdMWeSpmezpNAN1NX7KNhf5Q74JQfI9w76DUmgsKyqSfFEdJRwTN8EhvVL4pyxaQzrl0R6/0T33C+JgSnxR12c0pXEx0QTnxzNgA6Uf4OrkKyu83VqojOmu7Kk0AXU+5Q95VXsLPbO7r3nhgN/wf6qJhWOIjCkdwLp/ZM4dXSqd7BPZFh/9zy4d0LENPvrDCJiRTjGeCwphNj2fZW8/HFBk4P/rtKD1NY3LbAe1Due9H5JnDS832EH/SF9EomLsYO+MabzWVIIsQdeyWHJ+kIGJMcxtF8Sxw3tw4zjhjCsfyLp/ZIY1i+RY/om2pmrMSYsLCmE2MbCMs6fOJg/XHlSuEMxxpjDWBlECFVW17Gj6ADjB/cOdyjGGNMiSwohtGl3OQDjhlhSMMZ0TZYUQiinwEsKg1PCHIkxxrQsqElBRGaIyCYRyRWRO1t4f7iIvC4iH4nImyKSHsx4wi2nsIzk+BjS+yWGOxRjjGlR0JKCiEQD84CZwATgchGZ0GyxXwFPquok4D7g/mDF0xXkFJQzbnCKdZAyxnRZwbxSmALkqupWVa0BFgIXNltmAvCG93pZC+/3GKrKxsIyxg2xoiNjTNcVzCapQ4GdftP5wCnNlvkQuBj4HXARkCIiqapa5L+QiFwPXA+QkZERtICD6bP9VZRX1THWWh6Zrm7Hu7DxRYjvDYl9IaGv33O/Q69jbdC6nijc/RR+ADwiItcCy4FdwGHDdKrqfGA+QHZ2dvDvMhEEOQVlAJwUsx0+eBuOvxyirJ7fdCGq8M5v4fX7ICoG6mvaXj4m4fBEkehNN0kkLcyLiQvNPpkjFsyksAsY5jed7s1rpKqf4a4UEJFk4CuqWhrEmMImp9C1PBrz8a9gx3LIeRku+iMk9AlzZMYAVfvhuRth08tw7EUw6/cQm+TmHyyBqlI4WHrouaV5+/Nh9yfudU1529uLTQoseQydDKmjQ/MdGCC4SWE1kCUiI3HJYDZwhf8CIjIAKFZVH/Aj4LEgxhNWOYXlDO8bS8xna2HgBPh0CfzlXLjs7zBwfLjDM5Gs8BN45ioozYPz7oepN7pRFwGS+rvHkaqv9RJKabPkUdLyvJLtUODNq608tB6JhinXw9l3ukRhgi5oSUFV60TkZmApEA08pqrrReQ+YI2qLgLOBu4XEcUVH90UrHjCLaegjM/33wufVcLnfggpg+GZa1xiuPAROO4r4Q7RRKIP/wkv3uquWK95CYaf2jnrjY6FXgPc40jV1bhkcaAI3vszvPcn+ORZmHYPnHBlZBa7Vu6D5b+Ck78BAzKDuqmg1imo6mJgcbN5d/u9fhZ4NpgxdAVVtfVs3VfJj/tvdjMypkLvY+CG5fCva+HZ6yB/LUyf6/6ZIsmeje65/2grZw6lumpY8iNY8ygMPwMueQxSBoU7KicmDpIHuscFv4WTroVXbodFN8Pax2HmLyE9QsYOq66AlX+Adx52V1ADx3XvpGCc3D0V1PuUcTXroe9wlxAAeg+Ba16EV38KK+dBwTq45PGu888ZTAdL3X5/8Dc3LdGu7DhtLKSN8x5jITXLWrl0tv357ip11xo47Tsw7V6I7sKHgmNOgOuWwkf/hNfuhr+eCyd+zcWdnBbu6IKjvhbWPgFvPQSVe2D8BXDu3ZA2Juib7sK/hJ7DVTIraSUfQNa0pm/GxMH5D8HQk9xl/Pyz4KsLIKN5690eZNMSeOm7ULEHTr8VBk2EvTnusScHchaDeo3QJAr6jTiUJNLGu+cBYyAuKay70S1tWQb/nuOKaC79G0yYFe6IAiMCx8+GsefD8odg5R9hw4twzo9ckUpPucL2+WDDc/DG/0HxVhh+Osx+CoadHLIQLCmEQE5BGVkxe4k5uNcVHbXk+Mtg0LHwz6/BE1+EGfe7H3tP6v18oBheuQM+fgYGHut+7EMnH75cXTUUbfESxaZDz5tfA1+tt5BA3wy/ZNFwdTEG4q2D4GF8Pnj7/8EbP3ff02V/gwFZ4Y7qyCX0hi/8H5x4NSy5A5bcCWsXuBOrkZ8Ld3RHZ8sy+O+9rsRg4LFwxb8ga3rIjwGWFEIgp7CcL/XdDhVARhsVeYOPg+uXwXPfgsU/gPzV8KXf9owz4vXPu306WAJn3Qlnfr/1OoSYeBg0wT381de6s6fmyWLrm1BffWi53ul+icLvOVJbrxwsdb+pT1+B4y6BC34H8cnhjuropI2Br/3HNe1e+iNYcAFM+LJLGH2Htf/5ruSzdS4ZbF0GfYbBRX+GiV+FqPDcaMuSQgjkFJZxS/Knrg32gHbKBBP7weynYfkv4c37Yfd6d1bXf1Rogu1sFXtcMtjwAgw5Hq563iW/joiO9Q7yY5vOr6+D0h2HiqAaEsaax6Du4KHlkgc3TRIDx7vXHWly2V0UfOSam+7PdxW0U77Zc64+RWD8lyBzmquIffvX8OlSd8Jx2ne6fl1U8VZXTPTJvyGxP5z3C8ieE/a4RbV7dRDOzs7WNWvWhDuMgO0tr+bkn/+Xdf1/RN/0CXDFwsA/vPk1+Pc3AIWL/wJjzgtanJ1OFT7+l2s1UnPAtTM/7ZbQVmj6fLA/r+lVRcNzTcWh5XqlueRw/Gx3hhYTH7oYg2ndU/DSbe6Ac+kCGDYl3BEFV2keLP0JbFzk6qHOux/Gzux6SbBij6tAXvs4RMfB1G/D6bcEvSOriKxV1ex2l7OkEFz/27yX7z76GmsTboTPz4UzvntkKyje5s70Cj92xS5n3dH122mXfeYORp8ugfST4cJ5h5/dh5MqlO1qmih2rnLPyYPhlBsg+7ruW9xUV+2S8donYMSZrkVbT22l05Ity1zd1b5NkPl5mPFg0JtxBqS6HN79Pbz7CNRVwUnXuP/nlMEh2bwlhS7ir//byupXnuTPcb+BOa917Gyt9iC89D348CnI+oIrc+yKRR6q8MHf3dlafQ1Mu9sdYMNUNnpEVGHLG+6fdusyiEuGydfA1G+5Cu3uojQPnrkaPvsATv8unHtX125uGiz1tbBqPrz5gPv/OfXbrtNoOBoh1NW4q4K3HoID+1zdx7l3hTxRWVLoIr7/zIdMzvklV0a9BnfmdbxoQtV1NHrlTtfP4bK/w5BJnRvs0SjNg0W3uAPq8DNg1sPdd8yago9gxSOurFcVjrvYlVEPOT7ckbUt97+uuNFXD1/+oytvj3QVe1wl7rp/uKvAL/zMFRGGokjJ53O/oTd+5uq8RpzpOqgODU/Hu0CTQhcvh+j+cgrLmBqz2f0QjqasWsQ1Uf36K+4s6NHpsO7pzgu0o3w+WPUX+MOprrXUF/+f65DXXRMCuGR78Xy49UM3DtCmJfDnz8GCWe7A29VOpHw+ePNB+PslkHIMXP+mJYQGyQPhy3+AOf91nUX/8014fKZL/MGi6n4n8z8H//mGa0b7tX+7/4swJYQjYVcKQVRX7+Oku1/g/bg5RJ/xXVec0hkq9rihMbb/zyWK8+4PzxARRVtg0Xdgxzsw+lzX1LE7FbUEqmq/K59f+UcoL3BtyE/7jhuvKtxDcxwohudugM2vwqTLek4T5mDw+VwP+tfnuqbRJ30dzv1p5xbF7lrrrky2LXejF5x7l/uddIF6QLtS6AK27atkgm4mWuvb7p9wpJIHuqadp30HVv/VdXYr+6zz1t8eXz2smAd/PN2NsHnhPNdmvCcmBHCtQk6/FW79yBXLoPD8t+B3x8M7v3NJIxw+W+d6wG9Z5q7QLvqzJYS2REW5yt3vrIWTv+kS/e8nw+pH3W/6aBRtOTTA5e71MPMhuHkNTPpql0gIR8KuFIJo0YefsfmZu/he7L+RO7YHpzXL+ufg+ZvcweCrT8CIMzp/G/72boIXbnJFRWNmwpd+4y7LI4kq5L4O7z4M296CuBR3sJl6I/RJD00M7z8JL//AjUJ66ZOQ3u4JoGlu93pYfDvseBsGT4Lzf9n6iAOtKS+Etx50vapjEuC0m+HUm12RURdjFc1dwENLcjj13W9y+uB6or79bvA2tCfHDY9RvBWm3wen3tT5FWn1te4g+OYDENfLdYSaeEnXawMeap+tcy2W1j/nvovjvuKu4AZPDM72aqtcZ8AP/gajzoavPNqx4amNowrr/wOv3uWaKU+6zP0PtddMtGq/6zC38g+upd1JX4ezbndX8V2UJYUu4BuPr+ThHV8m6eQr3eV9MFWVwfM3Qs5L3p2zHum8oQwKP3ZXBwUfuuZ05/+yS//4w6I0z9U5rF3ghjgefa5LDqPO6bzEWbLdNTct+BDO/AGc8+Pu0dy3O6iphP/9P5fgo+PcAf6UGw+vM6qrdkW2y38FB4vdScA5P+kWDSssKXQB1/78LzxR+wN3NjfxkuBv0P8euwPGuGarRzPoWV0N/O9X7p8lsZ9LbBMu7Lx4e6KDJbDmcXdjmIrd7orhtFtcoj6akTwbererwsV/dj11Tecr2gJLf+w6XqZmwcwHXAc4X73rof/Gz10v+VHnwOfvgWNODHfEAesSFc0iMkNENolIrojc2cL7GSKyTEQ+EJGPROT8YMYTSvsP1DL8wMdu4kjLKTtKBM64Da56Dir3wvxzYOOLHVvXrvddJeZbD7qzoZtWWUIIRGI/OPN78N2PXQV8fa1rBvm7E1xP1qqyI1ufrx6W/QL+8VU3WNoNb1pCCKbU0XDFP90IpVoPf/8KPHWZa5L83A2updJVz8PVz3erhHAkgnalICLRwKfAdCAfd8/my1V1g98y84EPVPWPIjIBWKyqI9pab3e5UnhvaxF7H7+Cz6fkkXD7xtAHULrT69n6vksU594VWFFDbZUbiO/dhyF5kGviOHZG8OPtqXw+12b93YddE+L4PpB9LZzyrUM3W2rNgWKXUHL/C8dfAV/6NcQmhiRsgysqWjHPFRUlD4Rpd8GEi7pda6IGgV4pBLP/+xQgV1W3egEtBC4ENvgto0BDNX0fIITtKoNrU2EZX4jahGacE54A+g5zHd2W3AFv/8YNe9BepWTee67uoGgzTL4apv+s+47/01VERcGYL7jHrve9sW9+Dyv+4HrWnnazu49Gc7ved00cKwpdYj7pWqvUD7WYeHfVd+rN7oQqQupvgpnyhgI7/abzvXn+7gW+JiL5uHs5f6elFYnI9SKyRkTW7N27NxixdrrCHZsYLCUkjDo9fEHEJrgOZbMegR0r4M9nuc41zdVUuuEzHjvPnR1d9RzM+r0lhM42dDJ89XG45QM4eQ5seB7+eJorotj6lqsvUHV1Eo+dByhctwSyv24JIZxi4iImIUD4O69dDjyhqunA+cDfROSwmFR1vqpmq2p2Wlr3GO0x/rNVAMjwTuy01lGTr3IHFxF4bIZrIdNg23J3YHrvj6539LdXuJYzJnj6jYCZD8Jt612xXsFH8OQsV4fzr2vcrUpHnAk3LO8WwyKYniWYxUe7AP9bIKV78/zNAWYAqOoKEUkABgB7ghhX0Pl8ypCydVTFJJOQNj7c4ThDJ8P1b7n78754i+t8Fh3rbkTTfxRcuxhGhPGqJhIl9YfP/cAVT3z0TzcI34YX3HDKZ90RUWenputoNymIyHeAv6tqyRGuezWQJSIjcclgNnBFs2XygGnAEyIyHkgAukf5UBt2lhzgRM2huP+JHNOVKqV6pbqBuZb93DUzlSh3QDrnJzY8QjjFJrge0SdeBZV7Qja+vjEtCeRKYRCwWkTeBx4DlmoATZZUtU5EbgaWAtHAY6q6XkTuA9ao6iLg+8BfROQ2XKXztYGsu6vL3b6DaVG7KBj+tXCHcrioaDcw3+hzXc/kHtqsrluKirKEYMKu3aSgqj8VkbuALwBfBx4RkWeAR1V1SzufXYyrQPafd7ff6w1AjyuzqMx9B4D+488KcyRtCPYYScaYbimgsg3v7L3Qe9QB/YBnReShIMbWbSUUrKaGGOIzrJLQGNO9tJsURORWEVkLPAS8A0xU1RuBk4CvBDm+bumYsnXkJYxzZcXGGNONBFKn0B+4WFV3+M9UVZ+I2O2dmqmsKGdMfS4fDeiC9QnGGNOOQIqPXgGKGyZEpLeInAKgqmEYv6Fr27XhHeKknqiu0D/BGGOOUCBJ4Y9Ahd90hTfPtOCgV8k8cMLnwhyJMcYcuUCSgvg3E1VVH8Ht9NatJRWuZrOmc8yQdgY7M8aYLiiQpLBVRG4RkVjvcSuwNdiBdUu+eo4p/4itiROJirKxaowx3U8gSeFbwGm4Xsn5wCnA9cEMqrvSPRvopZXsH2j3yzXGdE+BdF7bgxuiwrSj7NP/0QeIGWGVzMaY7imQsY8ScAPXHYsbmwgAVb0uiHF1S1W573BA+zNs5Lhwh2KMMR0SSPHR34DBwHnAW7jRTsuDGVS3pEqv3atZ4xvD2CG921/eGGO6oECSQqaq3gVUquoC4Iu4egXjb/9Okqt3szl+Ir0TjuIG7cYYE0aBJIVa77lURI7D3TZzYPBC6qbyVgJQPtDGOzLGdF+B9DeYLyL9gJ8Ci4Bk4K6gRtUN1W9/lwOaSHLG8eEOxRhjOqzNpODdGrPMu8HOcmBUSKLqhmq3vcv7vizGDLH7Ghtjuq82i4+83su3hyiW7utgCQklm1jlG8f4ISnhjsYYYzoskDqF/4rID0RkmIj0b3gEsnIRmSEim0QkV0TubOH934jIOu/xqYiUHvEedAU7VwGwTsYxIrVXmIMxxpiOC6RO4TLv+Sa/eUo7RUkiEg3MA6bjekKvFpFF3t3W3EpUb/Nb/jtA97w3ZN4K6oihauAkYqK70D2ZjTHmCAXSo3lkB9c9BchV1a0AIrIQuBDY0MrylwP3dHBb4ZW3kg0yipFDrFGWMaZ7C6RH89W0qbv+AAAgAElEQVQtzVfVJ9v56FBgp990w7hJLW1jODASeKOV96/HG28pIyOjnc2GWG0VumstK2qnM26w1ScYY7q3QIqPTvZ7nQBMA94H2ksKR2I28Kyq1rf0pqrOB+YDZGdna0vLhE3BOqS+hjW+sVxrPZmNMd1cIMVH3/GfFpG+wMIA1r0LGOY3ne7Na8lsmtZZdB95KwBY6xvDA3alYIzp5jpSK1qJK+ppz2ogS0RGikgc7sC/qPlCIjIO6Aes6EAs4Ze3kt1xGUSnpJGaHB/uaIwx5qgEUqfwIq61EbgkMgF4pr3PqWqdiNwMLAWigcdUdb2I3AesUdWGBDEbWOh/d7duw+eDvJWskymMG2RXCcaY7i+QOoVf+b2uA3aoan4gK1fVxcDiZvPubjZ9byDr6pL2bYKqUt6oH814q08wxvQAgSSFPKBAVasARCRRREao6vagRtYdePUJK+rGcKtdKRhjeoBA6hT+Bfj8puu9eWbHCqriB5CnAxlnw1sYY3qAQK4UYlS1pmFCVWu8imOTt5IdvSYRXR5F5sDkcEdjTKerra0lPz+fqqqqcIdiApSQkEB6ejqxsR27r0sgSWGviMxqqBgWkQuBfR3aWk+yPx/25/F+/y8xOq0X8THR4Y7ImE6Xn59PSkoKI0aMQETCHY5ph6pSVFREfn4+I0d2bDCKQIqPvgX8WETyRCQPuAO4oUNb60m8m+q8VjGKcYOtktn0TFVVVaSmplpC6CZEhNTU1KO6sguk89oWYKqIJHvTFR3eWk+StxKN68VbZYP4vtUnmB7MEkL3crR/r3avFETkFyLSV1UrVLVCRPqJyP8d1VZ7gryVlA04kXqiGW9XCsaYHiKQ4qOZqtp4nwPvLmznBy+kbqBqP+z+hG2JEwGs5ZExQZScHLpGHEuWLGHs2LFkZmbywAMPtLhMdXU1l112GZmZmZxyyils37698b3777+fzMxMxo4dy9KlS9td77Zt2zjllFPIzMzksssuo6bGtelZvnw5kydPJiYmhmeffbbJ9hcsWEBWVhZZWVksWLCgE/feCSQpRItI4/gNIpIIRPZ4DjtXA8oaHUefxFgG904Id0TGmKNUX1/PTTfdxCuvvMKGDRt4+umn2bDh8JH+H330Ufr160dubi633XYbd9xxBwAbNmxg4cKFrF+/niVLlvDtb3+b+vr6Ntd7xx13cNttt5Gbm0u/fv149NFHATca9BNPPMEVV1zRZNvFxcXMnTuX9957j1WrVjF37lxKSko69XsIJCn8A3hdROaIyBzgNTp3hNTuJ28FSDSvlw9j3OAUK3M1JgQqKiqYNm0akydPZuLEibzwwgsAbN++nXHjxnHttdcyZswYrrzySv773/9y+umnk5WVxapVqwJa/6pVq8jMzGTUqFHExcUxe/bsxm34e+GFF7jmmmsAuOSSS3j99ddRVV544QVmz55NfHw8I0eOJDMzk1WrVrW6XlXljTfe4JJLLgHgmmuu4fnnnwdgxIgRTJo0iaiopofopUuXMn36dPr370+/fv2YPn06S5Ys6fB32pJAKpofFJEPgc97s36mqkvb+kyPl7cSHXI8H+XX8tVsq08wkWHui+vZ8FlZp65zwjG9ueeCYwNaNiEhgeeee47evXuzb98+pk6dyqxZswDIzc3lX//6F4899hgnn3wyTz31FG+//TaLFi3iF7/4Bc8//zzLli3jtttuO2y9SUlJvPvuu+zatYthww4N7Jyens5777132PL+y8XExNCnTx+KiorYtWsXU6dObfL5XbvcwNAtrbeoqIi+ffsSExNz2PKtaSnG9j5zpALpp4CqLgGWAIjIGSIyT1W751DXR6uuBnatofy4q6ncWs9YGy7bmJBQVX784x+zfPlyoqKi2LVrF7t37wZg5MiRTJzo6viOPfZYpk2bhogwceLExjL/c845h3Xr1oUr/G4joKQgIifibpd5KbAN+E8wg+rSCj6Euiq2NFQyW1IwESLQM/pg+cc//sHevXtZu3YtsbGxjBgxorE9fnz8oWrOqKioxumoqCjq6uoA2r1SGDp0KDt3HrpZZH5+PkOHDj1s+Ybl0tPTqaurY//+/aSmprb5+Zbmp6amUlpaSl1dHTExMa1ur/m233zzzSbrOvvss9v8zJFqtU5BRMaIyD0ikgP8HndrTVHVc1T1950aRXfiDYK3xjcGERhjA+EZExL79+9n4MCBxMbGsmzZMnbs2HFEn2+4Umj+ePfddwE4+eST2bx5M9u2baOmpoaFCxc2Fk/5mzVrVmOrn2effZZzzz0XEWHWrFksXLiQ6upqtm3bxubNm5kyZUqr6xURzjnnnMbWRQsWLODCCy9scx/OO+88Xn31VUpKSigpKeHVV1/lvPPOO6LvoT1tXSnkAP8DvqSquQAicniajTR5K6H/aD4ojmV4/yR6xQd0sWWMOUpXXnklF1xwARMnTiQ7O5tx48Z16vpjYmJ45JFHOO+886ivr+e6667j2GPd1dHdd99NdnY2s2bNYs6cOVx11VVkZmbSv39/Fi50N6I89thjufTSS5kwYQIxMTHMmzeP6Gg3/E1r633wwQeZPXs2P/3pTznxxBOZM2cOAKtXr+aiiy6ipKSEF198kXvuuYf169fTv39/7rrrLk4++eTGuPr379+p34O0dm8bEfky7gY4p+PqExYCf1XVjg2o0Umys7N1zZo14dm4Kjw0Csaez7m5X2XMoBT+dNVJ4YnFmBDYuHEj48ePD3cY5gi19HcTkbWqmt3eZ1stPlLV51V1NjAOWAZ8FxgoIn8UkS8EEpiIzBCRTSKSKyJ3trLMpSKyQUTWi8hTgaw3bPZthoPF1AydwraiSuu0Zozpcdrtp6Cqlar6lKpeAKQDH+AGxWuTiEQD84CZuFt4Xi4iE5otkwX8CDhdVY/FJZ6uK8+VPW5JnIgqNhCeMabHCaTzWiNVLVHV+ao6LYDFpwC5qrrVux/DQqB5Lco3gXne0Bmo6p4jiSfk8lZC0gA+OpAKwHi7UjDG9DBHlBSO0FBci6UG+d48f2OAMSLyjoisFJEZQYzn6OWtgIypbCysICkummH9ksIdkTHGdKpgJoVAxABZwNm4fhB/EZG+zRcSketFZI2IrNm7d2+IQ/SUFUDJdsg4lZzCMsYOTiEqyoa3MMb0LMFMCruAYX7T6d48f/nAIlWtVdVtwKe4JNGEV2SVrarZaWlpQQu4TTvdTXU0Yyo5heVWn2CM6ZECuZ/CxSKyWUT2i0iZiJSLSCADoKwGskRkpHdP59nAombLPI+7SkBEBuCKk7Ye0R6ESt5KiE1id9JYSg/UWn2CMSHSE4bOvu666xg4cCDHHXdck3UVFxczffp0srKymD59euOIp6rKLbfcQmZmJpMmTeL9999v/Eywh85GVdt8ALnA+PaWa+Wz5+PO/rcAP/Hm3QfM8l4L8GtgA/AxMLu9dZ500kkaFn86U/WJL+kbObt1+B0v6cot+8IThzEhtGHDhnCHoL169QrJdurq6nTUqFG6ZcsWra6u1kmTJun69esPW27evHl6ww03qKrq008/rZdeeqmqqq5fv14nTZqkVVVVunXrVh01apTW1dWpqupbb72la9eu1WOPPbbJun74wx/q/fffr6qq999/v95+++2qqvryyy/rjBkz1Ofz6YoVK3TKlCmqqlpUVKQjR47UoqIiLS4u1pEjR2pxcfFhMbb0dwPWaADH7UCKj3ar6sYOJpzFqjpGVUer6s+9eXer6iLvtarq91R1gqpOVNWFHdlO0FWXQ+HHkHEqmwrLAWuOakyoddehswE+97nPtdjz2H9d/kNnv/DCC1x99dWICFOnTqW0tJSCgoKuMXQ2sEZE/okr6qlumKmqkTMoXv5qUB9kTCVndRnH9EmgT1JsuKMyJrReudOdHHWmwRNhZsvFNM1156GzW7N7926GDBnivorBgxtHfW1tiOyuMnR2b+AA4N+LWYmkkVLzVoJEQfrJ5Lz4AeOG2FWCMaGmPXzobBHpEjfsCuQmO18PRSBdWt4KGDyRmuhe5O6p4NxxA8MdkTGhF+AZfbB056GzWzNo0CAKCgoYMmQIBQUFDBw4sMk2mq8rrENnNxCRdBF5TkT2eI9/i0h6p0bRldXXQv4ayDiVLXsrqPOpXSkYEwbddejstvivy3/o7FmzZvHkk0+iqqxcuZI+ffowZMiQsA+d3eBx4Cngq97017x50zs1kq6q8COoPeDqEwpdS9zxdmMdY0KuOw+dffnll/Pmm2+yb98+0tPTmTt3LnPmzOHOO+/k0ksv5dFHH2X48OE888wzAJx//vksXryYzMxMkpKSePzxxwHCO3R24wIi61T1hPbmhUrIh85eMQ+W/hi+l8P9b5fy+Dvb2XDfecREh7szuDHBZ0Nnd09BGTrbT5GIfE1Eor3H14CiDsba/eStgH4joPcQNhaWkzUo2RKCMabHCuTodh3u3syFQAFwCRAZlc+qruVRxqkA5BSUWf8EY0yPFkjrox3A4bUtkaBoC1TuhYypFFVUs6e82oa3MMb0aK0mBRG5XVUfEpHf4/olNKGqtwQ1sq4gb4V79uvJPNYqmY0xPVhbVwoNQ1uE6YbIXUDeSkjsDwPGsHHTdsCGtzDG9GytJgVVfdF7eUBV/+X/noh8tYWP9DzeTXUQYVNhGQOS40hLiW//c8YY000FUtH8owDn9SwVe6B4i0sKYPdQMCZMutrQ2cuXL2fy5MnExMTw7LPPNnmvtWGt165dy8SJE8nMzOSWW25pGEW6Sw6d3WpSEJGZXn3CUBF52O/xBFDX6ZF0NXnupjpknEq9T9lUWM44q08wpseqr6/npptu4pVXXmHDhg08/fTTbNiw4bDlMjIyeOKJJ7jiiiuazC8uLmbu3Lm89957rFq1irlz5zYe5G+88Ub+8pe/sHnzZjZv3tw4sukDDzzAtGnT2Lx5M9OmTWtMRK+88krjsvPnz+fGG29sdxudpa0rhc9w9QlVwFq/xyKgc/tVd0V5KyEmAYacwPaiSqrrfDa8hTFh1FWGzh4xYgSTJk0iKqrp4bO1Ya0LCgooKytj6tSpiAhXX311kyGyu83Q2ar6IfChiDylqrWdutXuIG8FDM2GmDhyClxfPbtSMJHswVUPklOc06nrHNd/HHdMuSOgZbvK0NmtaWu46/T09MPmQ/cdOnuEiNwPTAASGmaq6qhOjaQrqamEgg/hDPcDyiksIzpKyBwYurJNY0xTNnR2aAQ6IN49wG+Ac3C9mQMa50FEZgC/A6KBv6rqA83evxb4JdCQ6h5R1b8GFHkw5a8BrW/sybyxoJxRA3qREBsd5sCMCZ9Az+iDpasMnd2a1oa1Hjp0KPn5+S2ut1sOnQ0kqurruMHzdqjqvcAX2/uQiEQD84CZuKuMy0VkQguL/lNVT/Ae4U8I4FUyCwxzIxHmFJZZfYIxYdZVhs5uTWvDWg8ZMoTevXuzcuVKVJUnn3yyyRDZ3XHo7GoRiQI2i8jNuLP6QMpRpgC5qroVQEQWAhcCh1fndzV5K2DQcZDQh7KqWvJLDnL5lIxwR2VMROsqQ2evXr2aiy66iJKSEl588UXuuece1q9f3+aw1n/4wx+49tprOXjwIDNnzmTmzJkA3Xbo7JNxvZv7Aj8D+gAPqerKdj53CTBDVb/hTV8FnKKqN/stcy1wP7AX+BS4TVV3trCu64HrATIyMk460jOEI1JfBw8Oh+Mvhy/+ijXbi7nkTyt49Jpspo0fFLztGtMF2dDZ3VNQh85W1dWqWqGq+ar6dVW9uL2EcAReBEao6iTgNaDFnhiqOl9Vs1U1Oy0trZM23Yrdn0BNRWOntY3emEdWfGSMiQRtDYj3Ii0MhNdAVdsrbNsFDPObTudQhXLDOvzvy/BX4KF21hl8fp3WADYVlpGSEMMxfRLa+JAxxvQMbdUp/Mp7vhgYDPzdm74c2B3AulcDWSIyEpcMZgNNugCKyBBVLfAmZ3FoEL7wyVsBfTKgj2sdkFNQzvjBvbtEUzFjwkFV7fffjbRXJdCetjqvvQUgIv+vWTnUiyLS7sipqlrnVUwvxTVJfUxV14vIfcAaVV0E3CIis3DDZhQD13Z8VzqBqksKI8/yJpWcwnIunhx4szRjepKEhASKiopITU21xNANqCpFRUUkJHS8ZCOQ1ke9RGSUXyuikUCvAANcDCxuNu9uv9c/oisNrleyDSp2N9Yn5JccpKK6zgbCMxErPT2d/Px89u7dG+5QTIASEhKa9KA+UoEkhduAN0VkKyDAcOCGDm+xK2tWn5DTWMlsw1uYyBQbG8vIkSPDHYYJoUBux7lERLKAhkbBOapaHdywwiRvBST0gTS3qzkFZQCMHWRJwRgTGdpqfXSuqr4hIhc3e2u0iKCq/wlybKGXtxKGTQVv9MOcwnKGpybRKz6QCypjjOn+2jranQW8AVzQwnsK9KykULkP9n3qOq15NhaW2cioxpiI0lbro3u856+HLpww2ukNkevVJxysqWf7vkoumHRMGIMyxpjQaqv46HttfVBVf9354YRR3gqIjoehkwHYvKccn9o9FIwxkaWt4qPIOhrmrXQJIcYNuZtTYMNbGGMiT1vFR3NDGUhY1RyAz9bBaY1j9ZFTWE5ibDQZ/ZPCGJgxxoRWu81qRCQBmAMcS9M7r10XxLhC67P3wVfbWJ8A7h4KYwanEB1lvTiNMZEjkJvs/A039tF5wFu4ge3KgxlUyOWtcM/DpgCuq/jGgjLGW32CMSbCBJIUMlX1LqBSVRfg7rp2SnDDCrG8lTBwAiT2A2BveTUlB2qtktkYE3ECSQq13nOpiByHu8nOwOCFFGK+eti5qnG8I7B7KBhjIlcgXXXni0g/4KfAItytOO8KalShtGcDVJc1rU/whrewKwVjTKRpq5/CYFUtVNW/erOWA6NCE1YINQ6Cd+hKIaewnCF9EuibFBemoIwxJjzaKj5aJyL/FZE5ItI3ZBGF2o53ofdQ6HPoJnEbC2x4C2NMZGorKQwFfgmcAWwSkRdEZLaIJIYmtBBouKlOxlTwbiBSU+djy94Kq08wxkSkVpOCqtar6lJv7KNhwGPAhcA2EflHICsXkRkisklEckXkzjaW+4qIqIhkt7ZMUJTmQXlBk/qErfsqqK1Xu1IwxkSkQFofoao1wAbcPZTLgPHtfUZEooF5wExgAnC5iExoYbkU4FbgvcDD7iQt1Sc0DG9hd1szxkSgNpOCiAwTkR+KyPvAS97ys1R1cgDrngLkqupWL6ksxF1pNPcz4EGg6shC7wR5KyC+t+uj4MkpLCc2WhiVFtAdR40xpkdpNSmIyLvA27g+Cd9U1bGqeq+q5gS47qHATr/pfG+e/zYmA8NU9eW2ViQi14vIGhFZ06n3is1b6XoxR0U3zsopLCNzYAqx0QFdRBljTI/S1pHvTmCEqv5QVdd29oZFJAr4NfD99pZV1fmqmq2q2WlpaZ0TwIFi2LuxSdERuOIjG97CGBOp2qpoXq6qehTr3oWroG6Q7s1rkAIcB7wpItuBqcCikFU271zlnjNOa5xVUllDYVkV44ZYUjDGRKZglpGsBrJEZKSIxAGzcT2iAVDV/ao6QFVHqOoIYCWuvmJNEGM6JG8FRMU23lQHXH0CWCWzMSZyBS0pqGodcDOwFNdq6RlVXS8i94nIrGBtN2B5K+GYEyH2ULeLnEJveAu7UjDGRKhAxj4CQESmAvfi7qnwW1V9vr3PqOpiYHGzeXe3suzZgcZy1Gqr3D0UTvlWk9k5BeWk9oojLTk+ZKEYY0xX0u7YR36zvgdcBAiuT0G7SaHL+uwDqK9p0mkN3JXCuCEpiNiNdYwxkamt4qM/icjd3p3XAEqBS3CJoSzokQVT4011Dt0Wot6nbNpdbvUJxpiI1lbroy8DHwAvicjVwHeBeCAV+HJowguSvJUwYCz0Sm2ctaOokqpanw1vYYyJaG1WNKvqi7jbcPYBngM+VdWHVbUTe5CFmM8HO1ce3j/BWh4ZY0ybPZpnicgyYAnwCXAZcKGILBSR0aEKsNPtzYGq/S3UJ5QTJZA1KDlMgRljTPi11fro/3DjFyUCS1V1CvB9EckCfo7rd9D95L3rng/ryVzGyAG9SIiNbuFDxhgTGdpKCvuBi4EkYE/DTFXdTHdNCODqE5IHQ78RTWbnFJYzMb1PeGIyxpguoq06hYtwlcoxwBWhCScE8lY2uakOQEV1HXnFB2zMI2NMxGv1SkFV9wG/D2EswVe6E/bvhFNvbjJ7k1UyG2MMENyxj7qend59fA5reWTDWxhjDERaUshbAXHJMOi4JrNzCspJiY9haN+ec/tpY4zpiAhLCish/WSIblpqZsNbGGOMEzlJ4WAp7F5/WP8EVSWnwIa3MMYYiKSkkL8aUBjeNCnsKj1IeXWd1ScYYwyRlBR2vQ9RMTD0pCazcwoaWh5ZUjDGmIDvp9DtnXU7nHA5xPVqMruh5dGYQZYUjDEmqFcKIjJDRDaJSK6I3NnC+98SkY9FZJ2IvC0iE4IYDPTNOGx2TmE5w/onkpIQG7RNG2NMdxG0pCAi0cA8YCYwAbi8hYP+U6o6UVVPAB4Cfh2seFqTU2iVzMYY0yCYVwpTgFxV3aqqNcBC4EL/BVTV/2Y9vQANYjyHqaqtZ+veChvewhhjPMGsUxgK7PSbzgdOab6QiNyEu9VnHHBuSysSkeuB6wEyMg4vAuqo3D0V+BTGDbErBWOMgS7Q+khV56nqaOAO4KetLDNfVbNVNTstLa3Ttr2xwBvewq4UjDEGCG5S2AUM85tO9+a1ZiEhvs1nTmE5CbFRDE/t1f7CxhgTAYKZFFYDWSIyUkTicPdgWOS/gHfDngZfBDYHMZ7D5BSWMXZQCtFRNryFMcZAEOsUVLVORG4GlgLRwGOqul5E7gPWqOoi4GYR+TxQC5QA1wQrnhbiY2NBOdPHDwrVJo0xpssLauc1VV0MLG42726/17cGc/tt2VtRTXFljQ1vYYwxfiKmR/PqwtW8vettMvtmktUvi117XDIYa5XMxhjTKGKSQk5xDk9ueJI6Xx0AQhS9RvXnH9sm8UH5GJcs+maR0TuDmKiI+VqMMaYJUQ1pf7Gjlp2drWvWrOnQZ2t9teSV5bG5dDN/eOcd8sq3kD6ojLyyPNTrNxcbFcvIPiMbrygy+2aS2TeTY5KPIUrC3oLXGGM6RETWqmp2e8tF1ClxbFQso/uOZnTf0Tz8Qi+OT4lnwUVTqKqrYtv+beSW5rK5dDO5Jbl8sOcDFm87VB2SGJPYmCAy+2aS2c9dWQxIHGA35zHG9BgRlRQa1Nb7yN1TwZljBgCQEJPA+NTxjE8d32S58ppytpRuIbc01z1Kcnkr/y2ey32ucZk+8X0Y3Wd0k6uKrH5Z9InvE9J9MsaYzhCRSWHbvkpq6n2Mb2cgvJS4FE4YeAInDDyhyfyig0VsKd3iriq8ZPHy1pepqK1oXCYtMa3JFUVm30xG9x1NUmxSUPbJGGM6Q0QmhcbhLTrYHDU1MZXUxFSmDJnSOE9V2X1gN5tLNjdeWWwu2cwzm56hur66cbmhyUMZ3GswghU5GWOOzNUTruacjHOCuo2ITAo5heXERgujBiR32jpFhMG9BjO412DOTD+zcX69r55dFbsa6yo2l26m6GBRp23XGGM6U2QmhYIyRqclExcT/NZE0VHRZPTOIKN3BtMypgV9e8YYczQiso1lTmE54224bGOMOUzEJYXSAzUU7K+y4bKNMaYFEZcUcgrLAbuxjjHGtCTykoLdWMcYY1oVcUlh0+5y+iXFMjAlPtyhGGNMlxNxSWFjQTnjBve2oSmMMaYFEZUUfD5lU2G53UPBGGNaEdSkICIzRGSTiOSKyJ0tvP89EdkgIh+JyOsiMjyY8eQVH+BgbX27w1sYY0ykClpSEJFoYB4wE5gAXC4iE5ot9gGQraqTgGeBh4IVD7h7MkPHh7cwxpieLphXClOAXFXdqqo1wELgQv8FVHWZqh7wJlcC6UGMh40F5UQJZA20pGCMMS0JZlIYCuz0m8735rVmDvBKEOMhp7CMEQN6kRgXHczNGGNMt9Ulxj4Ska8B2cBZrbx/PXA9QEZGRoe3k1NYznHH2H0OjDGmNcG8UtgFDPObTvfmNSEinwd+AsxS1erm7wOo6nxVzVbV7LS0tA4FU1ldx46iA9ZpzRhj2hDMpLAayBKRkSISB8wGFvkvICInAn/GJYQ9QYyFTbtteAtjjGlP0JKCqtYBNwNLgY3AM6q6XkTuE5FZ3mK/BJKBf4nIOhFZ1MrqjlpOgZcU7ErBGGNaFdQ6BVVdDCxuNu9uv9efD+b2/Q1IjmP6hEEM7ZsYqk0aY0y30yUqmkPhC8cO5gvHDg53GMYY06VF1DAXxhhj2mZJwRhjTCNLCsYYYxpZUjDGGNPIkoIxxphGlhSMMcY0sqRgjDGmkSUFY4wxjURVwx3DERGRvcCODn58ALCvE8PpDmyfI4Ptc2Q4mn0erqrtjija7ZLC0RCRNaqaHe44Qsn2OTLYPkeGUOyzFR8ZY4xpZEnBGGNMo0hLCvPDHUAY2D5HBtvnyBD0fY6oOgVjjDFti7QrBWOMMW2wpGCMMaZRxCQFEZkhIptEJFdE7gx3PMEgIo+JyB4R+cRvXn8ReU1ENnvP/cIZY2cSkWEiskxENojIehG51Zvfk/c5QURWiciH3j7P9eaPFJH3vN/3P737ovcoIhItIh+IyEvedI/eZxHZLiIfe7cqXuPNC/pvOyKSgohEA/OAmcAE4HIRmRDeqILiCWBGs3l3Aq+rahbwujfdU9QB31fVCcBU4Cbv79qT97kaOFdVjwdOAGaIyFTgQeA3qpoJlABzwhhjsNyKu997g0jY53NU9QS/vglB/21HRFIApgC5qrpVVWuAhcCFYY6p06nqcqC42ewLgQXe6wXAl0MaVBCpaoGqvu+9LscdMIbSs/dZVfWIuqUAAAVySURBVLXCm4z1HgqcCzzrze9R+wwgIunAF4G/etNCD9/nVgT9tx0pSWEosNNvOt+bFwkGqWqB97oQGBTOYIJFREYAJwLv0cP32StGWQfsAV4DtgClqlrnLdITf9+/BW4HfN50Kj1/nxV4VUTWisj13ryg/7ZjOnuFputSVRWRHtcGWUSSgX8D31XVMncS6fTEfVbVeuAEEekLPAeMC3NIQSUiXwL2qOpaETk73PGE0BmquktEBgKviUiO/5vB+m1HypXCLmCY33S6Ny8S7BaRIQDe854wx9OpRCQWlxD+oar/8Wb36H1uoKqlwDLgVKCviDSc5PW03/fpwCwR2Y4r+j0X+B09e59R1V3e8x5c8p9CCH7bkZIUVgNZXmuFOGA2sCjMMYXKIuAa7/U1wAthjKVTeeXKjwIbVfXXfm/15H1O864QEJFEYDquLmUZcIm3WI/aZ1X9kaqmq+oI3P/uG6p6JT14n0Wkl4ikNLwGvgB8Qgh+2xHTo1lEzseVS0YDj6nqz8McUqcTkaeBs3HD6+4G7gGeB54BMnBDjl+qqs0ro7slETkD+B/wMYfKmv9/e/cTYlUZh3H8+5TBFDMoZZuEJjIrQm1IaVWRFK50EbURI4aIqKQ2TpsggiIVbSdERBEuVKwwHBdZNhqaTSB5Y4rQnW6CigTpz4yk/Vq8v3vmOjreO2ODzb3PBwbuOfec97x3mOF3zvtenvcVyrxCu37mpZQJxmspN3UfRsTrkm6n3EXfCNSAJyPi7NXr6czI4aOBiFjVzp85P9snuTkH2BERb0q6iRn+2+6YomBmZs11yvCRmZm1wEXBzMwqLgpmZlZxUTAzs4qLgpmZVVwUbFaTdD5TJH+QtLf+Hf7/+BoP15M5p3DOLZI+bn7kRefNk/TClbZjNl0uCjbbjWaK5GJKGOC6q90hSXMi4qeIeKL50ReZB1RF4QraMZsWFwVrJ8M0hKJJelnSUUkj9XUHcv+rubbGV5J2ShrI/V9KWp6v52eswgUk3S9pOHP9v5Z0V+7vlzQo6QAwJOk25boWkt7Lp5nvJP0q6TVJ3ZKGJB3LzPx6au8mYGEeu2VCO12SPsjja5JWNFx7t6R9mbO/eQZ+t9YhHIhnbSHXzHiEEnuBpJXAIkpejIBBSQ8Bo8DjwL2U2OljwLdTuNRx4MGIOCfpUWBDtgdwH7A0Ik5naisAEfFM9qkX2EdZ92IMeCwD/OYD30gapOTjL46IvjynaofyFBQRsUTS3ZQEzTvzvT5KSuxZ4ISkrRHRmAxs1hIXBZvtrs8Y6QWUDKD9uX9l/tRyu5tSJHqAPRExBoxJ2jvF680FtklaRIk2vq7hvf2TRQ5I6gI+Al6MiFMZ5LchC9U/2f9mMcgPAFsBIuK4pFNAvSgMRcSZvNaPQC8XxsWbtcTDRzbbjeZddS/liaA+pyBgY8439EXEHRHxfpO2zjH+P9E1yTFvAAdzDmP1hOP+vEzb7wC7I+KL3F4L3Awsy/7/fJlrtqIx8+c8vuGzaXJRsLYQEX8BLwHrM075M+DpXGsBSQsyl/4IsDrH57uBVQ3NnASW5evJJnfnMh7R3N9K3yStA3oiYtOEdn6JiL9zbqA39/9OeZq5lMOUYkIOG90KnGilD2atclGwthERNWAEWBMRnwM7gGFJ31OWbeyJiKOU+OER4FNKwuqZbOIt4HlJNUrS7KVsBjbmMa3ejQ8ASxomm58DtgPLs29PUeYqiIjfgCP5FdstE9p5G7gmz9kF9LdLKqj9fzgl1TqOpO6I+EPSDcAh4Nn6Ws9mnc7jjtaJ3pV0D2UMf5sLgtk4PymYmVnFcwpmZlZxUTAzs4qLgpmZVVwUzMys4qJgZmaVfwGRlfVj5hR3wQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x1135d7978"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"for ll in [10e-6, 10e-4, 10e-2]:\n",
" print('On Lambda', str(ll))\n",
" bigger_network = Network([441, 100, 4])\n",
" bigger_network.train(X_train, y_train, X_valid, y_valid, eta=1.5,lam=ll, num_epochs=50, isPrint=False) \n",
" plt.plot(np.linspace(0,50, 10), bigger_network.valid_accuracy, label='lam=%.6f' %ll)\n",
"\n",
"plt.xlabel('Regularization')\n",
"plt.ylabel('% Validation Accuracy')\n",
"plt.title('Regularization vs Validation')\n",
"plt.legend() \n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Part E**: Now let's see if we can get better results with different network architectures. On a single set of axes, plot the **validation accuracy** vs epoch for networks trained on the full training set for at least 50 epochs using the architecture from **Part D** as well as two other architectures. Which architecture seems to perform the best? What is the best accuracy achieved on the validation set? "
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"On lambda Network\n",
"epoch 1/ 50: train acc: 0.895 valid acc: 0.876\n",
"epoch 11/ 50: train acc: 0.969 valid acc: 0.951\n",
"epoch 21/ 50: train acc: 0.983 valid acc: 0.963\n",
"epoch 31/ 50: train acc: 0.973 valid acc: 0.948\n",
"epoch 41/ 50: train acc: 0.915 valid acc: 0.890\n",
"epoch 50/ 50: train acc: 0.988 valid acc: 0.963\n",
"On layer Network\n",
"On neuron Network\n",
"Done.\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4VOXZ+PHvPTPJZIeQhDVAWGQXXFBwoRVrBbWCigu21l0pb61o69uirYrWWqu2P19tq8WlarW4VsW6o6ISQEgQkE12yMISEsi+zHL//jgnYRKSMAmZTJJ5Ptd1rpmzzJl7hnDuOc8qqophGIZhADjCHYBhGIbRcZikYBiGYdQxScEwDMOoY5KCYRiGUcckBcMwDKOOSQqGYRhGHZMUjJARkadE5O6A9dkisk9EykQkRUTOEJEt9vpFIY4lQ0RURFyhfJ/20PCziMgHInJNMMe24r3uEpFnjiVeo3MR00/BaA0R2Qn0AryAD9gAvAjMV1V/I8dHASXARFVdY2/7FFioqv93lPdaDIwDeqtqdSvjzQB2AFGq6m3NOdqKiHwIrFDVexpsnw78A0hvLsaWfJYWHnsW8JKqph/1QxhdlrlTMI7FhaqaCAwEHgJ+AzzbxLG9gBhgfcC2gQ3Wj2Bf1CYBCkw7yrHOYILuAF4ArhIRabD9p8DL4U5aRmQzScE4ZqparKoLgSuAa0RkDICIPC8iD4jIMOA7+/BDIvKZiGwDBgPv2sVH7iZOfzWwHHgeqFdEYp//SRF5X0TKgckiEisifxaRXSJSLCJLRCQ24GU/EZHdInJARH4bcK5TRSRLRErsIq6/NBaMiGwUkR8FrLtEpEBEThKRGBF5SUQKReSQiKwUkV6NnOZtIAUr2dWeJxn4EdbdFiJygYh8Y8eTIyLzmvh+EJHFInKj/dwpIo/an287cEGDY6+zP0OpiGwXkVn29njgA6Cv/e9RJiJ9RWSeiLwU8PppIrLe/nyLRWRkwL6dInKHiKy1v/tXRSSmqbiNjskkBaPNqOoKIJeAi529fTMw2l7trqpnq+oQYDfW3UZCM8VCVwMv28uURi6yPwb+ACQCS4BHgZOB04EewK+BwOKsM4HhwA+AewIuav8H/J+qJgFDgNeaiGcBcGXA+hTggKquwkpa3YD+WBf9nwGVDU+gqpX2+a8O2Hw5sKm2aA0ot/d3x7qwzw6y3uUmrORyIjAeuLTB/v32/iTgOuD/ichJqloOnAfk2/8eCaqaH/hCO7kvAG4D0oD3sZJ6dIPPMRUYBIwFrg0iZqMDMUnBaGv5WBfjYyYiZ2IVMb2mqtnANqwkEOgdVc206zFqgOuBOaqap6o+VV3aIOHcp6qV9sV3DVZdBYAHGCoiqapapqrLmwjr38A0EYmz13+MdaGsPUcKMNR+72xVLWniPC8Alwb8kr7a3gaAqi5W1W9V1a+qa+33+H4T5wp0OfCYquaoahHwx8Cdqvqeqm5TyxfAxzRI4s24AnhPVT9RVQ9WAo7FSsC1HlfVfPu93wVOCPLcRgdhkoLR1voBRW10rmuAj1X1gL3+bxoUIQE5Ac9TseottjVzzr0BzyuABPv5DcAwYJNd7POjI14JqOpWYCNwoZ0YptlxAfwL+Ah4RUTyReRhu4K9sfMsAQ4AF4nIEODUgPMgIhNE5HO7aKoY664jtZnPVasv9b+TXYE7ReQ8EVkuIkUicgg4P8jz1p677nx2Is7B+jev1dT3a3QSnb55ntFxiMgpWBeIJW1wrlisX71OEam90LiB7iIyLqCYJbD53AGgCqv4Zw0toKpbgCtFxAFcArwhIil2sUpDtUVIDmCDnSiwfz3fB9xnV5C/j1WX0lTl+4tYdwjDgY9UdV/Avn8DfwXOU9UqEXmM4C7ee7CKr2oNqH1i19u8ab/nO6rqEZG3gdoK76M1RcwHjg84n9jvlRdEXEYnYe4UjGMmIkn2L+tXsJo0ftsGp70Iq6nrKKwiiBOAkcBX1C+Lr2P/cn0O+ItdSeoUkdOaqcQO/AxXiUiafY5D9uYjmtbaXgHOBWZT/9f9ZBE53m4FVYJVnNTUOcBKCudg1QO80GBfIlBkJ4RTObLYrCmvAbeKSLpdeT03YF80VmItALwicp79OWrtA1JEpFsz575ARH5g3wH9CqgGlgYZm9EJmKRgHIt3RaQUqwjht8BfsCov28I1wD9Vdbeq7q1dsH49/0Sa7ox1B/AtsBKrGOtPBPd3PhVYLyJlWJXOM+0K4SOo6h5gGVZZ+qsBu3oDb2AlhI3AF1hFSo1S1Z1YF9R4YGGD3f8D3G9/v/fQdMV3Q09jFWGtAVYB/wl4v1LgVvtcB7ESzcKA/Zuw7oK2262L+jaI9zvgKuAJrLuyC7EaCtQEGZvRCZjOa4ZhGEYdc6dgGIZh1DFJwTAMw6hjkoJhGIZRJ2RJQUSeE5H9IrKuif0iIo+LyFa7W/xJoYrFMAzDCE4o+yk8j9VS5MUm9p8HHGcvE4An7cdmpaamakZGRttEaBiGESGys7MPqGra0Y4LWVJQ1S/tDjxNmQ68qFbzp+Ui0l1E+tjN/ZqUkZFBVlZWG0ZqGIbR9YnIrqMfFd46hX7U746fS/3u8nVE5GaxRrDMKigoaJfgDMMwIlGnqGhW1fmqOl5Vx6elHfXuxzAMw2ilcCaFPOqP0ZKOGUPFMAwjrMKZFBYCV9utkCYCxUerTzAMwzBCK2QVzSKyADgLSBWRXOBeIApAVZ/CGkHyfGAr1hC7bTVmjmEYhtFKoWx9dOVR9ivw81C9v2EYhtFynaKi2TAMw2gfZpIdwzCMDkJVqfBWcLDqIMXVxRyqPlS3FFcX8/307zM6dfTRT3QMTFIwDKPDUVVKqrxUe33ERDlxuxxEOx1Yk711Dl6/l+Lq4noX92Cee/3eJs+ZGptqkoJhGF1D7YX+QFk1BaXVHCir5kBpNQVl1RworbHW6/bVUOOrP2mdCLhdjrok0fDR7XISE2U9uqPqr7f20R3lINopVPmqWnZxrzpEqae0ye/C5XDR3d2d7u7udHN3Y2DSQMa5x9HN3a3e9uSY5LptSdFJuByhv2SbpGAYBgAVngpySnPYWbKT4upiuru70yOmBz1ietRdnBxSvxoy8EJ/+AJvXdTrLvxl9npZNTXeI2cndTqElPhoUhPcpCW6GdozkdTEaNIS3LijnFR7fFR7/XWPVU08VtR4OVgRuN1PtddHtcd/RIKxowdHJQ5XKeIqQVylOKKsx7r12ucOT5Pfm2gMTo3HRQJRJBAlfUmQYaQ4EolxJhHrTCTOmUScK4n4qCQSXd2Ii4oj2r77iXI5iHI6cNuPUQ4H0eogyiP4/A7KahxUVzgoclbRM8lNUkzUsf5TN8skBaPTq/H6Ka/2UlbtpaLGR1m1l/JqLxU1XsqqfQH7vJRX++qe1+4rt18H1gVKBBwiOOxHCXjucAQ8F+rvO9p+hzQ4b/33qX2ty+HA6RBcTsHlEJwOh/0oRDlbtu5yCC6no+45+DhYs599lTnsrcxhT3kOeeW7yCvPoaByX7Pfs+AgShJw+hNRXzxeTxzV1bF4PXGoLwH1xqO+eNSbgGgCPWK6k5YQS2qDC31aopvUhNolmuS4aByOtisWUlVKakrYX7GfgsoCCioK2F9RwN7y/da2igIOVB3gYNUBavxHziQa7YglwdWDOEcyMY7+uKU7UZKISxNwagIOjUP8Cag/DrxxeNWBx++nxmslH4/Pj8erlPn8FHmt9RqfH4/XT42vDI+v6TuIo/n9RWP46cSBx/L1HJVJCkabU1W8fsXnVzw+Pz5/8+uVNT7Kaw5fvMsbXNzLay/eNfbF3b6wl9dYzxv/FXgkp0OIj3aS4HYR53YR73aR4HbSIz6OuGgnAvgV/Kqo/Wgt1mfy+bXJ/T6/v26/6uHj6r828HX1zxN4bp9f8fr8dd+R19/SKXPV+pUbfSBgKUCiD+CILkTk8Pelvhj8NWn4q/virxlrPa9JRX3xiLPcWlzWozOqHIe7Emd0Bc6oChxR+3DElVCj5Y1GUY1Q5u5OVEwyjphkNKYHvqgeVJNMuSeZ0soeFGsPkj3JJMck093dvdniEVXlUPWhugt9Y48HKg9QUFHQ6MU+ISqB1NhUesb1ZGC3E+gZ17NuPTU2lbTYNNLi0oiPim/h990yqorHp3WJwuPzU20/enxal1xq6rYdTjhj07uHNDYwSaFL8/j8FJXXUFhWYz2WV1NUbj0/UFZDlceHt8EFqOFFu+HFKZj1Fl/DmuByiH3hdhHvdhLvdhEf7aJnorvuee2FPXA93m1f+KPrv9bt6lwVlbVqk0rg9+zzK8VVJewq2cXu0t3klO0it9T6xb+nYjdVvsq617skirSYdNJihpPqTifFnU5ydF+So/oSRSJ+xTqn/W/p9Ssuhxz+NW//wm/qF73H76G4upiiqiIOVh2kqKqo7vnBqoMcrD5IYWUhWw9trWtVoxz5RyJIXTl6sjuZHjE9ULTeBd/jP7IYJzE60bqgx6ZxYs8T6y7ugY+psanERcW17T9MK4kI0S4h2uUAd7ijOZJJCp1Ijddfd3E/fKGvoci+2B+wt1mJoJqSqsZbMTgEesRHE+921RUrOB0Ou+hB6ooeoqOcuGJcAUURDlz1jmn4GoddXHH09drzOR1CbLSz0Qu72+Vs52+446nx1ZBbmsvOkp3sKtnFrpJd7Cjewa6SXRRWFdYdJwh9E/qSkZTBaf1OJqNbBgOTBpKRlEHv+N5H1AW0pShHFKmxqaTGpgZ1vNfv5VD1obqkUVR9OIEEJpQdxTsASItLY3zS+CMu9Glx1sU+1hUbss8WiUxSCKMqj+/wRdy+kNdd6MvsbbW/7stqKK1u/CLvdAg94qNJiY+mR3w0o/smkZrgpoe9Xrs9JcFNSnw03WKj2rQM12gbFZ4K/rv9v+wo3sGOkh3sKt5Ffnk+fj1c3NMjpgcZSRl8L/179S786YnpuJ0d8GdnI1wOV4uSiNG+TFJoZ6+u3M3fPt9GYVk15XblZkMu+yJvXcij6Z/c/fDFPSGalHg3KQmHL/hJMeYi39kdqjrE7EWzWVe4jlhXLBlJGYxJHcMFgy8go1sGGUkZDEgaQFJ0UrhDNbo4kxTa2etZuXh8fi4/pT8p9q/3hr/mk2JcnbLs22idfeX7mPXJLHJKc3hs8mOc3f/s8P77b3gH1rwKUbHgToBoe2nsecNtUfHgMKPndGYmKbQjn185fs8bzI/9kB67EsAZDa5o69EZBU5309ucUeByN9je2LZmjnVG29vtbSKg/gaLgt/XyPYWLP7m9vus9wh8v4SekJwBcSlWTBEkpySHmz65iYNVB3nqh09xSu9TwheMKmQ+BovmQVI/6++lpgxqysFTEfx5ouIDkkU8uBMDnidAdKL9GG8nk8SA5/b+wGNNkmlXJim0ox0HypnOYmLECz1HgM8DvhrwVoO3BqrLrPXaxVvTYL0aGmm10WVExVvJobGl+wCIiglndG1uy8Et3PzJzXj8Hp6d8ixjUseELxifFz74NWQ9C2NmwEVPWj8gavl9hxNEdRnUlNqP5db26tKA52X2sWWHjynbG7DdPi4YrlhIGWIvQ+svcT1C811EOJMU2tHGnALOlV2UjbiJuIseat1JfN76ieKIBOIBX3XjSaXesdXW+cTRxCL2o/Mo++3F0fA4aeQ1jRwDULYPDu6CgzutpWg7bPsMvJX1P3ti30YSxkDrMaFXp7rLWFuwltmLZhPjjOH5Kc8zNHlo+IKpKYc3boDNH8AZt8EP7j3y17nDCTHdrKUt+P3gKa+fZOqe1yaZMijdC4VbYd962PQeBI4LFNsjIEkMqf88yrRIai2TFNpRwdZs3OLFNXRi60/idFkLHaPNdcioQtn+w4ni4E44ZCeOHV/AmgXUu2tyxR5OEEfcZQyE6I7zfS3fs5xbP7uVlJgUnj73adIT08MXTNl++PcVsGc1nP8onHpT+7yvw2EVG7kTITHI1/g8cGg3HNhiJYraZftiWPPv+sd269/43UX3AVaCM5pkkkJ7yl8FgLP/+DAH0gmIQGIvaxkw4cj9nioozqmfNA7utO44di45sngiodfhBNEwaST2abdy6892f8YdX9zBwKSBzP/hfNLi0trlfRt1YAu8NMNKDFe8DCPOD18swXBGHS5Kaqi6DIq22Yli2+GEsfZ1qC4OOEc0JA+qf3eRepz1GJ/Wqe42Q8UkhXaiqqQUr6PUlUJiUr9wh9P5RcVY/5lTjztynypUFNlJYkf9pLF7Oax7w6rkruWMhh6D4fu/tsrTQ+Tdbe9yd+bdjE4Zzd/P+Tvd3G1UFNMau5fDgplWkd6170H6yeGLpS24E6DPOGsJpAoVhUfeXRRug62fWMWpdedIavzuImWIdUcTIUxSaCc5RZWM8m+luMfxJJpfI6ElAvEp1tLYxc7nOfIuY/tieON62PwxnP9w25Wd217e+DIPrXiICb0n8PjZj4d3yIX1b8N/bobu/eEnb0CPQeGLJdREID7VWgaeVn+f32f9HTS8u8j5Gr59g/rFkzEQFWe1ioqKtZ5HxVnFkkd7HhVrvy5wX6zVsKL2uA5UpBXSpCAiU4H/A5zAM6r6UIP9A4HngDSgCLhKVXNDGVO4bNqVyzmSz/7+Pw53KIYzyroz6DH48DafF756FL54GHYthUv+AQNPP+a3UlXmr53PX1f/lcn9J/PI9x8JX89jVVj2N/j4d9D/VLjylchuweNwHi5CHHpO/X2eSijacThRVB60muV6Ku0mupXWekURePKsSnNPJdRU2M13W9hK0OlukDxi6yeg2ufHXw4ZZ7TRF9C4kCUFEXECfwN+COQCK0VkoapuCDjsUeBFVX1BRM4G/gj8NFQxhVPRlhU4ROkx/BgqmY3QcbrgrLkw5Afwn5vgn+fDmbfDWXda/UZaQVV5NOtRXtzwItOGTOO+0+9rl0lSGuX3wUd3wddPwchpcMl800KnOVGx0GuUtbSUKnirDicIT0X9ROKpaLAv4HnD9aoSKN1nt9SqgP4TgE6aFIBTga2quh1ARF4BpgOBSWEU8Ev7+efA2yGMJ6wcdiVzdP9OXnbb1fU/BX62BD6cC0v+YjWNveRpSBvWotP4/D7uW3Yfb219ix+P+DG/OfU3IR2UrlmeSnjzRtj0X5j4czj3AdMhLJRE7F/4sUBKuKNpsVD+ZfQDcgLWc+1tgdYAl9jPLwYSReSIb1FEbhaRLBHJKigoCEmwoaSqpBZ/S0F0OsQmhzsc42jcCTD9r3DFS1YTyH98D1Y+Y/0CDEKNr4b//fJ/eWvrW8waO4u5p84NX0IoPwAvXGi18Z/6EEx90CQEo1nh/uu4A/i+iHwDfB/IA44YJU5V56vqeFUdn5YWxiZ8rbS3pIqRupXSlLHhDsVoiZEXwv8ss+oW3vsV/Ptyq/lmMyo8Ffzis1/wya5PuGP8Hdxy4i3hG8eocBs8+0PY+y1c/iJMnB2eOIxOJZRJIQ/oH7Cebm+ro6r5qnqJqp4I/NbediiEMYXFlq1b6CNFuAaEcVwbo3USe8NVb8J5j8COL+Hvp8F3HzR6aElNCbM+mcXyPcu5//T7uWb0Ne0cbICclVZCqDwEVy+EUdPCF4vRqYQyKawEjhORQSISDcwEFgYeICKpInX31XditUTqcg5u+RqAniNOO8qRRockAhNuhpu/gKQ+Vvv+d2+zKg9tByoPcP2H17OucB2PfO8RLj7u4vDFu/G/VpGROwluXNR45z/DaELIkoKqeoFbgI+AjcBrqrpeRO4XkdqfLWcB34nIZqAX8IdQxRNOrj2r8OIkJv3EcIdiHIueI+DGT+GMOZD9PDw1CfKy2VO2h2s/vJZdJbv469l/5dyMc8MX49f/gFevslrN3PBJ471/DaMZokFWnnUU48eP16ysrHCH0SIr7ptEb3cVA+auDHcoRlvZ8RW89TO2VxVw84BBVDgc/O2cv3NizzAlfr8fPrkblv0Vhl8AM57pUOM9GeEnItmqetQxdsJd0dzlFZRUMsK/lXJTydyo59c9zxub36As2KGUO4pBk9g4859cl94PT00pz1XGcKKre3hi8VTBm9dbCeGUm+CKf5mEYLSaSQohtv27NSRJBVGmkvkIu0t28+fsP3PfsvuY/Npk7vrqLlbsWVFvTuKOKntfNtcvnkN0XCovjLmVEfu3WcVJq/8ddNPVNlFRBP+6CNa/ZfU/OP+RDjVkgtH5mLGPQqxkm1XJ3GtkaHshdkaZ+ZkAPPL9R1ixZwUf7PiAd7e/S7+EfkwfOp3pQ6bTN6FvmKM80le5X/HLxb+kd3xvnj73aXrH94Zh58Nbs+Ht2bD5Q/jRY6EfQuLgTnjpUmtI8UufC+lgfkbkMHcKIRa1ZxUVxJCY3oru8l1cZl4m/RP7MzVjKvecdg+fXf4Zf5z0R9IT0/n76r8z9c2p3PTxTby//X2qvFXhDheAD3d+yK2f3UpGtwyen/q8lRDAGqf/moVwzn2w6X148nSrN3So5K2CZ86B8gK4+h2TEIw2Y5JCiPUsXU9u7AhzS99Aja+GFXtXcHrfw4POxbpi+dHgH/HMuc/w4YwPmT1uNrtLdvObr37D2a+dzQPLH2DdgXWEq3HEG5vf4Ndf/JqxaWN5dsqzpMQ26HzvcMKZt8FNn1rNQf91MXx4p1Xm35a++xCev8AaRuGGj9tk4D7DqGWSQggVl5YzxLeDstRxRz84wnyz/xsqvZWc2e/MRvf3S+jH7BNm88GMD3jm3Gf4Xv/v8fbWt7nyvSu5ZOElvLD+BQorC9st3n+u+yf3LbuP0/udzlM/fIqk6KSmD+4zDmZ9AafOguV/h6cnw951bRNI1nPwypWQOgxuWARpw9vmvIZhM0khhHZu/Bq3eIkZaCqZG8rMy8TlcHFq71ObPc4hDib0mcBDkx7i88s/5+6JdxPniuPRrEc55/VzmPPZHD7f/Tkevyckcaoqj696nL9k/4UpGVN4YvITxLqCGF00Ktaal+Enb1iTvDw9GZb+1Wo62hp+PyyaB/+93Rrm+dr3rFnpDKONmYrmECrdugKAPqNMJXNDS/KXcHLPk1s02UxidCKXD7+cy4dfzrZD23h769ss3LaQz3I+IyUmhQuHXMhFQy9iSPe26bDlVz8Pfv0gr373KjOOm8HdE+/G2dJiwON+CLOXwrtz4OPfwpaP4KKnoFsLZt/zVsM7P4dvX4eTr4Xz/2zP020Ybc/cKYRQ1N5VFNKd5D5deGarVthfsZ8tB7dwer/Wl4UP6T6EX43/FYsuW8Tjkx9nbNpYXtrwEhe9cxE/ee8nvL75dUprSlt9fo/fw11L7uLV717lutHXce9p97Y8IdSKT7VGXJ32BORmw5Onwbr/BPfaykPWPMrfvg4/uMdq1WQSghFC5q8rhHqXrScnbiQpZvrNejLzrKaoZ/Q99juoKEcUkwdMZvKAyRRWFvLf7f/l7a1vc/+y+3l4xcOcM/AcLh56MeN7jw96+OpqXzV3LL6DxbmLmXPSHG4Yc8Oxj3QqAiddDQPPsKbCfOM62PxR81N/HsqBly+1Rju95GkYe/mxxWAYQTBJIUTKi4vo78sjL/VH4Q6lw8nMzyQtNo1hyS2buOZoUmJTuGb0NVw96mrWF67nrS1v8cGOD/jv9v9afR+GTGf60Ob7PpR7yvnFZ79g5d6V/HbCb5k5YmabxkjKELj+I/jyEWtpaurPPWvh5cusCXJ++h8Y9L22jcMwmmCKj0Ikd8NSHKLEDmq+IjXS+Pw+luUv44x+Z4RsngERYUzqGO4+7W4+u/wzHpr0kNX3YY3V9+HGj2/kve3vHdH34VDVIW786EZW7VvFHyf9se0TQi2nCybfCdd/aDVjff4C+PR+8NZY+7cugn+eBw6XdYxJCEY7MncKIVK6zapk7jfKtCEPtK5wHSU1JW1SdBSMGFcMFwy+gAsGX0B+WT7vbHuHd7a+w9yv5pIYlch5g87joqEX0TOuJ7M+mUVOaQ6PTX6Ms/qfFfrg+p8KP/vK6svw1Z9h66cwajp89oA1yumPX7eG6jaMdmSSQohE7/uGXfRhQM/e4Q6lQ8nMy8QhDib2mdju7903oS+zx81m1thZZO3N4q2tb7Fw20Je2/wa0Y5oXA4XT57zJKf2ace7O3eiNfXnsCmw8Fb49D4YcrY1U5o7sf3iMAybSQoh0qdsA1vixzHQVDLXk5mXyZjUMXSPCdOIolh9H07tcyqn9jmVuybcxYc7P2RJ7hJuGnsTY1LHhCeokRdC+inW3cLYy8EZFZ44jIhnkkIIVBXlkqYH+DbN9GQOdKjqEN8e+JbZ4zrOXMGJ0YlcNuwyLht2WbhDsab+PPEn4Y7CiHCmojkE9mxYCkDcINOTOdDyPctR9Jj6JxiGEVomKYRA+Y6v8aiT9JFmbtxAS/KWkBSdxJiUMBXRGIZxVCYphIB732q2yAD6pYV4PP1ORFXJzM/k9L6nt75nsGEYIXfUpCAivxCR5NacXESmish3IrJVROY2sn+AiHwuIt+IyFoROb8179Oh+P30Kd9IfvyokLXD74w2H9zMgcoDnNHPjANlGB1ZMHcKvYCVIvKafZEP6konIk7gb8B5wCjgShFpONPM74DXVPVEYCbw9+BD75g8B7aSoOVUhWsC9w6qdpa1wPkTDMPoeI6aFFT1d8BxwLPAtcAWEXlQRI42FOWpwFZV3a6qNcArwPSGpwdqB6bvBuS3IPYOaf9GU8ncmMy8TIYlD6NnXM9wh2IYRjOCqlNQa6qrvfbiBZKBN0Tk4WZe1g/ICVjPtbcFmgdcJSK5wPvALxo7kYjcLCJZIpJVUFAQTMhhU7Hja8o0hoHDzZ1CrXJPOav2rzJFR4bRCQRTpzBHRLKBh4FM4HhVnQ2cDBzrxLBXAs+rajpwPvAvkSOHslTV+ao6XlXHp6WlHeNbhlbM/tVsYDCD0pqZmSvCrNizAq/f225DWxiG0XrBdF7rAVyiqrsCN6qqX0SaGwI0D+gfsJ5ubwt0AzDVPt8yEYkBUoH9QcTV8Xhr6F2xmVUJF+FwmErmWpn5mcS6YjnR1LMYRocXTPHRB0BR7YqIJInIBABV3djM61YCx4nIIBGJxqpIXtjgmN3AD+xWNJL1AAAgAElEQVTzjgRigI5dPtQM3951ROGluucJ4Q6lw1BVluQtYULvCUQ7o8MdjmEYRxFMUngSKAtYL7O3NUtVvcAtwEfARqxWRutF5H4RmWYf9ivgJhFZAywArrXrLzqlws1WJXP8YNNprdbu0t3kleWZ+gTD6CSCKT6SwAu1XWwU1JhJqvo+VgVy4LZ7Ap5vALrM1aJqx0oKtBtDhg4PdygdxpK8JUDbzLJmGEboBXOnsF1EbhWRKHuZA2wPdWCdUWzBGr7VoQztaYY8rrU0fykDEgfQP6n/0Q82DCPsgkkKPwNOx6okzgUmADeHMqhOqaqElKqd5CeMwuU0o4eANdfxyr0rTdGRYXQiRy0GUtX9WJXERjM0/xscKJ7epoVNrVX7VlHpreTMfmeGOxTDMIJ01KRgNxO9ARiN1ToIAFW9PoRxdTqHtn5NMpA02MzJXCszL5MoRxTje40PdyiGYQQpmHKOfwG9gSnAF1j9DUpDGVRnVL1zBTv8vRiWMSDcoXQYmfmZnNTrJOKi4sIdimEYQQomKQxV1buBclV9AbgAq17BCBB3YA1rdSjDeieEO5QOYW/5XrYe2sqZfU3RkWF0JsEkBY/9eEhExmANXGdGNQtUsoekmv3sTRiF22XmCgCr1RFgKpkNo5MJpr/BfHs+hd9h9UhOAO4OaVSdjOZlI4Cnj6lkrrUkbwk943oytPvQcIdiGEYLNJsU7MHpSlT1IPAlMLhdoupkynasJEadJA8+OdyhdAhev5fle5ZzzoBzzERDhtHJNFt8pKp+4NftFEunVbNrJZu0PyP69wp3KB3CugPrKK0pNUVHhtEJBVOnsEhE7hCR/iLSo3YJeWSdhd9P/IG1fKtDGNnH9GQGq+jIIQ4m9pkY7lAMw2ihYOoUrrAffx6wTTFFSZai7cT4SsmPH0VcdFBDQnV5mXmZHJ96PN3c3cIdimEYLRRMj+ZB7RFIp5WXDYDXVDIDcLDqIOsL1zP7hNnhDsUwjFYIpkfz1Y1tV9UX2z6czqdi5wpU3aQNGhvuUDqEZfnLUNT0TzCMTiqY8o7AGehjsCbFWQWYpAB4d69kgw5mVD9TzQJWL+bu7u6MShkV7lAMw2iFYIqPfhG4LiLdgVdCFlFn4q0hrmgjq/3ncmVfMyezX/1k5mVyWp/TcDpMJz7D6IxaM8ZzOWDqGQD2r8elNeTHj6JbbFS4owm7zQc3U1hVaJqiGkYnFkydwrtYrY3ASiKjgNdCGVSnYVcy+/ucFOZAOobaWdZO73t6mCMxDKO1gqlTeDTguRfYpaq5IYqnU6nZlUWxJtF3oBnKAaymqCN6jCAtLi3coRiG0UrBFB/tBr5W1S9UNRMoFJGMYE4uIlNF5DsR2SoicxvZ//9EZLW9bBaRQy2KPsy8OVms8Q9hTL/u4Q4l7Mpqyli9f7W5SzCMTi6YpPA64A9Y99nbmiUiTuBvwHlYRU5Xiki9JimqeruqnqCqJwBPAP8JNvCwqyohtngra/xDGG0qmVmxdwVe9ZpZ1gyjkwsmKbhUtaZ2xX4eHcTrTgW2qup2+zWvANObOf5KYEEQ5+0Y9qxGUHLjRpKS4A53NGGXmZdJnCuOE9JOCHcohmEcg2CSQoGITKtdEZHpwIEgXtcPyAlYz7W3HUFEBmK1aPqsif03i0iWiGQVFBQE8dbtoLaSua+pZFZVMvMzmdBnAlFO0wrLMDqzYJLCz4C7RGS3iOwGfgPMauM4ZgJvqKqvsZ2qOl9Vx6vq+LS0jlGJ6c3JZoe/F4P79w93KGG3s2QneWV5nNHXNEU1jM4umM5r24CJIpJgr5cFee48IPCKmW5va8xM6g+41+H5crNZo0MY08/UJ9TOsnZ6P1PJbBid3VHvFETkQRHprqplqlomIski8kAQ514JHCcig0QkGuvCv7CR848AkoFlLQ0+bEr34i7Pt1semZFAl+QtISMpg/6J5q7JMDq7YIqPzlPVuqai9ixs5x/tRarqBW4BPgI2Aq+p6noRuT+wjgIrWbyiqtrYeTqkvFUA7IoZQc/EyK5krvJWkbU3y/RiNowuIpjOa04RcatqNYCIxAJBXQlV9X3g/Qbb7mmwPi+4UDuQvGx8OHD2HRfx002u2reKKl+V6Z9gGF1EMEnhZeBTEfmnvX4dET5Cqi83i43+AQxP7xnuUMIuMz+TaEc043uND3cohmG0gWAqmv8kImuAc+xNv1fVj0IbVgfm96N5q1jjP8VUMmP1Tzi518nERcWFOxTDMNpAUKOkquqHqnqHqt4BlIvI30IcV8dVtB1XTQmrdQij+0Z2JfOesj1sK95m6hMMowsJalJhETkRq8fx5cAOOtNwFG0t36pk3h41nPTk2DAHE16Z+ZkApn+CYXQhTSYFERmGlQiuxOrB/Cogqjq5nWLrmPKyqSSGmL4jI76SeWn+UnrF9WJI9yHhDsUwjDbS3J3CJuAr4EequhVARG5vl6g6MH9uNt/6BzE6PbKn3/T4PSzLX8aUjCkRnxwNoytprk7hEmAP8LmIPC0iPwAi+3+/twb2ruUb/+CIHxn124JvKfOUmfoEw+himkwKqvq2qs4ERgCfA7cBPUXkSRE5t70C7FD2r8fhqzY9mbF6MTvFyYQ+E8IdimEYbeiorY9UtVxV/62qF2KNX/QN1qB4kcceGXWzaxiDUuLDHEx4Lc1fyti0sSRFR/Ydk2F0NUE1Sa2lqgftEUt/EKqAOrS8VRxydCe5z2AcjsgtSSusLGR94XrT6sgwuqAWJYVIp3mr+MY3mNERPv3msj3W2IVmljXD6HpMUghWdSkUbGKVd3DE1ydk5mWS7E5mZMrIcIdiGEYbM0khWPnW9JuRPoeCX/0szV/KaX1PwyHmz8cwuppg5lO4RES2iEixiJSISKmIlLRHcB2KXcn8nWMoQ9MSwhxM+Gwq2kRRVZEpOjKMLiqYYS4eBi5U1Y2hDqZDy8tmr7MPvdP64XJG7i/kzDxraIvT+p4W5kgMwwiFYK5u+yI+IWBVMq/yDWZMhHdaW5K3hJE9RpIamxruUAzDCIFg7hSyRORV4G2gunajqkbOoHile5GSXLI8kyO6krm0ppS1BWu5dsy14Q7FMIwQCSYpJAEVQGAvZiWSRkq1p99c7R/CxRE8XPaKPSvwqtf0TzCMLiyYSXaua49AOrT8Vfhx8p0MYljvyK1kXpK/hPioeMb1HBfuUAzDCJFgWh+li8hbIrLfXt4UkfRgTi4iU0XkOxHZKiJzmzjmchHZICLrReTfLf0A7SIvm91RgxjYKwW3yxnuaMJCVcnMy2RC7wlEOaLCHY5hGCESTEXzP4GFQF97edfe1iwRcQJ/A84DRgFXisioBsccB9wJnKGqo7EG3etYVNG8bLK9gyK6f8KOkh3sKd9jRkU1jC4umKSQpqr/VFWvvTwPpAXxulOBraq6XVVrgFeA6Q2OuQn4m6oeBFDV/S2IvX0UbUeqivm6JiOiK5lrm6KapGAYXVswSaFQRK4SEae9XAUUBvG6fkBOwHquvS3QMGCYiGSKyHIRmdrYiUTkZhHJEpGsgoKCIN66Ddmd1tb4I3tO5sy8TDKSMuiX0PCf0DCMriSYpHA91tzMe7Em3bkUaKvKZxdwHHAW1rSfT4vIEaPN2SOzjlfV8WlpwdyktKG8bGocsWwlnZF9Etv3vTuIKm8VWfuyTC9mw4gAwbQ+2gVMa8W584D+Aevp9rZAucDXquoBdojIZqwksbIV7xcaednsiD6OQQlJxEUH04K368nel021r9oUHRlGBGjyKiciv1bVh0XkCax+CfWo6q1HOfdK4DgRGYSVDGYCP25wzNtYdwj/FJFUrOKk7S2IP7S8NbBnLSt1KmMyIreSeUneEtxON+N7jQ93KIZhhFhzP31rh7bIas2JVdUrIrcAHwFO4DlVXS8i9wNZqrrQ3neuiGwAfMD/qmow9RXtY/8G8FWztGYgJ0VyJXN+Jif3OpkYV0y4QzEMI8SaTAqq+q79tEJVXw/cJyKXBXNyVX0feL/BtnsCnivwS3vpeAIqmX8aoZXM+WX57CjewaXHXRruUAzDaAfBVDTfGeS2ridvFRVRyeSRyqgIHQgvM99qimoqmQ0jMjRXp3AecD7QT0QeD9iVBHhDHViHkJfNtqjhDEyJp1tsZPbizczLpE98HwZ1GxTuUAzDaAfN3SnkY9UnVAHZActCYEroQwsze/rNFZ5BjInQoiOP38PyPcs5ve/piEi4wzEMox00V6ewBlgjIv+2m4xGlvzVgPJl+QAmROjwFmv2r6HcU26KjgwjggRTp5AhIm/Yg9Ztr11CHlm41VUyD47YO4Wl+UtxipMJfSaEOxTDMNpJsAPiPYlVjzAZeBF4KZRBdQj5qyiO7c8hEhkdoZXMS/KWMC5tHInRkdmT2zAiUTBJIVZVPwVEVXep6jzggtCG1QHkrWJr1DD6doshJcEd7mja3YHKA2ws2mh6MRtGhAlm3IZqEXEAW+zOaHlA155ppnQfFOew3H0uoyO009qy/GWAGRXVMCJNMHcKc4A44FbgZOCnwDWhDCrs8q3pNz8vTY/Y+oTM/Ex6xPRgZI+R4Q7FMIx2FMyAeLWD05XRdqOjdmx52ag4WefPYHYEtjzyq5+leUs5o98ZOCSY3w2GYXQVzXVee5dGBsKrpaqtGTm1c8jLpihhKFWV7oicWGdj4UYOVh/k9L6nhzsUwzDaWXN3Co/aj5cAvTnc4uhKYF8ogworVcjLZnPs90hNcNMzMfIqmZfkLQEwScEwIlBznde+ABCRP6tq4JjJ74pIq0ZO7RSKtkNVMcslgzH9kiKyJ+/S/KWMShlFSmxKuEMxDKOdBVNgHC8ig2tX7PkR4kMXUpjlWZXMH5dEZiVzSU0JawrWcEZf0+rIMCJRME1SbwcW272YBRgIzAppVOGUl43PFcd3Vf2YE4GVzF/v+Rqf+kxTVMOIUMG0PvpQRI4DRtibNqlqdWjDCqO8bAoTR+AvczA6Au8UMvMySYhKYGza2HCHYhhGGDTX+uhsVf1MRC5psGuIiKCq/wlxbO3P54E9a9iUfDFJMS7Sk2PDHVG7UlUy8zOZ2GciUY7IHCrcMCJdc3cK3wc+Ay5sZJ8CXS8p7FsPvmqWVWcwpl+3iKtk3l68nb3le5k1tuuWDhqG0bzmWh/daz9GRoc1qBsZ9cOivvxwZOQVHdU2RTWVzIYRuZorPmp23mRV/cvRTi4iU4H/A5zAM6r6UIP91wKPYI2nBPBXVX3maOcNmbxVeGNS2FGVEpEjo2bmZTK422D6JPQJdyiGYYRJc8VHxzResog4gb8BPwRygZUislBVNzQ49FVVveVY3qvN5GWzP2k0HJKI68lc6a0ke182V4y4ItyhGIYRRs0VH913jOc+FdiqqtsBROQVYDrQMCl0DPb0m5t6n058tJNBKV23K0ZjsvZmUeOv4cy+ZpY1w4hkR22SKiIxwA3AaCCmdruqXn+Ul/YDcgLWc4HGpvCaISLfAzYDt6tqTsMDRORm4GaAAQMGHC3k1tmzBlAyqzIY1TcJhyOyKpkz8zNxO92c1OukcIdiGEYYBdOj+V9YYx9NAb4A0oHSNnr/d4EMVR0LfAK80NhBqjpfVcer6vi0tLQ2eusG7ErmD4p6R2z/hPG9xxPjijn6wYZhdFnBJIWhqno3UK6qL2DNuhbMpL15QP+A9XQOVygDoKqFAR3hnsGaryE88rLxJA0kvyY+4uoTcktz2Vmy0xQdGYYRVFLw2I+HRGQM0A3oGcTrVgLHicggEYkGZgILAw8QkcBmLtOAjUGcNzTyVrEvcTQAYyJseIul+UsBM8uaYRjBjX00X0SSgd9hXdQTgLuP9iJV9drTd36E1ST1OVVdLyL3A1mquhC4VUSmAV6gCLi2dR/jGNnTb25Muhi3y8HQtK4922hDS/KW0De+LxlJGeEOxTCMMGuun0JvVd0b0G/gS2BwU8c3RlXfB95vsO2egOd3Ane25JwhYU+/uaQigxF9knA5I2e2MY/Pw9d7vuaCwRdEXA9uwzCO1NzVb7WILBKRG0Ske7tFFA55q1Bx8n5hGmMirNPa6oLVVHgrTNGRYRhA80mhH1Zv4zOB70TkHRGZKSJdb5S4vGw8KSMpqHJGXCVzZl4mLnExoXcwbQcMw+jqmkwKqupT1Y/ssY/6A89hdT7bISIvt1eAIWdPv7k3YSRAxE2sk5mfybie40iIjqx6FMMwGhdU4bmq1mD1RN4IlAAjQxlUuyraDlWH2OA4DpdDGNY7ci6OByoPsKloE2f2M01RDcOwNJsURKS/iPyviKwC/msfP01Vu063V3v6za8qBjKsVyJulzPMAbWfuqaoZlRUwzBszbU+WopVr/AacJOqZrdbVO0pLxuNiuOT/d05a1RkVTIvyVtCSkwKw3sMD3cohmF0EM31U5gLfKWq2l7BhEVeNjU9x7J/my+iKpl9fh/L8pcxqd8kHBI5TXANw2hecxXNX3b5hODzwN617IkfBRBRcyhsKNzAoepDpimqYRj1RPZPxP0bwFvFehmKCIzsEzlJITM/E0E4re9p4Q7FMIwOJJhhLroue2TUL8sHMCQtgbjojv11qCpVvirKPeVUeCqsR29Fq9aLqooYlTKKHjE9wv2xDMPoQIK+CorIRGAe1pwKj6nq26EKqt3kZUNcCl/si2XikPa5Syj3lJO9L5tyT3ndUuGtCOqiXuGtwK/+oN7H7XQT54ojLiqO+Kh44lxxdHN3o09Cn7r1KRlTQvxpDcPobI469lHApl8CFwMCfA10gaSwiupeJ7J3Y3W7VDJ7fB6u+/A6NhYdORhsrCuWWFcs8VHxdRft5Jhk0qPSiXPZF/aAC3zteu3z2iXWFUtcVBxRjqiQfx7DMLqe5u4UnrL7JzysqlXAIeBSwI/Vga1zqy6F/RvJH3MOQLtMrPP3NX9nY9FG5p02jxN7nXj44u6Kw+mInP4RhmF0XM3N0XyRiFwI/FdEXgRuA34MxAEXtVN8oWNPv7mOoQCMCnHLo1X7VvHcuue4eOjFzBg2I6TvZRiG0VrNtj5S1XexpuHsBrwFbFbVx1W1oD2CCym7kvmL8v4MTImjW2zoilvKasq4a8ld9I3vy29O/U3I3scwDONYNZkURGSaiHwOfAisA64ApovIKyIypL0CDJm8bEjO4Ot9oR8E708r/8Se8j08OOlB4qPiQ/pehmEYx6K5O4UHgPOAy4E/qeohVf0V1qxrf2iP4EIq7xtqep1ITlElo0M4/eanuz7l7a1vc8OYGzix54khex/DMIy20FxFczFwCVYdwv7ajaq6BWu+5c6rbD8U7yZv6E+B0N0pHKg8wH3L7mNkj5HMHjc7JO9hGAAej4fc3FyqqqrCHYoRZjExMaSnpxMV1boi8eaSwsXAlYAHq4K567BHRv1WrVKwUAxvoarcnXk3Fd4KHpr0EFFO00TUCJ3c3FwSExPJyMgw06pGMFWlsLCQ3NxcBg0a1KpzNDf20QFVfUJVn1LVVjVBFZGpIvKdiGwVkbnNHDdDRFRExrfmfVosLxvEyVdlfejbLYaUBHebv8Vr373Gkrwl3H7y7Qzu3qKprQ2jxaqqqkhJSTEJIcKJCCkpKcd0xxiysY9ExAn8DateYhRwpYiMauS4RGAOVoe49pGXDT1HsWpPDaND0GltZ/FOHs16lNP7ns6VI65s8/MbRmNMQjDg2P8OQjkg3qnAVlXdbs/c9grWdJ4N/R74E9A+haH29JuePiey/UB5m9cnePwe7vzqTtwuN78/4/dmWGrDMDqVUF6x+gE5Aeu59rY6InIS0F9V32vuRCJys4hkiUhWQcExdpGwp9/MixuJKoxp45ZH89fOZ13hOu6ZeA8943q26bkNoyNzOp2ccMIJjBkzhssuu4yKiooWvf6xxx5r8jVnnXUWw4cPZ+zYsYwYMYJbbrmFQ4cONXmu1atXIyJ8+OGHLYoBYN68eTz66KMtfl1XEbafsSLiAP4C/Opox6rqfFUdr6rj09LSju2N878BYI3fqmRuyzGP1hSs4em1T3Ph4As5N+PcNjuvYXQGsbGxrF69mnXr1hEdHc1TTz0V9Gt9Pl+zSQHg5ZdfZu3ataxduxa328306Y0VPFgWLFjAmWeeyYIFCxrdr6r4/cENLhlpQjlWdB7QP2A93d5WKxEYAyy2y8B6AwtFZJqqZoUuqmyIiiOzJJXUhIP0TGybSuYKTwV3fXUXveJ6ceeEO9vknIbRGve9u54N+W07PNmovknce+HooI+fNGkSa9euBeCiiy4iJyeHqqoq5syZw8033wxAQkICs2bNYtGiRcyYMYP8/HwmT55Mamoqn3/+eZPnjo6O5uGHH2bo0KGsWbOGcePG1duvqrz++ut88sknTJo0iaqqKmJiYti5cydTpkxhwoQJZGdn8/7777Nx40buuusufD4fqampfPrppwBs2LCBs846i927d3Pbbbdx6623tvQr67RCmRRWAseJyCCsZDCTgKatqloMpNaui8hi4I6QJgSwkkKfcazNL2dMv6Q2q5x7JOsRckpzeHbKsyRGJ7bJOQ2jM/J6vXzwwQdMnToVgOeee44ePXpQWVnJKaecwowZM0hJSaG8vJwJEybw5z//ue64zz//nNTU1OZOD1hFVePGjWPTpk1HJIWlS5cyaNAghgwZwllnncV7773HjBnWeGNbtmzhhRdeYOLEiRQUFHDTTTfx5ZdfMmjQIIqKiurOsWnTJj7//HNKS0sZPnw4s2fPbnW7/84mZElBVb0icgvwEeAEnlPV9SJyP5ClqgtD9d5N8nlgzxq8J1/Plq1lnDOyV5ucdnHOYt7Y/AbXjb6OU3qf0ibnNIzWaskv+rZUWVnJCSecAFh3CjfccAMAjz/+OG+99RYAOTk5bNmyhZSUFJxOZ93FujWami14wYIFzJxp9a+dOXMmL774Yt37DBw4kIkTJwKwfPlyvve979W15+/R4/CEUxdccAFutxu3203Pnj3Zt28f6enprY61MwnpVGOq+j7wfoNt9zRx7FmhjAWom34zL24UPr+2Sae1wspC7l16L8OTh3PLibe0QZCG0TnV1ikEWrx4MYsWLWLZsmXExcVx1lln1bWhj4mJwels3ZDxPp+Pb7/9lpEjRx6x/c033+Sdd97hD3/4Q11nrtLSUgDi44Mbe8ztPlys7HQ68Xq9rYqzM4qs9pL2yKjf+K3OZMdayayqzFs2j7KaMv446Y9EO6OPOUTD6EqKi4tJTk4mLi6OTZs2sXz58iaPTUxMrLt4N8fj8XDnnXfSv39/xo4dW2/fp59+ytixY8nJyWHnzp3s2rWLGTNm1N2pBJo4cSJffvklO3bsAKhXfBTJIi8pxKXwdVEiSTEu0pNjj+l0/9nyHxbnLGbOSXM4Lvm4NgrSMLqOqVOn4vV6GTlyJHPnzq0rumnMzTffzNSpU5k8eXKj+3/yk58wduxYxowZQ3l5Oe+8884RxyxYsICLL7643rYZM2Y02gopLS2N+fPnc8kllzBu3DiuuOKKFn66rkmaKpfrqMaPH69ZWa2si/77adAtnWkH55DgdvHvm5r+Az2a3SW7ufTdSxmbOpb55843ndSMsNq4ceMRRSlG5Grs70FEslX1qEMJRc6VrLoMCjbh63MSm/aUHlPRkdfv5a4ld+FyuHjgzAdMQjAMo8sIaUVzh7JnDaifvLhR1Pj8x1TJ/Oy3z7KmYA1/mvQnesf3bsMgDcMwwityfuLalcyr/Fbzs9beKaw/sJ6n1jzFeRnncf7g89ssPMMwjI4gcu4URlwA8Wl8s8tBfLSTQSktnxaz0lvJ3K/mkhKbwm8n/jYEQRqGYYRX5CSFlCGQMoR1y5Yyqm8SDkfLezL/Jesv7CzZydPnPk03d2jndTYMwwiHyCk+Anx+ZUN+CaNbMVz2krwlvPLdK/x01E+Z2Kf1rZYMwzA6sohKCjsOlFPp8bW4PuFg1UHuzrybod2HMuekOSGKzjA6NxHhqquuqlv3er2kpaXxox/9KCTvl5CQEJLzHouuMGR3RCWF9fnFQMvmUFBV7l92P4eqD/HQpIdwO9t+6k7D6Ari4+NZt24dlZWVAHzyySf069fvKK+qryMOJ9GSmLrCkN2RU6cArMsrxu1yMDQt+F8YC7ctZNHuRdx+8u0M7zE8hNEZRhv5YC7s/bZtz9n7eDjvoaMedv755/Pee+9x6aWXsmDBAq688kq++uorwBpG4vrrr2f79u3ExcUxf/58xo4dy7x589i2bRvbt29nwIABvPTSS8ydO5fFixdTXV3Nz3/+c2bNmhVUmO+++y4PPPAANTU1pKSk8PLLL5OWlsbw4cNZunQpaWlp+P1+hg0bxrJlywD42c9+xu7duwFrop8zzjjjiJh+97vfcd1111FTU4Pf7+fNN9/kuOPqj2LQVYbsjqg7hXV5JYzok4TLGdzHzi3N5Y8r/shJPU/imlHXhDg6w+j8Zs6cySuvvEJVVRVr165lwoQJdfvuvfdeTjzxRNauXcuDDz7I1VdfXbdvw4YNLFq0iAULFvDss8/SrVs3Vq5cycqVK3n66afrxic6mjPPPJPly5fzzTffMHPmTB5++GEcDgdXXXUVL7/8MgCLFi1i3LhxpKWlMWfOHG6//XZWrlzJm2++yY033thoTE899RRz5sxh9erVZGVlNTpiamNDdtfasmUL//M//8P69euJi4vjpptu4s0332TNmjW8/vrrdcdt2rSJjz76iBUrVnDffffh8XiC//LbSMTcKagq6/KLmTaub1DH+/w+frvEanb64KQHcTpaN5qjYbS7IH7Rh8rYsWPZuXMnCxYs4Pzz6/fjWbJkCW+++SYAZ599NoWFhZSUWJMBTZs2jdhYayyyjz/+mLVr1/LGG28A1qB6W7ZsqRviujm5ublcccUV7Nmzh5qamrrXXH/99UyfPp3bbruN5557juuuu5fPYU8AABFPSURBVA6wEsSGDRvqXl9SUkJZWdkRMZ122mn84Q9/IDc3l0suueSIuwToOkN2R0xSyCmqpLTKG3Ql8/Prn2fV/lX84cw/0C+hZeWihhHJpk2bxh133MHixYspLCwM6jWBQ1qrKk888QRTpkxp8Xv/4he/4Je//CXTpk1j8eLFzJs3D4D+/fvTq1cvPvvsM1asWFF31+D3+1m+fDkxMTHNxvTjH/+YCRMm8N5773H++efzj3/8g7PPPrtuf1casjtiio/W1VYyB9EcdWPhRv66+q/8cOAPuXDwhaEOzTC6lOuvv557772X448/vt72SZMm1V2MFy9eTGpqKklJRzb6mDJlCk8++WRd0cnmzZspLy8P6r2Li4vrKrdfeOGFevtuvPFGrrrqKi677LK6eRzOPfdcnnjiibpjGs4HUWv79u0MHjyYW2+9lenTp9dNNVqrKw3ZHTFJIfdgBdFOB8N6N1/JXOWt4s6v7iTZncw9E+9ps+k6DSNSpKenN1pBOm/ePLKzsxk7dixz58494qJd68Ybb2TUqFGcdNJJjBkzhlmzZjX6i7miooL09PS65S//v727j66qOvM4/v0BwYD4ihlGBYtvjIUI0SAiLxpc4sKxgyBKYTlj1VmLWqEFBhRmZFXACRIXVXRQM1qUacFkxAGhsYpYHaiCvIggUEWEhhFRXkSp8Q2Jz/xx9r29hLyBublw7/NZ666cs88+5+xNDnfn7H3Osx94gIkTJ3LjjTeSn59/yLSe/fv3p6KiIt51BNGscKtXr6Zz58507NiR4uLiasv0zDPPkJubS15eHhs2bDhoPATSK2R3RoXO/mp/JS2a1z42ULSyiNnvzKb4qmJ6ntnziM7jXGPz0Nl1W716NaNHj44/DZXOvk/o7IwZUwDqbBCW71jO7HdmM/SCod4gOJdGpk6dymOPPRbvvnI1S2r3kaR+kjZJel/S+Gq23y5pvaS1kl6T1DGZ5anNvm/2MeH1CZx90tmMzh+dqmI455Jg/PjxbNu2jV69eqW6KEe9pDUKkpoCjwDXAB2BodV86T9tZheaWR5wP/BAsspTl8I3Ctn71V7u630fLZp9v2k6nXPuWJXMO4VuwPtmttXM9gOlwHWJGczsLwmrxwMpGeB4fuvzvFD+Aj/L+xmdWndKRRGcc+6okMwxhTOBDxLWtwOXVs0kaTjwL0Bz4Mqq20OeYcAwgLPOOqtBC/lRxUcUvlFIXk4et+Xe1qDHds65Y03KH0k1s0fM7FxgHDChhjyPm1lXM+uak5PTYOf+zr5jwusTqLRKpvSaQrMmGTXu7pxzh0hmo/Ah0C5hvW1Iq0kpMCCJ5TnEb//0W1Z+vJJx3cbR7sR2de/gnKtRY4fOlsSYMWPi69OmTYu/wXw0mz59OtnZ2ezbt++w9y0oKOBIH8mvr2Q2CquA8yWdLak5MARYmJhBUmIAkWuBzUksz0He+/Q9HlrzEH3a9WHgeQPr3sE5V6vGDp193HHHMW/ePPbs2XNY56hLssNbl5SUcMkllzBv3rxqt6c6fHjS+kvM7ICkEcAioCnwpJltlDQZWG1mC4ERkq4CvgU+BRolFOn+yv2M/+N4Tmh+Avdcdo+/tezSStHKIt7d+26DHvOCUy9gXLdxdeZrzNDZzZo1Y9iwYTz44IMUFhYetG337t01hsRu1aoVY8eOBSA3N5eysjKAQ8JbL1u2jClTpmBmXHvttRQVFQHR5D4jR46krKyMFi1asGDBAtq0acPcuXOZNGkSTZs25aSTTmLp0qWHlHnLli1UVFTw6KOPUlhYGH+7etasWcybN4+KigoqKytZsmQJRUVFzJ49myZNmnDNNdcwdWoU6HDu3LnccccdfPbZZ8ycOZPevXvX63dYX0kdUzCz35tZBzM718wKQ9ovQ4OAmY00s05mlmdmfcxsYzLLEzPjrRls/nQz9/a8l9YtWjfGKZ3LCI0dOnv48OHMmTPnkK6Y2kJi1yQxvHVWVhbjxo3jlVdeYe3ataxatYrnnnsOgC+++ILu3buzbt06Lr/8cp544gkAJk+ezKJFi1i3bh0LFy6s9hylpaUMGTKE3r17s2nTJnbu3BnftmbNGp599lmWLFnCCy+8wIIFC1ixYgXr1q3jrrvuiuc7cOAAK1euZPr06UyaNKnOeh2ujBtZXfXxKmZtnMWNHW7k8raXp7o4zjW4+vxFnyyNHTr7xBNP5Oabb+bhhx+O7w+1h8SuSWJ461WrVlFQUEDswZabbrqJpUuXMmDAAJo3bx4fJ8nPz2fx4sUA9OzZk1tuuYXBgwdz/fXXV3uOkpIS5s+fT5MmTRg0aBBz585lxIgRAPTt2zceRvvll1/m1ltvpWXLlsDB4bVjx87Pz6e8vLzWOh2JjGoUPt//OXe/djftTmjH2K5jU10c59JSY4fOHjVqFBdffPFBge5qCondrFmzg8YLvv7662rLUJusrKx4l3NieOvi4mJWrFjB888/T35+Pm+++SatW/+1J2L9+vVs3ryZvn37AsTne4g1CocbXjtZobVT/khqY5qyYgq7vtzFfb3vo2VWy1QXx7m01Nihs0899VQGDx7MzJkz42k1hcRu3749a9asAaLumpq6pbp168aSJUvYs2cPlZWVlJSUcMUVV9Ra7y1btnDppZcyefJkcnJy+OCDDw7aXlJSwsSJEykvL6e8vJwdO3awY8cOtm3bdsix+vbty1NPPcWXX34JNG547YxpFF4sf5GyrWUM6zyMzjmdU10c59JWY4XOTjRmzJiDnkKqKST2oEGD2Lt3L506dWLGjBl06NCh2uOdfvrpTJ06lT59+tClSxfy8/O57rrrqs0bc+edd3LhhReSm5tLjx496NKly0HbS0tLDwmvPXDgQEpLSw85Vr9+/ejfvz9du3YlLy+PadOm1XruhpQxobOX7VhG6bul/KrgV2Q1yUpCyZxLHQ+d7RJ56Ox66HFGD3qc0SPVxXDOuaNaxnQfOeecq5s3Cs6liWOtK9glx/e9DrxRcC4NZGdn88knn3jDkOHMjE8++eSQR3EPR8aMKTiXztq2bcv27dvZvXt3qoviUiw7O5u2bdse8f7eKDiXBrKysqp949e5w+XdR8455+K8UXDOORfnjYJzzrm4Y+6NZkm7gUODhdTPaUDDzshx9PM6Zwavc2b4PnX+gZnVOZ/xMdcofB+SVtfnNe904nXODF7nzNAYdfbuI+ecc3HeKDjnnIvLtEbh8VQXIAW8zpnB65wZkl7njBpTcM45V7tMu1NwzjlXC28UnHPOxWVMoyCpn6RNkt6XND7V5UkGSU9K2iVpQ0LaqZIWS9ocfp6SyjI2JEntJL0q6U+SNkoaGdLTuc7ZklZKWhfqPCmkny1pRbi+/1tS81SXtaFJairpLUllYT2t6yypXNJ6SWslrQ5pSb+2M6JRkNQUeAS4BugIDJXUMbWlSopZQL8qaeOBP5jZ+cAfwnq6OACMMbOOQHdgePi9pnOdvwGuNLMuQB7QT1J3oAh40MzOAz4F/jmFZUyWkcA7CeuZUOc+ZpaX8G5C0q/tjGgUgG7A+2a21cz2A6VA7bNwH4PMbCmwt0rydUBshvT/AgY0aqGSyMw+MrM1Yflzoi+MM0nvOpuZVYTVrPAx4Erg2ZCeVnUGkNQWuBb4dVgXaV7nGiT92s6URuFM4IOE9e0hLRO0MbOPwvLHQJtUFiZZJLUHLgJWkOZ1Dt0oa4FdwGJgC/CZmR0IWdLx+p4O3AV8F9Zbk/51NuAlSW9KGhbSkn5t+3wKGcTMTFLaPYMsqRXwP8AoM/tL9EdkJB3rbGaVQJ6kk4H5wAUpLlJSSfoRsMvM3pRUkOryNKJeZvahpL8BFkt6N3Fjsq7tTLlT+BBol7DeNqRlgp2STgcIP3eluDwNSlIWUYMwx8zmheS0rnOMmX0GvApcBpwsKfZHXrpd3z2B/pLKibp+rwQeIr3rjJl9GH7uImr8u9EI13amNAqrgPPD0wrNgSHAwhSXqbEsBH4Sln8CLEhhWRpU6FeeCbxjZg8kbErnOueEOwQktQD6Eo2lvArcELKlVZ3N7F/NrK2ZtSf6v/uKmd1EGtdZ0vGSTogtA1cDG2iEaztj3miW9PdE/ZJNgSfNrDDFRWpwkkqAAqLwujuBe4DngGeAs4hCjg82s6qD0cckSb2APwLr+Wtf878RjSuka507Ew0wNiX6o+4ZM5ss6Ryiv6JPBd4C/tHMvkldSZMjdB+NNbMfpXOdQ93mh9VmwNNmViipNUm+tjOmUXDOOVe3TOk+cs45Vw/eKDjnnIvzRsE551ycNwrOOefivFFwzjkX542CSypJlSHK48YQ2XOMpCZhW1dJD4fl4yS9HPL+WFLvsM/a8Dx+sspXIKlHHXmek/TGER6/oob02yXdHJZvkXTGER6/IBY1tJptoyS1PJLjuszlYS5csn1lZnkA4XX9p4ETgXvMbDWwOuS7CCAhbzFwn5nNrs9JwotsMrPv6sx8sAKgAlhWw3FPBvKBCknnmNnWavI0S4jBUy9mVpywegvRi0k7DucY9TAKmA18Wd8djqQuLr34nYJrNOF1/WHACEUKJJWFxmI2cEm4M/gpMBi4V9IcAEl3Slol6e2EOQTaK5oj4zdEX6rtJF0tabmkNZLmhrhIsdj0k0L6ekkXhCB6twOjw3l7V1Ps64HfEb0kNSSWKGmWpGJJK4D7JbWS9FQ49tuSBiXkLQx3SW9IahPSJkoaK+kGoCswJ3ZXJClf0hJFgdAWJYQ1OC/cTa0L9Tg3nKKVpGclvStpTvi3/QVwBvCqpFfD/hUJZbpB0qwa6nK8ork5Viqav+C6kK9TSFsb6nj+kVwH7ihnZv7xT9I+QEU1aZ8RRXcsAMpCWnw5rM8CbgjLVxNNWC6iP2TKgMuB9kRvMncP+U4DlgLHh/VxwC/Dcjnw87B8B/DrsDyR6A3Zmsq/GOgNdADWVylfGdA0rBcB0xO2nxJ+GvAPYfl+YELV8wL/C3QNy1lEdy05Yf3HRG/gQ/Sm9sCwnA20DP9u+4hi/zQBlhMFUovV+bTqfhdE4SFm1VCXKURvBwOcDLwHHA/8B3BTSG8OtEj19eWfhv9495E7FlwdPm+F9VbA+cD/AdvMLNbf351oEqXXo94kmhN9ScbEAua9SXQHUKvwV/35wGtmZpK+lZRrZrGZ7eZaFLEU4CoS7iTM7NOwuJ/oCzd23r51nPbvgFyiqJgQhbP4SFEcnDPNbH44/tehjAArzWx7WF9L1Fi+Vlf9qkisy9VEAejGhvVsorAKy4G7Fc1tMM/MNh/mOdwxwBsF16gUxXSpJIru+MP67kY0vvCfVY7VHviiSr7FZja0huPE4uJUUr9rfzBwCvDn8OV7IjAUuDts/6KG/RJ9a2axWDL1Oa+AjWZ22UGJIThaDRLj/dR2jsSYNtlVtlX9dxxkZpuq5HkndDFdC/xe0k/N7JVayuWOQT6m4BqNpBygGJiR8EVZH4uA2xLGB84M4xBVvQH0lHReyHe8pA51HPtzoKYv3KFAPzNrb1GEznwS7gaqWAwMj63o8ObOTSzDJiBH0mXhOFmSOlk0s9x2SQNC+nH1eLKoat12Svqhoqe/Btay3yLg52HwHkkXhZ/nAFvN7GGi6JydD6OO7hjhjYJLthZhYHIj8DLwEjDpcA5gZi8RPbW0XNJ6oikYD/kiN7PdRE/ylEh6m6i7o64JaH4HDKw60BzuQn5A1NDEjv9nYJ+kS6s5zr8Dp0jaIGkd0KfeFYz69ItD109Tov7+onCctUDskdl/An4R6rYM+Ns6jvs48GJsoJloPt+ysO9HNe4F9xKNbbwdfm/3hvTBwIZQzlzgN/WuoTtmeJRU55xzcX6n4JxzLs4bBeecc3HeKDjnnIvzRsE551ycNwrOOefivFFwzjkX542Cc865uP8Hd/LJ6q2RKyUAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x11586def0"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#part D architecture\n",
"lambda_network = Network([441, 100, 4])\n",
"print('On lambda Network')\n",
"lambda_network.train(X_train, y_train, X_valid, y_valid, eta=1.5,lam=10e-6, num_epochs=50, isPrint=True) \n",
"plt.plot(np.linspace(0,50, 10), lambda_network.valid_accuracy, label='Part D Arch')\n",
"\n",
"# More hidden layers architecture\n",
"layer_network = Network([441, 15, 20, 15, 4])\n",
"print('On layer Network')\n",
"layer_network.train(X_train, y_train, X_valid, y_valid, eta=1.5,lam=10e-6, num_epochs=50, isPrint=False) \n",
"plt.plot(np.linspace(0,50, 10), layer_network.valid_accuracy, label='More Layers Arch')\n",
"\n",
"# More Neurons architecture\n",
"neuron_network = Network([441, 500, 4])\n",
"print('On neuron Network')\n",
"neuron_network.train(X_train, y_train, X_valid, y_valid, eta=1.5,lam=10e-6, num_epochs=50, isPrint=False) \n",
"plt.plot(np.linspace(0,50, 10), neuron_network.valid_accuracy, label='More Neurons Arch')\n",
"print('Done.')\n",
"\n",
"plt.xlabel('Different Architechtures')\n",
"plt.ylabel('% Validation Accuracy')\n",
"plt.title('Diff Archs vs Validation')\n",
"plt.legend() \n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### The original network without changing any of the architectures has done the best as can be seen in the graph above. The heighest validation accuracy was 96%."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [max 10 points] Extra Credit: Improving Network Performance \n",
"***\n",
"\n",
"See if you can get better performance by exploring advanced techniques. Things you might try are: \n",
"\n",
"- Implementing **Mini-Batch** Stochastic Gradient Descent \n",
"- Experimenting with different activation functions (like tanh and ReLU)\n",
"- Experimenting with different loss functions (like cross-entropy or softmax) \n",
"\n",
"For more detailed discussion of these techniques it'll be helpful to look at Chapter 3 of [Nielsen](http://neuralnetworksanddeeplearning.com/chap3.html). \n",
"\n",
"To receive the extra credit you should try at least a couple of the above and clearly describe what worked and what did not. \n",
"\n",
"**Important Note**: Don't do any of these things in the original `Network` class, because you'll almost certainly break the unit tests. Copy the `Network` class from above and rename it `BetterNetwork` (or something) and modify the new class. \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}