Artificial Intelligence Assignment 3

ECE 473 : Introduction to Artificial Intelligence
Assignment 3
Instructions on programming assignments
You are asked to modify the code in hw3 submission.py between
############################################################
############################################################
# BEGIN YOUR CODE
# HINTS OR INSTRUCTIONS
pass ;
# END YOUR CODE
############################################################
############################################################
• Please use Python3 to do all assignments in the course.
• You are allowed to make your own helper functions outside the skeleton functions.
• Do not change other than hw3 submission.py. You are only allowed to use numpy package to compute
and construct functions. Do not use scikit-learn, pytorch, or tensorflow etc to implement your
functions.
• You can test your code with test functions in test.py. For your reference, you can compare your own
classifiers with the ones in sklearn package that is implemented in test.py.
• Homework will be graded based on the provided dataset and may be additionally tested on datasets
that are not released to students.
• You need to get no more than 5% difference of accuracy from the reference to get full score.
• Only submit hw3 submission.py
1
Problem 1: Gaussian Naive Bayes Classifier
Calculation of Bayes Theorem can be simplified by making some assumptions, such as independence between
all input variables. Although a dramatic and unrealistic assumption, this has the effect of making the
calculations of the conditional probability tractable and results in an effective classification model referred
to as Naive Bayes.
The Naive Bayes algorithm has proven effective and therefore is popular for text classification tasks. The
words in a document may be encoded as binary (word present), or count (word occurrence) input vectors
and binary, multinomial, or Gaussian probability distributions used.
In this problem, you will implement Gaussian Naive Bayes Classifier and test it on two datasets:
1. breast cancer dataset
2. digits dataset
Digits dataset consist of 1797 samples with 10 classes. Each sample has 64 features, which is originally 8×8
image. Your task for both dataset is to predict correct class for the test samples.
(Note: When you compute Bayes rule, denominator cannot be 0. In other words, a value inside log cannot
be equal or less than 0.)
Tasks in Problem 1:
1. Implement function Naive Bayes.mean
2. Implement function Naive Bayes.std
3. Implement function Naive Bayes.gen by class
4. Implement function Naive Bayes.calc gaussian dist
5. Implement function Naive Bayes.predict
2
Problem 2: Two-Layered Neural Network
Based on the structure of Logistic regression of Homework 2, you will extend the structure to two-layers
neural network in this problem. Note that the counting index of “layers” starts with the first hidden layer
up to the output layer. Thus, as illustrated in Fig 1, two-layered neural network has one hidden layer with
a certain number of hidden units. The number of hidden units can vary considering various aspects such as
how complex the input feature is or how deep the neural network is.
Figure 1: Illustration of two-layered neural network.
In this homework, you will implement a two-layered neural network with a hidden layer with 64 units
and ReLU activation before fed forward to the output layer. Whereas for the output layer, you will use
sigmoid function instead.
Denote x ∈ R
D as an input data with D dimensions and y ∈ {0, 1} as the corresponding true binary
label. We denote f(x; θ) : R
D → R as a two-layered neural network that we want to train. The goal is to
find f that minimizes the loss L in Eq.(1).
L(f(x), y) = LBCE(f(x), y) + λkθk
2
2
. (1)
This loss L consist of two terms: 1) binary cross entropy loss (LBCE) and 2) regularization. The first term
LBCE is identical to the logistic regression in hw2 as in Eq.(2). The second term is the regularization with
the weight λ which is for better generalization and to prevent the model from over-fitting to the training
data. Note that θ includes all trainable parameters including weights and biases in all layers.
To update the parameters with back propagation in the neural network, you will use mini-batch stochastic
gradient descent method as in Homework 2.
LBCE(f(x), y) = E(x,y)
h
y log(f(x)) + (1 − y) log(1 − f(x))i
. (2)
Note: hyperparameters are provided as follows. Do not change the hyperparameter setting.
• learning rate = 10−3
,
• batch size = 64,
• iterations number = 2000,
• regularization hyperparameter λ in Eq. (1) (i.e., reg in hw3 submission.py) = 10−3
.
In this problem, you will test the neural network on breast cancer dataset.
Tasks in Problem 2:
1. Implement function Neural Network.activation
2. Implement function Neural Network.sigmoid
3. Implement function Neural Network.loss
3
4. Implement function Neural Network.fit
5. Implement function Neural Network.predict
4