Assignment 3-Sentiment Classification

Sentiment Classification
In this assignment, we will design a sentiment classifier for classifying the sentiment of the reviews. This is a Natural Language Processing (NLP) task where the input is a natural language text and output is the sentiment label. We will consider two different review datasets: yelp reviews for restaurants and IMDB reviews for movies.
yelp dataset
The Yelp dataset consists of 7000 reviews in the training set, 1000 reviews in the validation set, and 2000 reviews in the test set. This is a 5 class problem where each review is classified into one of the five ratings with rating-5 being the best score and rating-1 being the worst score.
IMDB dataset
IMDB dataset consists of 15000 reviews in training set, 10000 reviews in validation set, and 25000 reviews in test set. This is a 2 class problem with class 1 being positive sentiment and class 0 being negative sentiment.
1. Most of the algorithms described in the class expects input as a vector. However, the reviews are natural language text of varying number of words. So the first step would be to convert this varying length movie review to a fixed length vector representation. We will consider two different ways of vectorizing the natural language text: binary bag-of-words representation and frequency bag-of-words representation (as explained in the end of the assignment). Convert both the datasets into both these representations and turn in the converted datasets. Instruction for dataset format is given in the end of the assignment (do not include the dataset in the printed report).
2. For this question, we will focus on the yelp dataset with binary bag-of-words representation. We will use the F1-measure as the evaluation metric for the entire assignment. As a baseline, report the performance of the random classifier (a classifier which classifies a review into an uniformly random class) and the majority-class classifier (a classifier which computes the majority class in the training set and classifies all test instances as that majority class). Now train Naive Bayes, Decision Trees, and Linear SVM for this task. You should do a thorough hyper-parameter tuning by using the given validation set. In the report, you should mention the list of hyper-parameters you considered for each classifier, the range of the individual hyper-parameters and the best value for this hyper-parameters chosen based on the validation set performance1. Report training, validation, and test F1-measure for all the classifiers (with best hyper-parameter configuration). Comment about the performance of different classifiers. Note that you should use the appropriate naive Bayes classifier for binary input features (also called Bernoulli naive Bayes)’.
3. Repeat question 2 but with frequency bag-of-words representation. Compare the performance with the binary bag-of-words based classifiers. Do you see a huge improvement? Note that you should use Gaussian Naive Bayes in this scenario since the input is a real vector.
4. Now repeat questions 2 and 3 but with IMDB dataset. Note that the IMDB dataset is a balanced dataset. Hence majority-class classifier doesn’t make sense for this dataset, and you can omit it for this question. Do your observations about the relative performance of the classifiers change as you change the dataset? Comment about it. 1Note that it is a good practise to discuss about your hyper-parameter tuning in any report or paper. Whenever you report the performance of an algorithm with hyper-parameters, you should mention all the hyper-parameters, the range of values you considered for each hyper-parameter and the final values you chose for the hyper-parameters to compute the test performance.
Page 2
Instruction for binary bag-of-words representation
1. First step is to construct the word vocabulary for a dataset. Given a dataset, we will only consider the training set and enumerate all unique words in the training set reviews2. To do this, we should do a bit of pre-processing to the raw text. For this assignment, we will do only 2 pre-processing steps: removal of punctuation, and lower-casing the words. For example, (How old are you?) would become (how old are you).
2. Once you have the vocabulary, count the frequency of each word in the training set and sort the words in the vocabulary based on frequency (in descending order). Now pick top 10,000 words in the vocabulary and ignore the rest of the words. These 10,000 words will form the feature set for our classification process.
3. For any example to be classified, generate a 10,000 dimensional feature vector as follows: for each of the top 10,000 words, there is one corresponding dimension in the feature vector that is 1 if the example contains the word, and 0 otherwise. Make sure to use the same pre-processing steps as before for validation and test reviews.
Instruction for frequency bag-of-words representation
1. Follow steps 1-2 of the instructions for the binary bag-of-words.
2. Now, for each of the 10,000 words, the corresponding feature is the frequency of occurrence of that word in the given review. You can calculate this by summing the occurrences of words in a review into a histogram, and than divide by the sum of occurrences of all 10,000 words so that the vector for each example sums to 1.
Instruction for dataset submission
For IMDB dataset, you should submit IMDB-train.txt, IMDB-valid.txt, IMDB-test.txt, IMDB-vocab.txt.
1. IMDB-vocab.txt file should contain the 10,000 words in the vocabulary, their corresonding id, and their frequency. Each line is a word, its numeric id, and its frequency all tab separated. Example:
the 1 20456
where the is the word, 1 is the id of the word, and 20456 is the frequency of the word.
2. For train/valid/test file, each line is a data point. Each review should be represented as space separates ids for corresponding words in the review3 and the class label in
2Think why should we not include validation and test set in the vocabulary construction process. 3You can igmore words that are not in the vocabulary.
Page 3
mentioned in the end of the review as tab separated. Example:
100 8 3 1034 0
Here 0 is the class label and rest of the numbers represent a 4 word review.
Follow the same instructions for the yelp dataset as well. Add all files into the same zipfile as your code (see below).
Instruction for code submission
1. Submit a single zipped folder with your McGill id as the name of the folder. For example if your McGill ID is 12345678, then the submission should be 12345678.zip.
2. If you are using python, you must submit your solution as a jupyter notebook.
3. Make sure all the data files needed to run your code is within the folder and loaded with relative path. We should be able to run your code without making any modifications.
Instruction for report submission
1. You report should be brief and to the point. When asked for comments, your comment should not be more than 3-4 lines.
2. Either print colored version of your plot or differentiate the plot items by using different symbols. Make sure that you are not submitting a black and white version of a colored plot which we cannot evaluate.
3. Do not include your code in the report!
4. If you report consists of more than one page, make sure the pages are stapled.