CS 5043: HW4: Convolutional Neural Networks

Assignment notes:

Deadline: Friday, March 15th @11:59pm.
Hand-in procedure: submit to the HW4 drop box on Canvas.
This work is to be done on your own. While general discussion about Python, Keras and Tensorflow is encouraged, sharing solution-specific code is inappropriate. Likewise, downloading solution-specific code is not allowed.
Do not submit zip or MSWord documents.

Data Set

The Core50 data set is a large database of videos of objects as they are being moved/rotated under a variety of different lighting and background conditions. Our task is to classify the object being shown in a single frame of one of these videos.

Data Organization

A subset of the database is available on MLFDS in /home2/fagg/datasets/core50
The database is partitioned into different conditions (s1, s2, ...)
Within the condition, you will find cans (o21 ... o25) and mugs (o41 .. o45), each contained within their own directory
Within each object directory is a sequence of PNG files. The last number of the file name is the image sequence number
Each image is 128 x 128 in size and is color (Red, Green, Blue channels)
I have provided a hw4-skel.ipynb file (/home2/fagg/notebooks) that contains the procedure for loading sets of images and constructing a data set for training.

Prediction Problem

We will focus on the distinction between mugs and cans, for which we only have five distinct example objects (though, for each, we have many different perspectives and conditions). Our goal is to construct a model that will be generally applicable: ideally, it will be able to distinguish any mug from any can. However, given the small number of objects, this is a challenge.

Primary Problem: Train/validate on the same set of objects (all 5), but different conditions. Training should involve most of the conditions, but validation must include at least three conditions.
Bonus Problem: Train on four objects; validate on one object. Use all of the conditions for both.
We will not be doing cross-validation rotations for this assignment. As such, it is probably possible to carefully select objects and conditions for your training/validation data sets that will work well for a given model. You should stay away from this level of tuning.

Architecture

You will create a convolutional neural network to distinguish the mugs and cans.

Here is my current state:

Three 5x5 convolutional filters with 2x2 strides (I am playing variably with max pooling)
Flatten
Two dense hidden layers with elu activation.
Single-output sigmoidal layer.

Since we are at the limits of data, it is important to take steps to address the overfitting problem. Here are the key tools hat you have:

Use as large a training set as possible (gives us variety), but use stochastic mini-batches to reduce the computation for a single training epoch step. Note that using more than every 5th image probably doesn't buy us much (compared to the memory usage).
Regularization
Dropout
Keep the number of parameters small (right now, I am sitting around 200K)

Hints / Notes

Start small: get the pipeline working first on a small, feasible problem (e.g., distinguish two different objects from a couple of conditions; validate on different images of the same objects/conditions).

matplotlib.imshow is useful:

plt.imshow(ins_neg[2,:,:,:].astype(np.float32))

For debugging purposes, I find it useful to examine the state of the early processing layers. I have provided this in my hw4-tools.py file
I do not suggest using dropout on your convolutional layers
I have a general function for creating networks that takes as input a set of parameters that define the configuration of the convolutional layers and dense layers. By changing these parameters, I can even change the number of layers. This makes it much easier to try a variety of things without having to re-implement or copy a lot of code.
Remember to check your model summary to make sure that it matches your expectations.

What to Hand In

Hand in your notebook containing all of your code + the PDF export of this notebook. This notebook must include:

Code for generating and training the network.
ROC curve (with AUC measure) for both the training and validation data sets.

It is fine if you want to hand in separate notebooks for the primary and bonus problems.

Evaluation

25 pts: Selection of proper training/validation data sets (following the constraints above)
50 pts: Model generation code. Is it correct? clean? documented?
25 pts: Model performance. Note that for this to count you must have an AUC of at least 0.6 for both the training and validation data sets
10 pts: You solved the bonus problem

andrewhfagg -- gmail.com

Last modified: Thu Mar 7 13:51:44 2019