CS 5043: HW0

Executing Deep Network Experiments on the Supercomputer

Objectives:

Using Tensorflow/Keras, implement a shallow network that is capable of learning to reproduce a function.
Implement experiment control code that executes a single instance of a learning run and reports the results to Weights and Biases.
Use the supercomputer to execute a set of experiments.
Using a separate program, bring the results together from the different experiments so they can be represented in a common set of figures.

Assignment notes:

Deadline: Tuesday, February 2nd @11:59pm. Solutions may be submitted up to 48 hours late, with a 10% penalty for each 24 hours.
This work is to be done on your own. While general discussion about Python, TensorFlow and Keras is okay, sharing solution-specific code is inappropriate. Likewise, you may not download code solutions to this problem from the network or a LLM.

Data Set

The data sets for this class are available on the supercomputer in the directory /home/fagg/datasets/. You do not need to keep copies of these data sets in your own home directory on the supercomputer (in fact, it is a bad idea).

For this homework assignment, we will be using hw0_dataset.pkl. This file contains one python object, a dictionary, that has two keys: ins and outs. Both are numpy arrays. The following code snippet will load the data into your python environment:

with open("hw0_dataset.pkl", "rb") as fp:
     foo = pickle.load(fp)

where foo is the dictionary.

Notes:

This code snippet assumes that this file is in the same directory as your python code (so, adjust the code, as needed).
There is only a training set (and no validation or test sets).
Take some time to visually examine the data.

Part 1: Network Training

Start from the xor_base.py code that we developed in class.
Your network training program must use the ArgumentParser to accept a variety of input arguments. These inputs could include network architecture details and learning rate.
Write a function that constructs a relatively shallow neural network that can regenerate the output from the corresponding inputs. This function should include parameters that specify the details of the network (including numbers of hidden units).
The first thing that you try will likely not work. You will need to think about the appropriate non-linearities to use and to play with the number of layers and neurons.
At the end of each run, compute the absolute prediction error for each example in the training set. Then, log the following to Weights and Biases:
- The max and the sum of these errors (two different log entries)
- A count of the number of examples that exceed an absolute error of 0.1
Make sure to save key data to a results file, including:
- The set of absolute errors
- Perhaps the measures that have also been logged to WandB
Once you have settled on an architecture that works well, then proceed to the next part.

Part 2: Multiple Runs

Use the supercomputer to perform 10 independent learning runs. You must use the supercomputer.
Produce a single figure that shows the learning curves (LOSS as a function of epoch) for all 10 runs on the same figure. (WandB will produce this for you)
Combine the absolute errors from across the runs to generate a single histogram of these errors.
Set the ylim of this histogram to the range 0..50 so we can see the small counts in the figure.
(you will need to write a Python program to generate this)
Plot a bar chart for each of the following (one bar per run):
- Max absolute error
- Sum of absolute errors
- Count of absolute errors larger than 0.1
(WandB produces this for you)
Reporting. You may use one of the following techniques to produce a report that contains these figures:
- Jupyter Notebook (submit as a notebook)
- Your favorite editor (submit as a pdf)
- Weights and Biases Report (only for the brave; submit as pdf)
In addition, describe your final network architecture.

Hints

Once a model is trained, you can ask it to predict output values for a set of examples using the model.predict() function
Numpy provides an absolute value operator: np.abs()
Matplotlib provides an easy way to generate histograms:
plt.hist(errors, 50)
This generates a histogram with 50 bins
You can set the visible limits of a figure using the xlim/ylim functions. For example, the following will set the visible limit of the vertical axis to 0 ... 50:
plt.ylim([0, 50])
A figure can be written to a file by Matplotlib:
plt.savefig('foo.png')
will generate a PNG file of the specified name.
You can log scalar statistics to Weights and Biases using:
```
       wandb.log(obj)
       
```
where obj is a dictionary of statistic names and their values (best of the value is a scalar).

You can also create plots with matplotlib and export them directly to wandb:

       fig = plt.figure()
       plt.hist(errors)
       plt.ylabel('Count')
       plt.xlabel('Absolute error')

       # Log the figure to wandb 
       wandb.log({'histogram figure': fig})

(this is for future use, and probably isn't helpful here)

At the command line, you can provide an entire list of values associated with a single argument. For example:
```
       parser.add_argument('--hidden', nargs='+', type=int, default=[5], help='Number of hidden units per layer (sequence of ints)')
       
```
args.hidden then becomes a list of integers. Think about how you could modify your model creation function to iterate over this list and create one layer per integer.

Scanning for a list of results files:

       # Directory containing results files
       dr = './results/'

       # File specification (normally bring this in as an argument)
       #  \d\d matches any 2-digit sequence
       file_spec = 'hw0_results_exp_\d\d.pkl'

       # Grab the list of matching files
       files = [f for f in os.listdir(dir) if re.match(r'%s'%(file_spec), f)]

       # Iterate over the files.  Open each and extract the errors
       objs = []
       for f in files:
            print(f)
            with open("%s/%s"%(dr,f), "rb") as fp:
                 obj = pickle.load(fp)
                 objs.append(obj['errors'])

Everyone is coming in with different experiences. Feel free to ask questions about Python / Keras / Numpy / Matplotlib. Ask questions early.
You won't finish this if you start the day before the deadline.

Expectations

Terminal MSE for the individual runs should be very low. If this is not the case, then go back to your network design.
It is very hard to learn a network that generates the correct output for every example. Getting a couple examples wrong in individual runs is okay.

What to Hand-In

Submit a zip file to the HW0 section of Gradescope (enter via Canvas) that contains:

your python code
if you use a Jupyter notebook, then submit the notebook itself
if you generated a pdf of your report, include that pdf
your batch file
A stdout and a stderr file from ONE experiments

Grading

30 pts: low MSE for every run
20 pts: few absolute prediction errors greater than 0.1
20 pts: well structured code
10 pts: appropriate documentation
10 pts: figures are well formatted and have appropriate axis labels
10 pts: Example stdout and stderr files
10 pts (bonus): all absolute prediction errors less than 0.1

andrewhfagg -- gmail.com

Last modified: Wed Jan 29 01:23:44 2025