Day 16 (7/29/19): Achieving Better Results by Adjusting Learning Rate

This morning I reviewed the results of the experiments I ran over the weekend. However, the tensorboard graphs showed that after a certain number of batches, every model appeared to stop learning on the training data, essentially becoming stagnant at the end. Below is a graphical representation of those models over the weekend.

I realized that the training script I was using didn't explicitly reset the optimizer and learning rate scheduler after each batch as I intended. Therefore, as the learning rate was continuously getting smaller, the model was no longer learning in the further batches. I adjusted my script to reset the parameters of both the optimizer and scheduler after each batch of 20 classes and restarted many of the experiments. I also added some more capabilities to load and save the models by creating dictionaries to track how the accuracy and omega values (against default values of offline model) change as the number of batches and classes increase. I hope to have results from the Offline, Full Rehearsal, and L2SP models with Baseline OOD Recognition tomorrow.