Day 9 (7/18/19): Incrementally Learning CUB200

Today I continued my work learning about incremental learning models by testing out different strategies on the CUB200 dataset. From what I understand from reading various articles, there seem to be five different approaches to mitigating catastrophic forgetting in lifelong learning models. These are regularization methods (adding constraints to a network's weights), ensemble methods (train multiple classifiers and combine them), rehearsal methods (mix old data with data from the current session), dual-memory methods (based off the human brain, includes a fast learner and a slow learner), and sparse-coding methods (reducing the interference with previously learned representations). 

All of these methods have their constraints and I don't believe it is yet clear what method (or what combination of different methods) is best. Full rehearsal obviously seems to be the most effective at making the model remember what it had previously learned but given that all training examples need to be stored in memory for this to work makes it not ideal in the real world. Therefore, some form of pseudo-rehearsal is probably necessary and could hopefully also be effective if the examples were stored in some compressed way.

Further reading on the challenges associated with incremental learning has given me a better understanding on the work that still needs to be done in the field. I am looking forward to being able to conduct my own experiments in this field soon.