A research team led by Professor Jae-Gil Lee from the School of Computing proposed a new method for training deep neural networks (DNNs) that improves accuracy, termed Task-Agnostic Undesirable Feature dEactivation (TAUFE). TAUFE was shown to invariably outperform existing systems, specifically outperforming the softmax-level calibrator by about 8~10% in certain tasks.

TAUFE is designed to combat problems caused by “undesirable features” that are irrelevant to the deep learning task. One example of such features is the background of an animal photograph when the task is categorizing animals. While the background may be correlated to the animal, as animals are more common in certain habitats, it is still irrelevant to categorizing the animal. It can be tricky for deep learning systems to ignore undesirable features when the size of the training data is limited. 

DNNs are particularly susceptible to such undesirable features as they are sensitive to trends undetectable by humans. One known way to combat undesirable features is with out-of-distribution (OOD) data — data that would not be part of the training set — that have the undesirable features of the in-distribution data used in training sets. For the above example, a picture of an animal would be in-distribution, while a picture of a chair would be OOD. Seeing OOD data allows models to recognize what information would be irrelevant to their task, which allows them to ignore these undesirable features. For example, the OOD pictures of chairs would allow DNNs to recognize that the presence of a chair is not relevant to the species of the animal.

Softmax-level calibration, a recently proposed method for combating undesirable features, incorporated OODs, but was limited in application and had lingering impacts on desirable features that are relevant to the task since it targets the classification layer. In contrast, TAUFE builds on the usage of OOD data without targeting the classification layer, instead preventing the activation of the task-specific layer for undesirable features by producing zero values for these features. This allows TAUFE to function for non-classification tasks and prevents desirable features from being affected. 

This study will be announced at the 2021 Conference on Neural Information Processing Systems (NeurIPS) held in December under the title “Task-Agnostic Undesirable Feature Deactivation Using Out-of-Distribution Data”. TAUFE is a novel method that provides exciting prospects as a possible way to combat lack of training data, its wide applicability allowing it to make ripples in the study of deep learning.