Learning Invariances in Neural Networks (nips2020)

Introduction：

• The ability to learn constraints or symmetries is a foundational property of intelligent systems
• we provide an extremely simple and practical approach to automatically discovering invariances and equivariances
• Our approach operates by learning a distribution over augmentations, then training with augmented data, leading to the name Augerino.
• Augerino (1) can learn both invariances and equivariances over a wide range of symmetry groups, including translations, rotations, scalings, and shears;
•                (2) can discover partial symmetries, such as rotations not spanning the full range from [-π, π];
•                (3) can be combined with any standard architectures, loss functions, or optimization algorithm with little overhead;
•                (4) performs well on regression, classifification, and segmentation tasks, for both image and molecular data

Method：

• A simple way of constructing a model invariant to a given group of transformations is to average the outputs of an arbitrary model for the inputs transformed with all the transformations in the group
• Augerino discovers invariances by learning θ from training data alone

Self-supervised Pre-training with Hard Examples Improves Visual Representations (arxiv2020)

idea：

1. present a modeling framework that unifies existing SSP methods as learning to predict pseudo-labels. 
2. we propose new data augmentation methods of generating training examples whose pseudo-labels are harder to predict than those generated via random image transformations
3. use adversarial training and CutMix to create hard examples (HEXA) to be used as augmented views for MoCo-v2 and DeepCluster-v2

• hard examples are instrumental in

  improving the generalization of the pre-trained models.
• 如何根据自监督任务去设计更好的data augmentation方法