NeurIPS 2020 Meetup Day 1a

By your attendance, you agree with NeurIPS Code of Conduct: https://neurips.cc/public/CodeOfConduct

1) Alexander Shekhovtsov: Explainable Training of Binary Neural Networks

There is now a crazy fashion for neural networks using low precision computations or even using mostly binary operations, which are much faster and need less energy. The performance of such binary neural networks on benchmarks like the ImageNet classification challenge stately improves while the number of unclear tricks and special ingredients involved in the training procedures grows. Many of these tricks are about the question of how to train with binary activations and or binary weights by somehow (ab)using backpropagation. We tried to derive learning methods that would be correct in some sense so that we would know what we are doing. Towards this end, we apply the stochastic relaxation method: each binary entity has a probability of taking a particular state, then the optimization and gradients can be performed with respect to these continuous probabilities. With some surprise, we have derived the popular straight-through estimator in a particular form and some popular weight update rules. We also have derived a more accurate new gradient estimation method [1]. In the publications, we devote a lot of attention to checking the accuracy of the gradient and comparing different estimators. The conceptual message of this research is that it is possible to train binary networks using explainable methods, although partially coinciding with previous empirical approaches but now free of guessing, with known properties and limitations, swap-in more accurate methods as needed and improve them further.

References:
[1] "Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks"
https://arxiv.org/abs/2006.03143
[2] "Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks"
https://arxiv.org/abs/2006.06880

Alex graduated from Kyiv Polytechnical Institute, Ukraine in applied mathematics. Received PhD from Czech Technical University in Prague in the area of discrete optimization methods applied to computer vision problems. 2013-2017 postdoc at Graz university of technology, continued working on discrete optimization methods. Since 2017 back to Czech Technical University. Presently, assistant professor at the department of cybernetics, FEE, Visual Recognition Group group headed by J. Matas. Fully switched the topic to deep learning. Particularly focusing on statistics in deep learning (uncertainty propagation, normalization, Bayesian methods, etc.) and training binary networks via stochastic relaxation methods.

2) Vít Růžička: Art and Machine Learning (10:45)

This talk presents approaches of using Machine Learning techniques such as Generative models in the domain of Art, for Creative AI applications and as a tool of creative expression. It will also describe how these techniques are used in other research areas, such as applications of Domain Adaptation, Data Augmentation and in visualizing and understanding representations learned by Neural Networks. Finally, we will look at some of the latest directions in Generative models research to have a glimpse into the world of tomorrow.

Vít Růžička is a DPhil student at the Department of Computer Science at the University of Oxford, supervised by Niki Trigoni and Andrew Markham. In 2019-20, he was working as a research assistant and lecturer at the University of the Arts London in the Creative Computing Institute. Before that, he was on a research internship at ETH Zurich in the EcoVision group (2019) focusing on Remote Sensing and at Carnegie Mellon University in Franz Franchetti’s group (2017-18) focusing on the fast performance of ML models. He has done his MSc and BSc at the Czech Technical University in Prague. His interests are Machine Learning research, Arts, literature, travelling and analogue photography. For more details: https://previtus.github.io/

• Vývoj softwaru a aplikací
• Meetup
• Pouze online