Title: "Deep learning and Bayesian inference for medical imaging" (Grant No. 702666)

The project was carried out from 2016/10/1 to 2018/09/30 within the Marie Skłodowska-Curie Action (Individual Fellowship), financed by the European Comission.

The aim of the project is to apply current state-of-the-art techniques of deep learning to medical imaging. Additionally, we aim at enriching deep learning models with Bayesian inference to better reflect uncertainty in medical data.

Volume-preserving Normalizing Flows

One approach is to apply a generative modelling to medical data. For this purpose we used variational auto-encoders (VAE) that utilize neural networks to model probability distributions between visible and latent variables. Further, in order to increase flexibility of the VAE we proposed a new volume-preserving flow using a series of Householder transformations. In other words, we transform a vector of latent variables to new vector of latent variables using the Householder transformation. As a result, we obtain a series (levels) of latent variables and the final one follows more flexible posterior distribution. The proposed approach was applied to grayscale image patches of histopathological data.
Another idea was to improve on the linear inverse auto-regressive flow (LinIAF). The LinIAF applies a linear transformation to latent variables with a lower-triangular matrix with ones on the diagonal. In order to increase the flexibility of this normalizing flow we proposed to use a convex combination of lower-triangular matrices. As a result, we obtained two levels of latent variables and the final one follows more flexible posterior distribution. The proposed approach was also applied to grayscale image patches of histopathological data.
See publications for details (NIPS Workshop on Bayesian Deep Learning 2016 & Benelearn 2017).

Sylvester Normalizing Flows

Volume-preserving normalizing flows allow to obtain a richer family of variational posteriors, however, these are limited to linear transformations. In order to overcome this problem, we proposed a non-linear transformation by utilizing the Sylvester's determinant identity. As a result, we obtained a class of non-linear inversible transformations. The main idea of the approach is to parameterize weights of the transformations using orthogonal matrices. In order to allow efficient calculations, we proposed to use Householder matrices and a numerical procedure to parameterize orthogonal matrices. The proposed approach was applied to different image benchmark datasets.
See publications for details (UAI 2018).

New prior for VAE

We proposed to extend the variational auto-encoder (VAE) framework with a new type of prior ("Variational Mixture of Posteriors" prior, or VampPrior for short). The VampPrior consists of a mixture distribution (e.g., a mixture of Gaussians) with components given by variational posteriors conditioned on learnable pseudo-inputs. We further extended this prior to a two layer hierarchical model and showed that this architecture, where prior and posterior are coupled, learns significantly better models. The model also avoids the usual local optima issues that plague VAEs related to useless latent dimensions. We provided empirical studies on six datasets, namely, static and dynamic MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces and Histopathology patches, and show that applying the hierarchical VampPrior delivers state-of-the-art results on all datasets in the unsupervised permutation invariant setting and the best or close to SOTA methods in the invariant setting.
Further, we utilized the VampPrior in the fair classification setting. Fairness is very statistical important property in many practical applications including medicine. We proposed a two-level hierarchical VAE with a class label and a sensitive variable. In order to enforce fairness, we used a regularization term based on the mutual information measure.
See publications for details (AISTATS 2018, ICML Workshop 2018).

Medical imaging using Deep Multiple Instance Learning

Training a whole slide imaging tool requires relatively large amount of computational resources. Similarly, providing pixel level annotations interferes pathologists daily work. In order to overcome these issues, we proposed to apply multi-instance learning combined with deep learning to histopathology classification. Our goal is to utilize weakly-labaled data to train deep learning models in an end-to-end fashion. In the research we discussed different permutation-invariant operators and proposed a new one basing on the attention mechanism that is fully trainable.
See publications for details (NIPS Workshop 2017, MIDL 2018, ICML 2018).

• Tim Davidson, Luca Falorsi, Nicola de Cao, Thomas Kipf, Jakub M. Tomczak, Hyperspherical Variational Auto-Encoders, UAI, Monterey, California, the USA, 2018, [arxiv], [PDF], [CODE], [BLOG]
• Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, Max Welling, Sylvester Normalizing Flow for Variational Inference, UAI, Monterey, California, the USA, 2018, [arxiv], [PDF]
• Philip Botros, Jakub M. Tomczak, Hierarchical VampPrior Variational Fair Auto-Encoder, ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, Stockholm, Sweden, 2018, [arxiv], [PDF]
• Maximilian Ilse, Jakub M. Tomczak, Max Welling, Attention-based Deep Multiple Instance Learning, ICML, Stockholm, Sweden, 2018, [arxiv], [PDF], [CODE]
• Nathan Ing, Beatrice S. Knudsen, Arkadiusz Gertych, Jakub M. Tomczak, Max Welling, Isla P. Garraway, Eric Miller, A deep multiple instance model to predict prostate cancer metastasis from nuclear morphology, Medical Imaging with Deep Learning (MIDL), Amsterdam, the Netherlands, 2018, [OpenReview], [PDF]
• Jakub M. Tomczak, Maximilian Ilse, Max Welling, Marnix Jansen, H.G Coleman, Marit Lucas, Kikki de Laat, Martijn de Bruin, Henk Marquering, Myrtle J. van der Wel, Onno J. de Boer, C. Dilara Savci Heijink, Sybren L. Meijer, Histopathological classification of precursor lesions of esophageal adenocarcinoma: A Deep Multiple Instance Learning Approach, Medical Imaging with Deep Learning (MIDL), Amsterdam, the Netherlands, 2018, [OpenReview], [PDF]
• Jakub M. Tomczak, Max Welling, VAE with a VampPrior, AISTATS, the Canary Islands, 2018, [arxiv], [PDF], [CODE]
• Jakub M. Tomczak, Maximilian Ilse, Max Welling, Deep Learning with Permutation-invariant Operator for Multi-instance Histopathology Classification, NIPS Workshop on Medical Imaging Meets NIPS, Long Beach, the USA, 2017 [arxiv], [PDF]
• Leonard Hasenclever, Jakub M. Tomczak, Rianne van den Berg, Max Welling, Variational Inference with Orthogonal Normalizing Flows, NIPS Workshop on Bayesian Deep Learning, Long Beach, the USA, 2017, [PDF]
• Jakub M. Tomczak, Max Welling, Improving Variational Auto-Encoders using convex combination linear Inverse Autoregressive Flow, Benelearn 2017, Eindhoven, the Netherlands, 2017 [arxiv], [PDF], [CODE]
• Jakub M. Tomczak, Max Welling, Improving Variational Auto-Encoders using Householder Flow, NIPS Workshop on Bayesian Deep Learning, Barcelona, Spain, 2016 [arxiv], [PDF], [CODE]

• An interview (together with Max Welling) for the promotional purposes of the University of Amsterdam.
  LINK to the brochure: [here].
• A description of the DeeBMED project in the newsletter of Wroclaw Univ. of Technology and Wroclaw Center of Technology Transfer.
  LINK to the newsletter (page 4): [here].
• A participation in the Open Dag at the Science Park in Amsterdam in October 2017. I had a chance to talk to Amsterdamers (from toddlers to seniors) and present them how AI will influence medicine.
  LINK to the event: [here].
  My short "press report": [here] and [here].
• An interview on the local radio (Radio Wroclaw) and a regional TV in May 2016. I had a chance to discuss the plan of DeeBMED and how AI will influence medical imaging.
• An interview and note in "Pryzmat" in April 2016 written by Iwona Szajner.
  LINK to the note: [here].

• Jakub M. Tomczak, a presentation at PASC 2018 Conference (2nd of July, 2018) and CERN (3rd of July, 2018)
  TITLE:
  The Success of Deep Generative Models
  ABSTRACT:
  Deep generative models allow to learn hidden representation of data and generate new examples. There are two major families of models that are exploited in current applications : Generative Adversarial Networks (GANs), and Variational Auto-Encoders (VAE). The principle of GANs is to train a generator that can generate examples from random noise, in adversary of a discrimanitive model that is forced to confused true samples from generated ones. Generated images by GANs are very sharp and detailed. The biggest disadvantage of GANs is that they are trained through solving a minimax optimization problem that causes significant learning instability issues. VAEs are based on a fully probabilistic perspective of the variational inference. The learning problem aims at maximizing the variational lower bound for a given family of variational posteriors. The model can be trained by backpropagation but it was noticed that the resulting generated images are rather blurry. However, VAEs are probabilistic models, thus, they could be incorporated in almost any probabilistic framework. We will discuss basics of both approaches and present recent extensions. We will point out advantages and disadvantages of GANs and VAE. Some of most promising applications of deep generative models will be shown.
  ANNOUNCMENT at CERN: [link]
  SLIDES: [PASC], [CERN]
  VIDEO: [link]
• Jakub M. Tomczak, a presentation for Tooploox company
  TITLE:
  Attention-based Deep Multiple Instance Learning
  ABSTRACT:
  Multiple instance learning (MIL) is a variation of supervised learning where a single class label is assigned to a bag of instances. In this paper, we state the MIL problem as learning the Bernoulli distribution of the bag label where the bag label probability is fully parameterized by neural networks. Furthermore, we propose a neural network-based permutation-invariant aggregation operator that corresponds to the attention mechanism. Notably, an application of the proposed attention-based operator provides insight into the contribution of each instance to the bag label. We show empirically that our approach achieves comparable performance to the best MIL methods on benchmark MIL datasets and it outperforms other methods on a MNIST-based MIL dataset and two real-life histopathology datasets without sacrificing interpretability.
  SLIDES: [here]

• Jakub M. Tomczak, a presentation in CWI (Dept. of Life Sciences and Health)
  TITLE:
  Deep Generative Modeling using Variational Auto-Encoders
  ABSTRACT:
  Learning generative models that are capable of capturing rich distributions from vast amounts of data like image collections remains one of the major challenges of artificial intelligence. In recent years, different approaches to achieve this goal were proposed by formulating alternative training objectives to the log-likelihood like the adversarial loss or by utilizing variational inference. The latter approach could be made especially efficient through the application of the reparameterization trick resulting in a highly scalable framework now known as the variational auto-encoders (VAE). VAEs are scalable and powerful generative models that can be easily utilized in any probabilistic framework. The tractability and the flexibility of the VAE follow from the choice of the variational posterior (the encoder), the prior over latent variables and the decoder.
  In this presentation I will outline different manners of improving the VAE. Moreover, I will discuss current applications and possible future directions.
  SLIDES: [here]
• Jakub M. Tomczak, an oral presentation at AISTATS (the Canary Islands)
  TITLE:
  VAE with a VampPrior
  ABSTRACT:
  Many different methods to train deep generative models have been introduced in the past. In this paper, we propose to extend the variational auto-encoder (VAE) framework with a new type of prior which we call "Variational Mixture of Posteriors" prior, or VampPrior for short. The VampPrior consists of a mixture distribution (e.g., a mixture of Gaussians) with components given by variational posteriors conditioned on learnable pseudo-inputs. We further extend this prior to a two layer hierarchical model and show that this architecture with a coupled prior and posterior, learns significantly better models. The model also avoids the usual local optima issues related to useless latent dimensions that plague VAEs. We provide empirical studies on six datasets, namely, static and binary MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces and Histopathology patches, and show that applying the hierarchical VampPrior delivers state-of-the-art results on all datasets in the unsupervised permutation invariant setting and the best results or comparable to SOTA methods for the approach with convolutional networks.
  SLIDES: [here]

• Jakub M. Tomczak, a presentation at Summer School on Data Science (Split, Croatia)
  TITLE:
  Deep Generative Models: GANs and VAE
  ABSTRACT:
  During this talk I present why generative modeling is important and what are the main trends in generative modeling. I focus on Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAE). I outline basics of these two approaches and point to recent developments. Moreover, I provice pros and cons of these methods.
  SLIDES: [here]
• Jakub M. Tomczak, a presentation at Falling Walls Lab in Brussels
  TITLE:
  Breaking the Wall of Medical Imaging
  Description:
  I had a great opportunity to present my work at Falling Walls Lab. I was chosen to be one of thirty presenters (MSCA fellows) that had a chance to describe their work within 3 minutes. This event was a part of "Science is wonder-ful" festival organized by the European Comission.
  SLIDES: [here]
  "Press report": [here] and [here]
  
• Jakub M. Tomczak, a presentation at Technische Universiteit Eindhoven
  TITLE:
  Variational Auto-Encoder: Deep Learning meets Generative Modeling
  ABSTRACT:
  Variational auto-encoder (VAE) is a scalable and powerful generative framework. The main advantage of the VAE is that it allows to model stochastic dependencies between random variables using deep neural networks that can be further trained by gradient-based methods (backpropagation). There are three main components within the VAE framework: (i) a decoder, (ii) an encoder, (iii) a prior. During the talk I will introduce basic ideas of the VAE and show how these three components could be formulated. I will especially focus on increasing flexibility of the encoder using the idea of normalizing flows. Further, I will present how to choose the prior for learning better latent representation. Eventually, I will outline possible extensions and future directions.
  SLIDES: [here]
• Jakub M. Tomczak, a presentation at National Cyber Security Summer School
  TITLE:
  Machine Learning: A Cybersecurity Perspective
  ABSTRACT:
  In this introductory presentation I present main threats of Big Data and show how machine learning could be used to prevent them. Further, I give a general introduction to main concepts of machine learning and a cutting-edge tool: deep learning.
  SLIDES: [here]