# How to teach our world knowledge (rather priors) to a neural network? Workshop at ODSC Europe, London 2022, https://odsc.com/europe/ Oliver Zeigermann Slides: https://bit.ly/2022-odsc-priors Colab Notebook: https://bit.ly/tf-lattice-beer _We will use Google Colab for the hands-on. Register here if you want to be active later: https://colab.research.google.com/_

### Where did the Smurf go? <video src="img/smurf/smurf-short.mp4" controls>

## This isn't so much about whether this is a great trick, but rather... * **object permanence** https://en.wikipedia.org/wiki/Object_permanence * is a knowledge about the world we are so certain of * we get stressed when it is challenged * and immediately start thinking: where did it go? * basis for a large number of magical tricks

### World knowledge * humans have a basic understanding of the world * be it about the physics of normal objects or about the properties of people * we also expect this from an intelligent automatic system * if a system does not fulfill this we are disappointed and our trust in the system decreases

## Modeling choices encode strong assumptions about the data <img src="img/world-knowledge/fchollet-model-priors.png"> https://twitter.com/fchollet/status/1439799099176357894

## Priors <div class="container"> <div class="col"> <img src="img/world-knowledge/weak-prior.jpeg"> </div> <div class="col"> <img src="img/world-knowledge/strong-prior.jpeg"> </div> </div> https://twitter.com/fchollet/status/1450871559803916290

## This is a technical talk _let us assume we (as humans) decide on and know the prior based on_ * Facts * Ethics / Fighting Unwanted Biases

## What is the true model for that data? <img src="img/world-knowledge/beer.png">

## This? <img src="img/world-knowledge/beer-linear.png"> linear model

## Or that? <img src="img/world-knowledge/beer-non-linear.png"> 3 layers, 1500 nodes each, relu acitvation

## Both models have issues * linear model is general, good at extrapolation, but ... * at the same time not very good at interpolation (underfits) * ... at first sight * do we really trust anybody to drink a negative amount of beer when it gets really cold? * complex model is good at interpolation * but violates basic world knowledge (beer consumption going up when temperature gets really low) * does not extrapolate well * Can we have the best of both worlds, please?

### Lattice based models * lattices are functions that interpolate between point on a n-d regular grid * works well on lower dim, up to 16 features shown to work * encode domain knowledge like monotonicity, convexity, feature relationships * avoid unexpected model behavior on data out of training distribution Concepts: https://www.tensorflow.org/lattice/overview <br> Paper: https://jmlr.org/papers/v17/15-243.html

### Calibrating features using piecewise linear functions <img src="img/world-knowledge/calibration.png"> Encoding world knowledge in constraints and regularizations <small> https://blog.tensorflow.org/2020/02/tensorflow-lattice-flexible-controlled-and-interpretable-ML.html</small>

### Mapping calibrated features to a lattice <img src="img/world-knowledge/laticefunction.png"> _A lattice is an interpolated look-up table that can approximate arbitrary input-output relationships in your data._ <small> https://blog.tensorflow.org/2020/02/tensorflow-lattice-flexible-controlled-and-interpretable-ML.html</small>

## Hands-On: Regression using Lattices <img src="img/world-knowledge/beer-calibrated.png"> https://bit.ly/tf-lattice-beer <!-- https://colab.research.google.com/github/DJCordhose/ml-resources/blob/main/notebooks/priors/extrapolate_lattice_beer.ipynb?hl=en -->

### Regularization * L1 as sparsity enforcing prior * Use L1 if you know your images only contain few pixels * L2 enforcing small values as prior * keep a certain parameter close a specific value * great visual example: https://twitter.com/matthen2/status/1520427990420791298 (L2 on θ₄ to keep final segment pretty stiff) * In Bayesian statistics, regularization corresponds to the choice of a prior * L1 regularization corresponds to a Laplacian prior * L2 is equivalent to a Gaussian Prior (https://towardsdatascience.com/bayesian-priors-and-regularization-penalties-6d0054d9747b)

### CNNs for Images * in images, adjacency of pixels has a meaning * objects are connected * features in an objects are translation invariant * use CNN to encode that knowledge into the network * same filter going over all parts of the image * much less parameters

### Autoencoders for Abstraction * assumption: there are underlying concepts that can be used for abstraction * project instances to latent representation * the less the capacity of the latent representation, the higher the abstraction * means to restrict capacity: * _undercomplete_ (capacity of latent representation << dim of inputs) * L2 to further compress latent a bit * _overcomplete_ (capacity of latent representation >= dim of inputs) * use L1 to increase sparsity of embedding when having dim > 2, to have an additional restriction (often gives better results) <small> https://www.deeplearningbook.org/contents/autoencoders.html</small>

### More Notebooks * Extrapolation in Regression and Time Series Data: https://colab.research.google.com/github/DJCordhose/ml-resources/blob/main/notebooks/priors/regression.ipynb?hl=en * Autoencoders: https://colab.research.google.com/github/DJCordhose/ml-resources/blob/main/notebooks/priors/autoencoder-denoising.ipynb?hl=en * Regularization * Autoencoders for Abstraction * CNNs using proximity as prior * Adversarial Learning (removing weird extrapolations): https://colab.research.google.com/github/DJCordhose/ml-resources/blob/main/notebooks/priors/extrapolate_nsl.ipynb?hl=en

## The High-Level View <img src="img/mikio/ml-wk-overview.png"> Courtesy of Mikio Braun https://twitter.com/mikiobraun

## What else is there?

### In a nutshell: Encoding World Knowledge What objective do you phrase in your learning experiment? - _type of learning_ - supervised learning, unsupervised learning, reinforcement learning - _input and output_ - feature extraction / preprocessing: what is really relevant? - bias: what features should you leave out? - _general architecture_ - type of model, network capacity, parametric - _reward and observation for RL_ - shape learning objective

### Finally, a promising alternative approach ## Foundation Models _trained on broad data at scale and are adaptable to a wide range of downstream tasks_ * promise to store the knowledge about the world in text form * have a high capacity for learning * can be trained on all data available in text form * one day a general abstraction for everything? https://arxiv.org/abs/2108.07258

### What do you gain by adding priors to your model? * Fairness * by fighting unwanted bias * Explainability / Interpretability * by stating your priors (global interpretability) * without necessarily sacrificing accuracy * sometimes even improving on that * Trust * by making better predictions

## Thanks for attending # How to teach our world knowledge to a neural network Slides: https://bit.ly/2022-odsc-priors Blog-Post: https://opendatascience.com/how-to-bring-our-world-knowledge-to-machine-learning/ ### Stay in Contact if you like Oliver Zeigermann, <a href='https://twitter.com/DJCordhose'>@DJCordhose</a> https://www.linkedin.com/in/oliver-zeigermann-34989773/