## MLOps: Monitoring and Managing Drift # Preparation 1. Wifi is on the back of your badge 1. These slides: https://bit.ly/odsc-west-mlops 1. Start project using Gitpod https://gitpod.io/#https://github.com/DJCordhose/mlops-drift 1. Login using Github Account when asked to 1. Choose the large machine when asked to 1. Open the story for you to follow: https://bit.ly/odsc-2023-mlops-story 1. Sources for Project: https://github.com/DJCordhose/mlops-drift 1. Introduce yourself to your neighbors and build teams of 2-4 people 1. Questions and discussions welcome at any time

1. Sources for Project: https://github.com/DJCordhose/mlops-drift 1. story: https://bit.ly/odsc-2023-mlops-story 1. Notebooks for part II * https://colab.research.google.com/github/djcordhose/mlops-drift/blob/main/notebooks/drift.ipynb?hl=en * https://colab.research.google.com/github/djcordhose/mlops-drift/blob/main/notebooks/analysis.ipynb?hl=en * https://colab.research.google.com/github/djcordhose/mlops-drift/blob/main/notebooks/train.ipynb?hl=en

# MLOps: Monitoring and Managing Drift ODSC West 2023, https://odsc.com/speakers/mlops-monitoring-and-managing-drift/ Monday, 30th October, 10:00 AM - 1:15 PM Oliver Zeigermann <img src="img/bit.ly_odsc-west-mlops.png" style="height: 150px;"><br> These Slides: https://bit.ly/odsc-west-mlops Project: https://github.com/DJCordhose/mlops-drift <!-- https://djcordhose.github.io/mlops-drift/2023-odsc.html -->

## What is Monitoring and Managing Drift all about? 1. _MonitoringDrift_: You want to know that there might be a problem with the quality of your model _before_ 1. your (internal or external) partners notice 1. your users notice 1. you lose trust 1. your business is impacted 1. you are in the news 2. _Managing Drift_: Once noticed, you want to be able to react quickly and effectively 1. analyze if there is a problem in the first place 1. if so, what to do about it

## How to prepare * install the project as described in the readme: https://github.com/DJCordhose/mlops-drift/blob/main/README.md#installation * do not stress yourself, if you can not get this up and running * there will be time for this in the workshop (even though not a lot) * you can also just watch and learn, you will get a lot of benefit even without running the code * the second part should work entirely without a local installation * when forming teams, it is sufficient to have one person with a working installation

## Who is Olli <div style="display: flex;"> <div style="flex: 50%;"> <a href='https://oreilly.de/produkt/machine-learning-kurz-gut-2/'> <img src='img/ml-buch-v2.jpg' height="400"> </a> </div> <div style="flex: 50%; font-size: x-large;"> <img src='img/olli-opa.jpeg'> </div> </div> <p> <a target="_blank" href="mailto:oliver@zeigermann.de">Oliver Zeigermann</a>: Machine Learning Engineer/Architect from Hamburg, Germany </p>

## Wait, are these all the slides? * the story for this workshop is here: https://github.com/DJCordhose/mlops-drift/blob/main/docs/story.md * the slides following are just a support for the story

### Use Case: Predict Risk <img src="img/zine/use-case.png" style="height: 530px;" class="fragment"> <small>Objective: Risk of potential customer as LOW, MEDIUM, HIGH</small>

<img src="img/mlops/swagger-sample-request.jpg" style="height: 40vh;"> *Result of example* ``` { "prediction": "LOW", "probabilities": { "HIGH": 0.0003246392006985843, "MEDIUM": 0.022435296326875687, "LOW": 0.9772400259971619 }, "predictor_type": "MODEL" } ```

## Our starting point Notebooks are the classic tool for ML development: https://colab.research.google.com/github/djcordhose/mlops-drift/blob/main/notebooks/train.ipynb?hl=en * Neural network with TensorFlow * 3 hidden layers, 100 neurons per layer * 1500 data sets in total * Training on 1200 data sets * Test/validation on 300 data sets * Accuracy training/test > 85%

### The quality of ML systems usually deteriorates over time <img src="img/zine/degrade.png" style="height: 530px;" class="fragment"> <small>This is not only true for ML applications, but it is more obvious with them</small>

### Updated models are necessary frequently <img src="img/zine/intervention.png" style="height: 530px;" class="fragment"> <small>Machine Learning applications need maintenance</small>

### Without Ground Truth you need a surrogate metric <img src="img/zine/drift.png" style="height: 450px;" class="fragment"> *Distribution of data of requests or predictions differs significantly from that of training*

### Distribution of Ages at the time of training <img src="img/causal-insurance/age-reference.png">

### does this drift? <img src="img/causal-insurance/age_no_drift_p75_no_clue.png">

### and this? <img src="img/causal-insurance/age_drift_p0_no_clue.png">

### Monitoring with Evidently, Prometheus und Grafana <img src="img/mlops/evidently_grafana_service.png"> <small>https://evidentlyai.com/blog/evidently-and-grafana-ml-monitoring-live-dashboards <br> https://docs.evidentlyai.com/integrations/evidently-and-grafana </small>

### Summary: Types of Drift _Understanding the difference between data and concept drift is useful when interpreting the changes. However, to detect them, we would typically use the same approach._ * _Covariate / Input / Data drift_: Distribution of the input has changed * _Prior / Label / Prediction drift_: Distribution of the prediction has changed * _Concept / Model drift_: Relationship between input and prediction has changed <img src="https://docs.seldon.io/projects/alibi-detect/en/stable/_images/bg_2d_drift.png" style="height: 100%;"> <small>https://docs.seldon.io/projects/alibi-detect/en/stable/cd/background.html#what-is-drift </small>

## What else can be monitored? * Quality of the data * how do missing or wrong fields change * plausibility * When you have Ground Truth * Metric for the quality of the prediction as used in training * Different typical quality metrics * Caution: no matter how good the metric, it is again only a surrogate for the quality of the prediction * Which model/fallback was used * Requests per minute/hour/day * Latency * Error Rate * User Feedback * Lastly, there is the business or product KPI: https://www.evidentlyai.com/blog/ml-monitoring-metrics#4-business-metrics-and-kpi

## Drift requires interpretation When the world changes, drift is to be expected and therefore ok | | Positive interpretation, no action required | Negative interpretation, action required | |---|---|---| | *Data and Prediction Drift* | important features have changed, model is fine and extrapolates well, e.g.: higher age, more risk | important features have changed, model does not extrapolate well | | *Data but no Prediction Drift* | no important features changed, model is robust enough for the drift | important features have changed, model does not extrapolate well | | *Prediction but no Data Drift* | ??? | probably Concept Drift, new analysis of the situation necessary | | | | |

## Which statistical test / which metric? there is unfortunately not the one suitable test * some only fit well for small (< 1000) data sets * our data sets are larger than 1000 * some can only work on numerical but not categorical data * we need both * some are normalized between 0 and 1 * we don't care about that * our metrics * Wasserstein metric for numerical data * Jensen-Shannon distance for categorical data * Drift algorithm: https://docs.evidentlyai.com/reference/data-drift-algorithm * When to use which metric: https://evidentlyai.com/blog/data-drift-detection-large-datasets

### Wasserstein Metric _If each distribution is considered as a pile of "earth" piled up on the metric space, then this metric describes the minimum "costs" of transforming one pile into the other._ * not too sensitive, only shows larger changes * normalized in changes in standard deviations * can (obviously) go over 1 * from 0.1 we assume a drift * only works for numerical data https://en.wikipedia.org/wiki/Wasserstein_metric

### Jensen-Shannon Divergence * Jensen-Shannon distance is square root of divergence, that is our metric * between 0 and 1 * from 0.05 we assume a drift * also works for categorical data * based on Kullback–Leibler divergence, relative entropy * histograms are compared, size of the sample therefore doesn't matter * binning for categorical data obvious * Intuition: how much information/entropy/surprise is in the difference of the two distributions? https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence

### A ML model does not go into production all by itself <img src="img/ml-system-postits.jpg" style="height: 550px;">

### What else / more material

### What else can be done * Outlier detection * Our model will not be able to extrapolate * Values outside the training range will often be predicted with an unrealistically high certainty * Outliers must be detected without ground truth * Adversarial Detection * Certain inputs can intentionally lead to a grossly wrong prediction * Such inputs can be detected and corrected * For this purpose, e.g. an autoencoder can be used, which corrects the input <small>https://docs.seldon.io/projects/alibi-detect/en/stable/od/methods.html <br> https://docs.seldon.io/projects/alibi-detect/en/stable/ad/methods.html </small>

### Drift Detection for Images _Reduce image to typical aggregated properties_ * Input Drift with histograms of low-level features (HSV): https://towardsdatascience.com/detecting-semantic-drift-within-image-data-6a59a0e768c6 * Input Drift with dimension reduction: https://docs.seldon.io/projects/alibi-detect/en/stable/examples/cd_ks_cifar10.html * Model Drift by comparing with distilled model: https://docs.seldon.io/projects/alibi-detect/en/stable/examples/cd_distillation_cifar10.html * A complete example with Alibi-Detect: https://docs.seldon.io/projects/alibi-detect/en/stable/examples/alibi_detect_deploy.html#4.-Drift-detection-with-Kolmogorov-Smirnov

### Drift Detection for Text 1. Text Descriptors Drift: compare the descriptive statistics of the text data, e.g. 1. length of text 1. share of out-of-vocabulary words 1. share of non-letter symbols 1. Domain Classifier / model-based drift detection * If a model can reliably identify the reviews that belong to the current or reference dataset, the two datasets are probably sufficiently different. * Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift: https://arxiv.org/abs/1810.11953 * can be applied to all kinds of data (including images) 1. Based on differences (eucledian, cosine, etc.) of averages of embeddings

### Drift Detection for Text - Links * https://docs.seldon.io/projects/alibi-detect/en/stable/examples/cd_text_imdb.html * https://www.evidentlyai.com/blog/unstructured-data-monitoring * https://www.evidentlyai.com/blog/embedding-drift-detection * https://www.evidentlyai.com/blog/how-deepl-monitors-ml-models * https://www.evidentlyai.com/blog/tutorial-detecting-drift-in-text-data * https://archive.ics.uci.edu/dataset/462/drug+review+dataset+drugs+com * https://colab.research.google.com/drive/10RBYBM5phCckMNZ0QRnOOcx4YyAou9uP * https://www.evidentlyai.com/blog/evidently-data-quality-monitoring-and-drift-detection-for-text-data

### When do we have the Ground Truth? *Human experts can* * *determine ground truth* * as soon as a human checks/revises the decisions * for a proposal system this can be very fast * for dark processing this should happen at regular intervals * *not determine ground truth* * we have to wait for realities * in time series, predictions are made for the near future, as soon as this occurs it can be checked * often such realities are only perceptible after some time and are subject to statistical fluctuations (as in this example)

### Our prediction itself creates drift * A deployment changes our role from observer to actor * We only insure people with a good risk forecast * If not, why should we make a forecast at all? * Our GT will have more and more good drivers * At least that is our hope (otherwise the forecast would not have worked) * If not (False Negative, Type II Error) * have people learned to trick our system? * "Then my daughter will insure the car" * Have we accidentally not insured good drivers (False Positive, Type I Error) * Possibility: *epsilon-greedy* mostly believe the prediction, but sometimes (epsilon) also insure a driver with a bad forecast * Prediction with all probabilities goes into our database https://twitter.com/ChristophMolnar/status/1569644089724764160

### Google Rubric Score * https://developers.google.com/machine-learning/guides/rules-of-ml * https://research.google/pubs/pub46555/ * https://research.google/pubs/pub45742/ * https://laszlo.substack.com/p/article-review-the-ml-test-score

### Challenges in Machine Learning Projects <img src="img/biggest_challenges_ML_MLCon_2023.jpg" style="height: 530px;" class="fragment"> <small>Survey MLConf Berlin 2023</small>

### Machine Learning Projecs can be thought as running in phases <img src='img/ml-phases-small-en.jpg' style="height: 100%;">