ConformalPrediction.jl

Predictive Uncertainty Quantification in Machine Learning

Patrick Altmeyer

Delft University of Technology

May 14, 2024

Talk Agenda1

  1. Introduction (5min)
  2. Applications (5min)
  3. Interactive demo of ConformalPrediction.jl (10min)
  4. Under Construction (5min)
  5. Q&A

Code along 💻 tinyurl.com/cpjcon2023

Pluto.jl 🎈

Feeling lucky? Use binder!

Conformal Prediction

Conformal Prediction involves turning heuristic measures of Predictive Uncertainty into rigorous ones.

A first crucial step towards building trustworthy AI systems is to be transparent about predictive uncertainty.

Example: Split CP

  1. Proper training set and separate calibration set: \(\mathcal{D}_n=\mathcal{D}^{\text{train}} \cup \mathcal{D}^{\text{cali}}\).
  2. Train model on proper training set: \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}(X_i,Y_i)\).
  3. Compute nonconformity scores, \(\mathcal{S}\), using calibration data \(\mathcal{D}^{\text{cali}}\) and fitted model \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}\).
  4. For user-specified coverage ratio \((1-\alpha)\) compute the corresponding quantile, \(\hat{q}\), of \(\mathcal{S}\).
  5. For the given quantile and test sample \(X_{\text{test}}\), form the corresponding conformal prediction set: \(C(X_{\text{test}})=\{y:s(X_{\text{test}},y) \le \hat{q}\}\).

Blog posts

Split CP illustrated

Figure 1: Softmax output for class 1 (top left); non-conformity scores for calibration set (top right); (1-α)-quantile (bottom left); non-conformity function applied to test point (bottom right). Solid bars make it into prediction set.

ConformalPrediction.jl

  • Built on top of MLJ.jl.
  • Compatible with Flux.jl through MLJFlux.jl.
  • Many state-of-the-art CP methods implemented for regression, classification and time series modelling.

🔍 Applications

CP meets SR

Remember SymbolicRegression.jl by Miles Cranmer?

# Standard MLJ workflow:
using MLJ
import SymbolicRegression: SRRegressor
model = SRRegressor(
  niterations=50,
  binary_operators=[+, -, *],
  unary_operators=[sin],
)

# Conformalize:
using ConformalPrediction
conf_model = conformal_model(model)
mach = machine(conf_model, X, y)
fit!(mach, rows=train)

Conformal Chatbot

Figure 2: High-level overview of Conformal Intent Classifier. Won 1st 🏆 at ING Global Experiment Week 2023.
Figure 3: Demo of a REPL-based conformalized intent classifier.

CICC substantially outperforms baseline approaches (e.g. top-\(K\)).

Blog post

Building a Conformal Chatbot in Julia (blog, TDS)

Conformal Image Classifier

A simple MNIST classifier.

# MLJFlux workflow:
using MLJFlux
ImageClassifier = @load ImageClassifier

# Conformalize:
using ConformalPrediction
conf_model = conformal_model(clf)
mach = machine(conf_model, X, y)
fit!(mach)
Figure 4: Probably a 7 🤔

Blog post

How to Conformalize a Deep Image Classifier (blog, TDS, Forem)

Time Series

Ensemble Batch Prediction Intervals (Xu and Xie 2021) contributed by Mojtaba Farmanbar 📣.

Tutorial

How to Conformalize a Time Series Model (docs)

Figure 5: EnbPI for Victoria electricity demand dataset.

🏃 Interactive Session

🚧 Under Construction

Conformal LaplaceRedux.jl

Contribute

LaplaceRedux.jl interfaced to MLJFlux.jl. Planning to add both ideas [#64].

Conformal Counterfactuals

Stutz et al. (2022) introduce Conformal Training: conformal predictions (left), set size (centre) and smooth set size loss (right).

Conformal CounterfactualExplanations.jl.

Contribute

Currently working on full conformal training implementation [#62].

🗨️ and finally …

Trustworthy AI in Julia

Taija collects Julia packages geared towards Trustworthy AI:

Contributions welcome! 😊

References

Angelopoulos, Anastasios N., and Stephen Bates. 2022. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” https://arxiv.org/abs/2107.07511.
Fong, Edwin, and Chris Holmes. 2021. “Conformal Bayesian Computation.” arXiv. https://doi.org/10.48550/arXiv.2106.06137.
Stutz, David, Krishnamurthy, Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. 2022. “Learning Optimal Conformal Classifiers.” https://arxiv.org/abs/2110.09192.
Xu, Chen, and Yao Xie. 2021. “Conformal Prediction Interval for Dynamic Time-Series.” In, 11559–69. PMLR. https://proceedings.mlr.press/v139/xu21h.html.