S\(\Omega\)I: Score-based O-Information Estimation

Mustapha Bounoua

EURECOM

Ampere Software Technology, France

Giulio Franzese

EURECOM

Pietro Michiardi

EURECOM

Introduction

  • Complex systems are often described by multivariate information
  • Understanding the relationships among multiple random variables is crucial

Brain regions

Sensors

How do the system components interact?

Extensions of Shannon Mutual Information

  • Shannon Mutual information: \(\mathcal{I}(X^1; X^2)\)
  • Not interpretable for large systems \(N>3\)

PID (Williams and Beer 2010):

  • Requires a partition into sources and target
  • Not scalable

O-information (Rosas et al. 2019):

  • No partition needed
  • Scalable

SOTA is limited to discrete or Gaussian distributions

  • S\(\Omega\)I estimates O-information without restrictions on data type or number of variables

Multivariate Interactions

Redundancy: The shared information between variables, recoverable from subsets

Synergy: Information arising from jointly observing all variables, inaccessible from individuals

High Dimensional Interaction Measures

Let \(X = \{X^1,\dots,X^N\}\), \(X^{>i} = \{X^{i+1},\dots,X^N\}\), \(X^{\setminus i} = X \setminus \{X^i\}\).

Total correlation: \[\mathcal{T}(X) = \sum_{i=1}^{N} \mathcal{I}(X^i;\, X^{>i})\] How much information \(X^i\) shares with \(X^{>i}\) — suggests redundancy

Dual total correlation: \[\mathcal{D}(X) = \sum_{i=1}^{N} \mathcal{I}(X^i;\, X^{\setminus i} \mid X^{<i})\] How much additional information \(X^i\) carries — suggests synergy

O-Information

\[\boxed{\Omega(X) = \mathcal{T}(X) - \mathcal{D}(X)} \qquad \begin{cases} \Omega(X)>0 & \text{Redundancy} \\ \Omega(X)<0 & \text{Synergy} \end{cases}\]

Gradient of O-information (Scagliarini et al. 2023): \[\boxed{\partial_i \Omega(X) = \Omega(X) - \Omega(X^{\setminus i})}\] Captures the individual influence of each variable

  • Hard to estimate: high dimensional with arbitrary data distribution

Score-based KL Divergence Estimation

Let \({\color{red}X_t} = {\color{green}X} + \sqrt{2t}\,{\color{red}W}\), \(t\in[0,\infty)\). Using (Franzese, M. Bounoua, and Michiardi 2024):

\[\mathrm{KL}(p \| q) = \int p(x)\log\frac{p(x)}{q(x)}\,\mathrm{d}x = \int p_t(x)\,\underbrace{\|\nabla\log p_t(x) - \nabla\log q_t(x)\|^2}_{\text{Difference of score functions}}\,\mathrm{d}x\,\mathrm{d}t\]

  • Learn \(\nabla\log p_t(\cdot)\) by denoising \({\color{red}X_t}\): \[\nabla\log p_t(x) = \frac{1}{2t}\Bigl(\underbrace{\mathbb{E}[X\mid{\color{red}X_t}]}_{\text{Denoiser}} - {\color{green}x}\Bigr)\]

Score-based O-Information Estimation — TC

\[\mathcal{T}(X) = \mathrm{KL}\!\left(p(x)\,\Big\|\,\prod_{i=1}^N p(x^i)\right) = \int \frac{1}{4t^2}\,\mathbb{E}\left\| \mathbb{E}[X\mid{\color{red}X_t}] - \begin{bmatrix}\mathbb{E}[X^i\mid{\color{red}X^i_t}]\end{bmatrix}_{i=1}^N \right\|^2\mathrm{d}t\]

  • Compare denoiser output: all variables together (Joint) vs separately (Marginals)

Score-based O-Information Estimation — DTC

\[\mathcal{D}(X) = \int \frac{1}{4t^2}\,\mathbb{E}\left\| \mathbb{E}[X\mid{\color{red}X_t}] - \left[\mathbb{E}\!\left[X^i\mid{\color{red}X^i_t},\,{\color{green}X^{\setminus i}}\right]\right]_{i=1}^N \right\|^2\mathrm{d}t\]

  • Compare joint denoising vs individual denoising conditioned on remaining clean variables (Conditionals)

\[\Omega(X) = \mathcal{T}(X) - \mathcal{D}(X)\]

Amortized Approach — Single Network

Architecture

Mode Forward pass
Joint \(\epsilon_\theta([{\color{red}X_t^1,\dots,X_t^N}],\,t)\)
Conditional \(\epsilon_\theta([{\color{green}X^1,..,}{\color{red}X_t^i}{\color{green},..,X^N}],\,t)\)
Marginal \(\epsilon_\theta({\color{red}X_t^i},\,t)\)

\[\hat\Omega(X) = \underbrace{\frac{1}{4t^2}\left\|\epsilon(X_t)-[\epsilon(X_t^i)]_{i=1}^N\right\|^2}_{\hat{\mathcal{T}}(X)} - \underbrace{\frac{1}{4t^2}\left\|\epsilon(X_t)-[\epsilon(X_t^i\mid X^{\setminus i})]_{i=1}^N\right\|^2}_{\hat{\mathcal{D}}(X)}\]

Experimental Validation

Synthetic benchmark:

  • Multivariate Gaussian with/without transformation
  • Redundancy, synergy, or a mix of both
  • Varying number of variables, per-variable dimension, and interaction strength

Baseline: MI neural estimators

Implementation:

  • MLP with skip connections — sufficient for simple settings
  • Transformer — needed for gradient of O-information

S\(\Omega\)I efficiently estimates O-information across all challenging settings

Synthetic Benchmarks — \(\Omega(X)\)

Redundancy:
\(R \sim \mathcal{N}(0,\mathbf{I})\)

\[X = \{R + \varepsilon_i\}_{i=1}^N,\quad \varepsilon_i \sim \mathcal{N}(0,\sigma\mathbf{I})\]

Synergy:
\(S \sim \mathcal{N}(0,\mathbf{I})\)

\[X^1 \sim \mathcal{N}(0,\mathbf{I}),\; X^2 = X^1+S,\; X^3 = S+\varepsilon\]

\(\varepsilon \sim \mathcal{N}(0,\sigma)\), \(\;X^1 \perp X^3\)

10 variables, dim=20. Top: Redundancy. Bottom: Synergy.

Synthetic Benchmarks — \(\partial\Omega(X)\)

6-variable mixed system ■ redundancy   ■ synergy

10-variable mixed system ■ redundancy   ■ synergy

Results with transformer-based architecture.

O-Information in the Mice Brain (Venkatesh et al. 2023)

Visual Behaviour Neuropixels (VBN). Spike recordings from 6 mouse brain regions. S\(\Omega\)I estimates O-information per 50 ms bin after stimulus flash.

6 brain regions

Higher redundant information in visual cortex is transmitted after a flash with a new scene

Conclusion

  • S\(\Omega\)I captures multivariate interactions for any data distribution and large number of variables — no restriction to discrete or Gaussian

  • The only required ingredient is the score function — extends to images, audio, fMRI, multimodal data, etc.

  • S\(\Omega\)I opens the door to many scientific applications (open for collaboration!)

Project repo
mustaphabounoua.github.io/soi_o_inf

Thank you!

References

Franzese, Giulio, M. Bounoua, and Pietro Michiardi. 2024. “MINDE: Mutual Information Neural Diffusion Estimation.” In ICLR. https://openreview.net/forum?id=0kWd8SJq8d.
Rosas, Fernando E., Pedro A. M. Mediano, Michael Gastpar, and Henrik Jeldtoft Jensen. 2019. “Quantifying High-Order Interdependencies via Multivariate Extensions of the Mutual Information.” Physical Review. E 100 3-1: 032305. https://api.semanticscholar.org/CorpusID:67855406.
Scagliarini, Tomas, D. Nuzzi, Yuri Antonacci, Luca Faes, Fernando Rosas, Daniele Marinazzo, and Sebastiano Stramaglia. 2023. “Gradients of o-Information: Low-Order Descriptors of High-Order Dependencies.” Physical Review Research 5.
Venkatesh, Praveen, Corbett Bennett, Sam Gale, Tamina Ramirez, Greggory Heller, Severine Durand, Shawn Olsen, and Stefan Mihalas. 2023. “Gaussian Partial Information Decomposition: Bias Correction and Application to High-Dimensional Data.” Neurips.
Williams, Paul L., and Randall D. Beer. 2010. “Nonnegative Decomposition of Multivariate Information.” https://arxiv.org/abs/1004.2515.