TENDE: Transfer Entropy Neural Data Estimator

Giulio Franzese — Eurecom, Data Science Department

Overview

TENDE (Transfer Entropy Neural Data Estimator) quantifies directed information flow between time series using neural conditional density estimators.

Transfer Entropy (TE)

\[TE_{Y \to X}(\tau) = I(X_{t+\tau}; Y_t^{(k)} \mid X_t^{(l)})\]

Measures how much knowing \(Y\)’s past reduces uncertainty about \(X\)’s future, beyond what \(X\)’s own past provides.

Biological applications

• Neural circuit connectivity
• Gene regulatory networks
• Cell signaling cascades
• Calcium imaging

Why Transfer Entropy?

Standard MI is symmetric: \(I(X;Y) = I(Y;X)\) — cannot detect direction.

Transfer entropy is asymmetric:

\[TE_{Y \to X} \neq TE_{X \to Y}\]

The net information flow from \(Y\) to \(X\):

\[\Delta TE = TE_{Y \to X} - TE_{X \to Y}\]

• \(\Delta TE > 0\): net flow \(Y \to X\) (Y drives X)
• \(\Delta TE < 0\): net flow \(X \to Y\) (X drives Y)
• \(\Delta TE \approx 0\): bidirectional / no coupling

TE as Conditional MI

\[TE_{Y \to X}(\tau) = I(X_{t+\tau}; Y_t^{(k)} \mid X_t^{(l)})\]

Expanding:

\[= H(X_{t+\tau} \mid X_t^{(l)}) - H(X_{t+\tau} \mid X_t^{(l)}, Y_t^{(k)})\]

Or equivalently (using log density ratios):

\[= \mathbb{E}\left[\log \frac{p(X_{t+\tau} \mid X_t^{(l)}, Y_t^{(k)})}{p(X_{t+\tau} \mid X_t^{(l)})}\right]\]

This is a KL divergence between two conditional distributions — estimable with neural networks!

Time Series Embedding

Given discrete-time series \(\{X_t\}\) and \(\{Y_t\}\):

def embed_series(x, history_len, tau=1):
    """Embed time series into (history, future) pairs."""
    T = len(x)
    history = np.stack(
        [x[t:t+history_len] for t in range(T - history_len - tau + 1)]
    )
    future = x[history_len + tau - 1 : T - tau + history_len - tau + 1]
    return history, future

# Usage
k, l, tau = 5, 5, 1
x_hist, x_fut = embed_series(X, l, tau)
y_hist, _     = embed_series(Y, k, tau)

Lag selection: domain knowledge, Takens’ theorem, or cross-validation.

Neural Conditional Density Estimation

TENDE fits two conditional models:

Full model: \(p_\theta(X_{t+\tau} \mid X_t^{(l)}, Y_t^{(k)})\)

Reduced model: \(p_\phi(X_{t+\tau} \mid X_t^{(l)})\)

class ConditionalGaussianNet(nn.Module):
    def __init__(self, context_dim, target_dim, hidden=128):
        super().__init__()
        self.enc  = nn.Sequential(
            nn.Linear(context_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU()
        )
        self.mu_head    = nn.Linear(hidden, target_dim)
        self.lsig_head  = nn.Linear(hidden, target_dim)

    def log_prob(self, context, target):
        h = self.enc(context)
        mu, lsig = self.mu_head(h), self.lsig_head(h).clamp(-5, 2)
        return Normal(mu, lsig.exp()).log_prob(target).sum(-1)

TENDE Estimator

class TENDE:
    def __init__(self, x_hist_dim, y_hist_dim, x_dim, hidden=128):
        full_dim    = x_hist_dim + y_hist_dim
        self.full   = ConditionalGaussianNet(full_dim, x_dim, hidden)
        self.reduced= ConditionalGaussianNet(x_hist_dim, x_dim, hidden)
        self.opt    = Adam(list(self.full.parameters()) +
                           list(self.reduced.parameters()))

    def estimate_te(self, x_hist, y_hist, x_fut):
        ctx_full = torch.cat([x_hist, y_hist], dim=-1)
        lp_full  = self.full.log_prob(ctx_full, x_fut)
        lp_red   = self.reduced.log_prob(x_hist,  x_fut)
        return (lp_full - lp_red).mean()

    def train_step(self, x_hist, y_hist, x_fut):
        ctx_full = torch.cat([x_hist, y_hist], dim=-1)
        loss = (-self.full.log_prob(ctx_full, x_fut).mean()
                -self.reduced.log_prob(x_hist, x_fut).mean())
        self.opt.zero_grad(); loss.backward(); self.opt.step()

Benchmark: Coupled Ornstein-Uhlenbeck

Model with known ground-truth TE:

\[dX_t = -X_t\,dt + c\, Y_t\,dt + \sigma\,dW^X_t\] \[dY_t = -Y_t\,dt + \sigma\,dW^Y_t\]

Only \(Y \to X\) coupling (via \(c\)). Analytical TE:

\[TE_{Y \to X} \propto c^2, \quad TE_{X \to Y} = 0\]

def simulate_ou(T=2000, c=0.5, dt=0.01, sigma=0.1):
    X, Y = np.zeros(T), np.zeros(T)
    for t in range(1, T):
        X[t] = X[t-1] + dt*(-X[t-1] + c*Y[t-1]) + sigma*np.sqrt(dt)*randn()
        Y[t] = Y[t-1] + dt*(-Y[t-1])             + sigma*np.sqrt(dt)*randn()
    return X, Y

Lag Sensitivity

TE as a function of lag \(\tau\):

• Peak TE at \(\tau^*\) indicates the dominant information transmission delay
• Relevant for: synaptic delays (ms), gene regulation (hours), ecological cycles (seasons)

lags = range(1, 20)
te_yx = [tende.estimate_te_at_lag(X, Y, tau) for tau in lags]
te_xy = [tende.estimate_te_at_lag(Y, X, tau) for tau in lags]

plt.plot(lags, te_yx, label=r'$TE_{Y\to X}$')
plt.plot(lags, te_xy, label=r'$TE_{X\to Y}$')
plt.xlabel('Lag $\\tau$'); plt.ylabel('TE (nats)')
plt.legend()

Application: Gene Regulatory Networks

Data: scRNA-seq time course (cells sampled at \(t = 0, 2, 4, 8, 24\) h after stimulation)

Pipeline:

1. Embed each gene’s expression profile as a time series
2. Estimate pairwise \(TE_{i \to j}\) for all gene pairs \((i, j)\)
3. Threshold to infer directed regulatory edges
4. Compare to known gold-standard networks

Practical note: With \(n\) genes, this is \(\mathcal{O}(n^2)\) estimations — parallelize!

Summary

• Transfer entropy = conditional MI between future of \(X\) and past of \(Y\) given past of \(X\)
• TENDE: neural conditional density estimation → computationally scalable
• Asymmetric by construction → detects directionality
• Key hyperparameters: history length \(k, l\), lag \(\tau\), hidden size

References:

• Schreiber (2000). Measuring Information Transfer.
• Bossomaier et al. (2016). An Introduction to Transfer Entropy.
• Franzese et al. (2024). TENDE.