Universal estimation of directed information via sequential probability assignments

Jiantao Jiao, Haim H. Permuter, Lei Zhao, Young Han Kim, Tsachy Weissman

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

We propose four approaches to estimating the directed information rate between a pair of jointly stationary ergodic processes with the help of universal probability assignments. The four approaches yield estimators with different merits such as nonnegativity and boundedness. We establish consistency of these estimators in various senses and derive near-optimal rates of convergence in the minimax sense under mild conditions. The estimators carry over directly to estimating other information measures of stationary ergodic processes, such as entropy rate and mutual information rate, and provide alternatives to classical approaches in the existing literature. Guided by the theoretical results, we use context tree weighting as the vehicle for the implementations of the proposed estimators. Experiments on synthetic and real data are presented, demonstrating the potential of the proposed schemes in practice and the efficacy of directed information estimation as a tool for detecting and measuring causality and delay.

Original languageEnglish
Title of host publication2012 IEEE International Symposium on Information Theory Proceedings, ISIT 2012
Pages523-527
Number of pages5
DOIs
StatePublished - 22 Oct 2012
Event2012 IEEE International Symposium on Information Theory, ISIT 2012 - Cambridge, MA, United States
Duration: 1 Jul 20126 Jul 2012

Publication series

NameIEEE International Symposium on Information Theory - Proceedings
ISSN (Print)2157-8105

Conference

Conference2012 IEEE International Symposium on Information Theory, ISIT 2012
Country/TerritoryUnited States
CityCambridge, MA
Period1/07/126/07/12

Keywords

  • Causal influence
  • context tree weighting
  • directed information
  • rate of convergence
  • universal probability assignment

Fingerprint

Dive into the research topics of 'Universal estimation of directed information via sequential probability assignments'. Together they form a unique fingerprint.

Cite this