Information-Directed Random Walk for Rare Event Detection in Hierarchical Processes

Chao Wang, Kobi Cohen, Qing Zhao

Research output: Contribution to journalArticlepeer-review

7 Scopus citations

Abstract

The problem of detecting a few anomalous processes among a large number of data streams is considered. At each time, aggregated observations can be taken from a chosen subset of the processes, where the chosen subset conforms to a given tree structure. The random observations are drawn from a general distribution that may depend on the size of the chosen subset and the number of anomalous processes in the subset. We propose a sequential search strategy by devising an information-directed random walk on the tree-structured observation hierarchy. The proposed policy is shown to be asymptotically optimal with respect to the detection accuracy and order-optimal with respect to the size of the search space. Effectively localizing the data processing to small subsets of the search space, the proposed strategy is also efficient in terms of computation and memory requirement.

Original languageEnglish
Article number9274405
Pages (from-to)1099-1116
Number of pages18
JournalIEEE Transactions on Information Theory
Volume67
Issue number2
DOIs
StatePublished - 1 Feb 2021

Keywords

  • Sequential design of experiments
  • active hypothesis testing
  • anomaly detection
  • channel coding with feedback
  • noisy group testing

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences

Fingerprint

Dive into the research topics of 'Information-Directed Random Walk for Rare Event Detection in Hierarchical Processes'. Together they form a unique fingerprint.

Cite this