Info-fuzzy algorithms for mining dynamic data streams

Lior Cohen, Gil Avrahami, Mark Last, Abraham Kandel

Research output: Contribution to journalArticlepeer-review

41 Scopus citations

Abstract

Most data-mining algorithms assume static behavior of the incoming data. In the real world, the situation is different and most continuously collected data streams are generated by dynamic processes, which may change over time, in some cases even drastically. The change in the underlying concept, also known as concept drift, causes the data-mining model generated from past examples to become less accurate and relevant for classifying the current data. Most online learning algorithms deal with concept drift by generating a new model every time a concept drift is detected. On one hand, this solution ensures accurate and relevant models at all times, thus implying an increase in the classification accuracy. On the other hand, this approach suffers from a major drawback, which is the high computational cost of generating new models. The problem is getting worse when a concept drift is detected more frequently and, hence, a compromise in terms of computational effort and accuracy is needed. This work describes a series of incremental algorithms that are shown empirically to produce more accurate classification models than the batch algorithms in the presence of a concept drift while being computationally cheaper than existing incremental methods. The proposed incremental algorithms are based on an advanced decision-tree learning methodology called "Info-Fuzzy Network" (IFN), which is capable to induce compact and accurate classification models. The algorithms are evaluated on real-world streams of traffic and intrusion-detection data.

Original languageEnglish
Pages (from-to)1283-1294
Number of pages12
JournalApplied Soft Computing Journal
Volume8
Issue number4
DOIs
StatePublished - 1 Sep 2008

Keywords

  • Concept drift
  • Data streams
  • Incremental learning
  • Info-Fuzzy Networks
  • Online learning
  • Real-time data mining

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'Info-fuzzy algorithms for mining dynamic data streams'. Together they form a unique fingerprint.

Cite this