Bridging the gaps towards advanced data discovery over semi-structured data

In this work we argue that two main gaps currently hinder the development of new applications requiring sophisticated data discovery capabilities over rich (semi-structured) entity-relationship data. The first gap exists at the conceptual level, and the second at the logical level. Aiming at fulfilling the identified gaps, we propose a novel methodology for developing data discovery applications. We first describe a data discovery extension to the classic ER conceptual model termed Entity Relationship Data Discovery (ERD²). We further present a novel logical model termed the Document Category Sets (DCS) model, used to represent entities and their relationships within an enhanced document model, and describe how data discovery requirements captured by the ERD² conceptual model can be translated into the DCS logical model. Finally, we propose an efficient data discovery system implementation, and share details of two different data discovery applications that were developed in IBM using the proposed methodology.