Using the information structure model to compare profile-based information filtering systems

In the IR field it is clear that the value of a system depends on the cost and benefit profiles of its users. It would seem obvious that different users would prefer different systems. In the TREC-9 filtering track, systems are evaluated by a utility measure specifying a given cost and benefit. However, in the study of decision systems it is known that, in some cases, one system may be unconditionally better than another. In this paper we employ a decision theoretic approach to find conditions under which an Information Filtering (IF) system is unconditionally superior to another for all users regardless of their cost and benefit profiles. It is well known that if two IF systems have equal precision the system with better recall will be preferred by all users. Similarly, with equal recall, better precision is universally preferred. We confirm these known results and discover an unexpected dominance relation in which a system with lower recall will be universally preferred provided its precision is sufficiently higher.