This paper presents an Extended Kalman filter (EKF) for relative position and attitude (pose) estimation in a mission around an asteroid. It compares two different ways of representing the pose: a conventional one (Cartesian coordinates for position and quaternions for attitude) and a novel approach (dual quaternions, which comprise both attitude and position in one eight-dimensional vector). Moreover, this paper presents a 'realistic' modelling for dynamics and hardware simulation for missions around small bodies. It includes a polyhedron gravity field modelling, polyhedron gravity gradient torque, navigation camera and laser ranger measurements. The results of the filters show that it is not only possible to estimate the relative states with high accuracy, but also parameters such as gyroscope drift and asteroid angular rates can be estimated. This, however, can only be achieved when the navigation camera detects landmarks in its field-of-view. Finally, the dual quaternion representation does not give any noticeable advantages over the conventional one; in fact, the two filters are identical in the steady state.