The comfort level of passengers is an important factor in measuring user experience in any form of transportation, including in autonomous vehicles. One of the main factors that determines user acceptance of autonomous vehicles is the passenger's level of discomfort in a 'control- and authority-less' experience. In this paper, we propose an approach for formulating discomfort through 'on-the-road' field studies, with human driven vehicles, while the passenger provides real-time explicit feedback on discomfort via a potentiometer. While previous studies focused on the association between vehicle dynamics and passenger discomfort, we demonstrate here how we can improve the classification ability of passenger discomfort by employing a multi-dimensional model that also takes into account the external scenario (contextual information). This is achieved by processing image data (e.g. distance from nearest bicycle) recorded through an outward looking camera in addition to location/route data obtained from other sensors like GPS. As such, the focus of this paper is on classification of external information.