A single phone call can make or break a valuable customer-organization relationship. Maintaining good quality of service can lead to customer loyalty, which affects profitability. Traditionally, customer feedback is mainly collected by interviews, questionnaires, and surveys; the major drawback of these data collection methods is in their limited scale. The growing amount of research conducted in the field of sentiment analysis, combined with advances in text processing and Artificial Intelligence, has led us be the first to present an intelligent system for mining sentiment from transcribed utterances—wherein the noisiness property and short length poses extra challenges to sentiment analysis. Our aim is to detect and process affective factors from multiple layers of information, and study the effectiveness and robustness of each factor type independently, by proposing a tailored machine learning paradigm. Three types of factors are related to the textual content while two overlook it. Experiments are carried out on two datasets of transcribed phone conversations, obtained from real-world telecommunication companies.