It has been suggested that the interference in the Stroop task arises due to both information conflict (color vs. word meaning) and task conflict (respond to the color vs. read the word). Interestingly, the task conflict (faster responses in the congruent condition compared to the neutral condition) is influenced by the readability of the stimulus. However, behavioral results indicate the influence of the contribution of readability to the task conflict appears only when vocal responses are required. We conducted a Stroop experiment with four readability levels of neutrals. Participants were required to respond manually, and both their response times and pupil dilation were measured. Our results showed the slowest responses for incongruent trials and the fastest responses for congruent trials. However, no differences in responses were found between the readability levels of the neutrals. In contrast, pupil dilation showed the largest dilation in the incongruent trials and smallest dilation in the neutral trials. In addition, the differences between congruent and neutral trials were influenced by the readability of the stimulus. Specifically, unreadable neutral stimuli showed meaningful differences early on, which also remained for longer time compared with readable neutral stimuli. These results are important and can help in specifying new requirements regarding the theory and modeling of the Stroop task; in particular, studies that want to control the neutrality of their neutral trials should take the readability into account.