Speaking is the most difficult as well as the most complex of all the four skills, as it requires expertise in, and exposure to, the target language. Different factors are found responsible for poor speaking skills among EFL learners in general and Saudi EFL learners in particular. The current study investigates the influence of various factors related to teachers, learners, and learning environment on the students’ attitude towards learning speaking skills. The questionnaire survey was employed to elicit responses from 184 undergraduate EFL male and female students in Taif University. Data analyzed through SPSS reveals that out of five variables only one was insignificant, whereas all other variables showed significant positive effect. In the light of the findings, it could be inferred that lack of measures on the part of teachers and learners as well as the classroom setting/environment do not fully facilitate both the male and female students to learn speaking skills in a better way. The poor level of their skills in English is attributed to the variety of teachers’, learners’, and environment related factors. And these factors affect negatively on the attitude of learners towards learning speaking skills.