Teacher characteristics and student learning.

Dills, Angela K. ; Placone, Dennis

INTRODUCTION

Teacher quality is an important input in student achievement although identifying quality teachers is difficult without detailed panel data (Rivkin, Hanushek, and Kain, 2005). Research in economic education finds that teacher knowledge is one factor affecting student learning (Allgood and Walstad, 1999). In addition, there is suggestive evidence that teacher attitudes matter (Marlin, 1999).

This paper addresses the question of what makes a high school teacher an effective economics instructor. We consider two main possibilities: subject matter knowledge and attitudes towards teaching economics. The innovative feature is to analyze both possibilities simultaneously with good measures of both teacher knowledge and teacher attitude. We separate the two effects and compare the importance of each.

Attitudes and knowledge are highly correlated. For example, Schober (1984) suggests that teacher achievement affects teacher opinions about economics. Allgood and Walstad (1999), albeit with a small sample of 12 teachers, convincingly demonstrate that a teacher's economic knowledge or a teacher's economic thinking positively affects student performance. Marlin (1991) finds that students with teachers more enthusiastic about teaching economics score higher on measures of economic knowledge. Although Marlin controls for teacher experience in teaching economics, he is unable to control for teacher knowledge. Given the correlation in Schober (1984) between teacher attitudes and achievement, one could easily interpret Marlin's estimates on teacher attitudes as an effect of teacher knowledge on student achievement. Relatedly, Boex (2000) provides evidence that college instructors' presentation skills, which includes enthusiasm for the subject, are more important than their intellectual or scholarly nature in affecting student ratings of teacher effectiveness. Untangling the two effects requires good information on teacher knowledge, teacher attitudes, and student learning.

To test these relationships directly, we survey high school economics teachers and their students. Focusing on high school students is important in two ways. First, seventeen states include economics as part of the high school curriculum (NCEE, 2005). Second, the more economic knowledge students have, the more likely they are to study economics in college (Ashworth and Evans, 2001).

Teachers were assessed on their knowledge of and attitude towards economics. Teachers tested their students at the beginning and end of an economics course to provide a measure of student learning. We then estimate an empirical model of student learning on teacher knowledge and teacher attitudes. We find that better teacher knowledge consistently improves student learning. Teacher attitude, as measured by a survey, has a statistically insignificant effect on student learning. We explore an alternate measure of teacher attitude towards economics: whether the teacher volunteered to teach the class. This revealed preference measure of teacher attitude has a positive and statistically significant effect on student learning. The effect size is similar to that of teacher knowledge.

EMPIRICAL METHOD

We measure teacher knowledge and teacher attitudes towards economics using standard assessments. We then test how teacher attitudes and teacher knowledge affect student learning. We estimate Ordinary Least Squares (OLS) regressions of the following for student i in class j:

[post-test.sub.ij] - [pre-test.sub.ij] = [gamma][knowledge.sub.j] + [phi][attitude.sub.j] + [error.sub.ij]

Our variables of interest are teacher knowledge and attitude. We expect both [gamma] and [phi] to be positive and are interested in their relative magnitudes. To assess relative magnitudes, we standardize the measures of teacher knowledge and attitude to variables with a zero mean and a variance of one. The coefficients then reflect the effect of a one standard deviation change in that teacher attribute.

Much of the discussion on specification of economic learning models focuses on econometric difficulties in regressing the change in test scores (post-test minus pre-test) on various attributes versus regressing the post-test score on the pretest score and various attributes (see Becker, et al, 1990 for example). Generally, the change score method is preferred; we follow this preferred approach.

Student gain scores are measured at the student level; teacher characteristics are measured at the teacher level. The regressions thus include a large number of observations for each teacher, exaggerating the amount of variation occurring in the teacher characteristics. To account for the correlation among students in the teacher characteristics, we cluster the standard errors by teacher. These regressions are similar to regressions using teacher-level data, weighted by the number of student observations.

DATA COLLECTION AND VARIABLE MEASUREMENT

We sent information packets to South Carolina teachers. We supplemented a list of economics teachers provided by the South Carolina Council on Economic Education with a search for social studies teachers at every high school in the state. In total, we mailed out 468 surveys to South Carolina teachers in April 2006. A second round of information packets were mailed in late July 2006 to 394 teachers. A total of 52 teachers agreed to participate; 41 of these fully completed their participation. Some teachers participated in more than one quarter or semester of classes. Teachers were compensated $100 upon completion of participation; some teachers were compensated more than once if they participated in more than one quarter or semester.

Teachers completed the Survey of Economic Attitudes (SEA) and the Test of Economic Literacy (TEL) at the beginning of their economics course. The TEL is a nationally-normed high school level assessment that provides a pre- and a posttest. Teachers administered Form A of the TEL to their students at or near the beginning of the course; they administered Form B of the TEL to their students at or near the end of the course. These two tests provide a measure of student learning during the course. Teachers also completed a demographic survey.

The first part of the SEA, the Attitudes towards Economics (ATE) section, measures teacher attitudes towards economics using fourteen questions with responses on a five-point Likert-type scale. These questions include such items as "Economics is dull" and "I would be willing to attend a lecture by an economist". Respondents note whether they strongly agree, agree, are undecided, disagree, or strongly disagree. Soper and Walstad (1983) provide reliability statistics for the ATE. The Cronbach Alpha of 0.88 for the ATE demonstrates good internal consistency; Soper and Walstad (1983) report a low standard error of measurement of 3.18.

Both assessments, the ATE and the TEL, are designed by the National Council on Economic Education (NCEE). They are used frequently in student and teacher assessment and research into student achievement (see, for example, Dutkowsky, Evensky and Edmonds, 2006; Walstad and Rebeck, 2001; or the summary in Becker, Greene, and Rosen, 1990). The raw scores for the teachers are presented in Table 1.

On average, teachers preformed quite well on the TEL averaging 35 questions right out of forty. There is a large variance in teachers' performance ranging from 16, below the average for students who've taken some economics, to a perfect score. The national average score on the TEL Form A for those without economics is about 19 out of 40; the average score with economics is about 25 out of 40 (Walstad and Rebeck, 2001).

For positive statements about economics the ATE is scored as follows: if a teacher strongly agrees with the statement, we assign that answer a 5; agrees receives a 4; undecided, 3; disagrees, 2; and strongly disagrees, 1. For negative statements, strongly disagrees receives a 5 and so on. Possible scores on the ATE range from 14 to 70. Teachers averaged a score of 60 ranging from a low of 42 to the maximum of 70.

We consider an alternate measure of teacher attitudes: how teachers are assigned to economics classes. (1) About half of the teachers in our sample (56%) report choosing to teach economics. The principal or department head assigned the remaining teachers to the economics course. Teachers' revealed preference in their choice to teach economics may better reflect their enthusiasm in the classroom, as compared to the survey-based measure of their attitude towards economics.

The previous research on teacher attitudes focuses on enthusiasm. Marlin (1991), for example, used a three-choice rating of teacher enthusiasm for teaching economics in his study of teacher attitudes. Patrick, Hisley, and Kempler (2000) discuss how more motivated teachers may be more effective at eliciting student motivation and thus, student learning. In a psychology experiment, they manipulate teacher enthusiasm and show that students lectured by more enthusiastic teachers are more interested in learning more about the topic.

Choosing to teach economics is positively correlated with teacher scores on the ATE. Teachers choosing to teach economics scored 62 on the ATE as compared to a score of 57.4 for those assigned to teach economics. This reflects a little more than one-half of a standard deviation change in the ATE.

Table 1 also presents summary statistics of the sampled teachers' characteristics. On average, each teacher submitted scores for 47 students. Many teachers submitted scores for more than one class during a semester or for more than one class during the school year. The average teacher is aged 46 with 18 years of experience teaching and 10 years of experience teaching economics. Teachers in the sample are 87.8 percent white and 44 percent male. 78 percent of the teachers have a Master's degree; one teacher has earned his doctorate.

The sampled teachers are somewhat comparable to an average social studies teacher in South Carolina (authors' calculations from the Schools and Staffing Survey, 2003-2004). On average, South Carolina social studies teachers have taught in public schools for 13.9 years and are 41 years old. Among these teachers, 88.8 percent are white and 58.3 percent male. 56.5 percent of South Carolina social studies teachers have a Master's degree.

Our sampled teachers are somewhat older, more educated, and more experienced than the relevant average teacher. Also, their high scores on both the ATE and TEL reflect the higher than average interest these teachers expressed by choosing to participate. We keep this revealed preference and sample selection in mind when considering our results.

Among our economics teachers, about 30 percent of them majored or minored in economics. Fully one-third have attended an NCEE or state training session. Again, this high rate of participation likely reflects the sampled teachers' interest in economics and willingness to participate. More than 40 percent use NCEE materials in their classroom.

For the 41 teachers in our sample, we standardize their scores on the ATE and TEL to be mean zero and variance one. This allows us to compare the magnitudes of the effects of these teacher qualities on student learning. Teachers used Form A of the TEL to assess students at the beginning of the course and Form B at the end of the course. We convert the Form A scores to a scale comparable to Form B as suggested by the TEL Examiner's Manual. The outcome of interest is student learning: the change in students' scores between the post-test and the pre-test. On the pre-test, students, on average, correctly answered 17 of the 40 questions; on the post-test, students, on average, correctly answered 21 of 40 questions. In our sample, both the pre-test and post-test scores are lower than the national averages. The national average on Form A of the TEL for students without economics is 19.05; the national average on Form B of the TEL for students with economics is 25.74 (Walstad and Rebeck, 2001). The average change in test scores for our sample was 3.89. The difference in the national averages is about 6.7 questions. The gain in test scores is also smaller than the difference between the national averages.

SPECIFICATION ISSUES

Other factors may affect student test scores. Becker, Greene, and Rosen (1990) enumerate the factors that may affect student learning of economics: student ability, teacher ability, course work, technology, demographics, and time usage. Females and blacks tend to perform worse than males and whites in college economics courses (Dynan and Rouse, 1997 and Borg and Stranahan, 2002). Class size may affect student performance (Arias and Walker, 2004). The evidence on the effects of instructor gender is mixed: Robb and Robb (1999) and Dynan and Rouse (1997) find no effect of gender on college student performance while Ashworth and Evans (2001) find that female secondary teachers increase the likelihood of studying economics in high school. Klopfenstein (2005) finds same-race effects for black math achievement.

Our limited sample of teachers precludes including many control variables including teacher characteristics. In addition, data limitations prevent us from including student characteristics such as race or sex. (2) For these exclusions to bias our estimates, the omitted variables must be correlated with teacher knowledge or teacher attitude. For example, if better prepared or more positive instructors are matched with students who learn more quickly, this would bias our estimates upwards.

We check for indications of this bias although the limited sample size makes it difficult to control for those factors. We split the sample at the median of the teachers' TEL scores. The average student's pre-test score for the top half of teachers is 18.62 versus 16.03 for the average student of a teacher from the bottom half. Students with higher pre-test scores tend to be taught by higher scoring teachers. Students with higher pre-test scores also tend to be taught by teachers with higher ATE scores. For a teacher scoring in the top half of teacher's attitudes, the average student pre-test score is 17.80; for a teacher in the bottom half, the average is 16.95.

If students that would have learned more regardless of their teacher are matched with higher quality teachers, this selection would bias the effects of teacher quality upwards. In this case, we observe that students with higher pre-test scores learn less, on average, than students with lower pre-test scores. The above median student experienced test scores gains of 2.6 points; the below median student gained 5.4 points. This difference isn't driven by ceiling effects; less than one percent of students correctly answered 38 or more of the 40 questions on the post-test.

Since high pre-test students are matched with higher knowledge teachers and high pre-test students have smaller growth in their test scores, this suggests that teacher-student matching may bias downwards the estimated teacher knowledge coefficients. This makes it more difficult to find effects of teacher knowledge on student learning. Similarly, teacher-student matching may bias downwards the estimated teacher attitude coefficients.

Higher pre-test scoring students tend to be matched to teachers who did not volunteer to teach economics, although this difference is not statistically significant. The average student pre-test score for a teacher choosing to teach economics is 17.32; the average student pre-test score for a teacher assigned to teach economics is 17.67. As lower pre-test students tend to have higher gain scores and are matched with assigned teachers, this suggests a potential downward bias in the coefficient estimate on whether a teacher choose to teach economics.

RESULTS

Figure 1a presents mean changes in student test scores by teacher scores on the TEL and ATE. Teachers with below median scores on the TEL taught students with smaller gains in test scores; this difference is statistically significant. Among the lower knowledge teachers, having a higher score on the ATE does not correlate with higher student test score gains. Among the higher knowledge teachers, a higher score on the ATE correlates with somewhat high student test score gains.

Figure 1b uses whether a teacher volunteered to teach economics as the measure of teacher attitude. Teachers choosing to teach economics taught students with significantly larger gains in test scores; this difference is statistically significant. This difference is particularly large for the lower knowledge teachers. However, for both the below and above median knowledge teachers, choosing to teach economics corresponds to statistically significantly higher test score gains.

We estimate a regression of the change in student test scores on the teachers' standardized TEL and ATE scores. Regressions include 1,946 students in 41 teachers' classes. Table 2 presents the results.

The first column presents the estimates of changes in test scores on the standardize measures of teacher knowledge and teacher attitude (from the ATE). This column echoes the pattern shown in Figure 1a. Students taught by a teacher with economic knowledge one standard deviation above the mean experience an additional increase in their scores of a little less than one question. This reflects about a 19 percent increase in their test score growth. Controlling for teacher knowledge, teacher attitude as measured by the ATE has a small and statistically insignificant effect on student test scores. The effect of teacher knowledge is about twice as large as that of teacher attitude. These estimates support previous research showing that teacher knowledge increases student learning. The existing evidence on teacher attitudes is thin; our evidence merely corroborates its thinness. Given the potential upward bias on teacher attitude, there seems to be little effect of teacher attitude, as measured by the ATE, on student learning. We focus on the other measure of teacher attitude in the remaining regressions.

The second column presents the estimates of changes in test scores on the teacher's score on the TEL and an indicator variable for whether the teacher chose to teach economics. The second column echoes the pattern shown in Figure 1b. Teachers with more knowledge correlate with increased student learning; teachers choosing to teach economics also correlates with increased student learning. The effect of a teacher volunteering is large, amounting to about 38 percent of average student test score gains. Both teacher characteristics have a statistically significant and economically relevant effect on student test score gains.

The third column adds two teacher demographic variables to the specification: an indicator for whether the teacher is male and one for whether the teacher is white. Male teachers experience significantly lower gains than female teachers with their students gaining, on average, one less question from pre-test to post-test. The coefficient on the white dummy variable is small, positive, and statistically insignificant. Further, including these demographics has little effect on the estimated coefficients on teacher knowledge or whether the teacher chose to teach economics.

We include teacher experience in column (4). Teachers with one more year of experience teaching economics have students with slightly higher test score gains; the estimate is small, less than a tenth of a question, and statistically insignificant.

The last three columns of Table 3 include controls that may reflect teacher knowledge or teacher interest in economics. In column (5) we include an indicator variable for whether the teacher was an economics major or minor in college. Economics majors or minors instruct students with smaller test score gains, although the difference is not statistically significant. Column (6) adds an indicator variable for whether the teacher has attended a NCEE or state training session. Column (7) includes an indicator for whether the teacher uses NCEE materials in his or her class. The coefficient on the training session is statistically insignificant and changes sign when we include the materials indicator variable. The students of teachers using NCEE materials experience much greater test score gains.

Including the last set of variables, particularly the use of NCEE materials, reduces both the magnitude and the significance of the estimated coefficient on whether the teacher chose to teach economics. These two variables are highly correlated ([rho] = 0.5342): teachers who chose to teach economics also typically use NCEE materials in their classroom. (3) Further, including these variables reflecting economics training reduces the magnitude and the significance of the male variable; any effect of teacher gender appears to be driven by their varying backgrounds in economics.

CONCLUSIONS AND POLICY IMPLICATIONS

We find that teacher knowledge of economics is an important determinant of student learning. Teacher attitude towards economics, as measured by the first part of the Survey of Economic Attitudes, has a small and statistically insignificant effect on student learning. However, teachers choosing to teach economics experience greater student test score gains. These volunteering teachers likely exhibit more enthusiasm for the class. The effect of teachers choosing to teach economics and teacher knowledge of economics are similarly sized and statistically significant.

These results suggest two things. First, that much emphasis, correctly, has been placed on improving teacher economic knowledge and providing teachers with materials for their economics courses. We encourage the continued development of materials and interventions focused on improving teacher understanding of economics. In fact, the use of NCEE's materials appears to improve student learning. Second, reconsidering how teachers are assigned to economics classes may alleviate some of their anecdotal distress and improve student learning in the process. To the extent allowed by resource constraints, our results suggest that allowing teachers increased control over the classes taught could improve student outcomes.

ACKNOWLEDGMENTS

This research was funded in part by an Excellence in Economic Education grant from the National Council on Economic Education. We thank Jeff Gibisch, Irina Ionescu, Hillary Morgan, Christy Redmon, and Aileen Sampson for their research assistance. We also thank Andrew I. Kohen and participants in the NCEE session at the 2008 ASSA meetings in New Orleans for their comments. All errors are, of course, our own.

REFERENCES

Allgood, S. & W.B. Walstad (1999). The longitudinal effects of economic education on teachers and their students. The Journal of Economic Education, 30(2), 99-111.

Anderson, G., D. Benjamin, & M.A. Fuss (1994). The determinants of success in university introductory economics courses. The Journal of Economic Education, 25(2), 99119.

Arias, J.J. & D.M. Walker (2004). Additional evidence on the relationship between class size and performance. The Journal of Economic Education. 35(4), 311-329.

Ashworth, J. & J.L. Evans (2001). Modeling student subject choice at secondary and tertiary level: A cross-section study. The Journal of Economic Education. 32(4), 311-320.

Becker, W., W. Greene, & S. Rosen (1990). Research on high school economic education. The Journal of Economic Education. 21(3), 231-245.

Boex, L.F.J. (2000). Attributes of effective economics instructors: An analysis of student evaluations. The Journal of Economic Education. 31(3), 211-227.

Borg, M.O. & H.A. Stanahan (2002). Personality type and student performance in upper-level economics courses: The importance of race and gender. The Journal of Economic Education. 33(1), 3-14.

Dutkowsky, D.H., J.M. Evensky, & G.S. Edmonds (2006). Teaching college economics in the high schools: The role of concurrent enrollment programs. The Journal of Economic Education. 37(4), 477-482.

Dynan, K. & C.E. Rouse (1997). The underrepresentation of women in economics: A study of undergraduate economics students. The Journal of Economic Education. 28(4), 350-368.

Klopfenstein, K. (2005). Beyond test scores: The impact of black teacher role models on rigorous math taking. Contemporary Economic Policy. 23, 416-428.

Schober, H.M. (1984). The effects of inservice training on participating teachers and students in their economics classes. The Journal of Economic Education. 15(4), 282-295.

Marlin, J.W. Jr. (1991). State-mandated economic education, teacher attitudes, and student learning. The Journal of Economic Education. 22(1), 5-14.

National Council on Economic Education. (2005). Survey of the states: Economic and personal finance education in our nation's schools in 2004. March.

Patrick, B.C., J. Hisley, & T. Kempler (2000). 'What's everybody so excited about?': The effects of teacher enthusiasm on student intrinsic motivation and vitality. Journal of Experimental Education. 68(3), 217-236.

Rivkin, S.G., E.A. Hanushek, & J.F. Kain (2005). Teachers, schools, and academic achievement. Econometrica. 73(2), 417-458.

Soper, J.C. & W.B. Walstad (1983). On measuring economic attitudes. The Journal of Economic Education. 14(4), 4-17.

Walstad, W.B. & K. Rebeck (2001). Test of Economic Literacy 3rd edition examiner's manual. National Council on Economic Education.

Walstad, W.B. & K. Rebeck (2001). Assessing the economic understanding of U.S. high school students. American Economic Review Papers and Proceedings. 91(2), 452-457.

Angela K. Dills, Mercer University

Dennis Placone, Clemson University and Center for Economic Education

ENDNOTES

(1) We thank Andrew I. Kohen for this suggestion.

(2) To satisfy the Institutional Review Board's Exemption category for human subjects research, we opted not to collect student-specific demographic information. A full IRB review outside the exempt categorization would require parental permission for each student, likely further limiting our sample size.

(3) Results are qualitatively similar when using the teachers' ATE scores instead of the teacher chose variable.

Table 1: Summary Statistics for Teachers and Students

 St.
 Mean Dev. Min Max
Teacher Characteristics
 (N=41)
TEL 35.02 6.03 16 40
ATE 59.95 7.54 42 70

Students taking economics 47.46 33.09 2 178
Teacher's Age 46.69 10.27 25 60
Years Teaching 18.14 10.77 2 38
Years Teaching Economics 10.31 6.85 0 26
Percent of teachers that:
 chose to teach economics 56.10%
are male 45.00%
are white 87.80%
have their highest
 degree as:
Masters 77.50%
Doctorate 2.50%
Student Characteristics
 (N=1,946)
DTEL 3.89 5.54 -16.511 23.729
pretest TEL 17.48 6.65 -0.009 36.711
posttest TEL 21.37 7.28 3 40

Table 2: Regression results of students' change in test scores on
teacher ability and attitude toward economics

 (1) (2) (3)

 students' change in test scores on TEL

teacher's TEL 0.753 * 0.655 * 0.684 *
(z-score)
 (1.79) (1.72) (1.71)
teacher's ATE 0.395
(z-score)
 (0.71)
teacher chose 1.477 ** 1.131 *
 (2.09) (1.69)
male teacher -1.165 *
 (-1.78)
white teacher 0.0681
 (0.07)
years teaching
economics

economics major
or minor?

Attended NCEE
or state training?

Use NCEE
materials?

Constant 3.713 *** 2.957 *** 3.613 ***
 (9.57) (7.08) (3.83)
R-squared 0.02 0.03 0.04

 (4) (5) (6)

 students' change in test scores on TEL

teacher's TEL 0.602 0.681 * 0.671 *
(z-score)
 (1.58) (1.72) (1.77)
teacher's ATE
(z-score)

teacher chose 1.110 * 1.202 * 1.169
 (1.70) (1.80) (1.68)
male teacher -1.142 * -0.765 -0.645
 (-1.78) (-1.17) (-0.85)
white teacher -0.0203 -0.755 -0.851
 (-0.019) (-0.67) (-0.76)
years teaching 0.0583 0.0561 0.0548
economics
 (1.30) (1.30) (1.28)
economics major -0.906 -0.933
or minor?
 (-1.30) (-1.31)
Attended NCEE 0.39
or state training?
 (0.43)
Use NCEE
materials?

Constant 3.039 ** 3.717 *** 3.667 ***
 (2.67) (3.28) (3.42)
R-squared 0.05 0.05 0.05

 (7)

 students' change
 in test scores on TEL

teacher's TEL 0.645 *
(z-score)
 (1.80)
teacher's ATE
(z-score)

teacher chose 0.223
 (0.43)
male teacher -0.392
 (-0.59)
white teacher -1.013
 (-0.87)
years teaching 0.0734 **
economics
 (2.03)
economics major -1.264
or minor?
 (-1.56)
Attended NCEE -0.518
or state training?
 (-0.71)
Use NCEE 2.266 ***
materials?
 (3.17)
Constant 3.530 ***
 (3.12)
R-squared 0.07

Robust t statistics in parentheses. Standard errors clustered
by teacher. * significant at 10%; ** significant at 5%;
*** significant at 1%. There are 1,946 students in the sample
and 41 teachers.

Figure 1a: Mean changes in test score by teacher characteristics

 above median ATE above median TEL

below median TEL 3.16 3.19
below median ATE 4.14 4.64

Note: Table made from bar graph.

Figure 1b: Mean changes in test score by teacher characteristics

 above median ATE above median TEL

below median TEL 2.07 4.57
below median ATE 4.03 4.69

Note: Table made from bar graph.