Avoiding bad teaching evaluations: Tricks of the trade

Standard

Student evaluations are the main method used to evaluate our teaching. These evaluations are, at best, an imperfect measuring tool.

Lots of irrelevant stuff affects evaluation scores. If you’re attractive or well dressed, this helps your scores. If you are a younger woman, you have to reckon with a distinct set of challenges and biases. If the weather is better out, you might get better evaluations, too. So, don’t feel bad about doing things to help your scores, even if they aren’t connected to teaching quality.

My university aptly calls these forms by their acronym, “PTE”: Perceived Teaching Effectiveness. Note the word: “perceived.” Actual effectiveness is moot.

People are aware whether or not they learned. However, superficial things can really affect perception. What our students think about the classroom experience is important. But evaluation forms are not really measuring teaching effectiveness. These evaluations measure student satisfaction more than learning outcomes. Since we are being held accountable for classroom performance based on student satisfaction, it is in our interest to pay attention to the things that can improve satisfaction.

Here are some ways I’ve approached evaluations with an effort to avoid getting bad ones.

  • I try to teach effectively. The best foundation of perception is reality. I put some trust in my students’ ability to assess performance. If I’m doing a good job, my students should know it.
  • I work hard to demonstrate that I respect my students. It’s easy to give in to the conceit that my time is more valuable than the time of my students. When I see myself going down that dark pathway, I try to follow the golden rule, and treat the time of my students with as much concern as I would like my own time to be treated. For example, I make sure class always ends on time.
  • I emphasize fairness. On the first day of class, I let students know that life isn’t fair, but I try hard to make sure that my class done as fairly as possible. Students often volunteer gripes about their other classes, and unfairness is always the common thread in these discussions. Even if students perform poorly in a class, if they think that it was conducted fairly, then they are still usually satisfied.
  • I recall Hanlon’s Razor: “Never ascribe to malice that which is adequately explained by incompetence.” None of my students are out to get me. Ideally, they’re out for themselves. Sometimes, I’m not clear enough about expectations. When a student needs something, I approach the interaction with the default assumption that it’s my fault. And if it’s not my fault, it’s not an intentional flaw, so I can’t give students a bad time about the shortcoming.
  • I don’t engage in debates about graded assignments. I tell my students that if there is a very simple mathematical error or something I missed, they can bring it to me immediately after class. Any other errors need to be addressed with a written request by the start of the next class meeting. I’ve only gotten a few of these, and in all cases, the students were correct.
  • When a student is persistent about points, I avoid the argument whenever possible. I don’t concede unearned credit, but I don’t dismiss the concern either. Nearly all requests for grade changes are so tiny, they have a negligible on the final grade. I show, numberwise, it doesn’t seem to make a difference. I tell them that if they are right on the borderline at the end of the semester, I’ll make a note of it and we could talk about it at that time. This prevents the student from waging a futile argument, and keeps me out of the business of catering to minutia.
  • I run a tight ship. I can get annoyed by inappropriate behavior, but the students are usually even more annoyed. When someone is facebooking in the front row or monopolizes discussion, the rest of the class is usually super-pleased that I shut it down, as long I do it with respect. Classroom management is a fine art that we are rarely taught. (I’ve learned some education faculty and K-12 teachers.) I think establishing the classroom environment in the first few days is critical. I don’t enforce rules, but I develop accepted norms of behavior collaboratively on the first day of class. When things happen outside the norm, I address them promptly and, I hope, gently. When anybody (including myself) is found to be outside the norm, we adjust quickly because we agreed to the guidelines on the first day of class. I’ve botched this and have been seen as too severe on occasion, but I’d prefer to err on that side then having an overly permissive environment in which students don’t give one another the respect of their attention.
  • A classic strategy is to start out the term with extreme rigor, and lessen up as time goes on. I don’t do this, at least not intentionally, but I don’t think it’s a bad idea as long as you finish with high expectations. In any circumstance, I imagine it would be disaster to increase the perceived level of difficulty during the term.
  • I use midterm evaluations, using the university form partway through the semester, for my own use. This gives me early evidence about perceptions with the opportunity to change course, if necessary. I am open and transparent about changes I make.
  • I often use a supplemental evaluation form at the end of the term. There are two competing functions of the evaluation. The first is to give you feedback for course improvement, and the second is to assess performance. What the students might think is constructive feedback might be seen as a negative critique by those not in the classroom. It’s in our interest to separate those two functions onto separate pieces of paper. Before we went digital, I used to hold up the university form and say: “This form [holding up the scantron] is being used by the school as a referendum on my continued employment. I won’t be able to access these forms until after the next semester already starts, so they won’t help me out that much.” Then I held up another piece of paper [an evaluation I wrote with specific questions about the course] and said, “This one is constructive feedback about what you liked and didn’t like about the course. If you have criticisms of the course that you want me to see, but don’t think that my bosses need to see them, then this is the place to do it. Note that this form has specific questions about our readings, homework, tests and lessons. I’m just collecting these for myself, and I’d prefer if you don’t put your names on them.” I find that students are far more likely to evaluate my teaching in broad strokes in the university form when I use this approach, and there are fewer little nitpicky negative comments.
  • I try to avoid doing evaluations when students are more anxious about their grade, like on the cusp of an exam or when I return graded assignments. When I hand out the very helpful final exam review sheet, which causes relief, then I might do evaluations.
  • I don’t bring in special treats on the day I administer evaluations. At least with my style, my students would find it cloying, and they wouldn’t appreciate a cheap bribe attempt. Once in a long while, I may bring in donuts or something else like that, but never on evaluation day.
  • I’ve had some sections in with chronic attendance problems, in which some students would skip or show up late. On those occasions, I made a point to administer evaluations at the start of class on a day that had low attendance. I imagined that the students who weren’t bothering to attend class were less likely to give a stellar rating. Moreover, the absent students weren’t as well qualified to evaluate my performance as those sitting in class. (Of course, those attendance problems indicated that I had a bigger problem on my hands.)
  • Being likable and approachable. Among all the things that influence evaluations, I think this is the biggest one. There are many ways to be liked by your students, as a human being, but I think being liked is prerequisite to really good scores. Especially with our students who face a lot of structural disadvantages, approachability is important for the ability to do the job well. I’m not successful enough on this front. It hasn’t tanked me in evaluations, because by the end of the semester the students are comfortable with me, but that doesn’t emerge as quickly as I’d like. This is the area I need to work on the most. I am to do all the professorly things with students with the greatest needs, they need to be able to talk to me.

Of course, some of these tips don’t apply if the evaluations are being administered online. This is a growing trend, and my university made the switch a couple years ago. (Thoughts and experiences with paper vs. online evaluations are in the ever-growing queue for future posts.)

Are there different or additional approaches that you use for the non-teaching-performance related aspects of student evaluations?

[update: be sure to read this comment. I think everything in this post is relevant to professors of both genders, but there are additional issues involving student biases that female professors need to deal with that I haven’t addressed. Professors need to be approachable to do their jobs. If students can’t talk to us, then that puts a low ceiling on what we can help our students achieve. However, what it means to be professional and “approachable” for a younger female professor might look really different than for an older guy. As I don’t have experience being a younger female professor, I’m not as well qualified to address this as some others. Another good reason to cruise over to Tenure, She Wrote.]

What happens when you don’t know anything about the subject you’re teaching?

Standard
Biologie & Anatomie & Mensch, via Wikimedia commons

Biologie & Anatomie & Mensch, via Wikimedia commons

Like many grad students in Ecology and Evolutionary Biology, I my made a living through grad school as a TA.

One semester, there were open positions in the Human Anatomy cadaver lab. I was foolish enough to allow myself to be assigned to this course. What were my qualifications for teaching an upper-division human anatomy laboratory? I took comparative vertebrate anatomy in college four years earlier. We dissected cats. I barely got a C.

You can imagine what a cadaver lab might be like. The point of the lab was to memorize lots of parts, as well as the parts to which those parts were connected. More happened in lecture, I guess, but in lab nearly the entire grade for students was generated from practical quizzes and exams. These assessments consisted of a series of labeled pins in cadavers.

My job was to work with the students so that they knew all the parts for quizzes and exams. (You might think that memorizing the names of parts is dumb, when you could just look them up in a book. But if you’re getting trained for a career in the health sciences, knowing exactly the names of all these parts and what they are connected to is actually a fundamental part of the job, and not too different from knowing vocabulary as a part of a foreign language.)

The hard part about teaching this class is: once you look inside a human being, we’ve got a helluva lotta parts, all of which have names. I was studying the biogeography of ants. Some of the other grad student TAs spent a huge amount of time prepping, to learn the content that we were teaching each week. Either I didn’t have the time, or didn’t choose to make the time. I also discovered that the odors of the preservatives gave me headaches, even when everything was ventilated properly. Regardless of the excuse that I can invent a posteriori, the bottom line is that I knew far less course material than was expected of the students.

Boy howdy, did I blow it that semester! At the end, my evaluation scores were in the basement. Most of the students thought I sucked. The reason that they thought I sucked is because I sucked. What would you think if you asked your instructor a basic question, like “Is this the Palmaris Longus or the Flexor Carpi Ulnaris?” and your instructor says:

I don’t know? Maybe you should look it up? Let’s figure out what page it is in the book?

The whole point of the lab was for students to learn where all the parts were and what they were called. And I didn’t know how to find the parts and didn’t know the names. I lacked confidence, and my students were far more interested in the subject. It was clear to the students that I didn’t invest the time in doing what was necessary to teach well. They could tell, correctly, that I had higher priorities.

Even though students were in separate lab sections, a big chunk of the grade was based on a single comprehensive practical exam that was administered to all lab sections by the lecture instructor. Even though I taught them all semester – or didn’t teach them at all – their total performance was measured against all other students, including those who were lucky enough to be in other lab sections taught by anatomy groupies. Even I at the time realized that my students drew the short straw.

One of my sections did okay, and was just above the average lab section. The other section – the first of the two – had the best score among all of the lab sections! My students, with the poor excuse of an ignoramus instructor, kicked the butts of all other sections. These are the very same students that gave me the most pathetic evaluation scores of all time. They aced the frickin’ final exam.

What the hell happened?

I inadvertently was using a so-called “best practice” called inquiry-based instruction. That semester, I taught the students nothing, and that’s why they learned.

Now, I know even less human anatomy than I did back then. (I remember the Palmaris Longus, though, because mine is missing.) I bet my students would learn even more now than mine did then, and I also bet that I’d get pretty good evals, to boot. Why is that?

I’d teach the same way I taught back then, but this time around, I’d do it with confidence. If a student asked me to tell the difference between the location of muscle A and muscle B, I’d say:

I don’t know. You should look it up. Find it in the book and let me know when you’ve figured it out.

The only difference between the hypothetical now, and the actual then, is confidence. Of course, there’s no way in heck that I’ll ever be assigned to teach human anatomy again, because the instructors really should have far greater mastery than the students. In this particular lab, I don’t think mastery by the instructor really mattered, as the instructor only needed to tell the students what they needed to know, and the memorization required very little guidance. (For Bloom’s taxonomy people this was all straight-up basic “knowledge.”)

I do not recommend having an ignorant professor teach a course. If a class requires anything more than memorizing a bunch of stuff, then, obviously, the instructor needs to know a lot more than the students. Aside from a laboratory in anatomy, few if any other labs require (or should require) only straight-up memorization of knowledge. Creating the most effective paths for discovery requires an intimate knowledge of the material, especially when working with underprepared students.

For contrasting example, when I’ve taught about the diversity, morphology and evolutionary history of animals, I tell my students the same amount of detail that I told my anatomy students back then: nothing. I provide a framework for learning, and it’s their job to sort it out. If a students asks about the differences between an annelid and a nemadote, I refrain from busting into hours of lecture. But I don’t just lead them to specimens and a book. I need to provide additional lines of inquiry that put their question into context. It’s not just memorizing a muscle. In this case, it’s about learning bigger concepts about evolutionary history and how we study attempt to reconstruct the evolutionary trees of life. I ask them to make specific comparisons and I ask leading questions to make sure that they’re considering certain concepts as they conduct their inquiry. That takes expertise and content knowledge on my part.

To answer the non-rhetorical question that is the title of this post, then I guess the answer is: It will be a disaster.

But if you act with confidence and don’t misrepresent your mastery, then it might be possible to get by with not knowing so much and still have your students learn. Then again, if you’re teaching anything other than an anatomy lab that involves only strict memorization, I’d guess that both you and your students are probably up a creek if you don’t know your stuff.

The semester after was the TA for human anatomy, I taught Insect Biology lab. That was better for everybody.

Extra credit is unfair to students

Standard

The old joke goes like this:

Q: Why did the undergraduate cross the road?

A: Extra credit!

I’ve known scores of students who would work their butts off for five extra points when they wouldn’t work nearly as hard for a normal 100-point assignment. It’s disheartening to witness such irrational behavior. However, this isn’t why I don’t offer extra credit.

I don’t offer extra credit because it’s inherently unfair.

I treat my students professionally. I respect their time and I expect the same courtesy from my students. When professors decide on an extra credit assignment in the middle of the course, this looks to me like poor planning. Even if the possibility or certainty of extra credit is placed in the syllabus at the start of the semester, that doesn’t make it fair to everybody in the course.

Just because all students are given equal opportunities, doesn’t mean that they are being treated fairly.

When students sign up for our classes, they are expected to attend class at the scheduled times, and complete the studying and assignments outside of class, though not at any particular time because they have other courses, jobs, and private lives. The syllabus says what is in the class and why the class exists. If a professor adds additional stuff to the course, at some point through the semester, then this provides a disadvantage to the student who has more commitments outside of regular class hours.

Maybe there are some ways that extra credit is used that is fair to everyone. I can’t think of any. If there is a regular part of the course that is used for points above 100%, that’s not extra credit, that’s just spreadsheet voodoo. Extra credit, as I consider it, is when students are given a chance to earn extra points by doing some stuff that is outside the typical curriculum of a course, or is scheduled outside class hours, or is connected to performance on an assignment in the middle of the semester.

Let me address different reasons that people might use extra credit, and why I view these reasons as unjustified:

1. Extra credit is a carrot to get students to do favors for their institutions. The most common one that I’ve seen that students get extra credit for attending seminars by visiting speakers. This drummed-up audience prevents anybody from being embarrassed by paltry attendance. This practice is manipulative, and doesn’t show adequate respect for time of students. Moreover, because not everyone may have equal availability to earn such extra credit, this gives some students an opportunity to earn more points than other students. (Assigning written assignments to students who cannot attend an extracurricular event to earn extra credit is punitive.) If students need to attend seminars for their courses, then this needs to be built into the course and scheduled during class hours, or placed in the course description. It’s not right to reward the students who have enough spare time to attend events while others might be working or have other commitments.

2. Extra credit is an opportunity for students to earn additional points if their exam scores were particularly low. I have seen some professors give students extra work, including an opportunity to revise exams, in order to improve their scores on exams. If a professor doesn’t like the mean score on an exam, the proper course of action is to give everybody a boost. If most students in the class performed below expectations, then offering extra credit to everybody is relatively punitive to the students who did perform higher than their peers. Some students did better than other students on an exam for a reason. To respect all of your students, honor those reasons and look to the future when students tank an exam. (For edu-folks: exams are summative assessments. Keep it that way.) The only way students should have a chance to revise an assignment or exam for additional credit is if it was structured that way in the first place and the students were aware of this policy at the outset. Anything else is unfair to those who did their best at the start.

3. Extra credit is assigned to motivate students. If students aren’t working hard enough, and extra credit is the incentive, then I humbly suggest that there is a suite of pedagogical approaches that will increase student effort and engagement that don’t involve the inherent unfairness in extra credit. Extra credit encourages students to obsess over their scores rather than focus on the content of the course. If you have students jump through hoops to get a higher grade than they think they would otherwise be getting, then how does this help them learn?

4. Extra credit keeps students happier. I’m doubt this is true. Does extra credit help professors out by boosting their evaluations? I’m not aware of any evidence along these lines and my anecdotal observations suggest that some students are aware that extra credit is manipulative. Even if extra credit would pacify some otherwise unhappy students, priority should be placed on fairness.

5. Extra credit is assigned because the professor overestimated or underestimated the difficulty of the curriculum. If students are underperforming because the course was harder than the professor intended, then the scale should be shifted. If students are overperforming and extra credit is required to give students enough material for learning, then other curricular changes within the bounds of the course should be implemented.

6. Extra credit is assigned to engage students with the community. If student involvement in the community through some extracurricular activity (such as a beach cleanup, or volunteer tutoring at a local elementary school) is desired by the professor, then it should be built into the required curriculum. It’s acceptable to integrate service learning in all kinds of courses. If you don’t want to require it, but want to provide the option, then you could make this activity one of a variety of things that are worth equal required points, or you could offer the possibility without giving student an academic reward for extracurricular activities.

Don’t get me wrong – I’m all in favor of an instructor calling an audible during the semester to change up all kinds of things. I often have a large amount of points to ‘homework and in class assignments.’ I often don’t know exactly what those are going to be when I start the semester. However, I’m not going to give extra points on top of those assignments. It’s simply unfair and doesn’t respect my students’ time.

When students come see me about extra credit during the semester, I explain that I don’t give extra credit because it’s unfair to students who have their time budgeted to other activities and to those who were able to perform well consistently throughout the semester. Nobody’s argued with any substance, other than, “Are you sure?” Yes, I am sure.

One strategic reason to be clear about not offering extra credit is that some students accustomed to the practice might not try hard to learn the course material in the first part of the course, hoping that extra credit might bail them out in the end. By not having extra credit, and making sure this is well known, then you might get a higher investment throughout the whole semester.

The professor-student relationship is structured by the power that the professor has over the student. By coming up with (seemingly) capricious ways to increase student scores throughout the semester, this looks like an abuse of that power to make things easier for the professor and (seemingly) harder on the students who don’t need the extra credit.

If you are providing a carrot to some students, then those who aren’t able to eat or fully digest the carrot will then see extra credit as a stick.  When I start my classes each semester, I tell my students: “The world isn’t fair. But in this classroom, I place a high value on endeavoring to be as fair as possible.” If I offered extra credit, then I’d be undermining that notion.

Many of my students work long hours outside of school, in addition to a full course load, and they also have families to care for. I’m not going to ask anything more of them other than what was in the course catalog and what I made very clear in the course syllabus. Even if I taught a bunch of students on a residential campus, who did not have major family obligations including a paying job, I still feel that extra credit would be an unprofessional manipulation that wouldn’t fairly treat those who did their best throughout the course.

What really is “excellence” in teaching ?

Standard

I’d like to extend a topic that I brought up recently – about the difficulty in evaluating excellence in teaching.

I danced around the central conundrum, by highlighting all the ways that we do, and don’t, evaluate teaching on our campuses. What I didn’t bring up is: what is it, exactly, that we are sizing up when we decide how good someone’s teaching is?

What I argued at the time is that most people use the template of their own teaching (or perhaps one’s aspirations for one’s own teaching) as a model for evaluating others. This operational definition of how excellence is measured doesn’t actually specify what characteristics define excellence.

Let’s set aside the evaluation problem, and ask: What is excellent teaching?

Nearly all of our evaluations focus on the process of teaching: what happens in class on a day to day basis.

What really matters is what happens in the end: what is the product of teaching?

If we focus on process, then we measure things like specific items in the curriculum, demeanor, what the assignments and exams look like, and classroom performance in the theatrical sense.

If we focus on product, then we look at outcomes: exam performance, student satisfaction, long-term professional growth, and — from the administration’s perspective — alumni giving.

You imagine that a quality process leads to a quality product, however a veneer of quality might not lead to a long-term high quality product. What educational practices that get high ranks in the evaluation system actually contribute to positive outcomes in the long term? How much of what we do in teaching doesn’t result in long-term learning but looks like quality teaching in the short term? For starters, the first thing that comes to mind is the use of overly detailed powerpoints that are shared with the class online for cramming studying for exams. Students love it, it raises satisfaction, the professor looks well prepared and: will students know any of it the next day, or how about five years later?

Just as we can design an individual class with backward planning, we can think of the whole college experience this way to inform what students really need to get what we want them to get out of the college experience. What constitutes an “excellent” college education? Whatever it is, then excellence in faculty is when they deliver on that goal.

In my view, when you graduate from college, the school has done its job if the product is:

  • articulate in thought, words and writing
  • able to differentiate between opinion and reason and has personal values informed by both
  • broadly interested in the world and all it has to offer
  • conversant in literature, art, history, geography, science, mathematics, philosophy, civics and more
  • able to separate style from substance
  • struggling to understand the faiths, or the lack thereof, of those in their midst
  • able to readily explain to any novice the basic tenets of their specialty
  • can perform admirably on a summative exam in the subject of one’s specialty (e.g., subject GRE)
  • reading the goddamn newspaper or its equivalent on a regular basis
  • able to whiff out pseudoacademic thinking (like Malcom Gladwell), pseudoscience and shabby reasoning
  • not taking oneself too seriously
  • treats other people as they would like to be treated

Those are my aspirations for what college should do for everyone. I wrote this on the fly but I imagine it would resemble to some degree what you think a “liberal arts education” is supposed to look like.

Just as it’s not the job of a K-12 teacher to merely teach the state standards, it’s not my job to cover the expected learning outcomes of my course. I’m working on providing one piece of a holistic college experience. Am I successful at this? No. But it’s a goal. Since I’m at a state comprehensive institution and all the talk is “career-training-this” and “job-preparation-that” we are really botching these fundamentals. My heart soared when our newish provost said, in his first address to the faculty, that he didn’t want to see any more business regarding what a college degree does for students, he wanted to hear more about what a college education does for students. In our environment, that idea is hard to deliver in deed, but I’m excited to see the emphasis nonetheless.

How do we measure teaching excellence? Foremost, students need to emerge from the class knowing the course material. In particular, they need to know it deeply: well beyond their graduation dates. Truly excellent teachers inspire lifelong learning, excitement tied to discovery in the discipline, and an academic ethos that pervades all aspects of life. There is a multiplicity of routes to this destination.

This is often why teaching is considered to be an art, because what works for one person might not work for another. The trickier part, though, is knowing what works. This might not be measurable at the end of a semester.

What it takes to get tenure: ambiguity of the teaching criterion

Standard

Getting tenure at a teaching university might be harder than getting tenure at a research institution.

If you don’t like that concept, then try this similar concept on: what you need to do to get tenure at a teaching-centered institution is far more ambiguous than what you need to do at a research university. One could argue that it’s easier to get tenure if you know specifically what you need to do. At most teaching schools, exactly what you need to do to get tenure is vague at best.

In one, you need to convince the faculty and administration of a teaching university that you are excellent at teaching. In the other, you need to convince the faculty and administration of a research university that you are excellent at research.

At research institutions, when you interview for a job, it is typically spelled out exactly what you need to do to get tenure: grants, publications, and train doctoral students. At most places, you’re given a neighborhood of a dollar amount, or a certain set of grant agencies and number of grants you need, and a number of publications in journals with a certain tier, as first and senior author. There may be some subtleties, but when you’re coming up for tenure, it’s clear based on the numbers whether you’re approaching that threshold, and you should be well aware if you are shy of the mark or have well exceeded it. If you’re marginal, then you know that you’re marginal.

The notion that teaching counts in hiring and tenure decisions at research universities is a sham, as recently pointed out by Alex Bond at The Lab and Field. If you’re at a research institution, being a horrible teacher won’t hurt your chances at tenure and being a fantabulous teacher won’t help your bid for tenure. (If you are unliked or extremely popular, however, and your case is marginal, then teaching performance could be inserted as a surrogate variable to help swing the review one way or another.)

At teaching institutions, the story is entirely different.

At your on-site job interview, I wish you luck trying to get a wholly quantitative description about what it takes to get tenure. Typically, you need to be “excellent” at teaching, and “excellent” at either research or service, and mighty good at the third. I think that’s the answer I got at every single one of the 10 or so teaching campuses where I’ve interviewed over the years.

You know how teaching doesn’t matter at research schools? Well, the converse isn’t entirely true. Research does matter at teaching schools, though there may be a lot of flexibility about what counts as research. At lower-ranked institutions, “research” might not necessarily involve publications or external funding, if you really like someone’s teaching. It could just involve keeping students busy in your lab outside of the curriculum and having some of them get into grad school.

Some teaching campuses put specific numbers on publications, which in my experience has ranged between 0-6, with no real specification of impact factor. The expected publication rate before tenure is negatively associated with teaching load, but this relationship has only a moderate correlation. Most places expect you to submit a grant but aren’t horribly put out if you aren’t funded. The research criterion is pretty clear-cut at teaching campuses, and there is also fudge room because it’s not the primary criterion.

Then, what constitutes excellent teaching? Most campuses go with a Supreme Court Justice Potter Stewart kind of definition.

Knowing excellent teaching when you see it isn’t a good way to decide whether someone gets tenure, is it?

How do most people judge excellent teaching, when they are required to make such a judgment? A student of human nature would suggest that it is identified by how much it resembles the practices of the observer. I’ve never met a full professor who didn’t have a moderately high opinion of their own teaching. And we’ve all met plenty of full professors who couldn’t teach their way out of a bag. This does not bode well for effective tenure decisionmaking. (By the way, is the Bush neologism ‘decider’ now a replacement for ‘decision-maker’?)

In practice, there are many factors that are included in the quantitative and qualitative measures of teaching performance at a teaching campus.

The drawback to all of these quantitative and qualitative measures is that they all suck, or at least have poor resolution.

Let’s go over them one at a time. Keep in mind that no school uses all of these measures in concert.

Teaching evaluation forms There is a whole subfield in education research on this topic, and I’m not going to let this site devolve into a bitch session about teaching evaluations. Really horrible instructors get horrible eval scores, and amazingly perfect instructors get high scores. What happens for most of us, the professors in the middle — ranging between not-so-good and run-of-the-mill excellent — is really murky.

At my university the forms are called PTEs: Perceived Teaching Effectiveness evaluations. The key word here is “perceived.” Are students good at knowing whether their instructor is effective? Often, yes. However, there are a huge number of systematic biases that go along with these forms, suggesting that we need to avoid using the numbers in a comparative fashion. Upper division courses have higher scores than lower division courses, which have higher scores than non-majors courses. This might be independent of teaching effectiveness. There are age and gender biases that affect student perceptions of effectiveness, and associations between the grades received by the students and the perceived effectiveness of the instructor are not necessarily causative. How you dress in the first weeks of class can really matter, too. From discipline to discipline, mean evaluation scores are quite variable. If you want to measure improvement in the same course, with the same professor, with the same student demographic (including time of day the course is taught), then this might be a good measure, at a coarse resolution.

If your tenure case is being evaluated at the level of the college or the university, and your scores are being compared against colleagues in other disciplines, or who teach different kinds of courses, that isn’t fair. I don’t know of a campus that specifies a specific threshold score for evaluations (at least officially), and that is a good thing. However, unofficially some campuses or committees are expecting scores to be above a certain level. If that’s the case, then faculty need to learn the little tricks to make sure they don’t do things that cause students to lower their scores. (That’s a whole other set of posts.)

Written remarks by students The voluntary responses by students on evaluation forms are potentially telling. Students can offer specific and useful praise, and also tell damning stories that very clearly can explain instructor performance. Recurring similar comments by multiple students are particularly valuable. However, most student responses are idiosyncratic and it’s very difficult to distinguish between a student with a legitimate grievance and one who is bitter about their own performance.

Classroom observations Faculty members in the department may be requested or required to sit in on a certain number of hours or lessons before offering a recommendation. These observations are effective so long as the observer is capable of identifying effective instruction. This is heavily subjected to the biases of the observer, especially as scientists typically have no training in teaching methods. For example, when I was a junior faculty member, I made sure to implement the methods of active learning in science instruction that I learned as a graduate student in the College Teaching Certification program and as a Preparing Future Faculty fellow. So what happened when I was observed by my senior departmental colleagues sizing me up for tenure? I’ll always remember this, word for word: “You need to be less Socratic and lecture more. You should be using powerpoint and use more detailed information.” Never mind the fact that all of the current research on science education told me to do the opposite of what they said. After all, these professors were the ones evaluating my tenure file. So, when they were in my classroom, I had to lecture, even though I knew this was an ineffective approach.

How could classroom observations be effective? The people doing the observing could know what they hell they are doing and could be well trained in evaluating effective teaching. This happens in public schools. In the state of California, to be come a fully credentialed K-12 teacher you need to go through an evaluative induction process, the Beginning Teacher Support and Assessment (BTSA, pronounced “bitsa.”) To be a BTSA evaluator, you need to be trained to observe and score the performance of teachers, and this training process involves a calibration of standards and a long list of specific criteria. One BTSA evaluator observing one set of instruction comes up with a score very similar to any other BTSA evaluator; that’s the way the system is built.

What about teaching-centered universities, how do senior faculty do their observations? They show up, if they care to spare the time, and they then may fill out a cursory form if one exists, and then include whatever observations they choose to include or not in their letters. I can’t think of an evaluation that is more subjective nor disconnected from whatever objective measure that could exist. (I’m not saying that I’m any less guilty than anybody else, mind you. Of course, most faculty would be peeved at the notion that they need to be trained to recognize good teaching.) Regardless, in some teaching schools, classroom observations aren’t a required or even optional component of the tenure portfolio. Oftentimes, the only thing that tenure committees know specifically about what happens in the classroom is by hearsay from students.

I was impressed that once, my all-time favorite dean chose to sit in on my classroom for half an hour, and when he wrote the letter for my file he referenced specifics from what he saw in my classroom. He didn’t do this for lack of being busy, and I appreciate the time he spent in directly evaluating me.

Assessment data Perhaps we could look at student performance using assessment data, looking at student knowledge before and after individuals pass through your course. These kind of assessment data aren’t common, and anyway, most science faculty are in full rebellion against regional accreditation agencies that are requiring assessment in curriculum design, and using assessment data like this could actually annoy some faculty members who might think that you’ve gone over to the dark side of assessment. I suppose you could use these numbers but just not call it assessment and maybe get away with it.

Student letters I think few campuses do this, but it happens in my undergraduate institution. I was asked by my college to write letters of evaluation for faculty members in whose courses I was enrolled. The college requests letters from some students who are listed by the faculty member, and also randomly (or perhaps haphazardly) selects other students from rosters of recent courses. I imagine that these letters would be a lot more informative than whatever would be in student evaluations. They do this for both tenure and promotion to full professor.

Hearsay It is stunning how students are willing to discuss my colleagues in front of me, as long as I’m not involved in the conversation. Just the other day, I was in my lab sorting ants, and some of my research students were going on and on in great detail about an instructor in our department, who is a close colleague of mine. There was a mix of criticism and praise. They were talking like movie reviewers or restaurant critics. I wasn’t involved in the conversation, but I was sitting easily within ear’s reach where they were saying all kinds of things about my colleague that they would never say directly to this person. This kind of overheard conversation happens all the time, especially if you’re teaching lab sections. It’s unprofessional of the students to do this in front of other professors, but I guess they’re not professional. I arguably have more indirect information about my colleagues’ teaching from this route than any other. If I believe most of what I hear, by the way, then most people in my department are incredibly awesome. Regardless, this isn’t a valid source of information for evaluating teaching performance, though I imagine that in some environments this is probably the source of information with the greatest sway.

External evaluations Research universities require external letters from experts in the subfield of the tenure candidate to evaluate their tenure file. So, teaching universities must get outside experts to evaluate the teaching of candidates in their subfields of expertise, right? Ha! That’s a good one!

What it takes to be “excellent” at teaching is being able to convince the other faculty in department that you fit that label. Faculty use a variety of information sources, including not not limited to the information above. Ultimately, the assessment is a holistic gestalt-based system. Kind of like how honey bee colonies use guard bees size up the pheromonal composition of bees landing at the nest to decide if they belong, academic departments work the same way. If you don’t fit in, then the guard bee will pounce on you.

The biggest way to not fit in is to not teach well.

However, another way is to teach well, but teach differently.

It’s often said that tenure is about “fit.” Some people say that’s vague: how do you define fit? It’s nothing that needs any special definition. Either you fit in or you don’t. Either you have the same values and the same approaches with respect to education, or you don’t.

This is why it’s sometimes hard to get tenure in a contentious department (read: snakepit) in a teaching institution. Even if you’re careful to not take sides in any weird departmental politics, everyone involved in the tenure process will be called upon to assess your teaching. This is going to involve a meeting where your teaching is discussed. If the department has divisions about teaching philosophy or approaches, this will emerge in the criteria used for evaluations. If one side really likes what you do and explains why, then the other side might end up in disagreement. This is not good. You can ameliorate this by how you sell your teaching approach in your tenure file. You don’t want to make the mistake of arguing that you have worked hard to find the most effective approaches to teaching and that your assessments show that students learn effectively. What you want to do is communicate that your teaching fits in with your department, and that you have worked collaboratively with your colleagues so that you have learned how to teach well from them. You don’t want to say anything that is overtly contrasting existing practice, because, ultimately, the people in charge of deciding whether your teaching is excellent will compare your work with the template of their own work. Just like guard honeybees that use their own smell to decide whether to reject outsiders.

Even if you have a history of demonstrating teaching excellence at a teaching institution, a fresh pair of eyes with a different perspective, or a different agenda, could look at the same record and come up with a credible argument that the record fails to demonstrate excellence. Without anything changing, the environment can shift so that what is perceived as “excellent” in one year might not be acceptable the next year.

This is different than research institutions, I think. It’s harder to argue against grant dollars and a list of publications on a CV. You could argue that the journals aren’t of a high enough impact or that the grants are from the wrong agency, but the research bar at a research institution is far, far more tangible than the teaching bar at a teaching institution.

I would guess that if you are unambiguously above the bar that’s been set for you for research productivity and funding, and you haven’t entirely botched something else, you should be golden. Even if there are academic disagreements about your work, if you’ve got the grants and published in the right journals, then that is likely to be fine. This is particularly the case if you’re at a unionized institution, in which the tenure process is more transparent than at an institution with an opaque process with secret information, because the faculty lack the power to make sure that the process is fair.

Of course, at nearly all universities, tenure rates are quite high, except for a few Ivies that have a de facto policy of hiring Assistant Professor positions as glorified 7-year postdocs. When people don’t get tenure, it might be because performance is not up to snuff, but it can also happen because the department is toxic or incompetent. Other crazy stuff can happen, too. Regardless, the lack of specific quantitative criteria in the teaching criterion create an element of hap into the process that makes it less predictable, which makes it a source for anxiety if not a source of difficulty.

In short, the amount of work it takes to be an excellent teacher doesn’t necessarily correspond to the amount of work you have to do to get tenure at a teaching institution. To do that, you (most likely) have to be an excellent teacher and you also have to do the work to convince your colleagues that you are. In some places, this is harder to do than others. In some places, you don’t even have to be an excellent teacher, as long as you are able to create that perception. There’s the rub.