Embedded Assessment

Embedded Assessment, 2006

Let’s start with a little free association, if I say assessment, what words/phrases come to mind?

Yes, assessment is connect to accreditation and mandates and people looking over your shoulder – and it can become a real burden.  But let me try to shift your thinking a little. My guess is that we all have had that moment when we were grading exams or papers and thought, my goodness, even my good students don’t seem to have gotten this part.  At that point, we’re likely to start thinking about how to approach whatever idea/concept they didn’t get differently.  We’re taking information about what students learned – or in this case didn’t learn – and using it to re-shape what we do in the classroom.

Assessment can be simply a somewhat more systematic version of this kind of reflection. 

Before I get to the central topic of how we can do this more systematic approach in a way that doesn’t drive us nuts, let share what I see as a few key principles for assessment.  These are generally drawn from a document produced by the American Association for Higher Education, but I have added, subtracted, and translated their language into something I find more accessible.   (handout)

1. We must focus on questions where the answers matter to us.  The “us” here can vary – it can be personal, but it can also be departmental or institutional.  Whoever is included, the bottom line is that the answers have to matter to the stakeholders – the people who will use the results.  It is therefore better to get rough answers to important questions than to measure easy things we don’t care about.   
   
2. If you haven’t thought through what matters to you, you can’t do meaningful assessment.  If you don’t know what questions or issues you want to address, you are just collecting data.
   
3. Good assessment acknowledges that learning is complex and occurs over time.  A one shot approach provides only a momentary glimpse, so the best assessment is on-going.  By that I don’t mean that you have to raise the same questions every year, but we do need to loop back periodically and see if anything has changed.
   
4. Knowing that students got the desired result isn’t nearly as helpful as knowing how they came to that understanding.  In most of our disciplines, what we are really looking for is that our students understand and can apply key concepts to new situations.  Assessment that looks at the reasoning process therefore is often more helpful than assessment that just looks at the result.
   
5. Assessment works best if broadly collaborative.  Two levels here.  First, even if you are doing the assessment for your own purposes, you can benefit from feedback from others on your process.  More importantly, if you are doing assessment for more than your own edification, you have to take the time to get buy in from the others involved.  This means agreement on goals, and on criteria for evaluation.  It also means shared experience in actually doing the evaluation. 
   
6. Assessment needs to matter in the institutional context.  If what you are doing isn’t valued, it isn’t going to have any impact, and so really isn’t worth the time.
   
7. Our central obligation is to become better teachers, and to help our students learn more.  Assessment is a way to do this.  It is systematic way of pushing us to think about whether our actions are having the results we desire.

So, how do we use assessment to become better teachers and help our students learn more?  The easiest way to link teaching and assessment is to think about what is commonly called course-embedded assessment.  This term refers to the use of a variety of techniques that measure student learning over a unit or course, but basically course embedded assessment takes grading – something we would be doing anyway – and extends/amplifies it for assessment.

And, how do we do this?  Let me start with the individual course level, and then I’ll talk about larger departmental efforts.

The first step is to decide on what you want to assess.  It could be something very immediate, like did they understand the concept we covered today.  Or it could be whether they have mastered something by the end of the course – say some specific content, understanding of broad concepts, or some particular skill.  Or it could be that you want to measure how much they improved over the semester – as opposed to where they are at the end.  In other words, value added.

The second step is to decide on what would constitute different levels of achievement.  What exactly do they have to be able to do to demonstrate excellence versus barely acceptable.  Sounds familiar, right – we all do this in some way when we grade.

The third step is to figure out what evaluation instrument would generate the information you want.  It may be a paper or test that you are already.  If not, you will need to create something new.  One word of warning: if you don’t make this assignment part of your normal grading sequence, you are asking for trouble.  First, you are adding work for yourself.  Second, and much more importantly, if the students don’t see this as something significant, they aren’t going to do their best work, and that will really mess up your assessment.

The next step, obviously, is to administer the evaluation instrument.  Could be at the end of a class, mid-semester, end of year – all depends on what your goal is.  After the students have done their part, you do yours.  You evaluate their performance using your pre-set criteria – and come up with two sets of results.  One is a grade for individual students.  The second result is assessment, which is where things get a little more complicated.

To turn grading of individuals into collective assessment of the class, we need two things to happen. First, the evaluation criteria must be explicit.  Rather than saying I know an A when I see it, we have to have carefully defined criteria that someone else in the field could come in and apply to the student work.  The other person might not come up with exactly the same evaluation of each student’s paper, but having clear criteria increases the chance of consistency because everyone is looking at the same aspects.  May disagree on how well a particular aspect was done, but clear criteria avoids the problem – for example – of one evaluator focusing on content and another on style, usually, clear criteria means creating a rubric.  (Session on web or I can help.) 

The second key thing that we need to do to move from grading to assessment is a feedback loop.   Two aspects.  First, ask yourself how well your assessment mechanism worked.   Did I get the information I wanted?  Do I need to change my evaluation criteria or the instrument?   Second, decide where you go from here.  What do the results tell me?  Are they learning what I hoped/planned?  Do the results suggest that I need to spend more time on certain aspects or be more explicit about certain things?  Do they suggest that I need to think about larger changes in my pedagogy? 

Okay, let me be a little more specific about various ways we can use embedded assessment, starting with some pretty low-cost approaches and then moving up. Imagine that you have just taught a difficult concept, and you want to know how well they have understood it before you go on.   A quick and simple assessment tool is to ask the students to do a minute paper – write – anonymously -- for 3-4 minutes at the end of class summarizing the key concept in their own words.  

Devices like this are known as Classroom Assessment Techniques or CAT’s -- developed by Angelo and Cross in late 1980s.  Dozens of varieties for different situations can be found their book, which is in the LES library – HAB 103.

Once the students turn in the CATS, you can evaluate their responses according to the scale you have created, and then move on to the feedback loop.  At the minimal end, the CAT results give you immediate feedback that you can use to guide what you do in the next class period.  If most of them didn’t get it at the level you wanted, you can do a review.

CAT results can also be used for larger forms of assessment.  If you are disappointed in the results, you can use that information to modify how you approach that material next time you teach the course.  If you do the same CAT over again and compare the results across years, you have the beginnings of on-going assessment.  The comparison can give you an imperfect but helpful guide on whether or not the changes worked.

CATS are great for grabbing a snapshot of where students are at a particular moment without much cost to anyone.  If you want to escalate, here are some options, depending on the questions you want answered.

Let’s say you are teaching a very content focused course where students have to really master specific information in order to move on to the next course.  Obviously, in that case you can use the same kind of quiz/exam you probably already use.  Turning it into assessment simply requires creating clear grading criteria and adding the reflective feedback loop where you think about the level of understanding achieved by the entire class rather than individual students.

Similarly, if you are looking for development of a particular skill – writing or the ability to frame questions in the discipline, or think critically – you can again use some assignment that fits your course goals to do assessment.  You just have to carefully design the assignment so it gets at what you want students to demonstrate, develop and apply your evaluation criteria, and then do the feedback loop.

One of the things that I’m particularly interested in is valued added.  Did the students learn something over the course of the semester?  Again, this can be specific content, concepts, or skills.   You can measure it by using rubrics to evaluate work done early in the class versus work done at the end of the semester, but I’ve also seen people using more quantitative approaches.  For example, you could set up a multiple choice test on a bunch of key terms/concepts you are going to address in the class, and do pre- and post test on those.  Again, when are done, you look where the most improvement occurred, and probably more importantly, what parts they still were weak on.  You can then use that information to adjust how you approach the class the next time.  You may decide you need to approach the difficult material differently, or you might decide to devote more time to it – and reduce the time spent on things everyone got.

Since lots of us are worried about assessing departmental goals – as opposed to individual course goals – let me talk about how embedded assessment can be used there.

At the simplest level, the process is pretty much the same.   If you look at Figure 1.1, you can see Barbara Walvoord’s schematic vision. (From Assessment Clear and Simple – great book).   As you can see, the criteria and evaluation are all done by the instructor, and the results come out in the form of grades for the students, help for the instructor, and in the bottom example, information for other groups.

Let me amplify that Simple Program Assessment diagram, using an example from Walvoord’s.   She explains a Biology Department that has as one of its learning goals for graduating seniors that they be able to do original scientific research and present it – and they have a senior capstone course that requires students to do that.  The teacher of the capstone develops criteria and a rubric for evaluation of the student product.  She then reports to the department on the results compared to prior years.  She may point to things she has changed and whether or not they are reflected in better scores in particular areas.  Or she may point to areas of weakness and ask the department to think about its curriculum and how to solve the problem.  The conversation and the resultant actions then become part of the assessment report.

This model is clearly pretty easy to administer – some additional work for the one faculty member, but that’s it.  The weakness is that it depends on the judgment of just one person.  If you want greater validity, you can expand the number of people involved. 

You can see some options in Walvoord’s Figure 1.2.  The “outsider” in this case could be one or more members of the department, or someone who is completely outside the department or even the institution.  The key point in any version of the expanded model is that you have to have lots of conversations to make sure that everyone involved is speaking the same language and sees the same things.  Obviously, this takes more time and energy, but there are big benefits in reliability and perhaps even more in a sense of a shared, clearly understood, goal.

To make this a little more tangible, let me quickly explain two things we have done in the history department.  One of the things we have as a goal for our entry-level courses is that students develop a more sophisticated understanding of what history is and how historians do their work.  Some of us have been doing course embedded assessment on this with a pre- and post writing assignment in our intro courses.  First day of class – what is history.  Last day or week of class – same assignment.  Several of us then read the results and score them for levels of sophistication/complexity based on criteria we have set. 

We’ve found that this gives us a pretty good measure of how well we are doing on this goal – plus our hope is that we can demonstrate even more significant change by comparing what students in intro level courses say with the responses of our senior majors.

We also have had for a long time a goal very similar to the Biology department in Walvoord’s example – we have said that we want our senior majors to both create and present an original piece of historical scholarship. We have argued that we addressed that goal in our senior research seminar where students write a major paper and make a formal oral presentation on the results. 

We did set up an assessment process for the oral presentations about ten years ago, but we hadn’t done anything with the papers, and more importantly, we hadn’t really talked about what we most wanted to know about our students’ abilities on graduation.

Over the last several years we have moved in fits and starts, with foot dragging and some disagreements – to a conclusion that what we most wanted to know was whether or students could make a convincing historical argument.

We then decided that we would assess that goal by looking at the papers our seniors produced in the research seminar.  That led to more conversations about what constituted a “convincing historical argument.”

What you see on the handout is the rubric we have at this point.  We are currently doing a small pilot with it to see how well it works.  Once we make some adjustments, we are going to have the entire department read and score a few papers collectively to enhance our consistency.  From there, we expect to have a small team of people read and score a sample each year, and then bring the results back for a departmental conversation as in the Biology model.  

Perhaps the ultimate form of embedded assessment at the departmental level is the student portfolio where students choose items from among the work they have done in classes to go into a portfolio that demonstrates their abilities. 

At least three things have to happen before you can get to the evaluation in this model.  First, the department has to decide on what it wants to assess.  Second, the faculty need to make sure there are sufficient assignments that are going to provided evidence on those goals.  If not, people are going to have to create them.  Third, the department has to communicate effectively with the students on what they are expected to include.

Once you collect the information, it needs to be evaluated by a departmental team, and the results reported back.  From there, the department must do two things.  First, adjust the goals and instrument if necessary.  Second, decide what to do with the information.  It may be that you say, great, our students are doing what we want in that area, so let’s look at another area next year and come back to this one in two, three, four years.  Or it may that you say, things are not as good as we would like, so you start a conversation on how to get better results.

Clearly, this approach requires a lot of faculty time and energy.  It also requires more student engagement. The greater student engagement is not only in assembling the data, but also in thinking about why they are going to choose certain pieces.  Good portfolios include a reflective piece that talks in a meta-cognitive way about why they have included this piece in the portfolio and what it shows about the progress of their learning. 

On the plus side, that meta-cognitive reflection has been shown to enhance student learning, and the faculty effort should result in reliable results on multiple learning goals.

In summary –

• If you are going to move from the kind of work we are already doing – evaluating our students’ work and assigning grades – to assessment of student learning on a broader scale, we need to do several things.
• First, we have to think about the intended outcomes.  What is it that we most care about?  What do we want students to learn/be able to do – AND what information would be most helpful to us.
• Second, we need to decide on criteria for evaluation, and we need to find appropriate instruments.  If you are doing this for departmental purposes, it gets much more complicated.  Not only do you have to get everyone to agree, but you need to practice the evaluation so you have consistent scoring.  Furthermore, in departmental assessment, you need to do it in multiple courses over time.
• Finally, we need to evaluate what we have learned AND apply it.   We need to experiment with changes in pedagogical techniques, adjustments in course content, or changes in the curriculum.  And then we need to keep testing to see if the changes have produced the desired results.

 Key sources
           Angelo, Tom and Patricia Cross, Classroom Assessment Techniques (1993)
           Walvoord, Barbara, Assessment Clear and Simple  (2004)

Embedded Assessment: Using Classroom Assignments to Achieve Our Assessment Goals
Presented by Ken Jones
February 14, 16, 2006

Principles of Assessment (loosely drawn from American Association for Higher Education)

1. Assessment should focus on questions that matter to you.
2. If you haven’t decided what matters to you, you can’t do meaningful assessment.
3.  Good assessment acknowledges that learning is complex and occurs over time.
4. It is better to know how students reached their conclusions than it is to know if they reached the right answer
5. Assessment works best if broadly collaborative.
6. Assessment must matter in the institutional context.
7. Assessment is a way to fulfill our obligation to help our students learn.

Basic Steps in Course Embedded Assessment

• Decide what you want to assess 
  What do I care about? What information would be most helpful? Did they understand a specific concept?  Did they master the course content? Did they improve significantly over the semester?
• Differentiate between levels of achievement
• Choose or design an evaluation instrument
  Make sure it evokes real effort from students
• Administer the instrument
• Evaluate individual students (grades)
• Do assessment of class learning as a whole
  Follow clear criteria that someone else could apply
  Feedback loop
         Did I get the information I wanted?
              If not, redesign
         Evaluate the implications of the result
         Apply understanding gained from results - make changes as necessary
  Repeat assessment over time
         Evaluate Connections between changes, intended results, and performance

Applying Course Embedded Assessment

• CAT's (Classroom Assessment Techniques - from Angelo and Cross)
  Immediate feedback on understanding of a single class presentation
          Comparison of results of pedagogical changes on different sets of students of time
• Content Mastery
  Use appropriate quiz/exam
• Skill Mastery
  Design assignment that reflects desired skills
• Value Added
  Pre- and post quizzes or papers

Using Course Embedded Assessment for Department Ends

Simplest Form (Walvoord)

• Department sets goals
      Instructor sets assessment took, criteria, and evaluation
      Instructor reports results to Department
• Department and/or instructor completes feedback loop
• Relatively easy, but rests on judgment of one person

More Complex Options (Walvoord)

• Outsiders (from department or elsewhere) are included
      Plan, set criteria, evaluate
      Must generate agreement on criteria and apply consistently
• Feedback loop through shared departmental conversation (with or with outsiders)
      More cumbersome and more costly in time and energy for faculty
      More reliable results on one or more questions, generates sense of shared goal

Portfolios

• Students choose what to include from course assignments according to departmental criteria
• Department must decide what to assess (can be multiple goals)
• Department must decide if appropriate assignments exist
• Department must communitcate with students on what to include
• Work is evaluated holistically by departmental team
• Shared departmental conversation to complete feedback
• Involves significant faculty time and energy
• Requires student effort as well
• Inclusion of a meta-cognitive prompt increases student reflection and learning
• Can provide reliable results on multiple learning goals

 

Key Sources
       Angelo, Tom and Cross, Patricia, Classroom Assessment Techniques (1993)
       Walvoord, Barbara, Assessment Clear and Simple (2004)