Michael F. Shaughnessy
Senior Columnist EducationNews.org
Eastern New Mexico University

Catherine Millett is Senior Research Scientist at Educational Testing Service (ETS), and co-author of A Culture of Evidence: An Evidenced-Centered Approach to Accountability for Student Learning Outcomes

1) Catherine, I understand that ETS has just released a report on Culture of Evidence. What is this all about?

A Culture of Evidence: An Evidence-Centered Approach to Accountability for Student Learning Outcomes presents a practical approach to help the educational community meet the call for accountability while respecting the diverse attributes of students, faculty and the institutions themselves.For this report, ETS researchers developed a framework to improve, revise and introduce comprehensive systems for the collection and dissemination of information on student learning outcomes.

2) You describe this framework as a juggling act of balancing the need for accountability while respecting a lot of variability in our higher ed system. What is your take on this?

Across the nation, the impetus to create a stronger culture of accountability for student achievement has been met with both enthusiasm and apprehension. For faculty and institutional leaders who are grappling with the often delicate nuances inherent in assessing student learning, a sensible solution to meet accountability demands is essential.

Schools are looking for practical measures to advance student learning that reflect and respect the diversity of students, faculty and the institutions themselves. In the first report of the series, A Culture of Evidence: Postsecondary Assessment and Learning Outcomes: An Evidence-Centered Approach to Accountability for Student Learning Outcomes, we noted three important hallmarks of the U.S. educational system. The first is open access to a postsecondary education for all citizens. The second is the relatively large degree of autonomy that institutions of higher education have in this country. The third is that, in thinking about the features of the U.S. system, the image of a series of continua comes to mind.

With this picture in mind, ETS, along with other institutions at the postsecondary level, is working to develop an approach to accountability that balances the need for having information about the entire postsecondary system, while providing opportunities for each institution to report on the accomplishments of its own students and whether its students are achieving these goals.

3) What exactly is the framework and what are the seven steps?

The COE III framework was developed by ETS and describes a systematic approach to the development and refinement of a campus-based approach for assessing student learning outcomes. The framework is consistent with the calls for accountability, while respecting variations in achieving learning goals that characterize the U.S. higher education system.

file

4) What assessment tool does ETS provide to help schools measure accountability?

As the only integrated test of general education skills, the Measure of Academic Proficiency and Progress (MAPP) test assesses four core skill areas — critical thinking, reading, writing and mathematics. MAPP has also been selected by the Voluntary System of Accountability (VSA) as a gauge of general education outcomes.

MAPP helps schools gain a unified picture of the effectiveness of their general education program to meet requirements for accreditation and performance funding. It also promotes curriculum improvement with actionable score reports that are used to pinpoint strengths and areas of improvement.

5) Parents who are paying tuition obviously want to make sure that their kids are getting a good education. But how involved should parents be in the monitoring of their kids' education AND the curriculum their children pursue?

The first year of college marks a significant transition for students and their parents. In high school, students and parents typically craft an educational plan together. In college, students begin to come into their own in thinking about their education. While adult students must be given the freedom and responsibility to make educational choices, it's only natural for parents to want to take an active role in their children's education. Ultimately, this is a very personal decision that must be made between the student and parents.

6) Let's talk about entry level FORMATIVE EVALUATION for a minute. I put formative evaluation in capital letters, because I think colleges and universities should be doing some kind of formative evaluation at the beginning of the four year process. Or is this not practical?

We encourage schools to conduct formative evaluation at the start of college.Institutions will need to have in place not only a process for administering the formative evaluation, but also one that explains the results to students and faculty as well as formulates a plan to take the next steps.

7) How can schools demonstrate the effectiveness of their program and evaluate students' mastery of a specific field of study?

The general answer is that there are two approaches that institutions can take. First, they can utilize assessment tools that have been developed locally and used within the major. This might include portfolios, performance in a capstone course, senior theses, and so on. These assessment approaches can be very helpful and they can reflect the specific interests and orientations of each university's academic programs.

One shortcoming of locally developed (or "home grown") measures is that they do not provide comparative data, and as a result the institution can measure directional change (e.g., "Are our students getting better over time?"), but they do not allow the institution or program know how their students compare to students from other, similar, programs and institutions.

To provide comparative data, the only currently available approach is to use standardized tests. We reviewed these in the A Culture of Evidence: Critical Features of Assessments for Postsecondary Student Learning, but let's consider one of these in a bit of detail.

The ETS Major Field Tests (MFT) are comprehensive undergraduate and MBA outcomes assessments designed to measure the basic, critical knowledge obtained and understanding achieved by students in a major field of study. MFT goes beyond the measurement of factual knowledge by helping educators evaluate their students' ability to analyze and solve problems, understand relationships and interpret material from their major field of study.

Best of all, ETS offers comprehensive national comparative data gathered from all Major Field Tests taken, enabling schools to evaluate student performance and compare their program's effectiveness to those at similar institutions nationwide.

8) A lot of students "shop around" and take their first year at a community college, transfer to another community college, and then finish their four-year degree at still ANOTHER college. Now – I am going to be real tough here – is it fair that we evaluate that final resting place of these students for those first two years spent at some two-year school or schools?

The first issue is who should be included in the analyses that are done to measure student learning outcomes. In our latest report, the "Evidence Centered Design Section" provides information that institutions can use to formulate their reference groups. Your readers may also find the information on Pre- & Post-testing in the tables in A Culture of Evidence: Critical Features of Assessments for Postsecondary Student Learning to be helpful in discussions about how to approach this measurement issue.

A second way to view your question is to think about which postsecondary institutions may actually be doing the best job of educating transfer students.We suspect that some institutions are actually excelling in this area.

9) Again – I am going to be somewhat vicious here – you and I both know of proverbial students who spend 5, 6, 7 even 8 years pursuing their college degrees. Should these students be evaluated in the same manner as those excellent students who finish in 4 years?

This is an excellent question. You are correct that students are taking longer to complete their degrees. According to the National Center for Education Statistics' Beginning Postsecondary Study 1995-1996, five years after students began postsecondary education, 47% had earned a degree. For purposes of measuring student learning outcomes, we should not think of this as a race. Our effort should be directed toward ensuring that each graduate of a postsecondary institution has achieved the student learning outcomes articulated by the institution. Some students may do this in four years while others may do it in six.

10) Again, we live in changing times, or as Bob Dylan used to sing, "The Times they are a-changin'."  How can you evaluate students' four year career when some of the classes they took were on-line or perhaps delivered over Instructional Television?

Instructional mediums and formats such as large lecture classes versus small seminars have not been the focus of our work. Institutions may want to examine if students who take an on-line version of a course perform equally well as students who take it face-to-face with a faculty member.

11) Culture of Evidence II reviewed the 12 major assessment tools in use today. Would you name these 12 tests for me and our readers?

A Culture of Evidence: Critical Features of Assessments for Postsecondary Student Learning  provides a high-level overview of twelve of the major assessments in higher education. The assessments are grouped under four major headings.

General Education and Workforce Readiness Assessments:

1.College Basic Academic Subjects Examination (College BASE)
2.Collegiate Assessment of Academic Proficiency (CAAP)
3.Collegiate Learning Assessment (CLA)
4.iSkills (formerly ICT Literacy)
5.Measure of Academic Proficiency and Progress (MAPP)
6.Standardized Assessment of Information Literacy Skills (SAILS)
7.WorkKeys

Subject-Area Knowledge and Skills Assessments:

8.Area Concentration Achievement Tests (ACAT)
9.Major Field Tests (MFT)

Soft Skills Assessments:

10.WorkKeys (Teamwork)

Student Engagement Assessments:

11.Community College Survey of Student Engagement (CCSSE)
12.National Survey of Student Engagement (NSSE)

12) Does your "Culture of Evidence" examine the writing skills of college students, or their thinking skills, or library or research skills?

Let's think about how your readers may find The Culture of Evidence series to be helpful in thinking about the writing skills of their college students. For this question, it may be most useful to think about this in reverse order of the series.

Will we use College A for this example. In The Evidence of Student Learning Outcomes Model in COE III, Step 1, Articulate Their Desired Student Learning Outcome, College A would state that having its graduates be able to demonstrate a certain level of writing skills is a desired outcome. In Step 2, Assessment Audit, College A would determine if an assessment is already in use that measures students' writing skills. This could be a locally developed measure, or one from an external source.At this point in the process, College A may determine that it does not have a writing assessment in place. In Step 3, Assessment Augmentation, College A could consider developing a measure to assess student writing or it could consider one of the assessments in COE II that includes an essay form.

Finally, we can take this question about writing skills back to COE I. When we consider that the U.S. has more than 4,000 postsecondary institutions, today it is highly probable that how institutions gauge the writing ability of their students is likely to vary depending upon characteristics such as the profile of their student population and their institutional mission.

13) What else would you like to add?

I'd like to remind your readers that ETS is driven by a mission to advance quality and equity in education through our assessments, research and services. ETS supports reasonable accountability in higher education and hopes that the Culture of Evidence Series and the seven-step framework described in the latest Culture of Evidence report are helpful guides for schools addressing this issue. Ultimately, it's about improving the overall learning experience for students. And that's something on which everyone can agree.

How can our readers review the reports in the Culture of Evidence series?

For more information on the Culture of Evidence Series please visit www.ets.org/cultureofevidence.org

Published March 10, 2008