AHSS Summary Report for calendar year 2006

The primary assessment-related activities of AHSS in 2006-07 consisted of 1) core assessment reports, filed online prior to the university’s Assessment Expo in October; 2) assessment posters (both Core and Program) exhibited at the Expo; and 3) writing and evaluation of program Plans.

As regards #1, most of the AHSS core courses submitted reports, and most report writers participated in the evaluation process. As regards #2, six AHSS posters were exhibited, and four of these (History Secondary Ed/Program, Music/Core, Sociology/Core, and BA Rhetoric and Writing/Program) were among the fifteen university-wide award winners.

As regards #3, 23 AHSS programs (almost all of the total; Studio Art, German, and MA Public History were granted reprieves, and several Secondary Ed programs did not submit Plans—see below for details) submitted Plans. The AHSS Assessment Team, which as usual was composed of eleven members representing the various departments within the college, divided into four sub-committees, and each sub-committee evaluated approximately six Plans. The review process followed the one used for the last several years. Each plan writer and department chair received a draft of the plan evaluation. The writers had the opportunity to meet with the sub-committee so that information in the report or in the evaluation could be clarified. Each program could submit a revised report, and the sub-committee could determine if the evaluation and/or comments needed to be revised. Finally, on April 30 the Team met to review and discuss the individual reports and the report evaluations.  The table below indicates the final evaluations as reported by the sub-committees.

Table A: Summary of AHSS Program Evaluations

Plan	Program		Rating
16	BA Spanish		Meets Expectations
13	BA Art History		Between Meets Expectations and Exceeds Expectations
14	BA English		Meets/Exceeds
15	BA French		Meets
17	BA History		Exceeds Expectations
18	BA Music		Meets
20	BA Philosophy		Between Meets Expectations and Needs Revision
21	BA Political Science		Meets
22	BA Psychology		Needs Revision
23a	BA Sociology		Meets/Exceeds
23b	BA Anthropology		Meets
24	BA Theatre Arts		Meets
26	BA International Studies		Meets
27	BA Liberal Arts		Meets/Exceeds
45b	BA Prof & Tech Writing II		Exceeds
65	MA Art		Meets/Needs
12a	BA Art Secondary Ed		Meets/Needs
67	MA Applied Psychology		Meets/Exceeds
68	MA Tech & Expo Writing		Meets/Exceeds
110	MA Liberal Studies		Meets/Exceeds
115	MA Second Languages		Meets
18a	BA Music Secondary Ed		Meets (conditional)
17a              BA History Secondary Ed		Meets/Needs

Summary of Results and Comparison Over The Years:

Of the twenty-three plans that were rated this year, two were judged in the highest tier, “Exceeds Expectations.”  Seven were judged “Meets/Exceeds,” and nine more were judged “Meets Expectations”. Only one was judged “Needs Revision”, and four more received “Meets/Needs”.

It is difficult to compare ratings this year with ratings from past years, since 1) this year we wrote Plans, not Annual Progress Reports; and 2) we used a new rubric, devised by the Provost’s Assessment Advisory Group (PAAG) in 2006. We no longer rank Plans or Progress Reports on the four-point Excellent-Reasonable-Limited-Inadequate scale, nor do we give separate ratings for Approach, Use of Findings, Stakeholder Involvement, and Overall. However, the general trend does seem to be one of continued excellence; in the past several years (before we changed rubrics), almost all programs received ratings of “Meets Expectations” or above.

Analysis of Results:

The ratings were slightly skewed by the controversy surrounding Secondary Ed programs. In the past several years there have been questions as to whether these programs should submit their own Plans/Progress Reports, or rather whether their data should merely be inserted into their parent content programs’ Plans/Progress Reports. At the end of this academic year (at the PAAG meeting, 5/2), after the submission—evaluation—feedback---revision process had ended, it was determined that the latter should prevail. There had been problems this year with the Secondary Ed Plans, as several programs (English, French, Spanish, and German) did not submit full Plans (of course we now realize they did not have to), and others used the formats and/or jargon of the Education programs’ accrediting agency, the National Council for the Accreditation of Teacher Education (NCATE), instead of PAAG’s formats and jargon. This is probably why the three Secondary Ed programs that did submit Plans this year (Art, Music, History) received mediocre ratings (two Meets/Needs, one conditional Meets). If we review the above ratings and eliminate the three Secondary Ed programs (which, we now realize, did not have to submit Plans anyway), the overall ratings look even better: two Exceeds, seven Meets/Exceeds, eight Meets, two Meets/Needs, and one Needs. 

Questions continue to be raised about inter-rater reliability. To this end, we held two “norming” sessions, in late February and March, in which PAAG’s new rubrics, forms and instruction sheets, and checklists (devised by PAAG during spring 2006) were discussed. However, only a small number of Team members were able to attend. In the future we will try to continue to address this issue; however, assessment at UALR will never reach a point at which inter-rater consistency is so high that comments, as well as ratings, will be nearly identical from subcommittee to subcommittee. Some fluctuation in each from year to year, given consistency in the quality of a given report, is inevitable. We are all individuals, not clones of each other, and assessment continues to be more art than science.

The team continued last year's practice of soliciting program response to a draft evaluation. After receiving its evaluation from the subcommittee, a program could revise or clarify information in its report, in a face-to-face meeting with the sub-committee or via email, and in response the subcommittee could revise its comments and/or its rating/s. This year programs and sub-committees communicated entirely by email, when there was communication, and did not have face-to-face meetings.   The process resulted in revisions of a few reports and evaluations. The team recommends that this practice be continued.

Strengths:
As stated, the vast majority of programs are engaged in successful assessment. A few of the plans’ highlights include:

· Some programs' efforts to involve a variety of stakeholders, in some cases soliciting student input (BA and MA Professional and Technical Writing, and Sociology);

· Some programs’ efforts to achieve reliability and/or validity (MA Psychology), especially through the use of norming sessions (MA Professional and Technical Writing);

· Some programs' commitment to tracking student achievement over time (such as History and English);

· Some programs' use of external reviewers and consultants (especially the MA in Psychology and Art History); and

· MALS’s thorough description of all of its artifacts and rubrics.

Areas of concern:
The curriculum maps, which were new to the plan forms this year (and will be new to the progress report forms next year), posed problems. The main issue was that most programs treated them as Procrustean beds. Some of the best plans were those that tweaked the maps to match the particular program’s needs. We are still learning how to use the new forms.

Much of the evaluation, feedback, and revision took place at the end of the semester. The decision to make the deadline for plan submission March 5 turned out not to work, as most of us have far heavier loads at the end of the semester than at the beginning. Next year we may move the deadline back to February.

There was some dispute about the new call to link Goals and Objectives to Blue Ribbon (Core) Competencies (“BRCs”)(in Program Assessment as well as Core Assessment). Most evaluating subcommittees were impressed by the good work many programs did in achieving this linkage. But one member asserted that too many programs merely stated such links without demonstrating them.

We debated whether a given department is responsible for ensuring that its majors achieve skill in objectives linked to all ten of the BRCs. The argument for this is that many students transfer from two-year and other colleges in which they take core courses outside of their major fields, and others take many of these courses in their first two years at UALR; students’ efforts at the ends of their careers, however, are often concentrated in their majors. Thus, so the argument goes, for us to ascertain that students have achieved Objectives related to all 10 BRCs by graduation, the departments are responsible for measuring their majors’ achievement of all of these Objectives, not just disciplinary knowledge. Psychology, so the argument goes, must assess its majors’ competence in Aesthetic Experience, Mathematics, and International Awareness, not just their understanding of Psychology. But others argued that Objectives related to the 10 BRCs are achieved in core courses in a variety of departments, and that one department should not be required to cover all ten in its major-level courses.

There continues to be ennui about the transdisciplinary nature of assessment (i.e., that assessment skills, discourse, and jargon are not located in any one AHSS discipline, rather in the single field of Assessment Studies), especially given the intradisciplinary sensibilities of most AHSS faculty (e.g., members of the History department tend to define themselves as historians, etc.). Some members question whether Political Science, to take a random example, can deliver trenchant commentary on Theater, or vice versa, given the vast differences between the two disciplines. That said, we understand that assessment (not just at UALR, but internationally) is conceived transdisciplinarily.

One of the big problems, as stated, was in the Secondary Ed programs. We are all glad that that issue has now been settled, and that henceforth Secondary Ed programs’ data will be incorporated into their parent content programs’ Plans and Progress Reports.

Plans for Next Year

Studio Art received its reprieve this year as a result of their discovery that, since they have eleven (!!!!) emphases (Photography, Painting, Sculpture etc.), they would have to devise eleven sets of learning objectives and eleven curriculum maps. They felt that they would need a year to devise an uber-plan to cover these. They are confident that they will have such a plan ready in early 2008. Similarly, German received its reprieve as a result of its undergoing a transition period (one faculty left, a replacement did not stay long); with more stability in this department, it is likely that they will be able to produce a plan in early 2008. The rest of the programs will begin their assessment cycles with progress reports in early 2008, using the just completed plans as foundations and the new PAAG-mandated formats as templates.

Appendix:   AHSS Assessment Team, 2006-2007
Rolf Groesbeck, team chair (Music)
Andrew Deiser (DISLS)
Roslyn Knutson (English)
Moira Maguire (History)
Carey Roberson (Art)
Angela Hunter (Philosophy and Liberal Studies)
Jacek Lubecki (Political Science)
Roger Webb (Psychology)
Cheryl Harris (Rhetoric and Writing)
Terry Trevino Richard (Soc. and Anthro.)
Yslan Hicks (Theater Arts)