How Do Librarians Learn Assessment ?

Library leaders across the nation are engaged in a continuous effort to leverage their resources while providing library services amidst shrinking budgets and the increased demands for accountability to their stakeholders. The ability to demonstrate and effectively articulate the impact of library programs and services on the larger community is emerging as a key competency for library administrators and librarians. Beyond that of traditional input/output metrics, there is a growing emphasis for libraries to make impactful connections between their contributions and the goals of their parent organization, based on data-driven assessment. The importance of this skill is clearly reflected throughout the ALA’s Core Competencies of Librarianship, a document that defines the “basic knowledge to be possessed by all persons graduating from an ALA-accredited master’s program in library and information studies...” and was adopted as policy by the ALA Council in 2009. The document outlines eight competency areas, of which four specifically state the need to have knowledge of the “principles” of or “methods” of assessment.


Introduction
Library leaders across the nation are engaged in a continuous effort to leverage their resources while providing library services amidst shrinking budgets and the increased demands for accountability to their stakeholders.The ability to demonstrate and effectively articulate the impact of library programs and services on the larger community is emerging as a key competency for library administrators and librarians.Beyond that of traditional input/output metrics, there is a growing emphasis for libraries to make impactful connections between their contributions and the goals of their parent organization, based on data-driven assessment.The importance of this skill is clearly reflected throughout the ALA's Core Competencies of Librarianship, a document that defines the "basic knowledge to be possessed by all persons graduating from an ALA-accredited master's program in library and information studies…" and was adopted as policy by the ALA Council in 2009. 1 The document outlines eight competency areas, of which four specifically state the need to have knowledge of the "principles" of or "methods" of assessment.
As in the field of education, assessment has been a growing subspecialty area in librarianship.The significance of assessment in the field beyond that which involves information literacy instruction and student learning, is underscored by the number of conferences and institutes devoted to educating and promoting assessment efforts and methods used in libraries.Events such as the Library Assessment Conference, the Southeastern Libraries Assessment Conference and the Assessment Institute hosted by library and library related organizations, such as the Association of Research Libraries' (ARL), Counting Opinions, and Indiana University -Purdue University Indianapolis, are growing in prominence.In recent years, there have been discussion groups (electronic and in person) devoted to the topic of assessment by the Association of College and Research Libraries (ACRL) and ARL.The increasing significance of assessment in the library science and/or information science field is also reflected by the creation of new positions for this emerging area of specialty.In a 2007 ARL study on assessment respondents indicated that nearly 60% of the assessment positions in their libraries had been created within the previous two years. 2More recently, a 2012 study concerning entry-level reference job advertisements showed that assessment appeared as a job duty for almost a third of the number (n=192) of advertisements studied.The authors note, "…this set of ads reflects the growing emphasis on assessment in the field." 3 All of the aforementioned indicates that there is a growing expectation that professional librarians should either have knowledge of assessment methods or the skills to implement them.
The expectations set forth by ALA for library practitioners to demonstrate these competencies, makes it even more vital that Library and Information Studies (LIS) programs become an integral part of this educational effort.Ideally, the assessment training occurring in the field should supplement that which is more formally provided in library schools, allowing librarians the opportunity to put into immediate practice the research and assessment methods learned as a graduate student.What appears to be happening is the exact opposite, with most librarians gaining their assessment knowledge and experience while on the job, not during their graduate studies.This observation led the researchers to ask the question -is assessment being included in the library school curricula?A limitation to this research concerns the terminology associated with assessment.As Hufford notes, in the field of library science there is a tendency to use the terms "assessment" and "evaluation" interchangeably. 4To find a more concise definition for assessment, the researchers turned to the literature produced by assessment experts in the field of higher education, such as Trudy Banta and Thomas Angelo.Higher education faculty and administrators have responded to the call of accountability from national, regional, and state organizations and accrediting bodies over the past two decades by concentrating their assessment efforts on the individual, classroom and programmatic levels. 5While doing so they have created an area of specialization in their field along with a substantial body of scholarship on this topic from which to draw upon, including standardized assessment terminology.

Definitions
The "terminological consensus" of assessment in higher education has evolved through the years to center on multiple methods (i.e., qualitative or quantitative) for program improvement. 6herefore a shared understanding of the purpose for our assessment efforts in libraries will enable the library profession to establish a clear nomenclature, or "terminological consensus" for assessment in the library profession.Using Thomas Angelo's general working definition as a foundation for the assessment activities that are occurring in the field of library science, assessment for this study is defined as "a means for focusing collective attention, examining assumptions, and creating a shared culture dedicated to understanding and continuously improving the quality of library services." 7Evaluation will follow the definition provided by Peter Gray, as that which "provides explicit information through objective tests and measures guided by precisely specified, if not behavioral objectives to make a value judgment". 8In summary, assessment tends to be a cyclical process involving identifying goals and objectives, developing appropriate instruments, collecting and evaluating data, sharing the results and implementing improvements.Evaluation can be more concisely described as a finite process to determine the value or worth of a program, course or initiative.This study was conducted with these definitions as guiding parameters.

Literature Review
The literature review concentrates on publications in librarianship from the most recent five years.There was quite a bit of literature produced by practitioners reflecting the growing importance and development of assessment skills.Most of the literature either concentrated on the assessment of information literacy skills, or library assessment instruments, such as LibQUAL+®.Other publications focused on the assessment of operations and services.However, there was very little literature focusing on assessment education for librarians in training, or practitioners.
The study that most closely matches the intentions of the researchers' effort was published by Edgar Bailey in 2010.Bailey conducted a content analysis of course syllabi in academic librarianship, using the web sites of forty-nine ALA accredited library schools in the United States. 9He noted that his study was the first of its kind in the field stating, "To date…there has been no examination of the actual content of courses to determine how closely that content matches…the proficiencies identified in the literature." 10 Bailey reviewed nine course syllabi appearing on the Web with the underlying assumption that the syllabus would not change significantly in the near future.In cases where the syllabus on the Web did not appear current, he requested the current syllabi from the instructors.Bailey examined factors such as who taught the course, what subjects were covered, the required textbooks, assignments and pedagogical methods used.From the syllabi, Baily created a log of the various subjects and their frequency of appearance.
His results revealed that "Collection Management/Development" was the most frequently mentioned subject (26) in the course syllabi.The subjects of "Budgeting/Finance", and "Information Literacy/Instruction" were the second most frequently identified, each with twentyfour mentions."Organization" along with "Personnel/Staffing/Human Resources" (23) rounding out the top three frequently mentioned subject areas."Assessment/Evaluation/Standards" appeared somewhat lower in the rankings -in sixth place -with eighteen mentions. 11He found that the list of subjects covered by these courses were very similar to a list of subjects generated by an informal survey of practitioners who felt these subjects should be included in an academic librarianship course.In addition, Bailey noted that these subjects also spoke to the desirable skills and competencies identified for academic librarians in the library literature.
Although admittedly limited, Bailey's study indicates that assessment/evaluation was not necessarily a popular area of study for academic librarianship.The number of mentions for this subject ranked even lower than "Scholarly Communication" (20) and "Management/Administration" (20).However, additional research using a systematic approach to identifying focal areas of study included in academic librarianship by reviewing the course syllabi across all LIS programs is warranted.
Assessment efforts employ quantitative, qualitative or mixed research methodologies.The foundational skills provided by research methods are important to successful assessment and evaluation efforts.Lily Luo's research provides an analysis of the effect of the research methods course on LIS practice and suggests how LIS program curricula can be improved to better meet the needs of practitioners. 12Implementing a Web-based survey using professional listservs such as LIBREF-L, she received responses from 555 LIS practitioners across library types, the majority of which were from academic librarians (78.9%).Luo's findings suggest that librarians are more likely to use results from existing research to improve their work than actually designing new tools and implementing them to evaluate programs and services.She discovered that an individual's position within an organization was irrelevant to their research involvement.For example, a front line librarian was just as likely to conduct research activities as a library manager or administrator.While over half (51.4%) of the respondents indicated they had taken a research methods course while working on their Master's in Library and Information Science (MLIS), 31.6%indicated they had not. 13Almost 73% of the participants felt research methods should be a required component of the MLIS degree.
Luo's results showed that there was no relationship between the supporters of having a mandated research methods course and their having taken a research methods course during their LIS studies.Additionally, her results reveal that the respondents' current work settings had no impact on their perceptions of a mandatory research methods course.Further, she found that academic librarians were more apt than public librarians to pursue knowledge of research methods through continuing education opportunities.However, the interpretation that led to the latter finding is most likely skewed because of the large percentage of respondents from academic libraries.
Important points emphasized by Luo's study as it pertains to this research effort are as follows: a) the majority of librarians tend to use previously published research rather than conducting original research of their own; and, b) one's (librarian) status within the organizational structure has no bearing on who is most likely to conduct research.An interesting follow up to Luo's work would be to determine how many of the respondents who indicated they did not take a research methods course while in library school resulted from their LIS program not offering such a course(s), or by choice.
The research by Chow, et al, provides a holistic view of important LIS program curriculum components from the perspectives of LIS administrators and faculty, students, and local practitioners.This case study concentrates on a LIS program placed on conditional accreditation status. 14One of the research questions addressed by the study is how the LIS program should evolve to most appropriately prepare students for a career in library and information studies.The participant group (n=117) was comprised of the LIS program administrators (3), faculty (10) and students (52), along with local library leaders (53); the majority of the latter being from public libraries. 15Using a mixed methods approach, these researchers gathered and analyzed data from participant observations, interviews, online surveys and qualitative documentation.Although there was much discussed in the findings from this study, the researchers noted early on that much of the results are not generalizable because the study focused on one program.
Based on their results, the need for including and emphasizing library skills seemed not to be the focus of the LIS program administrators; however it was a focus of the LIS faculty and students, as well as the local library leaders.The student participants indicated there was a need to learn how to integrate research skills into the LIS curriculum in order to enable them to meet professional standards.Responses from the local library leaders which have already been noted as being heavily skewed towards public librarianship, cited interpersonal and communication skills as being the most valuable skills needed by library practitioners.Research methods, assessment, nor evaluation surfaced as an important skill by these participants yet LIS faculty participants recognized that "engaging in research was an essential component of a successful LIS program…." 16verall, there seems to be very little found in the literature concentrating on assessment education.What is available illustrates a notable dissonance between LIS educators and library practitioners' views on the importance of integrating assessment (and research methods) more fully into LIS program curricula.Practitioners recognize the importance of these skills, which is signified by the creation of new library positions with assessment as the primary responsibility and the conference and institutes dedicated to the topic.Federal and State legislators, regional accrediting bodies and local constituencies all require a demonstrable return on investment.The piece that seems to be missing in LIS education is a focus on the pedagogical approach to teaching librarians-in-training to effectively conduct assessment.

Limitations
This study is a content analysis of course descriptions from fifty-eight ALA accredited Library and Information Studies (LIS) graduate programs in North America listed on the ALA website, (http://www.ala.org/ala/accreditedprograms/directory/alphalist/index.cfm) as of January 2012.The MLIS courses included in the study were selected based on the following criteria: the descriptions appear on the Web; the descriptions are available in the English language; courses were at the Masters level; and, they were about, related to, or contained elements of assessment and evaluation.It is important to note that this study consists solely of analyzing the terms presented in the course descriptions and therefore interpretations are subjective.

Methodology
A total of 329 course descriptions from forty-four institutions were copied and pasted from the institutional websites and compiled in an Excel spreadsheet.Collection management and collection development courses were then filtered from the remaining course descriptions, since the assessment or evaluative aspects of these courses do not pertain to individual or specific developmental outcomes.The remaining sample of 210 (n) courses was used for further analysis.
Each of the 210 course descriptions was reviewed and categorized by both researchers using a numbering system.The primary categorization method was based on the following words or phrases being present in the description: "research methods", "assessment", "evaluation", "analysis", "assessment and evaluation".Secondary to this, the researchers also took into account the described goals or intent of the course.In the rare instances where the researchers categorizations differed, a common agreement was reached.Research methods courses were included in this study, as these courses provide the theoretical and practical foundation for assessment/evaluation design, implementation, and interpretation.

Results
The overall results of this study based on the remaining 210 courses are illustrated in the table below.In general, course descriptions indicating evaluation as the intended goal are five times more likely to appear than those indicating assessment and a little under twice as likely as research methods.Course descriptions concerning Evaluation appear most frequently across the descriptions and programs reviewed (103 courses: 49%).The category with the second highest number of courses was that for Research Methods (55 courses: 26%), which includes the study of both qualitative and quantitative methods.Rounding out the top three is that category of courses pertaining to Assessment with 21 (10%) courses.
Further review and classification of the courses categorized as assessment and evaluation was completed using the following sub-categories: information literacy and instruction, library operations and services, information technology, and user needs.The results of this analysis indicate that 57% (12) of the assessment courses pertain to information literacy and 24% (5) concern library services and operations.The results for the evaluation courses proved to be the exact opposite with 52% (55) of courses focusing on library services and operations and 29% (30) of these pertaining to information literacy and instruction.

Conclusions and Recommendations
The purpose of this research was to determine if LIS programs were teaching assessment.The unintended results of this study served to reinforce the assertion that the library profession needs to develop and reinforce through education and practice precise nomenclature pertaining to assessment.It is clear from these results that more in depth research is needed to parse out the meanings behind the terms assessment and evaluation used in the course descriptions and it would be best carried out in collaboration with LIS faculty.
Based on terminology alone, courses pertaining to evaluation were more prevalent in LIS programs than those for assessment.Yet the context of many of these descriptions seems to indicate that assessment would have been more appropriate and the term evaluation was being used erroneously.This is particularly true for those course descriptions referring to the "evaluation" of library services and operations.If the course descriptions containing the terms assessment and evaluation are combined they constitute more than half (59%) the total descriptions studied.If the research methods courses are included in this number as well, the percentage increases to 85%.The results seem to indicate that while the LIS programs studied are addressing these skill sets in the curricula, the terminology employed is inconsistent and therefore the results of this study cannot be generalized.
Given the limitations of this study, it would be worth updating this research in collaboration with LIS faculty to see what, if any, significant changes in the results would occur.By collaborating with the faculty who are teaching these assessment and evaluation courses, researchers will be able to identify more distinctly the courses appropriate to each category.Moreover, further studies addressing the gap in the literature concerning assessment and library science education are warranted and can help inform LIS program curricula as well as professional development needs of practitioners in this area.
Such studies could focus on the following:  Clarify how the terms assessment and evaluation are being used within the course descriptions to establish a shared assessment nomenclature across the profession. Investigate LIS programs affiliated with ARL member libraries to determine to what extent their curricula focuses on assessment. More deeply explore the research methods courses offered by LIS programs to see if they include assessment and evaluation design. Conduct a qualitative study of LIS program faculty to ascertain their views on the importance of assessment skills in the practice of librarianship. Conduct a qualitative survey of Deans/Directors of libraries to determine how skills related to assessment, evaluation and research methods are sought when hiring and how these skills are valued in the organization.
In order to strengthen the role, visibility and effectiveness of our libraries, all information professionals need to be knowledgeable of assessment methods to some extent, but most need to share a common nomenclature.This consensus of terminology will enable the library profession to grow a body of experts who will contribute to the professions' body of knowledge pertaining to assessment.This will enable future librarians and library leaders to become skilled in data driven decision making while enhancing the skills of current practitioners as they navigate the complexities of our professional landscape.
Consuella A. Askew (caskew@fiu.edu) is Associate Dean for Public Services at Florida International University Libraries.Ellen Theodore-Shusta (theodore@ohio.edu) is Director, Planning, Assessment and Organizational Effectiveness for the University Libraries at Ohio University.

Table 2 : LIS Programs with largest concentration of courses
Looking at the number of course descriptions containing the terms assessment, evaluation and research methods, it is apparent where the program emphasis for each might be and at which institutions.Simmons College and the University of Washington LIS programs both have the largest number of courses (7) pertaining to evaluation.The University of California at Los Angeles (UCLA) has the largest number of courses(8)pertaining to research methods.The largest concentration of courses pertaining to assessment is shared equally between the University of Arizona, Emporia State and the University of Illinois.From the data provided in Table2and with the exception of the UCLA program, it appears that if research methods courses are part of the LIS program curricula, then it is likely that assessment and evaluation are not.As a side note, of the 10 LIS programs appearing in Table2, six (60%) are ARL member libraries.Over 79% of the research methods courses offered by the LIS programs listed are in LIS programs with access to an ARL member library.