Core Competencies for Assessment in Libraries A Review and Analysis of Job Postings

The authors analyzed 231 job postings that appeared on library job lists over the span of 18 months. ATLAS.ti, a textual coding software, was used to facilitate the contextualization of the information. Jobs were included in the study if the postings addressed evaluation or assessment responsibilities. Key skills and knowledge areas required for assessment are identified. Opportunities for developing training are suggested. Areas of librarianship showing the most growth in assessment are identified.


Introduction
Assessment continues to grow within the library profession.Over the past 10 years the profession has seen an increase in the number of conferences and other training programs devoted to assessment.While the work of LLAMA's Measurement, Assessment, and Evaluation Section (MAES) has always encompassed assessment in some way, other library associations have more recently adopted it as a focus.
The Association of College and Research Libraries' Immersion program, begun in 1999, 1 included assessment early on in its training of teaching librarians.More recently the program has added an assessment track that provides an even greater focus on developing a framework for the measurement of student learning and demonstrating institutional value.The Association of Research Libraries (ARL) began its Library Assessment Conference in 2006.As part of that conference, ARL has started a "boot camp" for beginning assessment librarians.The Northumbria International Conference on Performance Measurement in Libraries and Information Services and the International Conference on Qualitative and Quantitative Methods in Libraries provide global opportunities.Within the United States, regional offerings, such as the Southeastern Library Assessment Conference, have emerged and state library associations have begun offering assessment workshops in the past few years.
Assessment has an increasing presence in our professional literature as well.The researchers searched for the term "assessment" in the EBSCO databases Library and Information Science Full Text and Library and Information Science Abstracts.For the years 1984 through 1998, the number of articles retrieved was 3,261; less than 50% of those articles (1,276) were peer reviewed.For the years 1999 through 2013, the number of articles retrieved was 10,275; 6,686 of those, or a little more than 65%, were peer-reviewed articles.This growth in peer-reviewed literature shows not only an increase in scholarship but also suggests a maturing of the field and the increasing importance of assessment in library planning.In order to determine the impact on library jobs, the LLAMA-MAES Education Committee decided to collect and analyze assessment jobs that were posted in the past one and a half years.This paper is a summary of those findings.An examination of assessment jobs will be preceded by an analysis of job postings with the term "assessment" or "evaluation" in the descriptions.This paper complements Consuella Askew and Eleanor Theodore-Shusta's (2013) examination of the presence of assessment in the curriculum of Master's programs in information and library science.

Literature Review
The literature on library assessment competencies is limited.Scott Walter and Megan Oakleaf presented an initial analysis of job postings related to assessment at the 2010 ARL Library Assessment Conference.The questions they ask that parallel the research in this paper are: 1. "Are academic libraries recruiting for assessment skills?" 2. "If they are, are they doing so in a meaningful way?" Their paper analyzed job postings and found that the skills required to do assessment were generalized or soft skills, such as the ability to "identify data needs" or to "communicate and report assessment results". 2The jobs that described a concrete assessment skills, such as working with LibQUAL+ ® were preferred qualifications not essential.The writers noted that assessment job descriptions lacked details or a clear scope compared to job postings for subject or instruction librarians.Based on the lack of clarity of assessment skills in the positions analyzed from 2004 to 2009, the authors concluded that the profession had not yet developed a culture of assessment.
Oakleaf continued exploring assessment librarian qualifications, proposing a guild approach to the skill development of assessment librarians. 3evious research analyzing job postings spans a broad array of library position types.The author of an analysis of digital librarian job postings searched titles and descriptions in College and Research Library News (CRLN) using the term "digital." 4This study of digital librarian postings found that "current awareness and appropriate technological skills and experience in the digital library environment, knowledge and experience in creation and management of digital information, and metadata are the most required qualifications for digital librarian positions with high emphasis on management skills." 5Yingqi Tang studied job postings for distance education librarians from 1970 and 2010.This study gathered job postings from "Career Lead" in the journal American Libraries and organized them into three categories: position profile; duties/responsibilities; and qualifications.After analyzing the data in an Excel spreadsheet, the researcher found that the job postings were diverse but leaned towards public services.The author concluded that technology skills will be increasingly important and required for these types of positions in the future. 6A study of music librarian job postings from 2002 to 2010 analyzed jobs from the Music Library Association job website.The author developed criteria for determining which job postings would be included in the study and included both professional and nonprofessional positions. 7Margaret Butler's 2008 study of interlibrary loan, document delivery and electronic reserves positions included job descriptions from a variety of library types, including large and small public libraries, a large academic library, and academic law libraries. 8The author found that a variety of competencies were touched upon and that the description of copyright and ethics was a component to some of the job descriptions in this pool.Wang, Tang and Knight conducted a content analysis of reference job descriptions found in CRLN covering a 44 year span.The jobs were categorized by frequency of positions, educational background, duties/responsibilities and variety of job titles.This study showed that this area of the profession is still strong and perhaps even growing in importance, with the inclusion of skills such as liaison activity and outreach. 9ere have been several research studies on school library job competencies.Two studies found that library media specialists indicated skills needed that were not addressed in their library school programs. 10Buttlar and Du Mont analyzed library school alumni attitudes towards the usefulness of the competencies in their MLIS program.They solicited information through a questionnaire rating the usefulness of a competencies using a scale: 1-essential; 2-very useful; 3-somewhat useful; 4-not very useful.The researchers emphasized the dynamic nature of the library profession and the need for library school curricula to address a variety of work environments.Their study calls for greater attention to communication skills and a focus on service orientation. 11

Terminology
Askew and Theodore-Shusta argue that having a clear "terminological consensus" for assessment is important to the profession. 12Within the library profession, Thomas Angelo defines assessment "as a means for focusing collective attention, examining assumptions, and creating a shared culture dedicated to understanding and continuously improving the quality of library services. 13The Academic Research Libraries Assessment blog defines assessment as: "any activities that seek to measure the library's impact on teaching, learning and research as well as initiatives that seek to identify user needs or gauge user perceptions or satisfaction with the overall goal being the data-based and user-centered continuous improvement of our collections and services." 14ter Gray's definition of evaluation is a method that "provides explicit information through objective tests and measures guided by precisely specified, if not behavioral objectives to make a value judgment." 15kew and Theodore-Shusta argue that evaluation is a more finite process, while assessment is more cyclical in that practitioners collect data to inform continuous improvement. 16This review of library job postings will examine whether these definitions in fact coincide with the required and desired skills found in the descriptions.

Methodology
This study's researchers reviewed several library job-posting websites.Different terms were used when searching the job list sites to capture the various ways assessment positions might be described.The jobs relevant to the scope of this research fell into two major categories with each category further divided into sub-categories.The first category grouped all jobs with the term "assessment" in the position title or with assessment as the main focus of the position.The second category grouped all postings that had the term "assessment" and/or related terms mentioned in the job description.The use of ATLAS.tiqualitative analysis software facilitated the content analysis of the terms within the job postings.

Job Posting Sites
Informed by the results of their literature review, the researchers searched six library job sites periodically from summer 2012 through winter 2014.More than one job-listing site was used because unlike many of the previous library job description studies, the scope of this research project included a review of all types of libraries.While two sites focus on academic libraries, the other four include multiple library types and therefore yielded job descriptions in a variety of library settings.

Terms
The researchers used the terms "assessment," "evaluation," "metrics," and "strategic" to search all the job sites.By keeping the focus on these specific terms, the researchers kept the project manageable, while at the same time casting a broad enough net to ensure that few job listings related to assessment would be missed.

Content Analysis
The writers used ATLAS.tisoftware for the content analysis because it is designed to facilitate a clear and deep understanding of the text analyzed for a research project.ATLAS.tiforces a grounded theory approach, meaning that the conclusions are drawn from the data itself rather than from the researchers preconceived notions.In this project, ATLAS.tihelped identify textual codes to represent the skills listed as necessary for assessment librarians in the job descriptions.An analysis of the co-occurrences of codes illuminated whether these different skills related to each other in a meaningful manner.This would help answer Walter and Oakleaf's question about whether assessment librarians were being recruited in a meaningful way.The researchers also used an Excel spreadsheet to create tables and pivot tables, which facilitated the counting of terms and the creation of illustrative graphs.
The terms specifically searched and contextualized, within the job postings mainly focused on assessment are shown in figure 1: Terms and phrases that became more common with later job descriptions, including "datadriven decisions," "evidence-based decisions," and "Balanced Scorecard" were retrospectively analyzed.Other duties and years of experience (where applicable) were also analyzed.A content analysis averaged an hour per assessment job in the first category.

Assessment jobs
There were 44 job postings found in which assessment was the main focus of the job.Of the 44 job postings three were not used because the postings were old and the full descriptions were not available.
ATLAS.ti provides a way to quantify textual data.The researchers created more than 50 codes using ATLAS.ti.The most common code, assessment was mentioned 333 times.In examining the context for the term "assessment," the researchers found that the term encompassed a variety of meanings including evaluation, analysis, communication, and/or program development.Out of the codes (like assessment), code families, super codes and finally networks were created.The number of times that codes overlapped or co-occurred influenced the development of code families and networks.
Figure 2 shows the ATLAS.tiwork space.Number 1, in the upper left corner shows the code manager button, which provides access to all of the codes.Below that, number 2 identifies some code families and super codes, which were created through the analysis.Number 3, in the center of the figure, shows the development of an early network.

Building a Framework for Assessment Core Competencies
A network grew from the analysis using ATLAS.ti.Networks are important because they show the relationship of different terms and ideas to each other.The network in figure 3 shows three main branches of an assessment librarian's job responsibilities and competencies, as well as the sub-groups or code families.The skills or methods most often mentioned in the job postings are shown in figure 5. Usability testing was mentioned 12 times.The broad phrase "knowledge of qualitative methods" was mentioned 27 times, while "knowledge of quantitative methods" was mentioned 24 times.

Organizational Assessment
Other areas that are not directly related to skills but comprise an important part of assessment include strategic planning, program improvement, and developing a culture of assessment.While specific skills are important, assessment is a forwarding thinking, big picture process that looks to external impacts and stakeholders.Organizational assessment is defined as systematic and relating to the library organization from both an internal and external (campuswide) perspective.The concepts of program improvement and building a culture of assessment are key strategies in the library assessment community.These meta-ideas cannot be reduced to core skills, but for an assessment librarian to be successful in developing a program, these ideas must be understood.The researchers categorized these codes found in ATLAS.ti as Assessment-Big Picture.The code family in this group included the codes program improvement, culture of assessment, and strategic planning.Figure 7 provides an itemized list with the corresponding frequency for each item.Evidence-based Decisions 7 The results in figure 7 show that less than 50 percent of the job postings tied assessment to strategic planning.Even fewer job postings mentioned developing a culture of assessment.At the same time, the phrases "data-driven decisions" and "evidence-based decisions" were either in the same sentence as the term "strategic planning" or in nearby sentences in the same paragraph.

Details Related to Skills
The experience of the researchers affirmed that an effective assessment librarian must understand some core concepts, such as rigor and validity in the context of social science research methodology.The ability to understand these concepts is essential to developing substantial tools, worthwhile data, and solid analyses.However, these concepts were only mentioned once in the same job posting.
The concept of showing the value or the added-value that libraries bring to a campus was only mentioned six times.The concept of showing the impact the library has on the campus was only mentioned six times in four job postings.The types of jobs that required showing impact included a wide range of jobs from entry level to administrative (associate dean).

Non-assessment Jobs with the term Assessment or Evaluation in the Job Posting
The second major data set examined was comprised of jobs that mentioned "assessment," "evaluation," or "strategic" in the job description, but whose job titles focused on something other than assessment.The final tally of this category included 187 jobs.This category was further divided into eleven sub-categories.Figure 8 depicts the sub-categories.

Details on Sub-categories
Access services included jobs with the phrase "access services" in the job title or with circulation or stacks responsibilities in the job description.These jobs usually included assessment tasks related to collection usage and circulation statistics.Administrative positions included Associate Deans, Directors, Deans, or Library Systems Heads.These jobs usually included skills in assessment related to strategic goals and planning.A few jobs in this category included the term "evaluation" in the context of job performance, workflows, and programs.One position required knowledge of SAS or SPSS.Another identified an understanding of the Balanced Scorecard as important.Cataloging positions often included terms such as "access" and "discovery"; metadata or records were emphasized in this job category.Assessment duties included collections usage statistics, and web analytics.Collections included jobs identified as having responsibility for digital collections.Information Technology (IT) included general information technology jobs, digital library jobs, a user testing job and a library systems job.
Often the digital library jobs, and user testing jobs blended skills in collections, technical services, and public services areas but were located in IT or had a greater focus on IT.The assessment duties for these jobs related to usage statistics, and knowledge of user research needs.There were only a few emerging technology positions.
Archives/Special Collections/Preservation jobs included traditional archivist jobs and digital archivists, a few special collections department heads and a digital library-preservation job.Public services/reference included reference jobs that included responsibilities for collection development, reference services, subject area specialty and in-class instruction.Assessment responsibilities in public services included review of electronic and in-person reference statistics, understanding collection development data, and basic instruction evaluation.User behavior services were a diverse category because of the descriptions.These jobs included usability testing, understanding user behaviors, and web design.The assessment aspect of the needed skills included knowledge of methods for evaluating user behaviors and usability testing.
Most jobs fell under public services, but a few fell under information technology or in a web design department.Skills included usability testing, user studies, and ethnographic approaches, but no one job in this category required all of these skills.The other category included a library consulting business for school libraries and jobs involving scholarly communications that appeared to be outside of the public services area and were more collections oriented and/or copyright focused.
Teaching librarians formed the largest segment of non-assessment librarian jobs that had assessment as part of the job responsibilities.Most of these jobs fell under public services.Many of the jobs included some reference responsibilities.Assessment skills related to these included knowledge of learning outcomes assessment, or developing learning assessment with campus faculty, and faculty development assessments.Jobs in this category that were department heads (instruction coordinators, etc.) also included assessment skills related to strategic planning.A few teaching librarian jobs identified skills in developing online learning or distance education assessment.
Assessment skills were more important and detailed more fully in administrative and instruction jobs compared to other non-assessment jobs.Additionally, these two job categories identified strategic planning and strategic assessment as important.
On average, instruction jobs mentioned assessment seven times.Two user behavior services jobs each mentioned assessment four times.A cataloging job, a public services position, a reference job, and an instruction job each mentioned assessment three times in their job postings.Twelve jobs from a variety of areas mentioned assessment two times.The rest of the jobs mentioned assessment only once in the job description.

University or College Job Postings versus Other Types of Libraries
All of the job postings with assessment in the job title were from libraries at four-year colleges or universities.The majority of jobs were in ARL member libraries.For jobs that included assessment as part of the job, but for which assessment was not the main focus, a wide array of college and university types were represented.Figure 9 illustrates the breakdown of the types of libraries that posted jobs with the term "assessment" or "evaluation" in the job description.The researchers found using the term "evaluation" more effective in finding a greater diversity of library types with job postings that identified some aspect of assessment as part of the skill set.
Of the 167 jobs found using the term "evaluation" two thirds also had the term "assessment"."Assessment" appears to be a term used primarily by academic libraries.

Assessment versus Evaluation
The researchers wanted to understand if the job postings differentiated between assessment and evaluation in a clear manner.In the 187 job postings reviewed, the term "assess" appeared 120 times.The term "evaluate" was mentioned 94 times.Most of the time, the use of the term "assess" did in fact refer to examining a program in a cyclical manner.The term "evaluate" was often used to imply a finite process, but it also often included undertones of assessment.Phrases such as "continuous evaluation" or "systematic evaluation" implied that the process was indeed cyclical and would meet the definition of assessment.At the same time, the use of the terms "assess" and "evaluate" were specific to a program or job, and not contextualized within the larger library landscape (at least within the job posting description).Therefore, this group of jobs did not have a "big picture" or larger organizational competency embedded in the evaluative or assessment competency.The instructional jobs and the administrative jobs were the exceptions, but the details of how these jobs were to assess or evaluate were vague at best."Outcomes assessment" was mentioned 11 times, with 10 of the occurrences referencing learning outcomes related to instructional positions.The other mention of outcomes related to technology for a director of technical and information systems/instructional services.Additionally, strategic planning was mentioned 44 times in 39 jobs.Almost all of these jobs were administrative or instructional.A significant minority were metadata librarians or librarians in charge of digital collections.These jobs, which were collections and access focused, often seemed to use evaluation and assessment interchangeably.Only instructional jobs seemed to focus on a well-defined concept of assessment.This view was learner centric and outward looking, using assessment to address campus initiatives or stakeholders concerns.

Discussion/Recommendations
Positions identified as "assessment librarians" are exclusively in the domain of four-year colleges and research universities.These positions range from entry level to high level administration, with most of the jobs requiring at least three years of experience.While a variety of library types were included in this study, it appears from the research that no other library type (public, special, school or community college) specifically recruited assessment librarians.At the same time, a variety of libraries include an assessment and/or evaluation component in many of their job postings."Evaluation" appears to be the predominant term used in special and public libraries.

Developing the Assessment Profession
From the findings of Askew and Theodore-Shusta, there is a clear need to provide more professional support to librarians who might be interested in obtaining an assessment job, given that assessment is not generally covered in library and information science curricula (Askew,  2013).Working with the professional library and information schools is an important strategy in growing the profession.Liaisons within ALA divisions could work collaboratively with the Association of Library and Information Science Education (ALISE) and communicate to library schools the importance of offering coursework on assessment topics.From the top eight competencies found in figure 1 it is possible to identify discrete skill development that could be offered through assessment workshops, certificate programs, or even a minor in library assessment.The educational programs could include modules on data analysis, user experience, basic statistics, and qualitative and quantitative data.Since communicating assessment results was a frequently cited qualification in job descriptions, the programs could provide opportunities to learn presentation skills, from basic graphic design rules to effective oral presentations.Some of the assessment job postings also mention major surveys such as LibQUAL+ ® , MINES and NSSE.Additionally, some job postings also mentioned familiarity with organizations such as EDUCAUSE, American Association of University Professors (AAUP), and ARL.Another important component of any assessment professional development for librarians should include background on the major library assessment initiatives, such as the Value of Academic Libraries proposition, Balanced Scorecard, and LibQUAL+ ® .Any assessment workshop or development opportunity should also include information about the major library assessment conferences.

Broadening the Scope of Assessment
Given that the assessment librarian positions found for this study all came from university or college libraries, opening a dialogue with leaders from public libraries, special libraries and school libraries might help develop a broader awareness of how assessment positions can impact a library or library system.Is it possible assessment is wrapped-up in job postings using terminology other than "assessment" or "evaluation" in the job title or position description?Might assessment be an assumed skillset for librarians in non-academic types of libraries?Further investigation with other library types would facilitate a better understanding of these questions and enrich the assessment conversation across all library types.LLAMA MAES is uniquely positioned to facilitate this conversation.

Limitations
It is possible that the terms used for this research were too limiting and did not capture job postings from some types of libraries.Both researchers come from academic library backgrounds.While information was sought through practitioners in public and special libraries, no clear path to locating these types of positions was identified.Additionally, a limited number of assessment librarian job postings were found and analyzed.The time period for the study was also a limiting factor since only jobs posted within an 18-month period were included in the study.While this research provides a detailed picture of the job postings found during the time period of the study, a larger body of postings would be needed to understand all the competencies a librarian might need to obtain an assessment job.This is a qualitative analysis that showed a pattern regarding required job skills, level of influence, and years of experience.At the same time, some of the job postings had additional skills and/or assignments.This makes it hard to say definitively what library administrators want from an assessment librarian.

Conclusion
This study expands on the research questions posed by Walter and Oakleaf (2010), while also looking at the library profession as whole, by analyzing job postings for public, special and school libraries.This study complements Askew and Theodore-Shusta's 2013 study of assessment coverage in course offerings in library school programs.Based on the analysis of the job descriptions in the current study, the researchers conclude that the profession is beginning to develop a culture of assessment.The job descriptions found and analyzed in the current study indicate that assessment is being incorporated into new areas of the field, such as emerging technologies as well as more traditional areas, such as cataloging.The clarity of assessment skills in the job descriptions analyzed in the current study has improved compared to those reviewed in the Walter and Oakleaf 2010 study.All of this suggests that the profession is becoming more intentional in its recruiting for assessment skills and that libraries are working towards building a culture of assessment in their organizations.
The researchers for this project reviewed the specific assessment skills and "soft" skills mentioned repeatedly in the job postings.From reviewing the data and incorporating both researchers' experiences as assessment professionals, they developed five essential knowledge and skill areas that could form the basis of educational modules.Appendix A illustrates these knowledge and skill areas.
Academic library initiatives are the main focus in the modules.Assessment trends and major tools from other libraries (public, special, schools) could be added or substituted.Note that the first module includes a historical perspective on the field of library assessment and the broader assessment context.While this is something that would never be articulated in job postings, the researchers feel that understanding the past and present growth of library assessment is imperative for new assessment librarians.The other modules reflect the general "soft" skills previously mentioned and reflect specific assessment-related skills (survey development, data analysis).Modules four and five detail broader skills that would benefit any professional.The reason for the inclusion of both of these modules is because the assessment job postings repeatedly mentioned both these skill sets as being important for hiring a successful assessment librarian.
The 187 jobs with some aspect of assessment in the description, but not in the job title, told a different story.These job postings were not recruiting in an intentional manner for assessment, and in fact the terms "assessment" and "evaluation" were used interchangeably.Assessment was not as prevalent in the responsibilities of traditional jobs, such as reference or access services positions (the one exception being instruction librarians).The inclusion of assessment in new jobs--digital scholarship, user behavior services, and programming-oriented jobs (XML)-were more common than expected.It was common to find some component of assessment or evaluation in the new non-traditional jobs.Could this mean that assessment is part of the new wave of jobs and the new landscape that libraries are a part of and at the same time forging?Are more assessment skills going to be explicitly embedded in all types of library jobs?If this is the case, general assessment skills identification is needed for different types of jobs.Only the administrative and instruction librarian jobs consistently identified the need for a set of assessment skills.For administrative job postings, the skills required tended to involve strategic planning and data analysis.For the instruction positions, the focus was on learning outcomes assessment.The move towards more accountability in higher education and state and local government has resulted in a greater need to demonstrate value and provide data as evidence of effectiveness.
Both instruction librarians and library administrators would assume responsibility for this accountability, and it stands to reason that this is where we would likely see a more clearly articulated understanding of library assessment.
Assessment is an important part of the growth and health of the library profession.The recommendations from this paper complement previous research by LLAMA MAES.It is important that as a profession we talk across library types to provide a more holistic view of evaluation and assessment in order to leverage educational opportunities to grow the skills needed in our librarians today.LLAMA MAES is perfectly situated in its support of all library types to continue this research.Partnership with other library associations (Public Library Association, Association of Specialized and Cooperative Library Agencies, ACRL, and ALISE) is recommended for the broadest input and greatest impact.

Figure 2 :Figure 3 :
Figure 2: A few of the codes are seen on the right side.The numbers on the left show the manager button, the names of the networks and a few early networks

Figure
Figure 4: Break down by frequency of each branch in the network

Figure
Figure 5: Top assessment methods/tools mentioned in assessment job postings

Figure 6 :
Figure 6: Software Packages Identified in Assessment Job Postings SPSS 13 Microsoft Excel 8 ATLAS.ti 5 Integrated Library Management Systems 3 SAS 3 Microsoft Access (database) 3 STATA 2

5: Top assessment methods/tools mentioned in assessment job postings
The job postings included a variety of software packages candidates should know.Most often the job postings required that the candidate know the identified software and be able to create reports, presentations or other data visualizations for the library as a whole or for individuals in a department.Quantitative packages, such as SPSS or SAS, were mentioned more often than qualitative software packages, such as ATLAS.ti.Software packages mentioned only once included Cognos Bl, Counter, Microsoft Access, NVivo, Quickbase, R, and SUSHI.This list includes a diversity of applications.SUSHI and COUNTER represent standard compliant applications that manage collection usage data.Cognos Bl is a robust IBM product.Quickbase is a database cloud application.Microsoft Access is desktop database software.R is a complex, free software often used for quantitative analysis.NVivo is a qualitative software package.The survey tools mentioned (LibQUAL+ ® , ClimateQUAL and MINES) are ARL