Reliability and Validity in Library Assessment

We all know the importance of assessment. We need to gather assessment data in order to determine how well our programs are meeting needs, how our collection is being used, what gaps we may need to fill, etc.

Gathering data can feel fairly straightforward, especially if the data is compiled from a computer system that runs a report. However, if we need to gather data from individuals on their knowledge (perhaps to see what they learned in an instruction session), or from individuals about their experiences with the library, the process can be more challenging. When we create assessments we need to consider both validity and reliability.

Validity refers “to the extent that a test measures what it is supposed to measure” (Center for Innovation in Teaching and Learning, 2021, para. 1). The Standards for Educational and Psychological Testing (2014) defines validity as the “degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (p. 11).

Reliability refers to the extent that an assessment produces consistent results.

In order to gather accurate data it is necessary to determine how you will ensure the validity and reliability of your assessment tools. To do this I suggest the following:

Validity:

• Clearly define the goals for the assessment
• Match assessment questions to each goal
• Ask a group of individuals to review the assessment and explain to you what they think each question means. This will help identify anything that is not clear. 
• Triangulate data–compare the data gathered in the assessment to other data that may be available

Reliability:

• Ask 4-6 questions for each assessment goal (Shrock & Coscarelli, 2008)
• Use a tool to gather assessment data that participants are familiar with (i.e. paper and pencil, Qualtrics, etc.)
• Give participants a similar environment (i.e. the same length of time to take the assessment, a comfortable place, etc.)
• If you include short-answer questions or other types of questions that are not multiple choice or true-false, you will need to ensure that all reviewers use the same scoring system and are trained in what they should look for.

As you embark on library assessment, I hope you consider the validity and reliability of your measures. Taking time to plan for both validity and reliability will help ensure that the data you gather is accurate data you can use.

Many papers and articles have been written about assessment in libraries. I listed some resources below that may be beneficial if you want to explore library assessment in more detail. 

Library Assessment Resources

Library Assessment Conference (Virtual, November)

Assessment and Evaluation in Special Libraries (previous Lucidea post from me)

Library Assessment Fundamentals (from CORE, part of ALA)

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.). (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.apa.org/science/programs/testing/standards

Center for Innovation in Teaching and Learning. (2021). Assessment validity and alignment. University of Illinois at Urbana-Champaign. https://citl.illinois.edu/citl-101/teaching-learning/resources/teaching-across-modalities/teaching-tips-articles/teaching-tips/2021/08/11/assessment-validity-and-alignment

Phelan, C., & Wren, J. (2006). Reliability and validity. UNI Office of Assessment. https://chfasoa.uni.edu/reliabilityandvalidity.htm

Shrock, Sa. A., & Coscarelli, W. C. (2008). Criterion-referenced test development: Technical and legal guidelines for corporate training (3rd ed.). Wiley.