ValidityandValidationMethods

《ValidityandValidationMethods》由会员分享，可在线阅读，更多相关《ValidityandValidationMethods（29页珍藏版）》请在装配图网上搜索。

1、Validityand Validation MethodsWorkshop FlowThe construct of MKT Gain familiarity with the construct of MKT Examine available MKT instruments in the fieldAssessment Design Gain familiarity with the Evidence-Centered Design approach Begin to design a framework for your own assessment Assessment Develo

2、pment Begin to create your own assessment items in line with your frameworkAssessment Validation Learn basic tools for how to refine and validate an assessment Plan next steps for using assessmentsDomain Modeling (Design Pattern)(Define Test Specs)Domain AnalysisDefine item TemplateDefine item Specs

3、Develop Pool of itemsCollect/ Analyze ValidityDataRefineitemsRefineitemsAssemble TestDocument TechnicalInfoAssessment Development ProcessValidity: The Cardinal Virtue of AssessmentThe degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences

4、 and actions based on test scores or other modes of assessment. - Mislevy, Steinberg, and Almond, 2003Validation is a process of accumulating evidence to provide a scientifically sound validity argument to support the intended interpretation of test scores - Standards for Educational and Psychologic

5、al Testing (AERA / APA / NCME, 1999)Jargon Note: Two kinds of “evidence”Assessment ReliabilityThe extent to which an instrument yields consistent, stable, and uniform results over repeated administrations under the same conditions each time Figure obtained from the website: http:/ Steps of item Vali

6、dationStepMethod1. Expert Panel Review (Formative)Alignment and Ratings of items2. Feasibility of itemsThink-Alouds3. Field testingTesting with a large sample4. Expert Panel Review (Summative)Alignment and Ratings of itemsIterative Refinement1. Expert Panel Review(Formative) Are the items aligned wi

7、th The test specifications? Content covered in the curriculum? State or national standards? Is the complexity level aligned with intended use (e.g., target population, grade-level)? Are the items prompts and rubrics aligned?2. Feasibility of Items (Think-Alouds)Does the item make sense to the teache

8、r?Does the item elicit the cognitive processes intended?Can the item be completed in the available time?Can respondents use the diagrams, charts, tables as intended?Is the language clear?Are there differences in approaches by experts and novices (or teachers exposed or not to the relevant instructio

9、n)?SimCalc Example:Think-AloudsSimCalcExpected proportional reasoning:3.5 white x white- = - 3 dark 5 darkFound:Just draw the bars!Proportional Reasoning Problem #3Conducting Think-AloudsSample N: You learn the most in the first 3-6 Who Experts and Novices Low, Medium, and High Achievers Varying in

10、proficiency in EnglishData capture and analysis Data can be extremely rich analyzed with varying levels of detail Often sufficient to do real-time note-taking Videotaping can be helpful Document Problems with item clarity (language, graphics) Response processes What strategies are they using?3. Fiel

11、d TestingItem-level concerns Are there ceiling or floor effects? What is the range of responses we can expect from a variety of teachers? Is the amount of variation in responses sufficient to support statistical analysis? What is the distribution of responses across distracters? Do the items discrim

12、inate among teachers performing at different levels?Assessment-level concerns Are there biases among subgroups? Does the assessment have high internal reliability? What is the factor structure of the test?Key Item Statistic: Percent Correct What percent of people get it correct? Gives us a sense of:

13、The item difficultyThe range of responses Alerts you to potential problems:Floor = roughly 0-10%Ceiling = roughly 85-100%SimCalc Example:Exploratory Results for item #201234Ability Level12345Distracter050100150Count of Teachers Who Chose DistracterN=179SimCalcQuartiles of total test scoreSimCalc Exa

14、mple:Exploratory Results for item #431234Ability Level12345Distracter10203040Count of Teachers Who Chose DistracterN=179SkipSimCalcSimCalc Example:Exploratory Results for item #6ResponseCountCorrect (12)160 (70%)Additive error (8)42 (18%)Other20 (9%)Skip8 (3%)SimCalcConducting a Field Test Test unde

15、r conditions as close to “real” as possible Analogous population of teachers Administration conditions Formatting Scoring Gather and use demographic data Determine sample size based on The number of teachers you can get The kinds of statistical analyses you decide to conduct e.g., 5-10 respondents p

16、er item for fancy statistics Can use simple and fancy statisticsField Testing with Teachers by Mail Purchasing national mailing lists of teachershttp:/ Best practices mailing sequence (Cook et al., 2000)An introductory postcard announcing that a survey will be sentAbout a week later, a packet contai

17、ning the surveyAbout two weeks later, a reminder postcardAbout two weeks later, a second packet containing the survey and a reminder letter About three weeks later, a third appeal postcardSteps of item ValidationStepMethod1. Expert Panel Review (Formative)Alignment and Ratings of items2. Feasibility

18、 of itemsThink-Alouds3. Field testing for psychometric informationTesting with a large sample4. Expert Panel Review (Summative)Alignment and Ratings of itemsIterative Refinement4. Expert Panel Review(Summative) Similar questions as in Step 1 (Formative review) Same or different panel of experts Rati

19、ngs and alignment collected after items are fully refined Results of summative expert panel review provide evidence of alignment of items with standards/curriculum, content validity, and grade-level appropriateness This could be reported in technical documentationSteps of item ValidationStepMethod1.

20、 Expert Panel Review (Formative)Alignment and Ratings of items2. Feasibility of itemsThink-Alouds3. Field testing for psychometric informationTesting with a large sample4. Expert Panel Review (Summative)Alignment and Ratings of itemsIterative RefinementCreating a Validity Argument Integrates all evi

21、dence into a coherent account of the degree to which existing evidence and theory support the intended interpretation of test scoresFor a Sound Validity Argument,at Minimum, Pay Attention toSources of EvidenceProcedures1. Test contentConduct alignment of items to standards/curriculum by content expe

22、rts2. Response processesHave at least one or two teachers do think-aloudsAdminister test to at least one group3. Relationships to other variablesIf possible, conduct one or more of the following:Conduct instructional sensitivity studyCorrelate with existing measuresCorrelate with construct-irrelevan

23、t variables4. Internal structureEstablish internal reliability (alpha)Assess inter-scorer reliability, if there is a rubric5. Consequences of testingBe aware of the limitations of your test, not going beyond intended purposes and its intended role on your projectActivity #5Conduct Think-AloudBreak i

24、nto groups of 3 and select roles1 interviewer1 interviewee1 observer to complete observation recording sheetSelect set of 2 itemsConduct think-alouds. Interviewer and observers take notes on the form in the protocol.Repeat two more times, switching roles, with new items.Revise your own items. Follow

25、ing, we will have a discussion aboutInsights about development of assessment itemsQuestions and challengesBe the observer for your own items!Activity #5Think-Aloud Pointers Find out how long problems take to do Uncover issues of item clarity and level of difficulty Derive a model of the knowledge an

26、d thinking that the students engage when solving each problem. In observation notes, describe: How problems are solved, focusing on the underlying knowledge, skills, and structures of item performance Actions, thought processes, and strategiesActivity #5Think-Aloud Pointers Interviewers SHOULDPrompt

27、 the teacher to keep talkingAsk clarifying questions about what teachers are saying (but not as scaffolding) Interviewers SHOULD NOTHelp teachers in any way during the interview (e.g., no hints, tips, or scaffolding). Be sure to avoid unintentional hints by being more encouraging when answers are co

28、rrect.Steps of item ValidationStepMethod1. Expert Panel Review (Formative)Alignment and Ratings of items2. Feasibility of itemsThink-Alouds3. Field testing for psychometric informationTesting with a large sample4. Expert Panel Review (Summative)Alignment and Ratings of itemsIterative RefinementSome

29、Useful ReferencesValidationAERA, APA, & NCME (1999). Standards for educational and psychological testing. Washington, DC: AERA.Baxter, G. P., Shavelson, R. J., Herman, S. J., Brown, K. A., & Valadez, J. R. (1993). Mathematics performance assessment: Technical quality and diverse student impa

30、ct. Journal for Research in Mathematics Education, 24(3), 190-216.Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington, DC: American Council on Education.Hoag, R. D., Meginbir, L., Khan, Y., & Weatherall, D. (1985). A multi

31、trait-multimethod analysis of the Preschool Behavior Questionnaire. Journal of Abnormal Child Psychology, 13, 119-127.Mehta, P. D., Foorman, B. R., Branum-Martin, L., & Taylor, W. P. (2005). Literacy and a unidimensional multilevel construct: Validation, sources of influence, and implications in

32、 a longitudinal study in Grades 1 to 4. Scientific Studies of Reading, 9, 85-116.Some Useful ReferencesValidation contd Messick, S. (1989). Validity.(In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13103). Messick, S. (1994). The interplay of evidence and consequences in the validation of

33、 performance assessments. Educational Researcher, 23(2), 13-23. Pellegrino, J., Chudowsky, N., Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press. Tremblay, R. E., Vitaro, F., Gagnon, C., Piche, C. & Roy

34、er, N. (1992). A prosocial scale for the Preschool Behavior Questionnaire: Concurrent and predictive correlates. International Journal of Behavioral Development, 15, 227-245. Weir, K., & Duveen, G. (1981). Further development and validation of the Prosocial Behavior Questionnaire for use by teac

35、hers. Journal of Child Psychology and Psychiatry, 22, 357-374.Some Useful ReferencesExpert Panel ReviewWebb, N. L. (2002). Alignment: Powerful tool for focusing instruction, curricula, and assessment. Presentation at the CCSSO State Collaborative on Assessment and Students Standards, San Diego, CA.W

36、ebb, N. L. (2005). Alignment, depth of knowledge, and change. Paper presented at the 50th Annual Meeting of the Florida Educational Research Association, Miami, FL.Think-AloudsEricsson, K. A., & Simon, H. A. (1993). Protocol Analysis: Verbal reports on data. Cambridge, MA: MIT Press.Flaherty, E. G. (1974). The thinking aloud technique and problem-solving ability. Journal of Educational Research, 68, 223-225.PsychometricsCrocker, L., & Algina, J. (1986). Introduction to classical & modern test theory. Orlando, FL: Harcourt Brace Jovanovich, Inc.