Donald R. Woods, "Problem-based Learning: resources to gain the most from PBL," D.R. Woods, Waterdown, ON, ISBN 0-9698725-2-6, revised 1996

E. How to... select instruments for assessment and program evaluation

A variety of instruments have been developed. Some instruments can be used primarily to evaluate the program. These are given the symbol @ in Table E-1. Usually, the results from these instruments are not given to the students. Some instruments can be used to help students understand themselves and to evaluate the program. These are given an * in Table E-1.

Two good resources to help you locate other instruments are:

Buros, O.K., (each year) "The Mental Measurements Yearbook," Gryphon Press, Highland Park, NJ.

Murphy, Linda L. (1994) "Tests in Print IV: an index to tests, test reviews and the literature on specific tests," Buros Institute of Mental Measurements, University of Nebraska-Lincoln, Lincoln, Neb.

Table E-1 lists some of the tests I am familiar with, their availability, specific comments about the test and a citation of who has used this test.

I find it helpful to have some idea as to the types of results I might expect. Although I have used only a few of the instruments, I include in Table E-2 some of the data. The number in [ ] represents the sample standard deviation.

Table E-1 Some instruments used to evaluate program effectiveness

Target skills, ability	Instrument: name and availability		Specifics	Cited or used by
Subject knowledge	Create your own, usual exams.	Use some of the options given in Table 5-1.
Problem solving skills	Heppner's PSI	P.P Heppner (1986). Available as Problem solving inventory from CPP, PO Box 10096, Palo Alto CA 94303-0979	Problem solving confidence, avoidance of engaging in solving difficult problems and a sense of control and a total PSI.	McMaster MPS program [@]
	Billings- Moos [@]	A.G. Billings and R.H. Moos (1981) Available as the Coping Responses Inventory CRI from Psychological Assessment Resources, PO Box 998, Odessa FL 33556.	Identifies 3 methods of coping (active cognitive coping, active behavioral coping and avoidance); and 5 coping indices (logical analysis, information seeking, problem solving, affective regulation and emotional discharge)	McMaster MPS program [@]
	Basadur's creativity [*]	M. Basadur and C.T. Finkbeiner (1985)	scores a "preference for ideation" and "premature closure."	McMaster MPS program [*]; M. Basadur et al. (1986)
	Create your own		Examples given for each process skill in Section D.3	McMaster MPS program
Metacognitive skills (interest, confidence, motivation, study/learning strategy) [see also Problem solving and self-directed learning]	CGF Cognitive Strategies Framework	Weinstein and Mayer (1986); Weinstein (1994a)		Kumar (1994)
	Oddi Continuing Learning Inventory	Oddi (1986)	intelligence, self-confidence, endurance and affiliation	Kumar (1994)
	LASSI [@]	Weinstein et al. (1987) Weinstein (1994b). Available from H&H Publishing Co., 1231 Kapp Drive, Clearwater FL 34625-2116; phone 813 442-7760	77 questions about attitude toward school, motivation to perform certain tasks, time management skill, anxiety toward academic tasks, ability to concentrate, elaboration and organization to foster understanding and recall, ability to select the main ideas, skill in using study aids, self testing strategies and strategies for studying for tests.	Kumar (1994); Nist et al. (1990)
Coping with change	Rotter Locus of control [*]	J. Rotter (1966)	Described in Whetten and Cameron (1991) p. 46. Also can use Heppner's PSI.	McMaster MPS program
	PEEK [@]	Weinstein et al. (1995). Available from H&H Publishing Co., 1231 Kapp Drive, Clearwater FL 34625-2116; phone 813 442-7760	30 questions to explore the students perceptions about the Academic environment in which they will be learning, the emotional reaction to college and the anticipated social pressures.
	Tolerance for ambiguity [@]	Budner (1962)	Described in Whetten and Cameron (1991) p. 48
Ethical decision-making	Defining issues test	James Rest (1979)	Described in Whetten and Cameron (1991) p 39	Mentkowski and Doherty (1984)
Interpersonal group skills	Listening and responding	Johnson (1986) p. 145 ff	Validity and reliability not reported. Unsure as to whether shift occurs with training.
	Trust	Johnson (1986) p. 51 ff
	Conflict resolution	Johnson (1986) p 206 or Johnson and Johnson (1982) p. 282.	Select from list of proverbs those that you prefer. Validity and reliability not reported. Unsure as to whether shift occurs with training.
		Kindler's MODI-self	Kindler (1988) p. 27. relative weighting for 36 approaches. Validity and reliability not reported. Unsure as to whether shift occurs with training.
	Chairperson	MPS inventory with peer feedback; measures individual improvement relative to group norms.	validity and reliability not measured.	MPS program
	Group skills	MPS inventory with peer feedback;	validity and reliability not measured.	MPS program
Self-assessment skills	personal interview based on evidence from student journals.	comparison of self-assessment with tutor assessment.	see Table 5-3. Less than 2% difference.	McMaster MPS program
	see Personal enrichment program: PEP
Lifetime learning skills	Personal enrichment program: PEP	Students set personal goals, criteria, decide on type of evidence and then work to achieve goal		McMaster MPS program
	LASSI and PEEK	Claire Weinstein et al. (1987) and Weinstein et al. (1995)		cited by Nist (1990)
	Teaching feedback plus journal writing	peers give feedback about quality of teaching and students write reflective journal about evidence; include handouts and materials used to teach.
	Branda's PBEE		Divide the self-directed learning program into three parts: issues raising and prioritization; self-study; use of the new knowledge.	Branda (1993) for large classes; many medical schools for individual "stations"
	Approaches to studying [*]	Adelaide Diagnostic Learning Inventory ADLIMS	Biggs (1985)	Newble et al. (1986); Shelley Johannsson
		Ramsden, Paul Lancaster Approaches to Studying Questionnaire LASQ	Entwistle and Ramsden (1981); Ramsden (1983)	Gledhill (1989)
		Knapper's short version of LASQ	18 item inventory	McMaster MPS program; Bertrand and Knapper (1994)
	Learning environment [@]	Knapper short CEQ	24 item 5-point Likert scale identifies 8 dimensions	McMaster MPS program; Bertrand and Knapper (1994)
		Ramsden: CPQ Ramsden (1983)	40 item five point Likert scale identifies 8 dimensions
	Self-directed learning readiness	Guglielmino		Shelley Johannsson
	Perry level [*]	W.S. Moore, Center for the Applications of Developmental Instruction	attitude toward role in learning process	Mentkowski and Doherty (1984); MPS program McMaster,
		Alverno's internal version
		J. Kurfiss/Gainen, see HTGTM
		SID-IV: Developmental Analysis, PO 855, Harrisonburg VA 22801
		Fitch Moore, see HTGTM
	Piaget's level; Test of Cognitive Development	John Renner, School of Education, U. of Oklahoma, Norman, OK 73069	Piaget's levels' shift from concrete to formal thinking	Mentkowski and Doherty (1984)
Critical thinking	Watson-Glaser Critical thinking appraisal [@]	Harcourt, Brace Jovanovich, Inc., 757 3rd Ave., New York, NY 10017		Mentkowski and Doherty (1984)
	California Critical Thinking Skills Test, CCTST [@]	P.A. Facione and N.C. Facione: California Academic Press	Inductive, deductive, analysis, inference and evaluation	Shelley Johannsson
	California Critical Thinking Dispositions Inventory CCTDI [@]	N.C. Facione, P.A. Facione and C.A. Sanchez: California Academic Press	Open-mindedness, analyticity, cognitive maturity, truth seeking, systematicity, inquisitiveness and self-confidence	Shelley Johannsson
	OLSAT: Otis-Lennon School Ability Test	Otis-Lennon School ability; Otis-Lennon (1977)		Herrnstein et al. (1986)
	Cattell Culture Fair Test	Cattell and Cattell (1961)	visual, classification, matrices and inference rules	Herrnstein et al. (1986)
	Cognitive abilities test or DCAT [@]			PS News 64-10
	New Jersey Test of Reasoning skills [@]			PS News 64-10
	WASI: Whimbey analytical skills inventory [@]	Art Whimbey,	IQ reasoning

Table E-2 Some example data from the MPS program N= 30 to 60

Target skills, ability	Instrument: name and measures		Before	After
Problem solving skills	Heppner's PSI	Problem solving confidence (want decrease)	28.69 [5.65]	19.46 [2.54]
		Approach-avoidance (want decrease)	46.08 [11.15]	33.23 [4.94]
		Personal control (want decrease)	17.38 [4.57]	12.77 [4.60]
		PSI (want decrease)	92.15 [17.31]	65.46 [8.45]
	Billings- Moos	active cognitive coping (max 33; want increase)	17.28 [5.8]	20.6 [3.8]
		active behavioral coping (39; want increase)	20.2 [6.25]	23.65 [4.3]
		avoidance (24; want decrease)	5.03 [2.8]	3.8 [2.2]
		using logic (12; want increase)	6.65 [2.4]	9 [1.94]
		seeking information (21; want increase)	10.1 [4.2]	11 [4.3]
		problem solving (15; want increase)	8.53 [3.6]	10.75 [1.8]
		regulating affective behaviour (18; want increase)	9.1 [3.7]	10.5 [1.7]
		emotional discharge (18; want decrease)	4.06 [2.2]	3.95 [2.64]
	Basadur's creativity	Preference for ideation (want increase)	18 [3]	21 [3]
		Premature criticism (want decrease)	24 [5]	21 [5]
Coping with change	Rotter Locus of control	J. Rotter (want decrease)
	Tolerance for ambiguity
Ethical decision-making	Defining issues
Interpersonal group skills	Chairperson skills	Task dimension (max 8, want increase)	6.3	6.6
		Morale dimension (max 8; want increase)	5.8	6.1
Lifetime learning skills	LASSI
	Teaching feedback plus journal writing
	Branda's Triple jump
	Approaches to studying	ADLIMS
		Lancaster
		Knapper Achieving	15.33 [3.49]	16.32 [2.8]
		Reproducing (want decrease)	16.04 [3.1]	12.93 [3.73]
		Search for Meaning (want increase)	14.8 [4.1]	14.33 [3.9]
	Learning environment	Knapper: student centered (want increase)	9.7 [5.0]	13.37 [3.16]
		control centered (want decrease)	17.2 [3.34]	17.7 [3.25]
		good teaching (want increase)	5.4 [3]	8.53 [1.7]
		openness (want increase)	5.2 [2.7]	8.17 [1.8]
		freedom in learning (want increase)	4.3 [2.6]	5.2 [2.2]
		clear goals & assessment (want increase)	5.4 [2.8]	7.57 [2.1]
		vocational relevance (want increase)	6.5 [3]	8.11 [1.76]
		social climate (want increase)	4.8 [2.7]	7.69 [1.87]
		workload	11.5 [4.5]	12.9 [2.8]
		use of formal teaching methods (want decrease)	6.7 [2.3]	6.72 [1.85]
	Self-directed learning readiness	Guglielmino
		Perry 0 to 5 (want increase)	3.66 [0.49]
	Piaget's level; Test of Cognitive Development
Critical thinking	Watson-Glaser Critical thinking appraisal
	California Critical Thinking Skills Test, CCTST
	California Critical Thinking Dispositions Inventory CCTDI
	Cognitive abilities test or DCAT
	New Jersey Test of Reasoning skills
	WASI: Whimbey analytical skills inventory

References

Basadur, M., G.B. Graen and T.A. Scandura (1986) "Training Effects on Attitudes Toward Divergent Thinking among Manufacturing Engineers," J. Applied Psychology 71, 4, 612-617.

Basadur, M. and C.T. Finkbeiner (1985) "Measuring Preference for ideation in Creative Problem Solving Training," J. Applied Behavioral Science, 21, 1, 37-49.

Bertrand, D. and C. Knapper (1994) "Contextual Influences on Student's Approaches to Learning in Three Academic Departments," TRACE, University of Waterloo, Waterloo, Ontario

Biggs, J.B. (1985) "Student Approaches to Learning and Studying," Australian Council for Educational research, Hawthorn, Victoria.

Billings, A.G., and R.H. Moos (1981) "The Role of Coping Responses and Social Resources in Attenuating the Stress of Life Events", J. Behavioral Medicine, 4, 2, 139-157.

Branda, Luis (1993) personal communication, McMaster University, Hamilton ON.

Budner, S. (1962) "Intolerance of ambiguity as a personality variable," J of Personality, 30, 29-50.

Christensen, C.A., D.R. Massey, P.J. Isaacs (1991) "Cognitive Strategies and Study Habits: an analysis of the measurement of tertiary students' learning," British J. of Educational Psychology, 61, 3, 290-299.

Clarke, R.M. (1986) "Students' Approaches to Learning in an Innovative Medical School: a Cross-sectional study," British J. of Educational Psychology, 56, 309-321.

Entwistle, N.E. and P. Ramsden (1983) "Understanding Student Learning, Croom Helm, London.

Gledhill, R.F. and C.A. van der Merwe (1989) "Gender as a Factor in Student Learning," Medical Education, 23, 2, 201-204.

Guglielmino, L.M. and Associates, 734 Marble Way, Bacon Raton, Florida 33432.

Heppner, P.P. (1986) "The PSI Manual", 210 McAlester Hall, University of Missouri-Columbia, Columbia, MO 65211;

Heppner, P.P and C.H. Petersen (1982) "The Development and Implications of a Personal Problem-solving Inventory," J. Counselling Psychology, 29 1, 66-75.

Herrnstein, R.J., R.S. Nickerson, M. de Sanchez and J.A. Swets (1986) "Teaching Thinking Skills," American Psychologist, 41, 11, 1279-1289.

Johannsson, Shelley, School of Nursing, University of Alberta, Edmonton. <sjohannssonn@uahson.ualberta.ca>

Johnson, D.W. and F.P. Johnson (1982) "Joining Together, 2nd edition, Prentice Hall, Englewood Cliffs, NJ

Johnson, D.W. (1986) "Reaching Out: interpersonal effectiveness and self-actualization," 3rd edition, Prentice-Hall

Johnson, J.A., J.A. Sample, W.J. Jones (1988) "Self-directed Learning and personality type in adult degree students," Psychology, A quarterly Journal of Human Behaviour, 25, 1, 32-36.

Kindler, H.S. (1988) "Managing Disagreement Constructively," Crisp Publications, Los Altos, CA.

Kumar, Lavinia, Coordinator Computer Learning Lab, Office of Education, New Jersey Medical School, e-mail post on PBLIST, 10 Nov 1994 <lkumar@undnj.edu>

Mentkowski, M. and A. Doherty (1984) "Careering after College: establishing the validity of abilities learning in college for later careering and professional performance, Alverno College, 3401 South 39th St., Milwaukee, WI 53215

Newble, D. and R.M. Clarke (1986) The Approaches to Learning of students in the traditional and in a innovative problem-based medical school," Medical Education, 20, 267-273.

Nist, S.L., D.L. Mealey, M.L. Simpson and R. Kroc (1990) "Measuring the affective and cognitive growth of regularly admitted and developmental studies students using the Learning and Study Strategies Inventory (LASSI)" Reading Research and Instruction, 30, 1, 44-49.

Oddi, L.F. (1986) "Development and validation of an instrument to identify self-directed continuing learners," Adult Education Quarterly, 36, 2, 97-107.

Ramsden, P. and N.J. Entwistle (1981) "Effects of Academic Departments on Students' Approaches to Studying," British J of Educational Psychology, 51, 368-383.

Ramsden, P. (1983) "The Lancaster Approaches to Studying and Course Perceptions Questionnaire: Lecturer's Handbook," Educational Methods Unit, Oxford Polytechnic, Oxford. UK.

Rest, James (1979) Minnesota Moral Research Projects, 330 Burton Hall, 178 Pillsbury Drive, SE, Minneapolis MN 55455

Roberts, Jackie et al. (1993) "Promoting Adjustment and Reducing Health Service Utilization: the effectiveness and efficiency of three types of Specialty Clinic Care for Chronically Ill Patients", McMaster University, Faculty of Health Sciences.

Rotter, J. (1966) Generalized Expectations for Internal versus External Control of Reinforcement," Psychological Monographs 80 (1, Whole No. 609).

Weinstein, C.E. and R.M. Mayer (1986) "The Teaching of Learning Strategies," in "Handbook of Research on Teaching and Learning, M. Whitrock, ed., MacMillan, NY, 315-327.

Weinstein, C.E., A. Schulte and D.R. Palmer (1987) "Learning and Studies Strategy Inventory (LASSI)" H & H Publishing, 1231 Capp Drive, Clearwater FL 34625 1-800-366-4097 (813-442-7760)

Weinstein, C.E. (1994a) "Strategic Learning Skills," in "Student Motivation, Cognition and Learning: essays in honor of W.J. McKeachie," Paul Pintrich, Don Brown and Claire Weinstein, eds. Erlbaum Publishers, Hillsdale, NJ.

Weinstein, C.E. (1994b) "Students at Risk of Academic Failure: learning to learn classes," in "Handbook of College Teaching: Theory and Application," Keith W. Prichard and R. McLaren Sawyer, eds., Greenwood Press, Westport, Connecticut.

Weinstein, C.E., D.R. Palmer and G.R. Hansen (1995) "Perceptions, Expectations, Emotions and Knowledge about College (PEEK)" H & H Publishing, 1231 Capp Drive, Clearwater FL 34625 1-800-366-4097 (813-442-7760)

Whetten, D.A. and K.S. Cameron (1991) "Developing Management Skills," 2nd ed. Harper Collins, New York, NY.