I. Course Description, Course Goal and Course Learning Objectives:

Modern quantitative evaluation techniques are built around counterfactual analysis, a process of understanding how the world changes due to an intervention or program. Counterfactual analysis requires the identification of a control group that represents the world in the absence of the intervention, a treatment group that represents the world after the intervention, and a comparison of the two to determine effect size associated with the program. The randomized control trial is a powerful form of counterfactual analysis, but it is also rare because it is usually expensive or unfeasible because of ethical or logistical constraints. As a result, evaluators have developed a collection of quasi-experimental designs to enable causal analysis in the absence of true experiments. Instead of leveraging randomization to create balanced treatment and control groups, the methods typically construct a comparison group in clever or careful processes in order to generate unbiased estimates of program impact. This course introduces students to the important concepts in research design and common quasi-experimental tools for causal analysis.

The three main learning objectives for the course are:

• Understand the tenants of counterfactual reasoning and potential weaknesses.
• The ability to identify attrition and non-compliance considerations which shape how we interpret treatment effects (the treatment-on-the-treated calculation versus the intention-to-treat calculation).
• Ability to recognize or apply the three main research designs used to identify program effects: difference-in-difference models, reflexive models, and post-test only models.

Course Prerequisites:

This course uses regression concepts covered in CPP 523.

II. Assessment of Student Learning Performance & Proficiency: Keys to Student Success

Students will demonstrate competency in understanding, producing and communicating results of their analyses through the following assignments:

• Weekly labs that provide opportunities to consolidate and apply material from the lectures.
• A mini examination that tests mastery of vocabulary and concepts from readings.
• A research design project.

Assigned work including the exam are used to measure comprehension and skill; the student’s course grade is a direct reflection of demonstrated performance. Students should take stated expectations seriously regarding preparation, conduct, and academic honesty in order to receive a grade reflective of outstanding performance.

Students should be aware that merely completing assigned work in no way guarantees an outstanding grade in the course. To receive an outstanding course grade (using the grading scheme described below and the performance assessment approach noted above) all assigned work should completed on time with careful attention to assignment details.

III. Course Structure and Operations; Performance Expectations

A. Format and Pedagogical Theory

Mastering advanced analytical techniques is like learning a language. You start by mastering basic vocabulary that is specific to statistics. Through your coursework you will become conversant in the domains of regression analysis, research design, and data analysis.

Progress might be slow at first as you work to master core concepts, integrate the building blocks into a coherent mental model of real-world problems, learn to translate technical results into clear narratives for non-technical audiences, and become comfortable with data programming skills. Over time you will find that your thought processes change as you approach problem-solving in a more structured and evidence-based manner, you apply counter-factual reasoning to performance problems, and you start reading the news and viewing scientific evidence differently. You begin to think and speak like a program evaluator.

By the end of this degree you will be conversant in statistics, research design, and data programming. Fluency takes time and will be developed through professional experience. It requires you to practice these skills to develop muscle memory. You can do this through participating in evaluations on the job and gaining experience building and cleaning data sets from scratch. Understand, though, that this degree focuses on building foundations for your career. Don't be nervous if it feels like it's impossible to master all of the material in this program – it is impossible to learn everything in this field in a year.

Similar to immersion in a language, the best way to learn the material is to be consistent in doing course work each day. The more frequently you revisit concepts and practice data programming the more you will absorb. The curriculum has been designed around this approach. Lectures are split into small units, and each unit includes questions to test your understanding of the material. Weekly labs allow you to spend some time applying the material to a specific problem. The final exam at the end of the semester is designed to help you make connections between concepts and consolidate knowledge. You will be much better off spending a small amount of time each day on the material instead of trying to cram everything into a couple of days a week.

Online discussion boards, when used, are design to accomplish three things: (1) allow students to interact with their peers and share ideas and interpretations of the assigned material, (2) such peer-to-peer discussion online helps build professional relationships with potential future colleagues in the field, and (3) the discussions permit the instructor to assess student engagement with the assigned material.

The online discussions are explicitly intended to meet the objectives stated above. They are not intended as another form of "lecture" where the instructor provides commentary and students simply react to that. Rather, the discussions are a chance for peer-to-peer interaction and proactive engagement by each individual student.

The purpose of all exams and assigned written work is also threefold: (1) the assignments and written exam afford students the opportunity to demonstrate substantive understanding of materials covered in course readings, lectures and online discussion, (2) the assignments and exam permit students to develop and demonstrate research, analytic and written communication skills, and (3) the written work permits the instructor to assess student knowledge, skills and ability within this subject domain.

B. Assigned Reading Materials

There is one assigned text for the class, which is available free online:

• Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2011). Impact Evaluation in Practice. The World Bank. Washington. Available free online.

Reference Texts

Each author approaches material in a slightly different way, so different textbooks work for different people. The following texts are recommended as good resources if you would like additional references:

• Bailey, M. A. (2016). Real Stats: Using Econometrics for Political Science and Public Policy. Oxford University Press.
• Bingham, R., & Felbinger, C. (2002). Evaluation in Practice: A Methodological Approach. CQ Press.
• William R.. Shadish, Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage learning.
• Cumming, G. (2013). Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. Routledge.
• Stock, J. H., & Watson, M. W. (2007). Introduction to Econometrics.
• Wooldridge, J. M. (2015). Introductory Econometrics: A Modern Approach. Nelson Education.

In addition to the required textbooks, the instructor will supplement the assigned unit readings with various journal articles, policy reports, or other related material. These will be made available in the course shell.

C. Course Grading System for Assigned Work, including Final Exam:

Your grade will be based on your performance in the following areas:

• Weekly labs
• A short final exam
• A final project

Letter grades comport with a traditional set of intervals:

• 100 – 99% = A+*
• 98 – 94% = A
• 93 – 90% = A-
• 89 – 87% = B+
• 86 – 84% = B
• 83 – 80% = B-

*A+ is given at the instructor’s discretion.

The assigned work for the term comes in the form of four elements, described below:

• Weekly Labs (50%): Each week you will receive a lab that will help you synthesize the lectures from the week though exercises that involve data, analysis, and important formulas from the lectures. These labs contain exercises that are similar in form or difficulty to what will be presented on the final exam. They are graded pass / fail by the instructors based upon an assessment of whether you have sincerely attempted the lab and answered over half of the questions correctly. This is designed to hold you accountable for the material, but not create anxiety about perfection. Each lab is worth 10 points so that you can drop one lab.

• Final Paper - Research Design (40%): You will design a hypothetical evaluation of a real-world program by applying concepts from this class. You do not need to collect and analyze data for the assignment, but you do need to provide background material on the program you decide to evaluate. Your grade will be based upon how complete the assignment is and how you apply concepts from this class. To assist with the direction of this assignment there will be weekly submissions beginning from week 2 through the end of the course that will allow you to receive feedback from me on the final assignment step-by-step. These weekly submissions will not be graded but they will allow you to make progress to your final assignment and receive direct feedback from me.

• Final Exam (10%): The exam will be review of topics covered on the labs. It will test for key vocabular (like the difference between the Intention to Treat measure and the Treatment on the Treated measure), and concepts like tests for non-random attrition.

D. General Grading Rubric for Written Work

In general, any submitted work written work (assignments and/or exams) is assessed on these evaluative criteria:

• Assignment completeness – all elements of the assignment are addressed
• Quality of analysis – substantively rigorous in addressing the assignment
• Demonstrated synthesis of core concepts from lecture notes and ability to apply to new problems

Most assignments in this course are labs that are graded pass-fail based upon completeness and correctness of responses (every attempt must be made to complete labs, and they must be more than 50% correct to receive credit). Discussion boards that accumulate points through each activity on the board.

The final project will be accompanied by a rubric describing the allocation of points and criteria for evaluation.

E. Late and Missing Assignments

Grades for the course are largely based on weekly labs. Assigned work is accompanied by detailed instructions, adequate time for completion and opportunities to consult the instructor with questions. As a result, each assignment element in the course is expected to be completed in a timely fashion by the due date. Once solutions are posted it is no longer possible to receive points for assignments.

F. Course Communications and Instructor Feedback:

Course content is hosted on this website. Lecture files, assignments and other course communications will be transmitted via this site and/or through the class email list. All assignment submissions will be made through the Canvas shell.

Please post lab questions on the Get Help page on this site, schedule individual office hours using the Calendly link provided above, and email the instructor directly instead of using the Canvas system.

Students should be aware that the course instructor will attempt to respond to any course-related email as quickly as possible. Students are asked to allow between 24 and 48 hours for replies to direct instructor emails, generally, as a reasonable time to reply to questions or other issues posed in an email. Additionally, the general timeline for instructor grading or other feedback on assignments, either writer work or online discussion work, is between 5 and 10 work days.

I. Student Learning Environment: Accommodations

Disability Accommodations: Students should be fully aware that the Arizona State University, the MA in EMHS program, and all program course instructors are committed to providing reasonable accommodation and access to programs and services to persons with disabilities. Students with disabilities who wish to seek academic accommodations must contact the ASU Disability Resources Center directly. Information on the Center's procedures, resources and how to contact its staff can be found here: https://eoss.asu.edu/drc/. The Disability Resources Center is responsible for reviewing any student's requests; once that review has taken place, the Center will provide the student with appropriate information on academic accommodations which in turn will be provided to the course instructor.

Religious accommodations: Students will not be penalized for missing an assignment due solely to a religious holiday/observance, but as this class operates with a fairly flexible schedule, all efforts should be made to complete work within the required timeframe. If this is not possible, students must notify the instructor as far in advance as possible in order to make an alternative arrangement.

Military Accommodations: A student who is a member of the National Guard, Reserve, or other branch of the armed forces and is unable to complete classes because of military activation may request complete or partial unrestricted administrative withdrawals or incompletes depending on the timing of the activation. For more information see ASU policy USI 201-18.

IV. Course Schedule and Unit-specific Learning Objectives

A. Schedule: Overview of Readings and Assignments

As students are all aware, ASU Online courses are typically offered on a seven and a half week schedule. A schedule for each week of the term is outlined here; the course is divided into seven units with specific learning objectives for each unit.

Please note: the course instructor may from time to time adjust assigned readings or adjust the due dates for assignment. The basic course content approach and learning objectives will not change, but slight modifications are possible if circumstances warrant an adjustment.

Couse Schedule

Unit 1 - Counterfactual Reasoning

Unit 2 - Varieties of the Counterfactual

Unit 3 - Campbell Scores

Unit 4 - Campbell Scores Continued

Unit 5 - Effect Size

Unit 6 - Work on Research Design Paper