Abstract:
Multilevel regression discontinuity designs have been increasingly used in education research to evaluate the
effectiveness of policy and programs. It is common to ignore a level of nesting in a three-level data structure
(students nested in classrooms/teachers nested in schools), whether unwittingly during data analysis or due to
resource constraints during the planning phase. This study investigates the consequences of ignoring
intermediate or top level in blocked three-level regression discontinuity designs (BIRD3; treatment is at level 1)
during data analysis and planning. Monte Carlo simulation results indicated that ignoring a level during analysis
did not affect the accuracy of treatment effect estimates; however, it affected the precision (standard errors,
power, and Type I error rates). Ignoring the intermediate level did not cause a significant problem. Power rates
were slightly underestimated, whereas Type I error rates were stable. In contrast, ignoring a top-level resulted in
overestimated power rates; however, severe inflation in Type I error deemed this strategy ineffective. As for the
design phase, when the intermediate level was ignored, it is viable to use parameters from a two-level blocked
regression discontinuity model (BIRD2) to plan a BIRD3 design. However, level 2 parameters from the BIRD2
model should be substituted for level 3 parameters in the BIRD3 design. When the top level was ignored, using
parameters from the BIRD2 model to plan a BIRD3 design should be avoided.
