The Effect of Policy on CS Education Success Metrics (Part One)

The State of Computer Science Report by Code.org, CSTA, & ECEP Alliance has tracked individual state progress on nine key policies that promote computer science education in schools across the nation for several years.   These nine policies are:

The State of Computer Science Education report also tracks states’ progress on several metrics indicating how well computer science education has spread throughout a state:

• The percentage of public high schools that offer a computer science course
• The percentage of students that are in a school that offers a computer science course
• The percentage of students that are enrolled in a computer science course (see post)
• The percentage of students that take an AP CS Exam (either AP CS A or AP CS Principles) (see post)

The percentage of public high schools that offer a computer science course is the principal metric that the report uses to measure success in a state.   The report makes an initial attempt to correlate the policies it tracks to this metric on page 28:

 

The data shows that policy does matter.  However, the r-squared value of 0.42 is less than what would be considered a strong relationship between the line of regression and the actual percentage of high schools offering CS.  Further, as the report notes: “The correlation between policies and schools offering does not imply causation.”  And because this study and those in this blog only look at a snapshot of data in time, we have no measure of how policy changed the success metrics in a particular state.  

While this finding may be less than compelling, this blog post will examine this metric and the relationship between the number of policies implemented and the other CS education success metrics.  The next post will investigate which of the nine policies correlate with the greatest improvement in each success metric.

States v. Policy Data

Figure 1 below shows how many states have implemented how many of the nine policies.  For a policy to be “implemented” in this study, the policy had to be an unqualified “Yes” in the snapshot of data that I took from the state-by-state details workbook from the 2021 State of CS Education website.    The average number of policies implemented per state is 5.16 policies.   The student-weighted average of 5.48 policies per state indicates that more populated states implemented more policies than less populated ones.    In contrast, while the national average for the percentage of HS offering CS is 51.7%, the per-state average is 57.4%.   More populous states tend to implement more pro-CSEd policies but have a lower percentage of HS offering CS.

Arkansas, Idaho, Indiana, Maryland, and Nevada have implemented all nine policies; Alabama also implemented all nine policies but showed up as eight in the snapshot of data I took in Feb 2022 from the state-by-state workbook associated with the report.  Oregon and Nebraska have implemented none of the policies, and Alaska one.

Figure 2 below shows how many states have implemented each CSEd policy.  P4 Certification and P2 Standards are the most common policies implemented, with almost 80% of states implementing each.  P1 State Plan, P5 Preservice Programs, and P9 Higher Education Admission are the least common policies implemented.  Only about 40% of states implemented P1, P5, and P9.

 

Regression with Number of Policies Implemented

Figure 3 shows a simple linear regression between the number of CSEd policies implemented and each success metric.  The sizes of the dots in the scatterplots reflect the state student population.  However, the student population is not part of the regression analysis.

 

Policy Difference % reflects the % of the state average affected by policy.  Policy Difference % = (State Average – Y-intercept from Regression) / State Average

The R-squared of 0.405 for the regression between the number of policies and % HS offering CS is different from the 0.42 reported in the 2021 State of CS Education report.   However, this difference is small enough to indicate that the techniques used to calculate the regressions and other statistics in this blog are in the spirit of the 2021 report.   Small differences in data retrieval dates and how policies were considered implemented may account for the R-squared differences.

The number of policies implemented better explains the % of HS offering CS than any other success metric.   The R-squared for the regression between the number of policies and % HS offering CS (.405) is higher than for other success metrics.   The R-squared for % of students enrolled in CS and for % of students taking the AP Exam are less than 0.13, indicating almost no relationship.   While the R-squared for % Students in a HS with CS is 0.33, the regression line’s slope is shallow.

Correlation Between Policies and Between Success Metrics

This blog post will finish by looking at how policy implementation correlates with each other and how each success metric correlates with the others.  For example, do states often implement specific policies together?    By focusing policy efforts on improving one success metric, do those efforts also improve other success metric results?   These are the types of questions a correlation analysis will help answer.  The next blog post will investigate how each policy affects each success metric.

Figure 4 above shows the correlation between the implementation of each policy.    Considering the type of data involved, it is comforting that each policy pair is slightly positively correlated.   It also makes sense that the strongest correlations are between:

• P7 Require HS to offer CS and P9 CS as Higher Ed Admission Requirement (0.44)
• P1 State Plan and P6 State CS Position (0.43)
• P4 Certification and P5 Preservice Incentives (0.42)

Interestingly, there is almost no correlation between P4 Certification and P6 State CS Position (.008) and between P2 Standards and P9 Higher Ed Admission Requirements (-.005).

Figure 4 also shows how each policy correlates to the student population within each state.   Interestingly, a larger student population negatively correlates to P6 State CS Position (-0.16), P1 State Plan (-0.06), and even P2 Standards (-0.03). Conversely, a large student population most positively correlates to P3 Funding (0.32).

Figure 5 above shows how each success metric (including the number of policies implemented) correlates.  Some observations from this study:

• The number of policies implemented strongly correlates to the Percentage of HS offering CS (.64) and somewhat to the Percentage of Students in an HS offering CS (.57). However, it is much less correlated to the Percentage of Students enrolled in a CS class or the percentage of students taking an AP CS Exam.
• The highest correlation between success metrics is between the Percentage of HS offering CS and the Percentage of Students in an HS offering CS (.85). This correlation makes sense since if the average size of HS in all states were the same, the correlation would be perfect.
• While the direct correlation between the Number of Policies and the Percentage of Student Enrolled in CS is relatively low (0.36), the percentage of student enrollment strongly correlates to the Percent of HS offering CS (0.60).
• The correlation between the Percentage of Students taking an AP exam and the other success metrics is relatively low (max: 0.40)

---

Please visit the CSEd Analytics page for the underlying data and reports behind this blog post and more nuanced information.  The attached reports will also show how your state compares to others in critical CSEd metrics.  The next post in the #CSEdAnlytics series will explore how each policy relates to each success metric.