Newspapers - Read Your Sources!

Update - Apologies for the bad proofreading job in the original blog post. Before writing this, I had spent the entire day writing a book proposal for a technical publisher. I was a bit burned-out.

It's an exciting time for the United States. In the forthcoming days (possibly tomorrow), the Supreme Court of the United States will issue a ruling on whether all or part of the Affordable Care Act is constitutional. Surrounding the hype is an attempt by the media to figure out--that is, predict--the court's ruling before it's released this week. One factoid passed around is that a group of former Supreme Court law clerks and attorneys surveyed after arguments a few months ago suggest that the law will be overturned.

In an article by Business Insider, author Brett LoGiurato writes:

Most former Supreme Court attorneys and clerks believe that the individual mandate to buy health insurance — the signature provision of President Barack Obama's Affordable Care Act — will soon be struck down as unconstitutional by the high court.

That's according to a joint poll by the American Action Forum, Center Forward and Purple Strategies. An astounding 57 percent of clerks and attorneys polled now think that the mandate will be ruled unconstitutional. That's up from just 35 percent in March, before the court held oral arguments on the case.

Read more: http://www.businessinsider.com/obamacare-unconstitutional-supreme-court-barack-obama-health-care-2012-6#ixzz1yko2eVxa

The pic below appears in the actual Business Insider article and is a snapshot from the actual survey.

I'm not sure how Business Insider misunderstood the study, especially because a picture of it (the very same picture above) appears in their article. Clearly the survey does not say that 57 percent of clerks and attorneys think the mandate will be ruled unconstitutional. Rather, it says the average probability given by all the respondents is 57 percent. But can we say that "most" of the survey respondents believe that the mandate will likely be struck down? No, that's wrong, too. We really don't have enough information to draw that conclusion. More on that point in a minute.

This article, by the Washington Post, gets it wrong too:

A new poll of 56 former Supreme Court clerks finds that 57 percent think the individual mandate will be overturned. That’s a 22-point jump from the last time the same group of clerks was surveyed, right before oral arguments. Back then, 35 percent thought the court would toss out the required purchase of health insurance.

Ezra Klien (who loves charts) gets it wrong.

Here, the Wall Street Journal's Joseph Rago, too.

Did anyone get it right? Yes. Laura Green, of the Palm Beach Post, writes:

A poll of former Supreme Court clerks and lawyers who have argued before the justices found that many switched their initial prediction that the court would uphold the sweeping law. The oral argument performance led them to now speculate that there is a 57 percent chance the court will strike down the heart of the law, which requires virtually every American to buy insurance.

Read more: http://www.wptv.com/dpp/news/national/health-care-act-studies-of-oral-arguments-predict-court-will-rule-against-reform-this-week#ixzz1yksclk3m

What can we say about the survey's results?

Not a ton. I question the usefulness of engaging folks on the probability that a dependent, nonrandom event will occur. When the poll's respondents came up with their "probabilities," they were really giving a score to how confident they are in their belief that the Supreme Court will strike down the mandate. Calling this a "probability" is misleading.

The average is above 50%--can't we say that most believe the mandate will be struck down? 

Not really. Consider this scenario. Of the 56 respondents, nine said the probability of the Supreme Court overturning the mandate was 100%; that's a rather slim, if certain, margin. Now consider that the rest -- 47 respondents -- giving the probability of the court overturning the mandate a 49%. In other words, the 47 feel the likelihood is slightly less certain than flipping a coin. Do the math; you'll see that it averages to about 57%. Do those nine respondents in this example constitute a "most believe"?

Survey Source:
Affordable Care Act Survey, June 2012
Sponsored by: American Action Forum, Center Forward, and Purple Strategies