Thursday, 19 March 2009

How not to do data analysis

io9 presents a chart that purports to show that shark-jumping has an effect on television ratings. I’ll freely concede that Battlestar Galactica has had its, er, weaker moments, but the chart doesn’t actually show that creatively weak episodes had any effect whatsoever on the ratings that can be distinguished from the underlying, secular downward trend in ratings.

Since I had about 300 more important things to do, I decided to analyze the data myself. First, I reentered the ratings data from here into an OpenOffice.org spreadsheet and then identified the “shark-jump” episodes with a dummy variable, with the help of IMDB. I then created two new variables: a simple ratings difference variable for each episode, and a dummy variable to indicate whether or not an episode immediately followed an identified shark-jump.

I then converted to a CSV file, opened R, and estimated a linear regression: Delta = a + b(FollowShark). While the effect of an episode following a shark-jump was negative (about 0.025 ratings points), the effect was not statistically significant (p ≈ 0.736, two-tailed). Throwing out “Razor” and “The Passage,” to focus on episodes io9 says showed ratings losses improves the coefficient to about -0.042 ratings points, but it is still not significant (p ≈ 0.613, two-tailed).

So, the moral of the story: the episodes identified may have been “shark jumps,” but they didn’t seem to have a discernible effect on the ratings of subsequent episodes. And, besides, any analysis that doesn’t identify the crapfest known as “Black Market” as a shark-jumping incident isn’t worth taking seriously to begin with.

Bear in mind that TV ratings themselves leave something to be desired; variations of several tenths of a ratings point are within the expected margin of measurement error.