Tom Maguire suggests that the National Journal finding that John Kerry is the most liberal member of the Senate isn’t supported by Poole and Rosenthal’s NOMINATE scores, at least not over the last two Congresses. He also quibbles:

Any fool can ask a question that ten wise men cannot answer: Dr. Poole bases his rankings on all recorded roll call votes, including the straight party-line organizational votes – for example, all Republicans voted for Bill Frist as Leader, and for the various Republican committee chairpersons. My suspicion is that the results give a good ranking within parties (so Kerry is really a centrist Dem), but the border between Republican and Democrat on substantive votes is blurrier than these results suggest. Objectivity and simplicity might suffer, but has this been looked at?

My (admittedly fuzzy) recollection of NOMINATE is that the results are fairly robust when you exclude pure party-line votes from the input data. A second approach to this question is a recent paper (released Monday!) by Joshua Clinton, Simon Jackman, and Doug Rivers that uses a Bayesian item-response theory model to approach the question (the same method used in their forthcoming APSR piece, a variant of which I used to measure political knowledge in my dissertation); the abstract follows:

We reanalyze the 62 key Senate roll calls of 2003, as identified by National Journal, using a statistical procedure that (1) is sensitive to different rates of abstention across senators and roll calls; (2) allows us to compute margins of errors on voting scores and the ranks of the legislators, as well as compute the probability that a given senator occupies a particular rank (e.g., is the “most liberal” senator). The three Democratic senators running for president in 2003 have markedly higher rates of abstention than the rest of the Senate, leading to considerable uncertainty as to their voting score (particularly for Senator Kerry). In turn, we find that contrary to recent media reports, Senator Kerry (D-MA) is not the “most liberal” senator, or at least not unambiguously; as many as three Senators could plausibly be considered the “most liberal“, with Kerry third on this list behind Senators Reed (D-RI) and Sarbanes (D-MD).

The note lacks any high-powered math, and should be accessible to anyone with an interest in politics and a modicum of statistical knowledge. Incidentally, their method does show a closer overlap between Democrats and Republicans than NOMINATE does (in part because they restricted the analysis to 62 “key” votes rather than all of the roll calls). One other thing to note: the whopping error bar around Kerry’s position, a direct result of his absenteeism from the Senate over the past year.

James Joyner isn’t quite convinced of Jeff Jenkins’ argument that John Kerry is more conservative relative to Democratic presidents (historically) than George Bush is liberal, using Keith Poole and Howard Rosenthal’s NOMINATE method. James writes:

The problem I have with Poole’s coding methodology is that it’s excessively time bound. To compare Bush 43 to Reagan or Kerry to Carter ignores massive shifts in public opinion during those time periods. The “center” is not a spot on a map; it’s a median of current attitudes.

There are actually two versions of Poole and Rosenthal’s methodology. The version Jenkins apparently used for his analysis (from the description in the article) is called W-NOMINATE, and only looks at a particular Congressional session (e.g. the 107th Congress, from 2001 to 2003). There’s a second version, called DW-NOMINATE, that allows comparisons over time between Congresses. In other words, using W-NOMINATE is inappropriate for comparisons over time.* James goes on to write:

I’d think the ACU/ADA ratings are much more useful than Poole’s, since the comparison is made against one’s contemporaries.

Actually, ACU and ADA ratings are essentially interchangeable with W-NOMINATE first dimension scores. But I think James is critiquing Jenkins for something that Jenkins actually didn’t do (even though the article might lead you to think he did).

It seems to me there are two related questions here: is Bush more extreme than Kerry? and, are Bush and Kerry more extreme relative to their partisan predecessors? The first question was pretty clearly answered by Jenkins in the article. The second can’t be answered by the W-NOMINATE method that Jenkins used—which, given his indication that he deliberately simplified the analysis (by using W-NOMINATE instead of DW-NOMINATE), makes it seem odd that he tried to make comparisons over time. The question I think Jenkins answered is “are Bush and Kerry more extreme relative to predecessor presidents vis à vis the Congresses they faced”—and, for that comparison, W-NOMINATE or ADA/ACU scores would work equally as well.

Update: Jeff Jenkins has a comment at Dan’s place that clarifies the situation; he did use DW-NOMINATE for the interyear comparisons, but that point was lost in the editing process. So ignore the above paragraph. ☺ He has some interesting points too in regard to Poole and Rosenthal’s book, Congress: A Political-Economic History of Roll Call Voting.

Also worth pointing out is the forthcoming APSR piece by Doug Rivers, Josh Clinton, and Simon Jackman, “The Statistical Analysis of Roll-Call Data”. There's also a recent issue of Political Analysis in which all of the articles were on ideal-point estimation (which is the technical term for NOMINATE and the Rivers-Clinton-Jackman approach). And, if you want to do it yourself, Andrew Martin and Kevin Quinn have included the Rivers-Clinton-Jackman procedure in their MCMCpack package for GNU R.

I previously discussed Kerry’s ideology here. Dan Drezner also discusses the article in question here.

Stephen Bainbridge (via Glenn Reynolds) isn’t impressed with the use of NOMINATE scores to cast John Kerry as more of a centrist; nor is he particularly thrilled with methods like NOMINATE to begin with:

Personally, I find the interest group scores much more accessible and transparent. For one thing, NOMINATE counts all nonunanimous roll calls, which can include a lot of procedural and uncontroversial (even nonpartisan) bills. The interest group rankings focus on bills that really tell us something about the political philosophy of the candidate in question. For another, the interest group ratings are widely used both by the media and, perhaps more important, by politicians themselves.

I’d respond that NOMINATE (and related methods) are preferable to interest group scores precisely because they count all nonunanimous roll calls; this avoids the selection effect where interest groups choose, say, twenty “key” votes as a litmus test for an entire session. And, presumably, those who vote on party lines on “nonpartisan” and “uncontroversial” bills are even more partisan than those who join with their natural opposition. Another worthy point in favor of NOMINATE: the “procedural” versus “substantive” distinction is largely subjective; cloture votes in the Senate, for example, are technically procedural motions to end debate (and potentially stop a filibuster), while procedural votes on rules in the House often have serious substantive consequences (by ruling certain amendments out-of-order, framing and controlling debate, and sometimes even amending the legislation in question).

Now, Clinton, Jackman, and Rivers are quite correct to point out that the statistical properties of NOMINATE are, at best, nebulous, although Lewis and Poole recently made a worthy effort to gain additional leverage on the bias and uncertainty of NOMINATE in Political Analysis. And, while some of the differences in the results of the techniques are the result of differences between the distributional assumptions of NOMINATE and the CJR scaling method* (which explains the differing positions of Kerry in years in which he missed a lot of roll calls), there are some good reasons to prefer the CJR technique—most notably, it’s significantly more tractable; you can estimate the model almost trivially using MCMCpack.

Anyway, for those with a morbid curiosity about these techniques, the latest American Political Science Review has an article by Clinton, Jackman, and Rivers called “The Statistical Analysis of Roll-Call Data,” which I recommend highly (and which you may or may not be able to access via this link).

I’ve posted previously on NOMINATE and related methods (again, in relation to John Kerry’s voting record) here and here. This is my entry in today’s Beltway Traffic Jam.