A Detailed Breakdown

To investigate how well Bernie Sanders’s poll numbers would hold up in a general election compared to his Democratic rivals, the authors designed an original online survey asking respondents how they would vote in head-to-head general election scenarios pitting Donald Trump against five different potential Democratic nominees: Bernie Sanders, Joe Biden, Pete Buttigieg, Michael Bloomberg, and Elizabeth Warren.

With a sample size of more than 40,000 respondents — dozens of times as large as most commercial polls — the survey allowed the authors to slice the data finely without introducing significant sampling error. Importantly, it also allowed them to avoid having to ask every respondent about every potential matchup — a practice likely to lead in many cases to insincere, strategic answers.

Instead, each respondent was asked about only one, randomly selected matchup, and answers from demographically similar respondents were pooled together to estimate how those voters’ election behavior would likely shift depending on which Democrat was on the general election ballot.

The authors stress that their study was not intended to predict the absolute levels of voter support for Democrats or the outcome of the November election, since polls so early in the election cycle are seen as unreliable predictors of the final vote. (The survey was conducted in January and February of this year.)

Rather, the question the paper asked was simply: is Bernie Sanders, the socialist, really as electable as his moderate rivals — as most polls seem to indicate?

The Spurious “11% Turnout Surge”

The authors’ argument unfolds in three logical steps.

First, they confirm the findings of the many public polls that seem to show Sanders to be at least as electable as the moderate candidates. In the authors’ survey, when responses are weighted to mirror the demographics of the entire eligible electorate, Sanders comes out on top in head-to-head matchups against Trump.

The margins against Trump that they find in this initial cut of the data are: Sanders: +4.3%; Biden: +4.3%; Bloomberg: +4.1%; Buttigieg: +1.9%; Warren: +1.1%:

Their second step is to show that these headline figures conceal two crucial patterns going on under the surface.

First, the presence of Sanders on the ballot drives a small but significant number of moderate voters into the Trump camp: about 2 percent of Republicans, for example, say they wouldn’t vote for Trump if his opponent were one of the moderate Democrats, but would vote to reelect him if Sanders were the alternative.

The other pattern that emerges from the data explains how Sanders manages to offset those losses: he attracts a significant bloc of young people — overwhelmingly Democrats and independents under the age of 35 — who boost his relative standing in the polls by telling pollsters that they will vote for Sanders if he’s the nominee but won’t vote for any of his Democratic rivals; that they’ll either cast their ballot for a third party, stay home, or are undecided.

According to the authors, this Bernie-or-Bust group comprises roughly 11% of respondents in the “under-35, non-GOP” demographic. This “11%” factoid then forms the basis for the third and final step of the argument, which is presented by the authors in the following passage and in the chart below:

Sanders’ electability case thus appears to rest on the proposition that nominating him will increase turnout among Democrats and Independents under 35 by approximately 11pp, offsetting his inferior performance among the rest of the electorate.

This chart is the most wildly misleading part of the paper. Anyone who looks at it without closely reading the body of the paper will undoubtedly assume it’s saying that a Sanders victory against Trump would require an 11 percentage point increase in the youth turnout rate, which would take the rate to never-before-seen levels — from 43% in 2016 to 54% in 2020.

But that is not what the study found. When the authors talk about an 11% “increase” in turnout, they are not referring to an increase over 2016 levels; they’re talking about an increase over the counterfactual turnout rate that Biden would get if he were the nominee instead of Sanders. Thus, the dotted line that zooms up to indicate unheard-of levels of youth turnout is pure fiction. Sanders does not need to attain any particular level of turnout for his turnout rate to be 11 points higher than what Biden would have gotten.

In addition, the 11% figure is simply wrong. On the one hand, the denominator is artificially truncated, because it includes not 18-34 year olds as a whole, but only 18-34 year old non-Republicans. When expressed as a percentage of all 18-34-year-olds, the “turnout surge” would be 7%, not 11%.

Second, the numerator is too large, because, as the authors admit in the caption to their Figure 5, it includes two categories of pro-Sanders respondents who did not say they would stay home in the event a moderate Democrat was on the ballot. (They said they were “undecided” or would vote for a third party.) The percentage of respondents who actually chose the “I would not vote” option is shown in the authors’ Figure 5: it is 9.2% (using the truncated denominator).

When both the numerator and denominator are corrected, therefore, the actual “turnout surge” number falls from 11% to 5.9%. For what it’s worth, this falls well within the normal range of over-time turnout shifts. According to the US Elections Project, the youth turnout rate (ages 18-29) rose by 10.5 percentage points between 2000 to 2004 and 16.3 percentage points between 2014 to 2018.

All we need to believe, therefore, to credit the scenario the authors claim is so improbable, is that there is a group of voters who make up less than 6% of young people (and less than 2% of the eligible electorate), who would turn out to vote for Sanders but not for Biden. Not only is that not an implausible scenario — it’s what those voters said they would do when they were asked in the authors’ survey.

The Authors Disconfirm Their Own Argument

To substantiate their claim that Sanders would need an implausibly large turnout, the authors’ next step is to explore how the candidate would fare in the absence of any “turnout surge.” To accomplish this, they make two sets of adjustments to their survey data.

In the first adjustment, they simply re-weight the survey responses to reflect the composition of the 2016 electorate rather than the overall adult population. What that means is that each respondent’s candidate preference (or lack of preference) is counted in the poll, but the weight that preference gets when calculating the final result now depends on the predicted probability of that individual actually showing up to vote, based on their demographic profile. Young people, with their lower relative turnout rates, now receive significantly less weight while older voters receive significantly more.

With this step, the authors completely eliminate the whole dilemma of how much to trust respondents’ self-reported turnout claims. We no longer have to worry, for example, about whether to believe a 25 year old when she claims that she will turn out for Sanders but won’t turn out for Biden. The authors have ruled out all such behavior by assumption. They have imposed identical turnout patterns on all candidates, eliminating the self-reported differential turnout that boosts Sanders’s relative standing in polls.

As you might imagine, this step significantly alters the performance of the different candidates in matchups with Trump. How does Sanders fare in this scenario?

As shown in the figure above, when this re-weighting is performed, Sanders’ vote-margin advantage over Biden in matchups against Trump falls from +1.4% to +0.6%. In other words, before the re-weighting, Sanders’s margin against Trump was 1.4 percentage points better than Biden’s; after the re-weighting it was 0.6 percentage points better.

We should pause to note what this means: despite loudly claiming to have found that Sanders’s relative electability depends on some wildly improbable “youth turnout surge,” what the authors actually found was that Sanders remains more electable than Biden even after all traces of such a self-reported turnout surge have been scrubbed from the data.

Torturing the Data Until It Confesses

How, then, do the authors still conclude that Sanders is less electable? The answer stems from a final — and completely illegitimate — adjustment the authors make to their data.

In all cases, the authors’ survey always shows a significant share of voters — roughly 11% to 15%, depending on which Democratic nominee is posited — who say they would vote for neither major-party candidate, or are undecided. The share is always smallest in scenarios where Sanders is the nominee, reflecting the significant number of Sanders supporters who say they will definitely vote for him if he’s nominated, but who give a different (but non-Trump) answer if the nominee is someone else. (The alternative choices in this survey are: “Undecided”; “A third-party candidate”; and “I will not vote.”)

In the authors’ final data adjustment, they choose to disregard such answers and simply impute a major-party vote choice to most of the respondents who didn’t give one.

Specifically, their procedure was the following: if, after refusing to choose Trump or the Democrat, a respondent answered a followup question (whose wording begins “if you had to choose. . .”) by conceding that they would “lean towards” one of the two major-party candidates; or if they answered a standard party identification question by saying they identified with or “leaned” toward one of the two parties, they are re-coded as having “voted” for the nominee of “their” party — even if they had actually just rejected that choice.

As justification for this move, the authors cite “evidence that self-reported intent to turn out correlates poorly with actual turnout decisions” (p.10) — but this is obviously a non-sequitur. The fact that many survey respondents claim they won’t vote but actually do, tells us nothing about who they will vote for. The vote-choice imputation the authors make would only be justified if they could point to empirical evidence showing that it corresponds to actual voting patterns documented in past elections. But they have no such data to point to.

On the contrary — their procedure is flatly contradicted by all available evidence. Existing research on so-called “thwarted partisans” (supporters of a losing primary candidate) is consistent in its findings: when such voters tell election-year surveys that they will not vote for “their” party’s nominee, 60% to70% follow through with that decision.

For example:

• In 2008, an Associated Press panel survey re-interviewed the same group of voters several times over the course of the 2008 election; its results were analyzed in a 2010 paper by Michael Henderson and Sunshine Hillygus of Harvard and Duke Universities. The survey showed that in June of 2008, roughly 48% of voters who had supported Hillary Clinton in the Democratic primary said they would not vote for Obama in November; when reinterviewed in November, two-thirds of those voters (30% of the total) confirmed that they had indeed voted for a different candidate or, in a minority of cases, had not voted at all.
• My own analysis of the 2016 Cooperative Congressional Election Study finds that when Democrats who had voted for Bernie Sanders in the primary (and whose primary turnout was confirmed in the official voter file) were asked in October how they would vote in November, 20% claimed they would not vote for Hillary Clinton (4% said they’d vote for Trump; 11% for a third party; 5% said they wouldn’t vote or were unsure). When reinterviewed after the election, three-fifths of these (12% of the total) reported that they had, in fact, voted for someone other than Clinton (6% for Trump and 6% for a third party; almost all turned out in the general election).
• University of Pennsylvania political scientist Dan Hopkins, who is involved with a long-running university-run panel survey, discussed the issue of “thwarted voters” in a recent FiveThirtyEight blog post and on Twitter late last year. He noted that in the spring of 2008, his survey found 13% of voters who said they would vote for Hillary Clinton but not Barack Obama in the general election. When reinterviewed in the fall of 2008, 62% of these voters still refused to back Clinton: 34% said they would vote for John McCain while another 28% backed neither major-party nominee. Likewise, in January 2016, 11% of the survey’s respondents said they would support Marco Rubio in the general election but not Trump. When recontacted in October, 73% still maintained that stance, with 26% backing Clinton and 48% undecided, planning not to vote, or supporting a third-party candidate. 

“The upshot,” Hopkins concluded, is that “if there are voters who say they would back one candidate from a party but not another, don’t assume that they will necessarily come around and back that party’s nominee come November.”

Yet the authors of the study considered here made that unfounded assumption about the vast majority of such voters in their survey. Only by spuriously adjusting the data in this way were they able to show Bernie Sanders falling behind Joe Biden in matchups against Trump — albeit by a mere 0.6%.

An Unelectable Candidate?

If political scientists did not immediately point out the irremediable flaws in this widely circulated paper, one reason was probably the fact that its thesis harmonizes with an existing empirical literature on “extreme” candidates. This literature generally finds that when more “extreme” candidates win primaries, they’re penalized in the general election.

Academics and wonkish commentators often point to this literature as evidence that if Sanders is nominated, he’s likely to perform poorly in the general election. But the literature does not actually provide evidence for that idea. The reason for this is obvious, and yet has profound implications for the way political scientists and other “politics wonks” comment on the election.

Like most quantitative research, the literature on extreme candidates deals in averages. It takes a large sample of cases, each of which is unique, and tries to distill a statistical tendency from them. The statistic is an abstraction. It is not meant to describe any single case; it is only meant to characterize the average tendency among hundreds or thousands of cases.

To illustrate, here is a chart from an influential paper in the extremist-candidate literature. Using a clever research design, it shows how an extremist who performs almost identically to a non-extremist in their respective primaries will, on average, pay a penalty, in terms of votes, in the general election, compared to the non-extremist.

The sudden break in the trend line at the center of the graph shows the penalty paid by the extremist candidates. According to the paper’s author, it amounts to around 8 to 12 percentage points of general-election vote share on average.

That is a large penalty, and Bernie Sanders would almost certainly count as an “extremist” by this author’s definition. Yet none of this tells us anything about Sanders’s likely general-election performance if he wins the nomination. That is because Sanders is not an average of N=233 candidates from 1980-2010. Nor, in fact, are any of the individual cases in the author’s sample.

Even the most cursory glance at the chart will show that there are lots of extremist candidates who, as individuals, are more electable than lots of “moderate” candidates, as individuals. That hardly contradicts the overall  findings of this literature, because the literature does not claim that extremism/moderation is the only attribute that matters for a candidate’s electability. Many other factors help determine “candidate quality,” and these, taken collectively, are almost certainly more predictive of election outcomes than ideology alone.

Totally aside from his ideological positioning, Bernie Sanders has shown himself to be a talented politician. In Vermont, conservative political operatives will freely tell visiting reporters that it would be a mistake to underestimate his skills. He regularly wins with double-digit margins in towns that voted for Trump.

In fact, it is quite likely that one of the dots in the chart above is Bernie Sanders. In which case, it’s probably to the right of the line and pretty high up.