Checking the Source
Chapter 2 of my series on How to Think
Ideas and arguments are not created in a vacuum. The argument is important but it comes from somewhere. Before you evaluate what a source is saying, it is important to know where that somewhere is.
Source evaluation is the first tool in the sequence, applied before the others, because it informs how to look at everything downstream. A claim assessed without knowing who is making it, why they might be making it, and what they have done with similar claims in the past is a claim you are evaluating blind. You may reach the right conclusion anyway. But you have made the work harder than it needs to be and abandoned the most accessible lever you have.
What are this source’s incentives — what do they gain from being believed, and what do they lose from being doubted? What is their track record on claims of this kind, in this domain, at this level of specificity? What expertise do they actually hold, as opposed to adjacent credentials they are implicitly borrowing? Are they describing something or advocating for something? And when they have been wrong before — as everyone is at some time — how did they respond?
That last question is the most important one, and the least asked.
That is source evaluation. It is the first tool because it is the cheapest — the information is almost always available, it costs nothing but the decision to look, and it tells you more than any other single move about what you are actually dealing with.
Sources that are biased are not automatically disqualified. Everyone is biased. But it is helpful to understand the ecosystem around the claim, whether there are any codewords that they use that might mean nothing to you but everything to their normal audience, and their ability or inability to correct themselves.
The Corrigibility Test
Sources behave differently when accountability arrives, and the behavior is more revealing than the original error. At one end of the spectrum is the immediate, direct, plainly worded correction. “We were wrong about X. Here is what we now know.” At the other end is vehement denial, often together with outrage that anyone can possibly doubt the claim and often blame a conspiracy. When the evidence is undeniable, this turns into: silence, or its close relative, the lawyer-drafted non-apology that technically acknowledges a problem while distributing responsibility so diffusely that no one is actually accountable for anything.
Between those poles sits a range of behaviors worth learning to recognize. A correction buried three days later in paragraph fourteen of an online version of the story tells you something different than a correction that runs where the original ran. A “clarification” that changes the meaning of a claim without acknowledging that the original meaning was wrong is reputation management dressed as transparency. An editor’s note reading “this story has been updated to reflect additional reporting” — without specifying what changed, why, or what the original said — is an institution signaling that the appearance of accountability matters more to it than the practice. And a large number of breaking news stories online get changed hourly without any acknowledgement of change at all.
Institutions under pressure make choices, and the choices reveal priorities. An outlet that corrects errors quickly, prominently, and in plain language has decided that accuracy matters more than the embarrassment of being wrong. An outlet that corrects reluctantly, minimally, and in language designed to obscure what happened has decided the opposite. The error itself tells you something; the correction behavior tells you more.
The New York Times has demonstrated both corrigibility and a stunning lack of the same in the past couple of decades. Unfortunately, it has been going in the wrong direction in recent years.
In May 2003, the Times published a 7,500-word front-page investigation of its own reporter, Jayson Blair, who had fabricated and plagiarized dozens of stories over years of employment. The piece named Blair, named the editors whose oversight had failed, described specific institutional failures at multiple levels, and did not attribute any of them to bad luck or misunderstanding. Executive Editor Howell Raines and Managing Editor Gerald Boyd resigned within weeks. The correction arrived while the story was still breaking — while the Times still had room to manage its response differently — and it was direct, self-implicating, and complete. An institution capable of that kind of accounting earns something real.
Sixteen years later, the same institution published the 1619 Project, whose lead essay included the claim that protecting slavery was among the primary motivations for the American Revolution. Prominent historians challenged the claim specifically and immediately, several of them with no ideological stake in resisting the project’s broader argument. The Times defended the claim, then quietly altered the language in the online version — “one of the primary reasons” becoming “one of the reasons” — without announcement or acknowledgment. The Times’ own fact-checker subsequently stated that the original claim had not been supported by the evidence. The project’s editor described the change as a clarification. The original language was not preserved in the published text for readers to compare.
The underlying historical dispute is a separate matter — historians can argue about the Revolution’s causes, and reasonable people examining the same evidence reach different conclusions. What the correction behavior reveals is independent of that argument. A significant factual claim was challenged by credentialed experts, the challenge proved well-founded, the language was altered without acknowledgment, and the institution described the alteration in terms chosen to minimize what had occurred. Set that beside the Blair response and you have a picture of an institution that can practice genuine accountability when the error is a reporter’s personal misconduct, and retreats to reputation management when the error is ideologically inconvenient. Both data points belong in your assessment of the Times as a source.
A third example from The New York Times suggests that the silent correction is now standard procedure. In December 2023, two Times reporters covering the Gaza war referred to the Israel Defense Forces as “Israeli occupation forces” — the standard formulation used by anti-Israel activists, not a neutral description — in a published online story. The editors caught the language and changed it to “Israeli forces” in a later version. No correction notice appeared. No editor’s note acknowledged that the original language had run. This episode indicates that Times reporters themselves are not as objective as they are supposed to be, and rather than apologize, the newspaper simply pretended it did not happen. A newspaper that insists on transparency from the institutions it covers has seemingly abandoned that standard for itself.
The spectrum runs further in both directions. At the far negative end are sources whose correction behavior has ceased to function as a signal at all — sources whose errors are so systematic, so consistent in their direction, and so resistant to any acknowledgment that the track record itself becomes disqualifying. The EuroMed Human Rights Monitor, a Hamas-linked, Geneva-based organization whose anti-Israel claims have been cited by major news organizations and United Nations bodies, has published claims that are not merely wrong but physically impossible: organ harvesting from decomposed bodies exhumed from graves, weapons that vaporize victims without leaving physical evidence, training dogs to rape on command. These are not claims that require sophisticated geopolitical analysis to evaluate. They require basic knowledge of how organ transplants are done, or how blast physics works, or how dog training works. The organization has not corrected them. It has continued publishing, and outlets have continued citing it.
A source at that point on the spectrum has told you something definitive. The prior does not simply go low — it reaches a floor below which no specific claim from that source can lift it. Citing EuroMed as a source is not caution hedged with appropriate skepticism. It is credulity, and when a credible outlet cites a disqualified source without disclosing the track record, some portion of the outlet’s own credibility attaches to the laundering.
The New York Times has cited EuroMed several times in Gaza war coverage, after it had made the claims of organ harvesting from dead bodies, including the recent column by Nicholas Kristof that repeated EuroMed’s absurd claim that Israeli soldiers had trained dogs to sexually assault Palestinian prisoners. The same institution, in October 2020, declined to cover the Hunter Biden laptop story on the grounds that the New York Post, which broke it, was not a sufficiently credible source.
The Post is a tabloid with a political orientation and a long record of accurate reporting on New York politics and crime. EuroMed is an organization that publishes physically impossible fabrications and has never corrected them. The outlet with the verifiable track record was dismissed, and the outlet with the fabrication record was cited. The laptop story was subsequently confirmed by independent reporting. The EuroMed claims were not - and the Times still stands by that story.
The gap between how the paper acted in 2003 with Jayson Blair and how it acts today could not be wider.
A source with a mixed or uncertain record warrants skepticism — examine their specific claims more carefully before accepting them. A source with a pattern of errors in one direction, on one category of subject, warrants heightened scrutiny — treat claims in that category as requiring independent corroboration before any weight is given them. A source with a documented pattern of fabrication, consistently serving a consistent agenda, with no meaningful correction behavior, warrants disqualification in the relevant domain, meaning their claims carry no evidential weight regardless of what they say.
The Incentive Structure Behind the Byline
Source credibility is not a binary, and sometimes sources with spotty records still get things right. The more precise tool is understanding what incentives a source has for making claims of the specific kind under examination — because those incentives vary considerably within a single source depending on the category of claim.
American mainstream journalism operates under a professional norm against paying sources for stories. The norm is framed as ethical, and it has an ethical dimension, but its underlying logic is epistemological: a source paid for their account has a financial incentive to produce an account whether or not one exists, which corrupts the testimony. That reasoning is sound as far as it goes. What it misses is that unpaid sources have their own incentives, no less distorting. A source who leaks to a reporter without payment is usually doing so because they want something — to damage a rival, to get ahead of a story they cannot suppress, to advance a narrative that serves their interests. Very often news media will quote a witness as if they had found them when in fact they work for a partisan NGO working with the reporter behind the scene on the story. Saying “I have no financial stake in this” is not the same as “I have no stake in this,” and the professional norm conflates the two.
The National Enquirer pays its sources. This practice places it outside the ethical pale of mainstream journalism and is routinely cited as grounds for dismissing its reporting. Examine the incentive structure that payment actually creates, though, and the picture becomes more complicated. A paid source has a financial stake in the story being true — a false accusation against a public figure with resources to litigate exposes both the source and the publication to serious legal liability. The Enquirer’s core business for decades was celebrity wrongdoing, which meant it needed its celebrity wrongdoing stories to be verifiable or it faced catastrophic defamation exposure. Its sources in that category were often direct insiders — employees, associates, people with firsthand knowledge — whose accounts could be checked against documentary evidence. The mainstream outlet relying on an unpaid source with a political axe to grind had no comparable accountability mechanism; the source faced no consequences for being wrong.
The National Enquirer may publish other stories that are unreliable. But what is documented is the Enquirer’s track record on celebrity wrongdoing stories involving public figures with legal resources. It broke the story on Gary Hart’s affair with Donna Rice, which ended his 1988 presidential campaign. In 2007, the Enquirer’s reported that Senator John Edwards — then a leading Democratic presidential candidate, public advocate of family values, husband of a cancer patient — was conducting an affair and had fathered a child with his mistress.
The mainstream press, had been tipped to the same story and declined to pursue it, and continued declining after the Enquirer published, partly on the grounds that the Enquirer had published it. Edwards denied the affair. The mainstream press largely accepted the denial. Edwards confessed in 2008; paternity was confirmed by DNA in 2010.
The outlets that dismissed the Edwards story based on source credibility were wrong about that specific claim, even if their general assessment of the Enquirer’s track record across all categories may be accurate. Different credibility profiles can and do live inside the same masthead and organization. A blanket discount applied to everything the Enquirer published was a meaningful tool applied without the precision the tool requires.
Where the Enquirer’s model falls short is equally worth naming: paying for stories creates pressure to have a story, which means borderline material gets published that a more cautious editor might hold. The model produces reliability in the category where legal exposure is highest and the facts are most verifiable. It produces less reliability where the claim is harder to verify, the subject less litigious, or the volume incentive dominates. The mainstream model produces reliability in categories where unpaid sources have aligned incentives and accountability mechanisms exist. Both models have systematic strengths in some categories and systematic blind spots in others, and the reader who applies a single credibility judgment to either one — in either direction — is using a blunt instrument where precision is available.
Claims vs. Evidence
Most people, when they imagine media distortion, picture a lie — a specific false claim, clearly stated, that can be identified and refuted. In any environment with functioning watchdogs — a free press, opposition researchers, fact-checking organizations, academics whose careers depend on catching errors — being caught in a direct provable falsehood carries serious reputational cost. Sophisticated actors with agendas generally work hard to avoid provable falsehoods. That does not make them honest. It makes them careful, and it means you are looking for the wrong thing if you are hunting for lies.
What you are actually looking for is subtler: omission, the true fact whose absence changes the meaning of the facts present; selective framing, the emphasis that directs attention to one part of the picture; the pattern of what a source consistently gets wrong, in one direction, on one category of subject.
The widely cited claim that 97% of climate scientists agree that human activity is causing global warming became one of the most repeated statistics in debates about climate policy. When you read the abstract of the actual 2013 paper by John Cook et. al you can see that this is not what was found: the authors only counted papers that specifically expressed an opinion on man-made global warming. Most climate papers didn’t say their opinion either way. It was not a survey of climate scientists. The real number of scientists who support the theory is still quite high - over 80% - but even that is somewhat self-selecting: the field of climate science has exploded since the 1990s and there are billions of dollars associated with the idea that it is primarily something humans are responsible for and therefore can solve. None of that necessarily destroys the underlying consensus claim but “the consensus is real and supported by multiple evidence types” is a different and more carefully bounded claim than a single percentage.
The claim routinely attached to the 97% figure in public discourse went considerably further than any of the underlying studies measured: that scientists agreed on specific emissions targets, on specific policy interventions, on the projections of particular models. Every individual source that cited the figure while attaching it to a stronger claim than the study supported was technically accurate about the number and functionally misleading about what it showed. No one was lying; but it was a consistent pattern of omission producing a false impression in the reader — which is how most distortion actually works.
The reader hunting for false claims will find nothing to hold onto. The reader asking whether the claim being made matches the evidence being cited will find the gap immediately. That second reader is doing the right thing, and it requires no specialized knowledge — only the habit of asking what the cited evidence actually shows before accepting what someone says it shows. Anyone reading the Cook et. al. abstract can see that his claim was much narrower than how columnists and politicians characterized it.
The Expertise Problem
Credentials are domain-specific. This is easy to state and sometimes difficult to apply, because expertise generates prestige that does not stay neatly inside its own domain.
Linus Pauling won the Nobel Prize in Chemistry in 1954 and the Nobel Peace Prize in 1962, making him one of only four people to have received two Nobel Prizes. Beginning in the 1970s, he became one of the most prominent advocates for high-dose Vitamin C as a treatment for cancer and a preventive for the common cold. Multiple well-designed clinical trials failed to replicate the effects he described, and the National Cancer Institute eventually concluded that the evidence did not support Vitamin C megadosing as a cancer treatment. Pauling’s Nobel was in chemistry. His claims were in nutritional medicine and oncology, domains where he had no particular standing and where his conclusions did not survive testing.
The mechanism is not unique to Pauling. Genuine expertise generates genuine prestige, and prestige is not domain-specific. An audience that has learned to trust a source in one domain extends that trust into adjacent domains without examining whether the extension is warranted. The source — accustomed to being trusted, habituated to being right, perhaps less current in the literature outside their specialty — may not notice the line they have crossed. Others may cross the line deliberately and use their expertise to silence critics.
The practical move: don’t whether a source is an expert, but what their expertise is in, specifically, relative to this specific claim. A cardiologist’s opinion on the cardiovascular effects of a drug is worth considerably more than their opinion on the drug’s effects on cognition. An economist’s analysis of labor market data is worth considerably more than their analysis of the epidemiological trajectory of an infectious disease. A human rights NGO might make claims about international law based on supporting their current campaign more than what the law actually says. Experts can have informed views outside their primary domain; the credential belongs to the domain, not to the person, and treating it otherwise leads you wrong.
The problem is most acute when experts speak to policy questions that touch their domain without being fully within it. An epidemiologist can model disease transmission with genuine authority. Whether the economic and social costs of a particular containment policy are worth paying is not an epidemiological question — it requires weighing values and evidence from multiple domains, and the epidemiologist’s credential does not travel there. During the COVID-19 period, this distinction collapsed almost entirely in public discourse. Experts in disease transmission were regularly treated as authorities on economic tradeoffs, educational policy, and civil liberties questions that their training gave them no particular standing to adjudicate. The audiences most deferential to expertise were, in this respect, the ones most systematically misled by it — not because the experts were lying, but because neither the experts nor their audiences asked whether the credential applied.
When the Institution Has Decided What It Will Find
Individual sources can be evaluated for expertise, incentives, and track record. Institutions present the same problem at scale, with an additional complication: when an institution’s conclusions consistently align with its funding sources or ideological commitments across many studies and many years, that pattern is itself evidence, independent of the quality of any individual piece of work.
The clearest documented case comes from the tobacco industry. Beginning in the 1950s and accelerating through the 1960s, tobacco companies funded research explicitly designed to manufacture doubt about the link between smoking and disease. Internal memoranda, later produced in litigation, showed that industry scientists understood the health effects of smoking and organized a campaign to obscure them — funding studies designed to produce ambiguous results, placing friendly scientists on editorial boards, and cultivating a public narrative of ongoing scientific uncertainty that the industry’s own private research had long since resolved. The Council for Tobacco Research, presented publicly as an independent scientific body, existed to produce the appearance of legitimate inquiry while its conclusions were predetermined. This is conspiracy in the strict sense: coordinated, intentional, documented.
The pharmaceutical industry’s publication bias problem has no comparable smoking gun but is at least as important. Clinical trials with positive results — that the drug works as hoped — get submitted for publication, accepted by journals eager for significant findings, and enter the medical literature. Trials with negative or null results — the drug does not perform, or performs worse than the comparison — often go unpublished. No central directive is required; the incentives are sufficient. A researcher whose trial produces a null result has a paper that journals are less likely to accept, a finding that does not advance their career, and a result their funder may prefer not to publicize. Multiply those individual decisions across thousands of researchers over decades and the published literature systematically overstates drug efficacy relative to the full evidence base — not because anyone chose to distort it, but because the selection pressure on what gets published is not neutral.
Below that sits a third layer, one that requires no funder pressure and no career calculation at all. Researchers generally do not publish studies that disprove their own hypotheses. A scientist who designs an experiment, runs it, and finds that their hypothesis was wrong faces a result with limited publication prospects, limited professional reward, and real psychological cost. The scientific community’s knowledge is built from the subset of research that gets written up, submitted, and published — a selected sample of all research conducted, skewed toward confirmation, toward positive findings, toward results clean enough to be publishable. The entire field’s collective understanding rests on an archive that contains the successes and buries the failures, which means the knowledge base is more confident and less accurate than the full evidence would warrant. Replication efforts, meta-analyses, and systematic reviews exist precisely to correct for this — but only in fields where those tools are applied rigorously and where null results, when they accumulate, can eventually force a reckoning with what the original literature obscured.
The reader’s job, when assessing institutional conclusions, is to ask what selection pressures apply to this institution’s output and what those pressures would predict about the direction of any distortion. A pattern of conclusions consistently favoring funders, consistently avoiding findings that would threaten institutional relationships, and consistently framing uncertainty in directions that serve preferred outcomes, is relevant information. Actual conspiracies on the level of the tobacco companies are rare, but institutional bias affecting what is presented as objective research is common.
Triangulation
When the media environment itself is compromised — when outlets on opposing sides are each amplifying their own narrative and suppressing the facts that complicate it — a method remains available, though it requires more work than reading a single trusted source.
Identify what each side says about the same event. Extract the facts they share despite their opposing agendas. Treat that overlap as the most reliable core of the story, because those are the facts that survived the most hostile possible scrutiny — each side had every incentive to suppress them and could not.
Beginning with the protests following George Floyd’s death in Minneapolis and extending through Portland, Kenosha, and dozens of other cities over the following months, coverage divided almost perfectly along ideological lines. Left-leaning outlets emphasized the scale and peaceful character of the demonstrations, the legitimacy of the underlying grievances, and the provocation preceding specific incidents of violence. Right-leaning outlets emphasized property destruction, fires, assaults on law enforcement, and the failure of local governments to maintain order. Each account was, within its own frame, defensible. Each account omitted what the other documented.
The reader who consumed only left-leaning coverage came away with a picture of overwhelmingly peaceful protest occasionally marred by isolated violence. That matched the statistical reality that most individual demonstrators at most events were not engaged in violence — and missed the sustained destruction in specific locations over sustained periods. The reader who consumed only right-leaning coverage came away with a picture of urban chaos and institutional capitulation to mob violence. That matched the documented reality of burned buildings and police precincts — and missed the peaceful majority that made the violent minority a minority rather than something representative.
The facts both sides had to acknowledge, because neither could suppress them without obvious bad faith: demonstrations occurred in hundreds of cities; a meaningful minority of locations experienced significant property damage and violence; the overwhelming majority of protesters at the overwhelming majority of events did not engage in violence; specific locations, particularly Portland over an extended period, saw sustained conflict that lasted well beyond any single night. A reader who extracted those shared acknowledgments had something closer to the actual picture than any reader who trusted either account alone.
The method has a critical limitation. Triangulation only works when the two sources are genuinely adversarial and independently reported. Two outlets drawing from the same wire service are one source with two mastheads. The number of news outlets that can afford their own national or international coverage has plummeted. A reader who found three news organizations reporting the same claim and concluded that three independent sources had confirmed it may have found one AP story republished in three places. The independence has to be real — separate reporting chains, separate sources, separate institutional incentives — or triangulation produces false confidence rather than genuine corroboration.
When Good Faith Is Not Warranted
The entire framework of this chapter assumes something worth making explicit: that the sources you are evaluating are at least attempting to engage with reality. Most of the tools — corrigibility assessment, incentive analysis, triangulation, pattern recognition — are instruments for calibrating how much to trust a source that is operating in good faith but imperfectly. They are less useful, and potentially counterproductive, when applied to sources that are not operating in good faith at all.
The Western intellectual tradition is built on an assumption of sincere engagement. Debate, peer review, journalism, and the legal adversarial system all generally presuppose that your interlocutor is genuinely trying to establish something true and is, in principle, open to being corrected by evidence. That assumption is foundational and mostly warranted. It is also, in specific contexts, a vulnerability.
In the 1990s, Holocaust deniers discovered Usenet — the early internet’s public discussion forums — and began spreading their lies.Well meaning experts responded showing their errors. What followed was an iterative development process for the deniers. Every careful refutation told the deniers which arguments were weak and which survived scrutiny. Every documented historical source they were shown allowed them to develop more sophisticated responses to that source. Every logical flaw identified in their reasoning was eventually patched. The people who spent hours meticulously dismantling denier claims were not just wasting their time on the deniers — the deniers were extracting value from them. The output, in only a few months of this process, was a considerably more sophisticated body of denial literature than had existed before the engagement began.
The deniers were not persuadable. They were not arguing in order to find the truth. They were arguing in order to become more convincing, and the good-faith engagement of their opponents was the resource that made them so.
A source that has established a documented pattern of fabrication — that has published claims shown to be physically impossible or outright lies and never corrected them, that continues producing similar claims in service of the same agenda — has forfeited the presumption of good faith that makes engagement worthwhile. Recognizing this isn’t cowardice or closed-mindedness; iIt is an accurate assessment of what you are dealing with, and acting on that assessment is the correct application of the tools this chapter describes.
When a source is disqualified, the productive response is to expose the track record, not to engage with the specific current claim. Showing that a source has published clear falsehoods and never corrected them is more useful to the watchful audience than dismantling its latest claims — because it equips the audience to handle the next claim, and the one after that, without requiring expert analysis each time. The track record is the argument. The specific claim is beside the point.
Engagement remains warranted under one condition: when the bad-faith argument is reaching people who might be persuaded, and who can be served more effectively by seeing the argument answered than by being told the source is disqualified. In that case the target of the argument is not the bad-faith source — it is the people being reached by the source, who may be entirely sincere and who deserve a real answer. The argument is for them, not for the source producing it. Trying to correct the source is a waste of time; inoculating the audience from believing that source is very worthwhile.
Back to Duke
How does any of this help with the David Duke question of Chapter 1?
We need to look at Duke’s entire history, not just his NPR interview in a vacuum.
Even if we accept that Duke has not associated himself with the Ku Klux Klan for decades, if you try to find any of his writings that specifically denounce the Klan’s racist beliefs you will come up short. His claim not to be a racist appears to depend on a strained definition of racism, which we will explore in later chapters.
If you look at his eponymous website, DavidDuke.com, you will find articles that have a completely different tone than his NPR interview. Headlines include “The Giant Jewish Vampire Squid Criminal Bank With Its Blood Funnel Stuck Into the Face of Humanity!” referring to Goldman Sachs, and “Purim: A Festival of Hate and an Insight into How Jewish Supremacists View Gentiles.” Duke and his fellow writers on the site are writing for their audience, and the message is quite different from what Duke says on national news media. He is using NPR as a recruitment tool. One can reasonably expect that the website named after him is closer to his real views.
His track record places him, on claims about Jewish people and racial hierarchy, at the disqualification end of the spectrum. His NPR argument, evaluated as a claim about those subjects, comes from a source with no meaningful presumption of good faith in that domain.
Duke’s argument requires an answer anyway — not for Duke’s benefit, but because it no longer belongs only to him. There are plenty of younger people who parrot Duke’s ideas even if they never heard of him. The arguments themselves are on social media. The people making the arguments may be entirely sincere.. Disqualifying Duke does not disqualify the argument from requiring a response, because the argument has escaped its source.
For now, this chapter argues that you do not have to respect David Duke or assume his argument is made in good faith. But his argument still needs to be answered. We will dismantle the actual argument in later chapters.
Great article! I'm glad I didn't have to wait to long for it.