hum-source-criticism

Overview

Source criticism is a systematic method for evaluating whether information is trustworthy. Originally from historical methodology, it's now essential for navigating an information environment flooded with misinformation, opinion-as-fact, and AI-generated content.

Framework

IRON LAW: No Source Is Automatically Trustworthy

Every source — including academic journals, government data, and news from
reputable outlets — has potential biases, errors, and limitations. Credibility
is assessed, not assumed. "It's from the New York Times / 中央社" is not
sufficient — WHAT are they reporting, based on WHAT evidence, and do other
sources corroborate it?

Source Classification

Primary sources: Direct evidence from the time/event (original documents, raw data, eyewitness accounts, original research, official records)

Secondary sources: Analysis or interpretation of primary sources (textbooks, review articles, news analysis, biographies)

Tertiary sources: Compilations of primary and secondary (encyclopedias, Wikipedia, databases) — starting points, not endpoints

Four Tests of Source Credibility

1. External Criticism — Is the source authentic?

• Who created it? Are they who they claim to be?
• When was it created? Is the date consistent?
• Is it the original or has it been altered?
• Is the publication/platform reputable?

2. Internal Criticism — Is the content reliable?

• Does the author have expertise in this topic?
• What is the author's potential bias or interest?
• Is the evidence cited? Can it be verified?
• Is the reasoning logical? Are conclusions supported by the evidence?

3. Triangulation — Do multiple independent sources agree?

• Check 3+ independent sources (not copies of the same original report)
• "Independent" means different authors, different organizations, different methods
• Agreement across independent sources strengthens confidence

4. Currency — Is the information current enough?

• When was it published? Has the situation changed since then?
• For fast-moving topics (AI, policy, markets), even 6-month-old sources may be outdated

Red Flags for Misinformation

Red Flag	Description
No author or organization identified	Who stands behind this claim?
Emotional language without evidence	Designed to provoke, not inform
No primary sources cited	Claims without traceable evidence
"Studies show" without naming the study	Vague appeals to authority
Single source amplified across many sites	Same claim copied, not independently verified
Too good to be true / too outrageous	Extreme claims require extreme evidence
URL/domain mimics reputable source	Fakecnn.com, bbc-news.co (not bbc.co.uk)

Output Format

# Source Evaluation: {Source/Claim}

## Source Identity
- Author/Organization: {who}
- Publication: {where}
- Date: {when}
- Type: Primary / Secondary / Tertiary

## Credibility Assessment
| Test | Assessment | Evidence |
|------|-----------|---------|
| External (authentic?) | ✓/⚠/✗ | {reasoning} |
| Internal (reliable?) | ✓/⚠/✗ | {reasoning} |
| Triangulation (corroborated?) | ✓/⚠/✗ | {other sources checked} |
| Currency (current?) | ✓/⚠/✗ | {relevance of date} |

## Red Flags
- {any detected red flags}

## Verdict
- Credibility: High / Moderate / Low
- Recommended action: {trust / verify further / discard}

Examples

Correct Application

Scenario: Evaluating a viral social media post claiming "Taiwan's GDP will surpass South Korea's by 2027"

Test	Assessment	Evidence
External	⚠	Anonymous account, no institutional affiliation, chart has no data source
Internal	✗	Uses nominal GDP (not PPP), cherry-picks semiconductor sector projection, ignores exchange rate volatility
Triangulation	✗	IMF and World Bank projections show no such convergence; no reputable analyst makes this claim
Currency	✓	Posted this month

Red flags: Emotional headline ("Taiwan DESTROYS Korea"), no primary data source cited, single unsourced chart Verdict: Low credibility — discard ✓

Incorrect Application

• "This is from Reuters, so it must be true" → Credibility assumed, not assessed. Even reputable sources can be wrong, outdated, or framing an issue in a particular way. Violates Iron Law.

Gotchas

• Bias ≠ unreliable: Every source has a perspective. A labor union's report on working conditions is biased but may contain accurate data. Assess bias AND accuracy separately.
• Wikipedia is a starting point: It's a tertiary source with references. Follow the references to primary/secondary sources. Don't cite Wikipedia as evidence — cite what Wikipedia cites.
• AI-generated content: AI can produce convincing but fabricated "sources" (fake papers, fake quotes, fake statistics). Verify that cited sources actually exist.
• Consensus ≠ truth, but it's a strong signal: Scientific consensus (climate change, vaccine safety) is the strongest available evidence. Lone dissenting "experts" who contradict consensus need extraordinary evidence.
• Source credibility is domain-specific: A cardiologist is a credible source on heart disease but not on economics. Match expertise to the claim.

References

• For CRAAP test (Currency, Relevance, Authority, Accuracy, Purpose), see references/craap-test.md
• For fact-checking tools and databases, see references/fact-check-tools.md