Level 5 — Advanced · Technical & Conceptual Reading
Lesson 75: Advanced Reading: Evidence Hierarchy, Source Quality & Triangulation
The fourth part of this course's argument-analysis arc — classifying evidence by type and source, telling source quality apart from evidence quality, checking whether cited sources are genuinely independent, and catching selection, survivorship, and publication bias before trusting a body of evidence.
Learning Objectives
By the end of this lesson, you should be able to:
- Classify evidence by its type and method — anecdote, case study, observational, experimental, administrative data, expert judgment — and distinguish primary sources from secondary ones
- Distinguish source quality (who produced the information) from evidence quality (how strong the underlying evidence is), and check whether multiple cited sources are genuinely independent or copies of one original
- Read reporting verbs — demonstrated, found, suggested, speculated, reported — as signals of how strongly a cited source made its own claim, and track 'according to' for whose claim is actually being reported
- Recognise proxy evidence and ask whether a measurement actually represents the concept a claim is about, and check evidence for selection bias, survivorship bias, and publication bias
- Synthesise conflicting studies into a more precise, conditional claim, rather than concluding that conflicting evidence means nothing can be known
Introduction
Lesson 74 asked whether an argument's evidence was sufficient for its conclusion. Today's lesson — the fourth part of this course's argument-analysis arc — goes one layer deeper still: when evidence for the same claim comes from several different places, how much should each piece actually be trusted, and how much does combining them actually help?
Why Evidence Quality Matters
Two sentences, both 'evidence,' with very different natures
A. Three employees said the new system is faster. B. System logs show that median processing time decreased by 28% over six months. A is self-reported evidence; B is system-generated quantitative evidence. Simply noting "there is evidence" isn't enough — the next question always has to be: what kind of evidence is it?
Types of Evidence, by Source and Method
Eight recurring categories, each with its own strengths and limits
Triangulation, Revisited
Connecting back to Lesson 74
Examining the same claim through several independent kinds of evidence — system logs, audit records, employee reports, cost records — and finding them point the same direction strengthens a claim considerably. This lesson takes triangulation further by asking a sharper question: how do you tell genuinely independent evidence from evidence that only looks independent?
Independent Evidence: Counting Sources Versus Counting Origins
The single most valuable new idea in this lesson
If four sources all trace back to the same original study, they are not four independent pieces of evidence — they're one piece of evidence repeated four times. Newspaper A reports a study. Newspaper B reports the same study. Website C repeats Newspaper A. Three articles, one underlying finding. Number of sources ≠ number of independent evidence sources. Before counting how many places support a claim, trace each one back to its actual origin.
Corroboration, Revisited
Connecting back to Lesson 74
Employee reports suggest fewer billing errors. Audit records independently show fewer billing corrections. The second, genuinely independent source reinforces the first — corroboration that meaningfully raises confidence, precisely because it comes from a different origin, not a repeated one.
Contradictory Evidence, Revisited
Connecting back to Lesson 74's conflicting evidence
Automation reduced processing time by 28%. But maintenance costs increased by 40%. One metric improved; another worsened — overall evaluation now needs more criteria than either fact alone supplies.
Source Quality Is Not Evidence Quality
Two separate questions, easy to collapse into one
A respected newspaper can report a weak study. An obscure technical document can contain extremely strong primary data. Source quality asks: who produced this information? Evidence quality asks: how strong is the underlying evidence itself? Keep both questions active — reputation alone answers neither.
Primary Versus Secondary Sources
Getting as close as possible to where the evidence actually originates
A primary source is where original information or data can be found: an original research paper, official statistics, system records, an audit report, an original interview, experiment results. A secondary source summarises or analyses someone else's evidence: a news article, a review article, a textbook, commentary, analysis. Secondary sources are useful for orientation — but whenever the exact evidence needs verifying, the reading move is always the same: go closer to the primary source.
"According to…" and Whose Claim Is This?
A phrase that flags where a claim actually originates
According to a recent industry report, automation can reduce processing costs. The author is naming their source explicitly — the industry report, not the author's own independent conclusion. Ask directly: whose claim is this — the author's, or someone else's being relayed?
Attribution Tracking
Connecting back to this course's earlier work on attribution and reported claims
Researchers argue that automation improves efficiency, although other studies have reported mixed results. "Researchers argue" — a reported claim. "Other studies" — counterevidence, also reported. The author is likely taking a balanced position, visible only if you track each claim back to whoever is actually making it. A reported claim is not automatically the author's own final belief.
Reporting Verb Strength
How a source's claim gets relayed changes how strong it sounds — track the verb
Stronger: researchers demonstrated that… Moderate: researchers found that… Cautious: researchers suggested that… Very cautious: researchers speculated that… Neutral: researchers reported that… The reporting verb an author chooses often reveals their own stance toward a cited claim — even when the underlying source material is identical.
Evidence Type Must Match Claim Type
A common, easy-to-miss mismatch
Claim: employees enjoy using the system — an employee survey is directly relevant evidence. Claim: the system reduces processing time — employee opinion alone is not enough; processing-time measurements are needed instead. Evidence must match the claim being evaluated — strong evidence for one kind of claim can be weak or irrelevant evidence for a different one, even about the same system.
Measurement-Claim Match
Connecting back to Lesson 74's reliability dimension
Users reported that the software was reliable is a subjective perception. The system experienced 99.95% uptime is a quantitative measure of operational reliability. Both can be useful, but they are not the same construct — always ask exactly what was measured before treating two "reliability" claims as interchangeable.
Proxy Evidence
When a concept can't be measured directly, something else stands in for it
"Operational efficiency" is hard to measure directly, so researchers often reach for a proxy: processing time, cost per transaction, error rate, throughput. But proxy ≠ the entire concept — processing time falling does not automatically mean overall efficiency has risen; it means the specific proxy moved, which is a narrower claim.
The Construct-Validity Intuition
One question that captures the core of formal construct validity, without the formal apparatus
Ask: does this measurement actually represent the concept being claimed? Employee happiness measured through number of hours worked is a questionable proxy — hours worked ≠ happiness. You don't need full research-methodology training to catch this; you just need to hold the question in mind every time a passage measures one thing and claims something broader.
Data Quality Checklist
Questions worth asking of any data, even system-generated data
A number that looks better than it is
Error rate fell from 8% to 4%. A strong-looking improvement — unless reporting rules changed during the same period, in which case the apparent improvement's interpretation shifts considerably. Even clean, system-generated numbers deserve this kind of scrutiny.
Selection Bias
Who ended up in the sample can quietly distort the whole picture
The company surveyed only employees who volunteered to use the new system. Then: 90% of participants liked the system. Ask: did employees who disliked the system simply avoid volunteering in the first place? If so, the sample is biased before a single response was even collected.
Survivorship Bias
A classic reasoning trap — only counting the survivors
The most successful companies all use technology X. Therefore, technology X causes business success. This counts only companies that survived and succeeded. Companies that used technology X and failed have been silently excluded from the count — precisely the mechanism behind survivorship bias, and a strong candidate explanation any time a "look at all these successes" argument appears without mentioning any failures.
Publication Bias
What gets published is not automatically what actually happened, in full
A study with an interesting, positive result is more likely to be published than a study finding "no meaningful effect." Published evidence ≠ all evidence that exists. This matters enormously for advanced research reading — a literature that looks uniformly supportive of a claim may simply reflect which findings were interesting enough to publish, not which findings actually occurred.
Replication
Connecting back to Lesson 74's evidence-weight hierarchy
One positive result doesn't settle a question on its own. When independent researchers study the same thing again and find a similar result, confidence rises — this is replication, and repeated findings are generally more persuasive than any single isolated one.
Conflicting Studies: Why Might an Author Omit Some?
Selective citation is worth noticing, not assuming malice about
Study A: automation improved productivity. Study B: automation had no significant effect. Study C: automation improved productivity only in highly standardized environments. If an author cites only Study A, ask: why were B and C left out? Possible reasons include selective evidence, genuine context differences, methodological differences, population differences, or measurement differences — not necessarily deception, but always worth naming explicitly.
Evidence Synthesis: Turning Conflict Into a Refined Claim
Conflicting evidence doesn't mean 'we know nothing'
Given Studies A, B, and C above, the productive question isn't "which study is right?" — it's why do the findings differ? Possible explanation: automation may work under certain conditions but not others. Refined claim: automation is beneficial under sufficiently standardized operating conditions — considerably more precise, and more defensible, than either "automation works" or "automation doesn't work" taken alone.
An Evidence Ladder, Revisited
Connecting back to Lesson 74's evidence-weight hierarchy — the same idea, restated as six levels
Level 1 — anecdotal ("I experienced this"). Level 2 — single case (one organization or project). Level 3 — small observational evidence. Level 4 — larger, more systematic observational evidence. Level 5 — comparative or controlled evidence. Level 6 — replicated, converging evidence. Not an absolute ranking, but a reliable default for judging weight while reading.
A Complete Worked Example
Primary and subjective evidence, a limitation, corroboration, and contradictory evidence, synthesised
A port authority introduced an automated billing system in four terminals. Six months later, system records showed a 28% reduction in average invoice-processing time. Employee surveys also indicated higher satisfaction with the billing workflow. However, the participating terminals had relatively standardized operations, and the study did not assess long-term maintenance costs. Similar improvements have been reported in two independent studies, although results from smaller ports have been mixed.
Primary evidence: system records, 28% reduction. Subjective evidence: employee surveys, satisfaction increased. Limitation: standardized terminals. Missing dimension: long-term maintenance costs. Corroboration: two independent studies found similar improvements. Contradictory evidence: smaller ports had mixed results. Refined interpretation: automation appears promising, especially in standardized environments, but its broader effectiveness and long-term cost-effectiveness remain uncertain — synthesis, not contradiction paralysis.
An Evidence Map
A mental structure for holding several pieces of evidence against one claim at once
A Source-Reading Algorithm
Twelve steps for any research-based passage
Lessons 72–75, Combined: A Master Framework
The full argument-and-evidence architecture built across this four-lesson arc
Vocabulary in Context
primary sourcenoun phraseformal
a source where original information or data can be found directly, such as a research paper, system record, or original interview (মূল উৎস)
Often with consult a primary source, rely on a primary source, primary source of
“The auditors went back to the primary source, the original gate log, rather than rely on the summary.”
secondary sourcenoun phraseformal
a source that summarises, analyses, or reports on someone else's original evidence, rather than presenting it firsthand (গৌণ উৎস)
Often with cite a secondary source, rely on a secondary source, secondary source such as
“A news article summarising the report is a secondary source, so check the original.”
selection biasnoun phrasetechnical
a distortion that arises when the group being studied or surveyed is not representative, often because of how participants came to be included (নির্বাচন পক্ষপাত)
Often with selection bias, risk of selection bias, correct for selection bias
“Surveying only customers who stayed with the company creates selection bias.”
survivorship biasnoun phrasetechnical
a reasoning error that draws conclusions only from cases that 'survived' or succeeded, while silently excluding the failures (টিকে থাকার পক্ষপাত)
Often with suffer from survivorship bias, avoid survivorship bias, survivorship bias in
“Studying only companies still trading introduces survivorship bias, since failed ones are missing.”
publication biasnoun phrasetechnical
the tendency for studies with interesting or positive results to be published more often than studies finding no meaningful effect (প্রকাশনা পক্ষপাত)
Often with publication bias, risk of publication bias, correct for publication bias
“Publication bias may inflate the evidence, since trials showing no effect are rarely published.”
proxy evidencenoun phrasetechnical
a measurable stand-in used for a concept that is difficult to measure directly, which may not fully capture that concept (প্রক্সি প্রমাণ)
Often with rely on proxy evidence, weak proxy evidence, proxy evidence for
“Complaint numbers are proxy evidence for customer satisfaction, though an imperfect measure.”
construct validitynoun phrasetechnical
the degree to which a measurement actually represents the concept it claims to measure (গঠন বৈধতা)
Often with high construct validity, question construct validity, construct validity of
“The survey has weak construct validity: it measures speed but claims to measure quality.”
reporting verbnoun phrasetechnical
the verb an author uses to relay a source's claim, such as found, suggested, or speculated; it shows how strong the claim is (প্রতিবেদনকারী ক্রিয়া)
Often with choose a reporting verb, strong reporting verb, reporting verb such as
“'The study demonstrated' is a stronger reporting verb than 'the study speculated'.”
Guided Reading Practice
Read this passage once, straight through. Then identify its evidence types, check whether its cited sources are genuinely independent, and name any bias risk it raises, before checking your reading against the notes that follow.
According to three separate news outlets, a recent industry report found that companies adopting predictive maintenance saw a 20% drop in equipment downtime. All three articles cited the same report, published by a vendor that sells predictive-maintenance software. The report's data came from a self-selected group of the vendor's own customers who volunteered to share their results. A separate, independently funded academic study of a broader set of manufacturers found more modest and inconsistent downtime improvements, with the largest gains concentrated among companies that already had strong existing maintenance data systems.
Evidence type: the "three separate" citations are not three independent pieces of evidence — all trace back to one vendor report, a case of source-counting, not origin-counting. Source quality concern: the report comes from a vendor with a direct commercial interest in a positive result. Selection bias: the data came from self-selected volunteer customers, likely already satisfied ones, not a representative sample of all adopters. Corroboration and conflict together: the independently funded academic study partly corroborates (some improvement, gains among certain adopters) and partly conflicts (more modest, inconsistent overall) with the vendor report — pointing toward a refined claim: predictive maintenance may help most where strong existing data systems already exist, not universally.
Golden Rule
Golden Rule
Do not ask only "is this source credible?" — ask what exactly this evidence establishes, how reliably it establishes it, and how far the conclusion can legitimately extend, since one piece of evidence can support a claim, but only genuinely independent, converging evidence can substantially strengthen it.
Lesson Summary
This lesson took the argument-analysis arc from Lessons 72–74 and extended it to the evidence itself: how to classify it, trace it to its origin, and weigh it honestly. You worked through evidence types by source and method — anecdote, case study, observational, experimental, administrative data, expert judgment — and learned to separate primary sources from secondary ones. You practised tracing cited sources back to their actual origin before counting them as independent, and separated source quality from evidence quality. You tracked reporting verbs (demonstrated, found, suggested, speculated, reported) as signals of a cited claim's own strength, checked whether a proxy measurement actually represents the concept a claim is about, and learned to watch for selection bias, survivorship bias, and publication bias. You closed by synthesising conflicting studies into a more precise, conditional claim, rather than treating disagreement between sources as a dead end.
What this four-lesson arc has built, start to finish
Lesson 72 gave you an argument's anatomy. Lesson 73 gave you a taxonomy for naming its missing bridges. Lesson 74 gave you a way to judge whether its evidence was sufficient. This lesson gave you the tools to evaluate that evidence's actual origin and quality before trusting it at all. Together, these four lessons replace "the author cited some evidence" with a precise, defensible account of what that evidence is, where it came from, how independent it really is, and exactly how far it can honestly carry a conclusion.
Practice: Test What You've Learned
Work through every question yourself before checking anything.
Before you start
For every scenario below, name the evidence type, the bias risk (if any), or the source-independence issue explicitly — "the evidence seems weak" is not yet a complete answer.
Part A — Evidence Types and Source Independence
- A blog post, a podcast, and a social media thread all cite the same unpublished internal company memo as their source. Explain why these three citations do not count as three independent pieces of evidence.
Show answer
All three trace back to the same single origin — the internal memo — so they are one piece of evidence repeated three times, not three independent confirmations. Counting sources instead of origins would create a false impression of broad corroboration when, in fact, nothing new has actually been added beyond the original memo's claim. - A company's own system logs show a drop in support tickets after a product update. Classify this evidence type, and explain one limitation worth checking even though it's system-generated.
Show answer
This is administrative or system data. Even though it's objective and system-generated, it's worth checking whether the ticket-logging or categorization method changed at the same time as the update, since a definitional or reporting-process change could produce an apparent drop that has nothing to do with the update itself. - An industry analyst predicts a technology will "revolutionize" a sector, without citing data. Classify this evidence type, and explain why it should not be treated as empirical proof.
Show answer
This is expert judgment. It should not be treated as empirical proof because it rests on the analyst's own reasoning and experience rather than measured, verifiable data — expert opinion can be useful and informed, but opinion is not the same evidentiary category as an actual empirical finding.
Part B — Primary/Secondary Sources and Reporting Verbs
- A magazine article summarizes a university's clinical trial results. Identify which is the primary source and which is the secondary source.
Show answer
The university's original clinical trial results are the primary source, since that's where the original data actually originates. The magazine article is the secondary source, since it summarizes and relays someone else's original findings rather than presenting them firsthand. - Compare: 'Researchers demonstrated that the drug reduces symptoms' versus 'Researchers speculated that the drug may reduce symptoms.' Explain how the reporting verb changes the claim's apparent strength.
Show answer
"Demonstrated" is a strong reporting verb, presenting the finding as close to established fact. "Speculated," paired with the hedge "may," is a very cautious reporting verb, presenting the same underlying claim as a tentative possibility rather than a confirmed result. The reporting verb alone can make an identical underlying study sound either settled or merely suggestive. - A report says, 'According to the manufacturer, the product is 99% effective.' Explain why this attribution matters for how the reader should weigh the claim.
Show answer
"According to the manufacturer" flags that this is the manufacturer's own claim, not the author's independently verified conclusion. Since the manufacturer has a direct commercial interest in the product looking effective, the reader should weigh this claim with more scrutiny and look for independent verification before accepting the 99% figure.
Part C — Proxy Evidence and Bias
- A school measures 'teaching quality' using only end-of-year test scores. Explain why test scores might be a questionable proxy for the broader concept of teaching quality.
Show answer
Test scores are a narrow, measurable stand-in for a much broader concept — teaching quality also includes student engagement, critical thinking, classroom management, and long-term learning, none of which a single end-of-year score captures. Test scores rising does not automatically mean the fuller concept of teaching quality has improved; it means that specific proxy moved. - A fitness app claims success because its most active, engaged users report high satisfaction, based on an in-app survey. Identify the bias risk in this evidence.
Show answer
This is selection bias. Only the most active, already-engaged users were surveyed, likely through the app itself, so users who disliked the app or stopped using it were never included — the sample was skewed toward satisfaction before a single response was collected. - A business book highlights ten highly successful entrepreneurs who all dropped out of college, concluding that dropping out helps success. Name the bias this argument commits, and explain what evidence is missing.
Show answer
This is survivorship bias — it counts only the "survivors" who succeeded after dropping out. Missing is data on the far larger group of people who dropped out and did not succeed, which would be needed to establish whether dropping out actually correlates with success rather than simply being present among a handful of famous winners.
Part D — Conflicting Evidence and Synthesis
- Study A finds a training program boosts performance; Study B finds no effect; Study C finds an effect only for new employees. Propose a refined claim that could account for all three findings.
Show answer
A refined claim: the training program tends to boost performance mainly for less-experienced employees, with limited or no additional effect for employees who are already experienced. This reconciles Study A's positive result, Study C's finding that the effect is concentrated among new employees, and Study B's null result, if Study B's sample was weighted toward experienced staff. - A widely cited statistic turns out to originate from a single small survey repeated across dozens of articles. Explain why this weakens, rather than strengthens, the claim's evidential support.
Show answer
Dozens of articles citing the same figure looks like broad support, but they all trace back to one small original survey — it is one piece of evidence repeated many times, not dozens of independent confirmations. Once traced to its origin, the statistic's actual evidential weight is exactly as limited as that single small survey, regardless of how often it gets repeated. - Explain why publication bias means a body of published studies can look more one-sided than the full picture actually is.
Show answer
Studies with interesting, positive results are more likely to get published than studies finding no meaningful effect, so the published literature systematically overrepresents supportive findings. A reader looking only at what got published can end up believing a claim is far more consistently supported than the complete body of research, including the unpublished null results, would actually show.
Part E — Full Passage Analysis
Return to this lesson's guided reading passage about predictive maintenance.
- Explain why the "three separate news outlets" do not constitute independent evidence.
Show answer
All three articles cited the same underlying vendor report, so they trace back to a single origin rather than three separate investigations. This is a clear case of source-counting rather than origin-counting — three citations, but only one actual piece of evidence. - Identify the selection bias in the vendor report's data collection.
Show answer
The vendor's data came from a self-selected group of its own customers who volunteered to share their results — likely already satisfied customers with successful outcomes, not a representative sample of everyone who adopted the technology, including those who saw poor or mixed results. - Identify how the academic study both corroborates and conflicts with the vendor report.
Show answer
It corroborates the vendor report by also finding some downtime improvement, particularly among companies with strong existing maintenance data systems. It conflicts with the vendor report by finding the overall improvements more modest and inconsistent across the broader set of manufacturers than the vendor's report claimed. - State the refined, conditional claim this lesson's notes propose, and explain why it is more defensible than either "predictive maintenance works" or "predictive maintenance doesn't work."
Show answer
The refined claim: predictive maintenance may help most where strong existing data systems already exist, not universally. It is more defensible because it accounts for both the vendor report's positive result and the academic study's more modest, inconsistent findings, rather than forcing the conflicting evidence into one flatly positive or flatly negative verdict that neither study alone actually supports.
Next: Lesson 76 — Advanced Reading: Bias, Framing, Selection & Missing Evidence: the fifth part of this course's argument-analysis arc, asking not just whether information is true, but what picture its selection, framing, and omissions create.