“Most people support the proposal,” says the headline. The linked survey received thousands of responses, which sounds substantial. But who had the opportunity to answer, and who chose to do so?
A large number of replies does not automatically describe the population you care about. Before arguing about the percentage, establish what group the survey was designed to represent and how the responses were obtained.
Find the population behind “people”
Does the study concern residents of one town, customers of one company, registered voters or visitors to a website? Those groups are not interchangeable. A survey of customers may be informative about customers while saying little about everyone who might use a service.
Then check the recruitment method. An open link shared on social media creates different selection issues from a carefully designed sample. People with strong opinions may be especially motivated to respond, and people without access to the channel may never see the invitation.
Do not reduce quality to one statistic
Pew Research Center’s survey-methodology overview describes several sources of error, including coverage, sampling, nonresponse and measurement. This is a useful reminder that neither sample size nor response rate alone settles whether a finding is reliable.
Weighting may help address some differences between respondents and the intended population, but it is not a magic label that removes every source of bias. Look for an explanation of what was adjusted and why, rather than assuming that the word “weighted” ends the enquiry.
Read the question in context
Check the exact wording, answer choices and fieldwork dates. “Would you support a park?” is different from asking about a specified park funded by a specified charge. Events during the collection period can also affect how a result should be described.
Imagine a neighbourhood poll circulated mainly through a cycling group. It may reveal useful views from that community. Calling it a vote by all residents would require evidence the recruitment method does not provide. The problem is the jump in interpretation, not that the participants’ opinions are unimportant.
Write a narrower, stronger conclusion
Try replacing “everyone thinks” with a description of who answered and under what conditions. Note uncertainty where the report does not supply enough information. Avoid dismissing a study solely because its result is inconvenient.
A careful reading asks whether the claim matches the method. That question is often more revealing than another argument over the largest percentage in the headline.