Subscribe to the monthly socialservice.sg newsletter and check out the socialservice.sg podcast!

A persistent trend in this “What it might have been?” section – with reviews of newspaper articles referencing research or data – is that articles often lead with eye-catching headlines of the research findings, yet disappoint with thin exposition of the research methodology. TODAY’s (Dec. 4) coverage of an index ostensibly measuring retirement readiness, in that familiar pattern, dove into the main findings without examining how they were derived, and in addition the supplementary information provided was inadequate.

Hence, the two related asks of the media are: First, not to cover such research publications and indices unless their full reports (or executive summaries, at the very least) are made available to the general public (with the links included in the article itself), not just to the journalists writing about them; and second – if the reports are available – to maintain scepticism and to consider other potential explanations for the findings. A good starting point, for instance, would be the research limitations identified by the researchers.

With the aforementioned retirement index, while retirement readiness – across four levels (high, adequate, low, or very low) is said to be measured by one’s “expected retirement lifestyle and needs, current income, accumulated savings, savings rates and investments, as well as home ownership” – it is not clear how the scores are initially calculated or subsequently added up. The TODAY report then considers differences across age and home ownership, but does not add more substantive details

Moving ahead, with the two above asks in mind, a short checklist – which approximates to the structure of a journal article – may help the media determine if they should report as well as how they should report on a research publication or an index:

• Does the public have access to the full report or the executive summary (not the data)? Can the article provide the relevant links?
• What is the sampling methodology (not just the sample size)? Is the sample representative, and if so how was it achieved?
• What were the questions asked of the respondents? As an added bonus: If quantitative surveys were used, what were the survey questions? And if qualitative interviews were conducted, what were the interview questions? It is not common to have the full surveys or interviews published, but even a sample of the questions asked (with the response options) – especially those which relate most directly to the research findings – should be shared.
• If quantitative differences are reported – that is, comparing between two different groups – it is important to determine if the differences are statistically significant.

Context – in terms of situating these research publications and indices with others – is critical too, and in this vein TODAY did a good job of highlighting that: “A financial wellness index compiled by OCBC bank in July found that Singaporeans are generally unsure of how to grow their wealth through investing and building up enough funds”. This is also an important component of the checklist and deserves greater media attention in the future.