When Your Paper Needs Methods Review

Peer review assigns reviewers by subject. That is the right default and it leaves a specific hole, which widens as methods in a field get more specialised than the field's median training.

A domain reviewer checks whether the question is worth asking, whether the framing is fair to the literature, whether the interpretation is defensible and whether the conclusions overreach. Those are real checks and they are not analytical ones.

A methods reviewer checks whether the design can answer the question at all, whether the analysis matches the design, whether the assumptions hold, whether the reported quantities are the ones being interpreted, and whether the result would survive a reasonable alternative specification.

The gap is widest where a field's methods have moved faster than its training. Fields that have absorbed machine learning, causal inference, multilevel modelling, Bayesian methods or complex survey designs in the last decade contain many excellent researchers who cannot audit those analyses, and know it.

Journals know this, which is why some run a separate statistical review track and why several major titles ask authors for a completed reporting checklist at submission. The EQUATOR Network collects those checklists by study type, and reading yours is the cheapest possible methods review.

Not every paper needs this. The following are the situations where a methods reader reliably finds something, and where their absence is what reviewers later complain about.

You are making a causal claim from data you did not randomise. Instrumental variables, difference in differences, regression discontinuity, propensity methods, synthetic control: all of them fail quietly when their identifying assumptions do not hold, and the failure is not visible in the results table.

You are using a predictive or machine learning model on scientific data. The recurring faults are leakage between training and test sets, validation that reuses the tuning data, class imbalance handled after the split, and performance reported without a meaningful baseline. Reporting frameworks such as TRIPOD exist for the clinical prediction case and name most of these.

Your data are clustered, repeated, nested or spatially correlated and your model assumes independence. This is the most common serious statistical error that domain reviewers do not catch, because the output looks entirely normal.

You are working with a complex survey design, weights, or a sample that is not what it appears to be. Weighted analyses go wrong in ways that produce plausible numbers.

You are using specialised qualitative methodology, such as grounded theory, discourse analysis, phenomenology or a mixed-methods integration design, in a venue or a team where the method is not standard. The failure mode is different from the statistical one, and a specialist reader is equally necessary.

The stakes or the scrutiny are high

a clinical or policy recommendation, a contested finding, a result that contradicts an established literature, or a dataset that others will reuse. Any of these justifies a methods read regardless of the technique.

Often the question is not whether to seek methods review but whether one has already happened. Review reports carry recognisable signatures, and reading them correctly determines how you should respond.

A report that comments only on the analysis, ignores the framing entirely and is signed as reviewer three or as statistical reviewer is a methods review. Treat every point in it as a condition rather than a suggestion, because the editor almost certainly will.

Phrases that name an assumption rather than a result

the independence assumption, the proportional hazards assumption, the exclusion restriction, the parallel trends assumption. Nobody writes these unless they intend the assumption to be tested and reported.

Requests for a sensitivity analysis, an alternative specification or a robustness check. This means the reviewer believes the result may depend on a choice you made and did not justify, and answering with a paragraph of reassurance rather than with a table will not close it.

A question about the sample size, the power calculation, or how the analysis plan was decided relative to seeing the data. This is a question about whether the analysis was specified in advance, and the honest answer, including that it was exploratory, is better than an invented rationale.

The absence of any such comment when your paper contains one of the triggers above. That is not reassurance; it means the analysis has not been checked, and how to answer a review you disagree with, or one that has missed something, is in our guide to responding to reviewers.

An unscoped request produces either a refusal or a vague reply. A methods reviewer needs to know which question they are answering, and the scope determines both the effort and the kind of person you need.

Name the question. Is my identification strategy valid, is this model appropriate for clustered data, is my validation procedure free of leakage, is this the right effect measure. One question per request works far better than an invitation to look at the paper.

Say which parts of the paper are in scope

usually the design, the analysis section, the tables and the code, and explicitly not the introduction, the framing or the literature. A specialist invited to read the whole paper will often decline on the ground that the topic is not theirs.

Say what you will provide

the analysis code, the data or a synthetic version of it, the analysis plan or protocol, the reporting checklist for your study type. What to attach and what not to is covered in our methods review request templates.

Methods review is a real cost in other people's time, and asking for it where it is not warranted uses up a resource you may need later. Four cases where the answer is no.

The analysis is standard for the field and standard for your data: descriptive statistics, a t test on independent groups, a straightforward regression on independent observations reported completely. A careful reading against a reporting checklist covers this.

You want reassurance rather than assessment. A methods reviewer who finds nothing wrong has still cost someone a day, and if you would not act on a negative answer there is no point asking the question.

The paper has a co-author who did the analysis and has the relevant expertise. Asking an external reviewer to audit your co-author's work without telling them is a collaboration problem rather than a methods problem.

The real problem is reporting rather than analysis. Missing sample sizes, absent measures of variability, unclear figure captions and undefined abbreviations are writing faults, and they are what our manuscript checker and structure checker are built to find.

Timing changes the value of methods review more than anything else about it. The same hour of a methodologist's attention is worth very different amounts depending on when you spend it.

Best

at design, before data collection. A methods conversation at that point can change the study into one that answers the question, and it is the only moment at which a fatal design flaw is cheap to fix.

Good

after analysis and before writing, when the results exist but the interpretation has not hardened. This is where most productive methods reviews happen and it is the point at which an alternative specification is still a day's work.

Useful

before submission, on a complete draft. This is what most people mean by methods review, and it catches reporting and specification faults while there is still time to run something.

Late but necessary

at revision, when a reviewer has raised a methods point you cannot answer alone. Say so plainly when you ask, since a deadline changes what someone can commit to.