Category Archives: Politics

From my point of view, this is big “P” politics of Nation States or Provinces within Nation States.

Performing health equity: A repeat prescription from yesteryear

In a recent paper in the International Journal for Health Equity, Berenson et al. deliver a hopeful vision of a world better able to take advantage of SDH research. Had the article been published in 2015, it would have landed in a world positioned to take full advantage of the prescription. Unfortunately, the very value frame that supports SDH interventions has fractured. They note this in the lightest way possible, citing a paper by Woolf et al. as evidence of “adverse political environments” for SDH. This characterisation completely undersells what Woolf et al. actually observed — a complete ban, a dismantling, of any research on SDH-style categories: equity, race, diversity, and inclusion. Woolf et al. provide what is close to a complete rebuttal of the point Berenson et al. seek to make.

One of the strongest conclusions from the authors is that “SDH researchers must aim to create and cultivate policy windows of opportunity by being active protagonists…” This assumes at least a world with something approaching shared values, not a world in which “social determinants” are not simply contested but openly reviled and its advocates cast as the enemy — where your notion of fairness is an attack on our cultural values, where your racial equality is advocacy for our genetic and cultural replacement. By their own adopted framework (Kingdon), policy windows rely on a conducive political context. That cannot be created by researchers and has not existed for over a decade! The current political context is not simply unconducive, it is actively hostile. If “cultivating policy windows” — something like a horticultural endeavour at the South Pole — were the solution, then climate regulation would have been resolved, and abortion laws would be universal and rights-protecting, but they are in retreat.

Berenson et al. warn against research that risks being too “technocratic and detached from real-world constraints”, that offers “descriptive documentation of inequities” without “confronting the interests and institutions that benefit from the status quo.” They will say, rightly, that they are not naive about politics — that they explicitly concede research alone cannot overcome entrenched power, and devote an entire section to asking whether the problem is politics rather than research conduct. All of that is true. But notice what the concession does in their hands. They file it. Then without any apparent guile or appreciation of irony, they propose a five-point technocratic corrective. What is needed, they say is more methodological pluralism, more multisectoral collaboration, better communication, reformed incentives, stronger “brokering institutions.” These echo the calls made in the health policy and systems research literature for at least the last fifteen years — including in several of the paper’s own citations, e.g., Rasanathan & Diaz 2016.

The paper is at its core an advocacy piece playing to an audience of the like-minded. As with so many pieces from the last twenty months, which have come out of a loose coterie of global health scholars, civil servants, philanthropists, and diplomats, it fundamentally preserves a shared delusion of global order (or its complete irrelevance) and the insights of yesterday’s science conducted under the aegis of that order. They want to preserve the findings as if they are real, but not the shared value system and international human-rights framework on which those findings — and their policy relevance — are contingent. That, finally, is the more interesting question the paper never asks: not why its fifteen-year-old prescriptions haven’t worked, but whether those contingent truths can ever reach beyond the frameworks that made them possible.

Berenson et al. call knowledge and science “a crucial bulwark to combat forces arraigned against solidarity.” But a bulwark is a defensive structure. It only matters if you still hold the ground behind it. And the ground — the shared value system, the international human-rights framework on which every one of their findings depends for its meaning — is precisely what is being overrun, and they do not seek to protect. This is the paper’s absurdity. Until universal values and rights are put back into the intellectual core, prescriptions like this one are not merely ineffective, they are complicit. They sustain the fiction that the problem is technique rather than the loss of the framework that made the technique meaningful. They keep wanting to refill the gas tank after the bandits have stolen the engine.

Global heating isn’t important–if you’re not suffering.

In the last weeks of June, Europe experienced a heatwave that broke temperature records from France through to Hungary. The heatwave combined high daytime and nighttime temperatures with high humidity. The result was more than 2,000 excess deaths in France alone.

The third heatwave of the European summer has started.

I have been fascinated by the political and technocratic response to the heatwave because, rhetorically at least, this summer has broken through the background hum of climate change conversations. There is a sense of urgency that I have not heard before. “Panic” is probably too strong a word, but governments are clearly worried. Will Europe be liveable in summer in 10 years time?

Having grown up with hot Australian summers and no air conditioning, I was familiar with the routine. Windows open at night to cool the house. Curtains and windows closed during the day to prevent it heating up. Fans helped. Evaporative cooling only worked if the humidity was not already too high. If it was too high, it didn’t feel like you were cooling, it felt like a sauna.

Where I now live in Europe, domestic air conditioning is not permitted, and the Australian strategies only worked partially in my apartment. Before the heatwave began, the internal wall temperatures were in the low 20s, by the end of the heatwave they were around 28.5. The thermal mass of the building had increased. The concrete and steel had absorbed the rolling heat of the day and radiated it out at night. Sleep was late in coming, patchy, and uncomfortable.

The thermal mass problem tracked something a colleague of mine at icddr,b and I had found in slums in Dhaka, Bangladesh. We had placed temperature data loggers in people’s dwelling and captured the rise and fall of temperature and humidity during the day and night. The construction materials were essentially tin and concrete. A tin roof and tin walls became an oven during the day, but cooled rapidly at night (low thermal mass). A concrete roof and concrete walls (high thermal mass) meant that the building’s internal temperature didn’t rise rapidly during the day, but nor did it cool rapidly at night. The concrete smoothed out the temperature variations, but in the absence of active cooling, it eventually rose to the average ambient temperature.

This is the challenge that Europe faces in a world of much hotter summers. Standard passive cooling techniques will get you only so far. Shade to avoid the direct rays of the sun or solar reflectance (albedo) surface to avoid the impact of direct exposure helps. High emissivity surfaces that radiate heat outwards in the cool of the night. Insulation (thermal resistance) to prevent the heat penetrating. Evaporative cooling if humidity permits and water is available.

In summers of regular heatwaves, however, these techniques are limited. Passive cooling can delay equilibrium, but it cannot overcome it. Eventually the thermal properties of the buildings catch-up. And it is difficult to design the building that sheds heat and never captures heat for a scorching summer, but retains and doesn’t lose heat in the winter. All of this assumes that there is even an option to make massive design changes. The vast majority of European housing stock was built with winters in mind, not summers. The oldest buildings, those that give the historic beauty to European cities, have poor thermal properties, and structural changes to them are heavily restricted by conservation laws.

If one cannot rely on passive cooling, then active cooling becomes the fallback. Historically, there have been very low rates of air conditioning in Europe compared to the United States. The latest heat wave, however, has opened discussion about allowing those in “genuine need” to install air conditioning. This, in itself, opens a can of worms. Who has genuine need? Who will pay for the active cooling for those who need it but cannot afford it. The elderly and those with chronic health conditions are often most vulnerable to heat stress. As groups of people, however, they tend to have less disposable income for the purchase and installation of active cooling. Unfortunately, in a continuous heat-stressed environment, everyone will be in need of cooling.

Widespread European air conditioning will also carry at least two unintended consequences. The first is that air conditioning needs manufacturing and maintaining, and power to run it. This will inevitably have greenhouse gas-production issues for a world that desperately needs to reduce greenhouse gas emissions. The second is that, if one is cooling the inside of a building, one is inevitably dumping heat into the outside. And it is slightly worse than that, you cannot move heat from inside to outside a building cost free, you must generate some additional heat to do that. Large scale air conditioning in cities can raise the external air temperature between 1 and 2 degrees, and it creates a feedback loop, because additional internal cooling is needed to overcome the increase in the rising external heat. The consequence for those who do not have air conditioning (those without “genuine need” or those without homes) is that they will experience even more extreme heat waves so that others may suffer less.

Will Europe be liveable in the summers in 10 years time? It really depends on what you mean by “liveable”. The dead trees I have seen– already unable to survive a June heatwave–do not augur well (see the related article on an iconic English oak tree). Passive cooling will help, but as the thermal mass argument demonstrates, it will be insufficient–the second law of thermodynamics prevents it. The answer Europeans seem to be reaching for is air conditioning–for those who can afford it. The heat it dumps outside falls on those who cannot, just as the emissions it produces fall on everyone, everywhere, including places already less able to cope. Widespread air conditioning does not solve Europe’s heat problem. Change nothing about how we live, and shift the cost onto those with the least capacity to bear it.

Campbell and Stanley explained replication rates in 1963

Over 60 years ago, Donald Campbell and Julian Stanley published their classic, slim volume Experimental and Quasi-Experimental Designs for Research. One of their earliest observations concerns the trade-off between internal and external validity. Specifically, the more precisely one can establish a causal relationship, the less one can say about its generality. In recent work, I show that simultaneously maximising internal and external validity is not merely a practical limitation to be mitigated, but a structural impossibility. The relationship is analogous to the Heisenberg uncertainty principle that shows one cannot simultaneously know both the position and momentum of a particle with arbitrary precision. In the context of the social and behavioural sciences, the more precisely one identifies a cause, the narrower the domain to which that knowledge applies.

I reviewed this problem in terms of the so-called “replication crisis”, the difficulty researchers have encountered in replicating published causal findings. Shortly after posting that paper, Nature published a series of articles on research credibility, including a large-scale investigation of replicability in the social and behavioural sciences. The empirical effort is extraordinary, involving hundreds of researchers and a substantial coordination infrastructure. The methods, results, and theoretical framing are all of considerable interest. However, the study has also generated headline figures that are readily misinterpreted—an outcome encouraged both by editorial framing and by the structure of the paper itself.

The central difficulty lies in two under-specified concepts that drive the research. The replication is of the “same question” and the “claim”. Whether a replication tests the “same question” is treated as a local, theory-laden judgement made by individual teams. Sameness is treated as constant at two levels simultaneously. First the multiple replications of a single study should be replicating the same thing, as if each attempt stood in an identical relationship to the original. And across all the original studies, the idea of sameness should stand in an identical relationship between a replication and its target regardless of which study is being replicated. If “same” does not mean the equivalent thing within and between replications, the target drifts meaninglessly

At the same time, replications are of “claims” which are scientific claims reduced to directional empirical statements, detached from the estimands, models, and analytic pipelines. That is, the claim is detached from the scientific meaning that gave it purchase in the original study. The same problem with “claims” arose in the team’s Nature paper on analytic robusteness. Abstracting scientific claims into more generic “claims” produces a mismatch between design and inference. Heterogeneous interpretations of what is actually being tested are collapsed into standardised statistical comparisons. Apparent agreement or disagreement may therefore reflect shifts in underlying targets rather than genuine replication or failure.

A related issue is that the study attempts to straddle internal and external validity without resolving their tension. It presents itself as assessing whether findings replicate, but in practice examines how results behave under modest variation in context, measurement, and implementation—something closer to robustness or transportability than strict replication. The use of multiple, non-equivalent metrics of “success” in the Nature article reinforces this ambiguity. Replication rates vary substantially depending on the criterion, yet a single headline figure is foregrounded: “Half of social-science studies fail replication test in years-long project“. The result is a study that is informative about the behaviour of findings (and researchers) under perturbation, but is easily—and predictably—read as making stronger claims about the reliability or truth of scientific results than its design can support.

Underlying both issues is a deeper disagreement about what replication is for. The paper’s opening paragraph explicitly reflects this tension. One reference is the National Academies of Sciences (NAS) report, which defines replication in procedural and statistical terms. Collect new data using similar methods and assess whether results are consistent, typically via effect sizes and uncertainty intervals. The other reference is a 2020 PLoS Biology article by Nosek and Errington (the two senior authors of this Nature paper), who argue that the NAS definition is not merely imprecise but conceptually mistaken. On the Nosek-Errington account, determining that a study is a replication is a theoretical commitment. Both confirming and disconfirming outcomes must be treated in advance as diagnostic of the original claim. The Nature paper adopts this language—replication teams were instructed to produce “good faith tests” of claims—but the article reports results entirely using metrics derived from the procedural-statistical tradition of NAS. This is not a superficial inconsistency. The two frameworks imply different standards of success, different interpretations of failure, and different meanings for any aggregated replication rate. The headline figures that have circulated are products of the latter framework; whether they would survive translation into the former is not addressed.

It is here that Campbell and Stanley’s observation, and its formalisation, becomes decisive. The procedural-statistical approach implicitly treats internal validity as primary and assumes that external validity can be inferred from it. That is, if results are consistent, the finding travels. The structural trade-off shows that this assumption cannot hold. The very steps taken to secure internal validity constrain the scope of generalisation. A high replication rate under this framework may therefore be simultaneously informative and misleading. It indicates that a result can be reproduced under sufficiently similar conditions, while obscuring how narrow those conditions may be. The Nosek-Errington framework recognises the need for theoretical commitment, but without a principled account of causal structure it cannot resolve the tension either. What the Nature paper ultimately demonstrates—perhaps inadvertently—is that replicability is not a property of findings alone. It is a property of the relationship between a finding and the conditions under which it is tested. This underscores a Cartwrightian notion of relationships tied to particular material configurations–nomological machines. Until that relationship is made explicit, headline replication rates will continue to invite overconfident conclusions in both directions and admonitions for better methods.

I did not have access to the published article which is behind the Springer-Nature paywall. Instead I relied on the publicly available preprint.

Analytic robustness could be a real problem

A recent article in Nature on the robustness of research findings in the social and behavioural sciences found that only 34% of re-analyses of the data yielded the same result as the original report. This sounds horrible. It sounds like two-thirds of the research that social and behavioural scientists are doing is low quality work, and certainly does not deserve to be published. One might reasonably ask if “confabulist” rather than “scientist” might not be a better job title.

Unfortunately, the edifice of “robust research” has been built on foundations of sand. The research shares many of the weaknesses of another article recently published in Science Advances, which I discuss here. There is little that can be concluded from the research that could actually inform scientific practice nor permit any observation about the quality or robustness of the original articles. It does, however, say something of interest for sociologists of science about the diversity of views that researchers have about how to re-analyse data to address conceptual claims.

The procedure followed in the Nature article was described thus.

To explore the robustness of published claims, we selected a key claim from each of our 100 studies, in which the authors provided evidence for a (directional) effect. We presented each empirical claim to at least five analysts along with the original data and asked them to analyse the data to examine the claim, following their best judgement and report only their main result. The analysts were encouraged to analyse those studies where they saw the greatest relevance of their expertise.

The word “claim” here does a lot of work. One might reasonably argue that a scientific claim in a published article is a statement of finding in the context of the hypothesis, the model, the analytic process, and the results. But this is not what is meant here. That full scientific sense of a claim is closer to what the Centre for Open Science team use as a starting point for a separate article on “reproducible” research. In the context of this article a “claim” is some vaguer statement of finding. It is an isolated single claim, has a direction of effect, and critically, is “phrased on a conceptual and not statistical level”.

The conceptual claim is closer to a vernacular claim. It is closer to the kind of thing you might say at a dinner party or read in the popular science section of a magazine. Something like, “did you hear that single female students report lower desired salaries when they think their classmates can see their preferences?” (Claim 025).

Under this framework, one should be able to abstract a full scientific claim into a conceptual claim, and if the conceptual claim is robust, independent scientists analysing the same data, making equally sensible choices about the analysis of the data, will converge on the conceptual claim. The challenge is that your pool of independent and equally sensible scientists need to agree with each other (without consultation) how that conceptual claim is to be translated into a scientific claim. A part of the science is deciding on the estimand for testing the claim, but the estimand is fixed by the analytic choice not by the conceptual claim. If two scientist analyse the same dataset but target different estimands through their analytic choices, they are not converging on the same conceptual claim. Against all logic, an analytic schema targeting a different estimand that nonetheless produces an estimate close to the estimate of the original paper, supports the robustness of the paper.

The framework, therefore, has a double incoherence. First, divergence of estimates (between the original analysis and re-analysis) is misread as fragility when it may simply reflect different estimands—different scientists sensibly translating the conceptual claim into different scientific claims. Second, and more damaging, convergence is misread as robustness when it may be entirely spurious—two analysts targeting different estimands who happen to produce similar point estimates are not confirming each other. They’re producing agreement by accident, across questions that aren’t the same question.

So the framework is wrong in both directions simultaneously. It penalises legitimate scientific pluralism and rewards numerical coincidence. A study could score as highly robust because several analysts happened to get similar numbers while asking entirely different questions. A study could score as fragile because several analysts made defensible but divergent estimand-constituting choices that led to genuinely different answers to genuinely different questions.

There is another an far more interesting reading of this paper, which has neither a click-bait quality nor the opportunity to remonstrate. Where the authors have identified fragility (or a lack of robustness), another could legitimately and positively see vitality and methodological pluralism. The social and behavioural sciences work in the messy space of self-referential agents actively interacting with and changing the environments in which they live and do science. It is hardly surprising that epistemic pluralism is a consequence of this. The 34% figure is not a scandal. It is valuable (under appreciated) data about the nature of social reality.

I did not have access to the published article which is behind the Springer-Nature paywall. Instead I relied on the publicly available preprint.