The Effectiveness Trap
TL:DR; Risk professionals in organisations with a broken risk culture need to navigate the effectiveness trap: how people inside powerful or corrupted institutions may tolerate actions they oppose to preserve access and influence. This includes exploring the moral uncertainty of working from within, the risks of dissent, and how a principled resignation can end up accelerating cultural norms that favour conformity over accountability.
The views and opinions expressed on this account are my own and do not reflect the official policy or position of my employer. Any content provided is for informational purposes only and should not be considered or relied upon as professional advice
I have followed the recent turmoil at OpenAI with interest, particularly the resignations of several high-profile employees on matters of principle. One of the most striking was David Robinson, who led the writing of the safety reports published with each major launch. In an open letter[i], Robinson argued that OpenAI’s preference for trial and error “by its very nature, guarantees periodic failures – and the scale of those failures is growing as systems become more capable”. By leaving and continuing to argue for a different approach, he hopes to persuade AI companies to adopt safety expertise developed in other industries where failure can be catastrophic.
There is an interesting question here about how organisations can change their risk culture. OpenAI and other frontier labs are especially interesting examples, because the culture that Robinson and others are challenging was also central to their early success.
But my focus is narrower: Robinson’s decision to quit rather than remain in a company whose actions he could no longer support and no longer believed he could shape from within.
This is certainly a conundrum that we risk professionals hope never to face. Even so, it is worth considering, if only as a thought experiment.
The Effectiveness Trap
In 2016, I attended a Guardian event coinciding with the publication of the Chilcot Inquiry into the UK’s involvement in the invasion of Iraq. The report[ii] was broadly critical of the British government and military. It concluded that the case for war was deficient; the legal basis was far from satisfactory; the UK had overestimated its ability to influence US decisions; planning and preparation were “wholly inadequate”; and military action failed to achieve its goals. In short, the findings were damning.
The panel of guests, all of whom had been involved in Iraq, were in a philosophical mood. Sir William Patey had served as UK Ambassador to Iraq and Afghanistan. Emma Sky had been a political adviser to the United States Army. Major General Tim Cross, a former Director General, had led the UK’s Joint Force Logistic Component and helped plan the invasion. All three had held senior and influential roles, but they remained subordinate to the US military, which controlled the strategy and conduct of the war and subsequent occupation.
The most compelling part of the discussion concerned their position as relative outsiders, at least in terms of formal decision-making power. They had to balance serious concerns about the direction of the war and occupation against a desire to remain useful and influential. Keeping a seat at the decision-making table sometimes meant facilitating actions they opposed and, in some cases, believed harmed both Iraq and the allied Western forces.
James Thompson, an American specialist in East Asia, identified a similar dynamic when examining power during the Vietnam War. He called it the effectiveness trap: “the inclination to remain silent or acquiesce in the presence of great men - to live to fight another day, to give on this issue so that you can be effective on later issues - is overwhelming. Nor is it the tendency of youth alone; some of our most senior officials, men of wealth and fame, whose place in history is secure, have remained silent lest their connection with power be terminated.”[iii]
I don’t wish to judge the panellists harshly. Without hindsight, it is difficult to know when (or whether) to take a principled stand against a compromised organisation. The choice is harder still when norms and strategic integrity erode gradually rather than collapse all at once.
It is possible that the US military included non-US and non-military allies partly to give its decisions greater legitimacy - the appearance of a democratic coalition at work. If so, participants outside the US command structure had little real power. Open dissent would reduce their value as symbols of legitimacy. Once people cease to be useful to an ingroup, they are often excluded or removed.
When your only decisive weapon is a career suicide vest, it is understandable to want to wait until using it will have a meaningful effect. The act must also be visible and unambiguous enough to protect the individual’s reputation. As with whistleblowers, people who resign on principle often face retaliation intended to undermine both their credibility and their concerns. In that context, David Robinson’s decision to hire a PR firm after resigning, but before publishing his open letter, was astute.
In practice, the effectiveness trap is rarely clear-cut. Emma Sky has described actions she took during the occupation that she believed served the Iraqi population. Carrying them out required real influence and therefore a close relationship with the US military. Yet her frustrations are also clear: her power depended on the military’s favour - or at least on its lack of interest in her work - and several initiatives were reversed or contradicted without her involvement. Her ability to act may also have depended on having first demonstrated loyalty to the ingroup and earned its trust. No handbook tells us when to limit harm from within and when to leave and become an external critic.
The Case For Sticking It Out
A fundamental assumption of those caught in the effectiveness trap is that an ostracised dissenter will be replaced by someone less willing or able to challenge power. Whether that happens depends to a large extent on how the replacement is chosen. The phenomenon of political self-selection suggests that applicants may be more likely to share the organisation’s ideology, and be less inclined to dissent. This has certainly borne out in the Trump administration where each principled resignee has been replaced by someone more compliant.
Yet whistleblowers - and perhaps dissenters more broadly - also tend to have spent less time in their organisations, suggesting that a replacement may be as likely to continue to challenge those in power.
Three conditions make a compliant replacement more likely:
The organisation’s ideology is clear to applicants.
The candidate pool is limited to people already known to the leadership.
Leaders make the appointment with little outside scrutiny.
Modern commercial recruitment often tries to avoid the second and third conditions, while organisational ideology may not be obvious to applicants. Military settings can be different: the prevailing outlook is often well known, and candidates are commonly drawn from within the institution and already known to senior leaders.
So far, I have assumed that principled people are trying—sincerely or otherwise—to maximise good or limit harm.
In a corrupted system, political self-selection may gradually push principled people out and replace them with “yes men”, or prisposoblenets. Joshua Yaffa defines the Russian word “prisposoblenets” as “a person skilled in the act of compromise and adaptation, who intuitively understands what is expected of him and adjusts his beliefs and conduct accordingly.” [iv]
The final and most severe stage of an organisation’s cultural corruption arrives when prisposoblenets dominate. These are people who have come to see their primary role as satisfying peers and managers, with their sense of purpose completely divorced from the result of the organisation’s actions.
The Case for Getting Outta Dodge
Angela Merkel’s approach to governing offers one possible model for engaging with an organisation whose risk culture is broken. Her most visible style of “Merkeln” - waiting, observing, and moving cautiously towards a workable consensus - reflects the worst cliches of risk management. That is not the aspect of her legacy I recommend!
More useful was her less visible discipline: do not draw red lines unless you are prepared to enforce them. Preserve room to manoeuvre, avoid unnecessary commitments, and manage crises through negotiation and compromise rather than public declarations of non-negotiable positions.
Using this approach, the act of leaving becomes the final step in a negotiation aimed at consensus and compromise. It is a step that will never be easy. But it can be done without regret when a carefully and soberly defined red line has clearly been crossed by the organisation.
Hopefully that has sparked some ideas. I am hoping to start a discussion with this blog. If you find it interesting, please share! If you would like to contribute or share feedback, please comment below or message me on LinkedIn!
[ii] The Report of the Iraq Inquiry, 6 July 2016, Report of a Committee of Privy Counsellors, https://www.gov.uk/government/publications/the-report-of-the-iraq-inquiry
[iii] How Could Vietnam Happen? An Autopsy, The Atlantic, APRIL 1968 ISSUE, James Thompson
[iv] Between Two Fires: Truth, Ambition, and Compromise in Putin's Russia, 2021, Joshua Yaffa, 978-1783783724
Comments