This article is written by Martin Stanley, editor of the Understanding Government websites, and former UK civil servant.

Public servants are often asked to design policies which are intended to change the behaviour of large organisations, or their employees. In practice, this is easier said than done.

Large organisations are not merely super-sized versions of smaller firms. Many senior managers do not know (and some do not want to know) what is going on outside their head offices, nor what risks (financial, environmental and other) are being taken by their staff. Internal pressure to report good results, and executives' aversion to taking responsibility for errors, mean that senior managers are often the last to learn what is going wrong.

This has in recent years led to a number of catastrophic failures involving large organisations, including the 2007/8 worldwide collapse of financial institutions, the explosion at BP’s Texas City Refinery, the accident at the Alton Towers resort, and the mismanagement of the Mid-Staffordshire Hospitals.

the principal-agent problem

Policymakers (and regulators) therefore need to take account of the culture and tensions that are typical of the large organisations whose behaviour they are seeking to influence or control.

In this first article in a three-part series, I will seek to explore these questions in some depth. This first note examines what management experts refer to as the principal-agent problem.

The principal-agent problem may be found in all large organisations.

There is a strong tendency for humans to align their goals and behaviours to those of the team or work group around them. As a result, local operations of large organisations are often in practice run in the interests of the firm's middle ranking managers and not their ultimate bosses. In other words, although the middle managers (the agents) are hired by the Board or shareholders (the principal) to pursue the principal's interests, the agents in practice develop priorities of their own.

It is almost universally common — and to some extent very sensible — for middle managers to focus on (what they see as) the long term good of their factory, office or other small part of the organisation. When asked to find efficiency savings they will fight hard to retain their budget and their staff numbers. They see no point in implementing all those “stupid directives” from their head office, and they find record keeping a bore. Boxes are meant to be ticked, whether or not the associated action has been carried out.

You need to be aware that it can be very hard to obtain reliable information from the intermediate levels in any hierarchy

Such managers will often enter into an implicit bargain with their teams, allowing the use of material for personal ends, and providing generous expense accounts and other perks, in order to generate a better (non-confrontational) climate within the team — often characterised as "high morale". Such teams resist change — especially if change might lead to greater efficiency (working harder and/or job losses).

Peering into the hierarchy

It is also the case, of course, that some middle managers are quite weak, and incapable of standing up to staff who are not doing a good or reliable job.

The problem gets even worse when the organisation's Board of Directors (or equivalent) is formed of out-of-touch mates of the Chief Executive and/or the “Great and Good”. Theranos is a good recent example. The company was set up, with the best of intentions, by Elizabeth Holmes who was determined to find a way to make blood testing more convenient for both doctor and patient. John Carreyrou's book Bad Blood describes how it all went terribly wrong.

He notes that Elizabeth Holmes had recruited the big names to her Board, including George Shultz, James Mattis and Henry Kissinger. Needless to say, none of them had the knowledge or experience to understand what the scientists in the company were doing, nor were they inclined to be critical of their apparently fabulous Chief Exec who quickly burned though $700 million of shareholder funds. When push came to shove, the whole house of cards came tumbling down, revealing an embarrassing fact: Theranos was nowhere near delivering a product that could live up to its over-hyped promises.

This principal-agent problem has two consequences for policymakers.

  • First, you need to be aware that it can be very hard to obtain reliable information from the intermediate levels in any hierarchy.
  • Second, you need to be aware that agents and their teams often resist the introduction of (what they see as) tedious protocols aimed at improving quality or safety. Rail and marine accident reports, for instance, include many examples of such behaviour. It is quite common for teams to falsify data, including quality and safety data. New recruits are told by experienced colleagues that formal instructions — or things they learned in an induction program — can safely be ignored as out-of-date or impractical. —

It can therefore be very hard for an outsider to find out what is really happening inside a regulated entity, not least because the entity's own senior executives probably do not themselves know what is happening and, if they do, will be very reluctant to admit their partial loss of control. Much the same applies to compliance officers, whose creation is too often regarded by their Board as a box which has been ticked, and so a good reason then to ignore regulatory issues.

When things go south

Here are some other examples of the principal-agent problem.

An explosion at the BP’s Texas City Refinery in 2005 killed 15 workers and injured more than 150 others. The plant had been poorly maintained, was badly managed, had a strong blame culture, and had been subject to several rounds of cost-cutting whose safety implications were not understood. This was a clear example of BP's “top brass” not having any understanding of what was happening deep down in their organisation.

Following the 2008 Financial Crisis, it became all too clear that the individual financial institutions — and in particular their Boards — did not appreciate or did not care about many risks including:

  • the riskiness of sub-prime lending (mainly in the US); nor..

  • the riskiness of certain commercial lending made by certain aggressive lenders; nor..

  • the fact that the apparently clever spreading of risk did not in fact reduce the total risk undertaken by the financial institutions when taken together, the fact that between 1986 and 2006 the average annual return on banking rose from its historical norm of 2% to 16%, as a result of the banks taking riskier bets, and rising leverage — which in practice amount to the same thing. There was no skill, efficiency, intelligence or judgement involved.

Alistair Darling was Chancellor of the Exchequer during the 2008 financial crisis and tells the nice story that, the night after Royal Bank of Scotland went belly up, he was taken aside by the chairman of one of Britain's biggest banks and offered the reassurance that "they had had a meeting last night and decided that, from now on, we will only take on risks that we understand".

Neither the Mid Staffordshire Hospital Board nor its regulators were fully aware of the depth of the problems in Stafford Hospital where

  • patients were left unwashed in their own filth for up to a month as nurses ignored their requests to use the toilet or change their sheets;

  • four members of one family, including a new-born baby girl, died within 18 months after blunders at the hospital; and

  • wards were left filthy with blood, discarded needles and used dressings while

  • bullying managers made whistle-blowers too frightened to come forward.

The most important lesson is that you should design new rules and regulations in such a way that their purpose is clear and sensible, and so that they can easily be followed by front line staff

Similar problems then came to light at Morecambe Bay NHS Foundation Trust, where 11 babies died as a result of a “them and us culture” in which midwives at the Furness General Hospital — keen on natural birth — called themselves "the musketeers" as they fought battles with doctors. Clinical records were destroyed and the Trust suppressed a critical report in order to get a clean bill of health from the regulators en route to achieving "Foundation" status. Whistle-blowers alerted several regulators, all of whom failed to take effective action. The Trust's Chief Executive left with a £225,000 payoff and set up a consultancy, boasting of "getting the best from teams". It would be funny of it weren't so tragic.

The consequences were commercial rather than damaging to health, but I was struck by Nick Butler's comment, in the FT in January 2014, about a profits warning by Royal Dutch Shell:

"At the heart of the .. problem seems to be the gap between operational reality and top management, including the board. Big problems capable of triggering a profits warning in a company this size do not arise over 80 days. They grow more slowly and are often invisible to boards meeting every six weeks who have to trust whatever data they are given. The rest of the industry should avoid gloating. It would be much more useful for the boards of the other majors to ask themselves if they really know what is happening in the companies for which they are legally responsible”.

The policy implication

So how should policymakers deal with this problem?

It is tempting to assume that when front line staff fail to comply with safety, financial and other protocols, it is because they are responding to pressure from above, such as meeting financial or other targets, to complete work quickly, to maintain production etc. However, leaders of rule-bending teams may themselves be keen to get through work quickly, and/or to impress seniors with their achievements, or maybe simply to get home on time, or to avoid a small amount of work running over into the next day.

It is truly very difficult for regulators and company directors to know that this is happening unless they impose robust and unpredictable inspection arrangements backed up by a strong compliance culture, and opportunities for whistle-blowing.

The principal-agent problem also means that policymakers need to take into account how the middle and lower reaches of an organisation might react to regulatory pressure. The need for change and/or improvement may be accepted by senior executives, but this does not mean that the necessary changes will be accepted or implemented at the working level. Any regulatory intervention needs to be designed with this problem in mind.

But the most important lesson is that you should design new rules and regulations in such a way that their purpose is clear and sensible, and so that they can easily be followed by front line staff. If not, then — once your attention is elsewhere — those clever laws and regulations will be subverted and slowly neutralised. — Martin Stanley

Further reading:

This and related subjects are discussed in more detail on the Understanding Policy Making and Understanding Regulation websites.

Jean Tirole's Hierarchies and Bureaucracies: On the Role of Collusion in Organizations offers a lengthier discussion and analysis of the Principal-Agent Problem.

(Picture credit: Death to the stock photo)

Make sure to share your own thoughts with the author by leaving a comment below