Unrecognised simplicities of effective action #2(b): the Apollo programme, the Tory train wreck, and advice to spads starting work today

A few months ago I put a paper on my blog: The unrecognised simplicities of effective action #2: ‘Systems engineering’ and ‘systems management’ — ideas from the Apollo programme for a ‘systems politics’.

It examined the history of the classified programme to build ICBMs and the way in which George Mueller turned the failing NASA bureaucracy into an organisation that could put man on the moon. The heart of the paper is about the principles behind effective management of complex projects. These principles are relevant to Government, politics, and campaigns.

The paper is long as I thought it worthwhile to tell some of the detailed story. At the suggestion of various spads, ministers, hacks and so on I have cut and pasted the conclusion below particularly for those starting new jobs today. This is in the form of a crude checklist that compares a) the principles of Mueller’s systems management and b) how Whitehall actually works.

You will see that Whitehall operates on exactly opposite principles to those organisations where high performance creates real value. You will also soon see that you are now in a culture in which almost nobody is aware of this and anybody who suggests it sinks their career. In your new department, failure is so normal it is not defined as ‘failure’. Officials lose millions and get a gong. There is little spirit of public service or culture of responsibility. The most political people are promoted and the most competent people, like Victoria Woodcock, leave. The very worst officials are often put in charge of training the next generation. For most powerful officials, the most important thing is preserving the system, closed and impregnable. Unlike for ministers, the TV blaring with DISASTER is of no concern – provided it is the Minister in the firing line not them – and the responsible officials will happily amble to the tube at 4pm while political careers hang in the balance and you draft statements taking ‘full responsibility’ for things you knew nothing about and would have been prohibited from fixing if you had.

For all those spads in particular who are moving into new jobs, it is worth reflecting on the deep principles that actually determine why things work and do not work. Nobody will explain these to you or talk to you about them. Sadly, few MPs these days understand the crucial role of management – they tend to think of it like science as a rather lowly skill beneath their Olympian status – so you will also probably have to cope with the fact that your minister is more interested in keeping one step ahead of Simon Walters (they won’t). The thing that officials will try hardest to do is convey to you that you have no role in personnel decisions and/or management.

If you accept that, you are accepting at the start that you will achieve very little. The reason why Gove’s team got much more done than ANY insider thought was possible – including Cameron and the Perm Sec – was because we bent or broke the rules and focused very hard on a) replacing rubbish officials and bringing in people from outside and b) project management.

You cannot reform the way the civil service works. Only a PM can do that and there is no chance of May doing it – she blew her chance and her reward is to be pushed around by Heywood and Sue Gray until her colleagues pull the plug and start the leadership campaign. You should assume that won’t be long so focus, manage a few priorities with daily and weekly timetables, and use embarrassing errors to negotiate secret deals with the Perm Sec to move rubbish officials out of your priority areas – trust me, Perm Secs understand this game and will do deals with alacrity to make their lives easier. Officials are less politically biased than you probably have been told – they are much more concerned with avoiding hard work and protecting the system than in resisting specific policies, and you can exploit this. Make alliances with the good officials who still have hope and have not been broken by the system, there are surprisingly many who will pop up if they think you actually care about the public rather than party interests.

You will also notice that fundamental issues of organisational culture described below explain the shambles of CCHQ over the past 8 weeks: the lack of information sharing, the lack of orientation, the culture of blaming juniors for the failures of overpaid senior people, bottlenecks preventing fast decisions, endless small errors compounding into a broken organisation because nobody knows who is responsible for what and so on. Every failing organisation has the same stories, people find it very hard to learn from the most successful organisations and people.

To the extent Vote Leave was successful, it was partly because I consciously tried to copy Mueller in various ways, though given my own severe limitations this was patchy. If you ever get the chance to exercise leadership, try to copy people like Mueller who tried to make the world better and build an organisation that people were proud to serve.

Finally, consider the basic condition that allows Westminster and Whitehall to be so rubbish and get away with it: they are not just monopolies, they set the rules of the game, and both the civil service and the parties make it almost impossible for outsiders to influence anything. But a) the combination of the 2008 crisis, Brexit, and extreme unhappiness about politics as usual provides a potentially powerful fuel for an insurgency, and b) technology provides opportunities for startups to catch public imagination and scale extremely fast. I’ve always been sceptical of the idea of a new UK party of any sort but I increasingly think there is a chance that a handful of entrepreneurs could start a sort of anti-party to exploit the broken system and create something which confounds the right/centre/left broken mental model that dominates SW1 and which combines Mueller’s principles with Silicon Valley technology.

If the Tory Party does not make some profound changes fast, then it faces being blamed for the disintegration of Brexit talks and the election of Corbyn after which it is possible that, rather than attempting a coup to take them over, entrepreneurs may decide it is more rational to build something that ploughs them into the earth next to Corbyn.

I said since last summer that if the Tory Party tried to carry on with Brexit and government using the same broken Downing Street operation, which spends its time on crap spin and has almost no capacity for serious management, and the same broken political operation, dominated by people who have failed to persuade the country convincingly for many years, then they would blow up. They failed to change Downing Street and they ran yet another fundamentally misconceived campaign that blew massive structural advantages. Kaboom.

[[Within minutes of publishing this blog I got the following email from someone I haven’t met but who I know was inside CCHQ with the para above highlighted and these words: ‘This is exactly my depressing experience – shit show run by people who don’t care about anything other than their jobs.’]]

MPs of all parties need to realise that the referendum makes it impossible to carry on with your usual bullshit – it forces changes upon you even though you want to carry on with the old games. The first set of MPs that realise this and change their operating principles will quickly overwhelm the others: there is a huge first-mover advantage especially in a field characterised by institutional incompetence that is susceptible to external shocks (terror, financial collapse) and which is opening up to technological disruption. And you will only get on top of Brexit if you realise that leaving the EU is a systems problem requiring a systems response and this means a radically different organisation of the UK negotiating team. The challenge is not far short of the political equivalent of the Apollo program and it needs similarly imaginative management.

For those who do want to do something better, the below will be useful. I encourage you to read the whole history HERE but for those rushing through a sandwich on Day 1 this summary will help you think of the big picture. If you want a detailed tutorial on how the civil service works then read The Hollow Men HERE

[Added later… It is also very instructive that despite the triumph of Mueller’s methods, NASA itself abandoned them after he left and has never recovered. Even spectacular success on a world-changing project is not enough to beat bureaucratic inertia. Also, the US Government passed so many laws that Mueller himself said in later life it would be impossible to repeat Apollo without making it a classified ‘black’ project to evade the regulations. JSOC, US classified special forces, has to run a lot of its standard procurement via ‘black’ procurement processes just to get anything done. The abysmal procurement rules imposed under the Single Market are just one of the good reasons for us to get out of the SM as well as the EU. I had to deal with them a lot in the DfE and had to find ways to cheat them a lot to get things done faster and cheaper. They add billions to costs every year and Whitehall refused for years even to assess this huge area to avoid undermining support for the EU.]

*

Excerpt from The unrecognised simplicities of effective action #2 (p.28ff) 

Core lessons [of Mueller’s systems management] for politics?

Finally, I will summarise some of the core lessons of systems management that could be applied to re-engineering political institutions such as Downing Street.  Mueller’s approach meant an extreme focus on some core principles:

  • Organisation-wide orientation. Everybody in a large organisation must understand as much about the goals and plans as possible. Whitehall now works on opposite principles: I doubt a single department has proper orientation across most of the organisation (few will have it even across the top 10 people), never mind a whole government. This is partly because most ministers fail at the first hurdle — developing coherent goals — so effective orientation is inherently impossible.
  • Integration. There must be an overall approach in which the most important elements fit together, including in policy, management, and communications. Failures in complex projects, from renovating your house to designing a new welfare system, often occur at interfaces between parts. Whitehall now works on opposite principles: for example, Cameron and Osborne approached important policy on immigration/welfare in the opposite way by 1) promising to reduce immigration to less than 100,000 while simultaneously 2) having no legal tools to do this (and even worse promising to change this then failing in the EU renegotiation) and 3) having welfare policies that incentivised more immigration then 4) announcing a new living wage thus increasing incentives further for immigration. They emphasised each element as part of short-term political games and got themselves into a long-term inescapable mess.
  • Extreme transparency and communication, horizontally as well as hierarchically. More, richer, deeper communication so that ‘all of us understand what was going on throughout the program… [C]ommunications on a level that is free and easy and not constrained by the fact that you’re the boss… [This was] the secret of the success of the program, because so many programs fail because everybody doesn’t know what it is they are supposed to do’ (Mueller). Break information and management silos — a denser network of information and commands is necessary and much of it must be decentralised and distributed between different teams, but with leadership having fast and clear information flow at the centre so problems are seen and tackled fast (a virtuous circle). There is very little that needs to be kept secret in government and different processes can easily be developed for that very small number of things. As McChrystal says of special forces operations generally the advantages of communication hugely outweigh the dangers of leaks. Whitehall now works on opposite principles: it keeps information secret that does not need to be secret in order to hide its own internal processes from scrutiny, thus adding to its own management failures and distrust (a vicious circle).
  • ‘Configuration management’. There must be a process whereby huge efforts go into the initial design of a complex system then there is a process whereby changes are made in a disciplined way such that a) interdependencies are tested where possible by relevant people before a change is agreed and b) then everybody relevant knows about the change. This ties together design, engineering, management, scheduling, cost, contracts, and allows the coordination of interdisciplinary teams. Test, learn, communicate results, change where needed, communicate… Whitehall now works on opposite principles: it does not put enough effort into the initial design then makes haphazard changes then fails to communicate changes effectively.
  • Physical and information structures should reinforce open communication. From Mueller’s NASA to JSOC, organisations that have coped well with complexity have built novel control centres to reinforce extreme communication. Spend money and time on new technologies and processes to help spread orientation and learning through the organisation. Whitehall now works on opposite principles: e.g. its antiquated committee structure and ‘red box’ system are ludicrously inefficient regarding management but are kept because they give officials huge control over ministers.
  • Long-term budgets. Long-term budgets save money. Whitehall now works on opposite principles: normal government budget processes do not value speed and savings from doing things fast. They are focused on what Parliament thinks this year. This makes it very hard to plan wisely and wastes money in the long-term (see below).
  • You need a complex mix of centralisation and decentralisation. While overall vision, goals, and strategy usually comes from the top, it is vital that extreme decentralisation dominates operationally so that decisions are fast and unbureaucratic. Information must be shared centrally and horizontally across the organisation — it is not either/or. Big complex projects must empower people throughout the network and cannot rely on issuing orders through a hierarchy. Whitehall now works on opposite principles: it is a centralising ratchet. E.g. Budgets and spending reviews are the exact opposite of Mueller’s approach. 1) They are short-term with almost no long-term elements. 2) They do not balance off priorities in any serious way. 3) They involve totally fake numbers — every department lies to the Treasury and provides fake numbers. Treasury officials dig into these. There are rounds of these games. Officials never stop lying. To maintain the charade the Chancellor never says to the SoS ‘stop your officials lying to us’ — candour would break the system. 4) The Treasury does not have the expertise to evaluate most of what they are looking at. The idea it is a department staffed by brilliant whiz kids is a joke. I saw DfE officials with very modest abilities routinely cheat the Treasury.
  • Extreme focus on errors. Schriever had ‘Black Saturdays’ and Mueller had similar meetings focused not on ‘reporting progress’ but making clear the problems. Simple as it sounds this is very unusual. Whitehall now works on opposite principles: routinely nobody is held responsible for errors and most management works on the basis of ‘give me good news not bad news’. Neither the culture nor incentives focus effort on eliminating errors. Most don’t care and you see those responsible for disaster ambling to the tube at 4pm or going on holiday amid meltdown.
  • Spending on redundancy to improve resilience. Whitehall now works on opposite principles: it tends to treat redundancy as ‘waste’ and its short-term budget processes reinforce decisions that mean out-of-control long-term budgets. By the time the long-term happens, the responsible people have all moved on to better paid jobs and nobody is accountable.
  • Important knowledge is discovered but then the innovation is standardised and codified so it can be easily learned and used by others. Whitehall now works on opposite principles: for example, in the Department for Education officials systematically destroyed its own library. The DfE operated with almost no institutional memory. By the time I left in 2014, after David Cameron banned me from entering any department officials would ask to meet me outside to find out why decisions had been taken in 2011 because three years later almost everybody had moved on to other things. The Foreign Office similarly destroyed its own library.
  • Systems management means lots of process and documentation but at its best it is fluid and purposeful — it is not process for ass-covering. The crucial ‘Gillette Procedures’ swept away red tape and Schriever battled the system to maintain freedom from normal government processes. When asked how he would do a similar programme to Apollo now (1990s) Mueller responded that the only way to do it would be as a classified ‘black’ project to escape the law on issues like procurement. Whitehall now works on opposite principles: its obsession is bullshit process for buck-passing and it fights with all its might against simplification and focus.
  • Saving time saves money. Schriever and Mueller focused on speed and saving time. Whitehall now works on opposite principles: its default mode is to go slower and those who advocate speed are denounced as reckless. Repeatedly in the DfE I was told it was ‘impossible’ to do things in the period I demanded — often less than half what senior officials wanted — yet we often achieved this and there was practically no example of failure that came because my time demands were inherently unreasonable. The system naturally pushes for the longest periods they can get away with to give themselves what they think of as a chance to beat ‘expectations’ but then they often fail on absurdly long timetables. In the DfE we often had a better record of hitting timetables that were ‘impossibly’ short than on those that were traditionally long. Also in many areas there is no downside to pushing fast — the worst that happens is minor and irrelevant embarrassment while the cumulative gains from trying to go fast are huge.
  • The ‘systems’ approach is inherently interdisciplinary ‘because its function is to integrate the specialized separate pieces of a complex of apparatus and people — the system — into a harmonious ensemble that optimally achieves the desired end’ (Ramo). Whitehall now works on opposite principles: it is hopeless at assembling interdisciplinary teams and elevates legal advice over everything in relation to practically any problem, causing huge delays and cost overruns.
  • The ‘matrix management’ system allowed coordination across different departments and different projects.  Whitehall now works on opposite principles. It is stuck with antiquated departments, an antiquated Cabinet Office system, and antiquated project management. Anything ‘cross-government’ is an immediate clue to the savvy that it is doomed and rarely worth wasting time on. A ‘matrix’ approach could and should be applied to break existing hierarchies and speed everything up.
  • People and ideas were more important than technology. Computers and other technologies can help but the main ideas came in the 1950s before personal computers. JSOC applied all sorts of technologies but Colonel Boyd’s dictum holds: people, ideas, technology — in that order. Whitehall now works on opposite principles: for example, the former Cabinet Secretary, Gus O’Donnell, recently blamed a ‘lack of investment’ in IT and a shortage of staff for a huge range of Whitehall blunders. This is really deluded. The central problem is known to all experts and is shown in almost every inquiry: IT projects fail repeatedly in the same ways because of failures of management, not ‘lack of investment’, and adding people to flawed projects is not a solution.

Ministers have little grip of departments and little power to change their direction. They can’t hire or fire and they can’t set incentives. They are almost never in a job long enough to acquire much useful knowledge and they almost never have the sort of management skills that provide alternative value to specific knowledge. They have little chance to change anything and officials ensure this little chance becomes almost no chance.

This story shows how to do things much better than normal. It shows that the principles underlying Mueller’s success are naturally in extreme competition with the principles of management that dominate all normal bureaucracies, public or private. People have been able to read about these principles for decades yet today in Whitehall almost everything runs on exactly the opposite principles: incentives operate to suppress learning. The institutional and policy changes inherent in leaving the EU are a systems problem requiring a systems response. Implementing Mueller’s principles would mean changes to most of the antiquated and failing foundations of Whitehall and bring big improvements and cost savings. Such changes are likely to be resisted by most MPs as well as Whitehall given few of them understand or have experience in high performance teams and would regard Mueller’s approach as a threat to their career prospects.

Because Whitehall is a system failure in which different failures are entangled, its inhabitants tend to potter around in an uncomprehending fog of confusion without understanding why things fail every day and therefore they do not support changes that could improve things even though these changes would be personally advantageous particularly for the first mover.

What is the minimum needed to break bureaucratic resistance and spark a virtuous circle?

How can people outside the system affect mission critical political institutions protected from market competition and resistant to major reforms?

How can we replace many traditional centralised bureaucracies with institutions that mimic successful biological systems such as the immune system that a) use distributed information processing to identify useful structure in the environment, b) find ‘good enough’ solutions in a vast search space of possibilities, and c) move at least ten times faster than existing systems?

[If you find this interesting and/or useful, then the PDF of the whole story is here. It involves some of the cleverest people of the 20th Century, such as John von Neumann.]

On the referendum #20: the campaign, physics and data science – Vote Leave’s ‘Voter Intention Collection System’ (VICS) now available for all

‘If you don’t get this elementary, but mildly unnatural, mathematics of elementary probability into your repertoire, then you go through a long life like a one-legged man in an ass-kicking contest. You’re giving a huge advantage to everybody else. One of the advantages of a fellow like Buffett … is that he automatically thinks in terms of decision trees and the elementary math of permutations and combinations… It’s not that hard to learn. What is hard is to get so you use it routinely almost everyday of your life. The Fermat/Pascal system is dramatically consonant with the way that the world works. And it’s fundamental truth. So you simply have to have the technique…

‘One of the things that influenced me greatly was studying physics… If I were running the world, people who are qualified to do physics would not be allowed to elect out of taking it. I think that even people who aren’t [expecting to] go near physics and engineering learn a thinking system in physics that is not learned so well anywhere else… The tradition of always looking for the answer in the most fundamental way available – that is a great tradition.’ Charlie Munger, Warren Buffet’s partner.

During the ten week official campaign the implied probability from Betfair odds of IN winning ranged between 60-83% (rarely below 66%) and the probability of OUT winning ranged between 17-40% (rarely above 33%). One of the reasons why so few in London saw the result coming was that the use by campaigns of data is hard to track even if you know what to look for and few in politics or the media know what to look for yet. Almost all of Vote Leave’s digital communication and data science was invisible even if you read every single news story or column ever produced in the campaign or any of the books so far published (written pre-Shipman’s book).

Today we have made a software product available for download – Vote Leave’s ‘Voter Intention Collection System’ (VICS) – click HERE. It was named after Victoria Woodcock, Operations Director, known as Vics, who was the most indispensable person in the campaign. If she’d gone under a bus, Remain would have won. When comparing many things in life the difference between average and best is say 30% but some people are 50 times more effective than others. She is one of them. She had ‘meetings in her head’ as people said of Steve Wozniak. If she had been Cameron’s chief of staff instead of Llewellyn and Paul Stephenson had been director of communications instead of Oliver and he’d listened to them, then other things being equal Cameron would still be on the No10 sofa with a glass of red and a James Bond flick. They were the operational/management and communications foundation of the campaign. Over and over again, those two – along with others, often very junior – saved us from the consequences of my mistakes and ignorance.

Among the many brilliant things Vics did was manage the creation of VICS. When we started the campaign I had many meetings on the subject of canvassing software. Amazingly there was essentially no web-based canvassing software system for the UK that allowed live use and live monitoring. There have been many attempts by political parties and others to build such systems. All failed, expensively and often disastrously.

Unfortunately, early on (summer 2015) Richard Murphy was hired to manage the ground campaign. He wanted to use an old rubbish system that assumed the internet did not exist. This was one of the factors behind his departure and he decided to throw in his lot with Farage et al. He then inflicted this rubbish system on Grassroots Out which is one of the reasons why it was an organisational/management disaster and let down its volunteers. After Vote Leave won the official designation, many GO activists defected, against official instructions from Farage, and plugged into VICS. Once Murphy was replaced by Stephen Parkinson (now in No10) and Nick Varley, the ground campaign took off.

We created new software. This was a gamble but the whole campaign was a huge gamble and we had to take many calculated risks. One of our central ideas was that the campaign had to do things in the field of data that have never been done before. This included a) integrating data from social media, online advertising, websites, apps, canvassing, direct mail, polls, online fundraising, activist feedback, and some new things we tried such as a new way to do polling (about which I will write another time) and b) having experts in physics and machine learning do proper data science in the way only they can – i.e. far beyond the normal skills applied in political campaigns. We were the first campaign in the UK to put almost all our money into digital communication then have it partly controlled by people whose normal work was subjects like quantum information (combined with political input from Paul Stephenson and Henry de Zoete, and digital specialists AIQ). We could only do this properly if we had proper canvassing software. We built it partly in-house and partly using an external engineer who we sat in our office for months.

Many bigshot traditional advertising characters told us we were making a huge error. They were wrong. It is one of the reasons we won. We outperformed the IN campaign on data despite them starting with vast mounts of data while we started with almost zero, they had support from political parties while we did not, they had early access to the electoral roll while we did not, and they had the Crosby/Messina data and models from the 2015 election while we had to build everything from scratch without even the money to buy standard commercial databases (we found ways to scrape equivalents off the web saving hundreds of thousands of pounds).

If you want to make big improvements in communication, my advice is – hire physicists, not communications people from normal companies and never believe what advertising companies tell you about ‘data’ unless you can independently verify it. Physics, mathematics, and computer science are domains in which there are real experts, unlike macro-economic forecasting which satisfies neither of the necessary conditions – 1) enough structure in the information to enable good predictions, 2) conditions for good fast feedback and learning. Physicists and mathematicians regularly invade other fields but other fields do not invade theirs so we can see which fields are hardest for very talented people. It is no surprise that they can successfully invade politics and devise things that rout those who wrongly think they know what they are doing. Vote Leave paid very close attention to real experts. (The theoretical physicist Steve Hsu has a great blog HERE which often has stuff on this theme, e.g. HERE.)

More important than technology is the mindset – the hard discipline of obeying Richard Feynman’s advice: ‘The most important thing is not to fool yourself and you are the easiest person to fool.’ They were a hard floor on ‘fooling yourself’ and I empowered them to challenge everybody including me. They saved me from many bad decisions even though they had zero experience in politics and they forced me to change how I made important decisions like what got what money. We either operated scientifically or knew we were not, which is itself very useful knowledge. (One of the things they did was review the entire literature to see what reliable studies have been done on ‘what works’ in politics and what numbers are reliable.) Charlie Munger is one half of the most successful investment partnership in world history. He advises people – hire physicists. It works and the real prize is not the technology but a culture of making decisions in a rational way and systematically avoiding normal ways of fooling yourself as much as possible. This is very far from normal politics.

(One of the many ways in which Whitehall and Downing Street should be revolutionised is to integrate physicist-dominated data science in decision-making. There are really vast improvements possible in Government that could save hundreds of billions and avoid many disasters. Leaving the EU also requires the destruction of the normal Whitehall/Downing Street system and the development of new methods. A dysfunctional broken system is hardly likely to achieve the most complex UK government project since beating Nazi Germany, and this realisation is spreading – a subject I will return to.)

In 2015 they said to me: ‘If the polls average 50-50 at the end you will win because of differential turnout and even if the average is slightly behind you could easily win because all the pollsters live in London and hang out with people who will vote IN and can’t imagine you winning so they might easily tweak their polls in a way they think is making them more accurate but is actually fooling themselves and everybody else.’ This is what happened. Almost all the pollsters tweaked their polls and according to Curtice all the tweaks made them less accurate. Good physicists are trained to look for such errors. (I do not mean to imply that on 23 June I was sure we would win. I was not. Nor was I as pessimistic as most on our side. I will write about this later.)

VICS allows data to be input centrally (the electoral roll, which in the UK is a nightmare to gather from all the LAs) and then managed at a local level, whether that be at street level, constituency or wider areas. Security levels can be set centrally to ensure that no-one can access the whole database. During the campaign we used VICS to upload data models which predicted where we thought Leave voters were likely to be so that we could focus our canvassing efforts, which was important given limited time and resources on the ground. The model produced star ratings so that local teams could target the streets more likely to contain Leave voters.

Data flowed in on the ground and was then analysed by the data science team and integrated with all the other data streaming in. Data models helped us target the ground campaign resources and in turn data from the ground campaign helped test and refine the models in a learning cycle – i.e. VICS was not only useful to the ground campaign but also helped improve the models used for other things. (This was the point of our £50 million prize for predicting the results of the European football championships, which gathered data from people who usually ignore politics – I’m still frustrated we couldn’t persuade someone to insure a £350 million prize which is what I wanted to do.) In the official 10 week campaign we served about one billion targeted digital adverts, mostly via Facebook and strongly weighted to the period around postal voting and the last 10 days of the campaign. We ran many different versions of ads, tested them, dropped the less effective and reinforced the most effective in a constant iterative process. We combined this feedback with polls (conventional and unconventional) and focus groups to get an overall sense of what was getting through. The models honed by VICS also were used to produce dozens of different versions of the referendum address (46 million leaflets) and we tweaked the language and look according to the most reliable experiments done in the world (e.g. hence our very plain unbranded ‘The Facts’ leaflet which the other side tested, found very effective, and tried to copy). I will blog more about this.

These canvassing events represented 80-90% of our ground effort in the last few months, hence some of the reports by political scientists derived from Events pages on the campaign websites, which did not include canvassing sessions, are completely misleading about what actually happened (this includes M Goodwin who is badly confused and confusing, and kept telling the media duff information after he was told it was duff). There was also a big disinformation campaign by Farage’s gang, including Bone and Pursglove, who told the media ‘Vote Leave has no interest in the ground campaign’. This was the opposite of the truth. By the last 10 weeks we had over 12,000 people doing things every week (we had many more volunteers than this but the 12,000 were regularly active). When Farage came to see me for the last time (as always fixated only on his role in the debates and not the actual campaign which he was sure was lost) he said that he had 7,000 activists who actually did anything. He was stunned when I said that we had over 12,000. I think Farage et al believe their own spin on this subject and were deluded not lying. (Obviously there was a lot of overlap between these two figures.) These volunteers delivered about 70 million leaflets out of a total ~125 million that were delivered one way or another.

While there were some fantastic MPs who made huge efforts on the ground – e.g. Anne Marie Trevelyan – it was interesting how many MPs, nominally very committed to Leave, did nothing useful in their areas nor had any interest in ground campaigning and data. Many were far more interested in trying to get on TV and yapping to hacks than in gathering useful data, including prominent MPs on our Board and Campaign Committee, some of whom contributed ZERO useful data in the entire campaign. Some spent much of the campaign having boozy lunches with Farage gossiping about what would happen after we lost. Because so many of them proved untrustworthy and leaked everything I kept the data science team far from prying eyes – when in the office, if asked what they did they replied ‘oh I’m just a junior web guy’. It would have been better if we could have shared more but this was impossible given some of the characters.

VICS is the first of its kind in the UK and provided new opportunities. It is, of course, far from ideal. It was developed very quickly, we had to cut many corners, and it could be improved on. But it worked. Many on the ground, victims of previous such attempts, assumed it would blow up under the pressure of GOTV. It did not. It worked smoothly right through peak demand. This was also because we solved the hardware problem by giving it to Rackspace which did a great job – they have a system that allows automatic scaling depending on the demand so you don’t have to worry about big surges overwhelming the system.

There were many things we could have done much better. Our biggest obstacle was not the IN campaign and its vast resources but the appalling infighting on our own side driven by all the normal human motivations described in Thucydides – fear, interest, the pursuit of glory and so on. Without this obstacle we would have done far more on digital/data. Having seen what is offered by London’s best communications companies, vast improvements in performance are clearly possible if you hire the right people. A basic problem for people in politics is that approximately none have the hard skills necessary to distinguish great people from charlatans. It was therefore great good fortune that I was friends with our team before the campaign started.

During the campaign many thousands of people donated to Vote Leave. They paid for VICS. Given we spent a lot of money developing it and there is nothing equivalent available on the market and Vote Leave is no more (barring a very improbable event), we thought that we would make VICS available for anybody to use and improve though strictly on the basis that nobody can claim any intellectual property rights over it. It is being made available in the spirit of the open source movement and use of it should be openly acknowledged. Thanks again to the thousands of people who made millions of sacrifices – because of you we won everywhere except London, Scotland and Northern Ireland against the whole Government machine supported by almost every organisation with power and money.

I will write more about the campaign once the first wave of books is published.

PS. Do not believe the rubbish peddled by Farage and the leave.EU team about social media. E.g. a) They boasted publicly that they paid hundreds of thousands of pounds for over half a million Facebook ‘Likes’ without realising that b) Facebook’s algorithms no longer optimised news feeds for Likes (it is optimised for paid advertising). Leave.EU wasted hundreds of thousands just as many big companies spent millions building armies of Likes that were rendered largely irrelevant by Facebook’s algorithmic changes. This is just one of their blunders. Vote Leave put our money into targeted paid adverts, not buying Likes to spin stories to gullible hacks, MPs, and donors. Media organisations should have someone on the political staff who is a specialist in data or have a route to talk to their organisation’s own data science teams to help spot snake oil merchants.

PPS. If you are young, smart, and interested in politics, think very hard before studying politics / ‘political science’ / PPE at university. You will be far better off if you study maths or physics. It will be easy to move into politics later if you want to and you will have more general skills with much wider application and greater market value. PPE does not give such useful skills – indeed, it actually causes huge problems as it encourages people like Cameron and Ed Balls to ‘fool themselves’ and spread bad ideas with lots of confidence and bluffing. You can always read history books later but you won’t always be able to learn maths. If you have these general skills, then you will be much more effective than the PPE-ers you will compete against. In a few years, this will be more obvious as data science will be much more visible. A new interdisciplinary degree is urgently needed to replace PPE for those who want to go into politics. It should include the basics of modelling and involve practical exposure to people who are brilliant at managing large complex organisations.

PPPS. One of the projects that the Gove team did in the DfE was funding the development of a ‘Maths for Presidents’ course, in the same spirit as the great Berkeley course ‘Physics for Presidents’, based on ideas of Fields Medallist Tim Gowers. The statistics of polling would be a good subject for this course. This course could have a big cultural effect over 20 years if it is supported wisely.