Operational Research Consulting & Data Journalism

As data becomes more and more accessible, together with visualisation tools becoming more available and user friendly, Data Journalism is heating up. I've been following the Guardian's Data Blog enthusiastically, it is full of interesting information relevant to current affairs, explained with much facts and data.

This article talks about the 10 point guide to data journalism. I particularly like point 5:

Data journalism is 80% perspiration, 10% great idea, 10% output

The Prezi under point 5 explains the process of how data is used to support news, the angles to consider when mashing datasets together, the technical challenges of working with data, iterative calculation and QA process, which finally get turned into the beautiful output with the various (mostly free) visualisation tools.

This is practically the same process that an Operational Research consulting project takes - or any application of OR or Science in general:

• Understand what the problem/question is
• Create a hypothesis to be proven or disproved
  
• Define what data is needed for the quest
• Get the data
• Clean it, and manipulate/wrangle with it so it's usable for analysis
• Analyse/calculate to come to some conclusion - hence proving or disproving the hypothesis
  
• Compare it to subject matter experts' view on what the likely answer should be (sanity check)
  
• Refine the analysis until satisfied
• Shape the output message so it can be easily understood by the audience
• Communicate the findings
• All throughout the process, keep communicating to the audience to make sure they are engaged and understand (principle-wise) what you're trying to do, so that they are not unpleasantly surprised when the final answer is presented
• Best yet, to ensure smooth change management if your solution is to be implemented, work closely with the end users from the start of designing the solution, and then implement and test, so that they believe in the solution because they were part of the creation process.

As the Flowing Data blog points out, this is what statisticians do. I will add that this is what Science does in general. I will also say that in practice, the first step, "understanding what the problem/question is", often takes 70-80% of the time. The technical 'doing' to follow, in practice, is relatively easy compared to what our academic institutions thoroughly prepare us for (which is needed).

For those interested in the how of data journalism, read this about the work that went into reporting on the 2011 London Riots. Fascinating social media analytics at work. Not easy. Impressive and very interdisciplinary.

P.S. Most of this post has been sitting as draft since the summer, hence referencing 'old' news. It's still relevant, so why not.

Operational Research considered 1 of 6 dsciplines in Social Sciences

Okay, so OR is grouped with Statistics as one of the six disciplines of social sciences, but still, I'm pleasantly surprised that OR is mentioned!

According to QS World University Rankings, the six disciplines considered as part of social sciences are:

• Finance
• Economics and Econometrics
• Law
• Politics and International Relations
• Sociology
• Statistics and Operational Research

Here you can download the full table (yeah, Google Doc!), and see the top 10 universities at a glance for each of the above subjects. For Stats and OR, here are your top 10:

Rank	Institution	Country
SOURCE: QS Quacquarelli Symonds 2011 - www.topuniversities.com
1	Stanford University	United States
2	Harvard University	United States
3	University of California, Berkeley (UCB)	United States
4	University of Cambridge	United Kingdom
5	Massachusetts Institute of Technology (MIT)	United States
6	University of Oxford	United Kingdom
7	National University of Singapore (NUS)	Singapore
8	University of Toronto	Canada
9	Imperial College London	United Kingdom
10	Princeton University	United States

P.S. If you haven't discovered it already, the Guardian's Data Blog is great!

An Alternative Way to Fly (as long as expectations are managed)

The purpose of this post is to share the discovery of an alternative way of operating an airline (flight schedule and route wise).



No matter how airlines degrade their service standards these days in the West, I think it's fair to say that most of us still believe that most airlines *intend* to:

• Take off on-time
• Land on-time
• Fly us from A to B as the ticket says, without surprise stops
  
• (Oh, and have toilets, of course)
  

On a recent trip to Ethiopia, we have been shown a rather different way of operating an airline. It contradicts with all of the above, but it works. We took 4 internal flights.

Here is how we experienced them first hand:

• 1 left on time as per the ticket, and even got us there early (bonus!), because...
• None of the 4 flights flew the original path it said it would: stopovers were skipped to go direct instead, or the direct flights got stopovers added onto it last minute
  
• None of them arrived late, because...
• Some of them took off earlier than stated
  
• Additionally, the air stewardesses were lovely, and they gave passengers snacks and drinks (*gasp* what novelty!)
• To their credit, they did try to inform passengers of the changes a couple of days ahead of the flight (in our case by email, which we only read after we got back to London).
• They also tell passengers to double check the flight times a couple of days before, to be aware of any late changes.
  

(For your curiosity: the international flights from London to Addis Ababa was quite standard. The only oddity was that they weighed everyone's carry-on luggage at the gate, because it's apparently a popular flight to take lots of stuff with you!)

IMHO, an airline would play this game, because: (we suspect - unconfirmed)

• It wants to minimise costs - mainly fuel in this case.
  
• It has 1-2 planes that fly in circles to cover off a handful of popular destinations.
• As the airline gets more and more requests for seats through the form of purchased tickets, it is faced with an optimisation problem to fly all its customers to their expressed destinations with minimum cost. The best way to do this is probably through re-shuffling the schedule. For instance, if a plane is hopping from A to B to C in sequence, where B is closer to A than C is, and if we discover 2 days before the flight that the plane is filled with 2/3 passengers going to C, and 1/3 going to B, then flying A->C->B is cheaper than A->B->C. What if there are customers wishing to go from B to C? We hear that the airline is known for cancelling flights as well. Luckily, we didn't experience this.
  

This way of operating an airline is possible, because:

• It is a monopoly.
• The number of flights are few, so it's easy to manage change.
• Customers expect it and adjust flying behaviour accordingly (i.e. always check the flight times before the day of flight, and always leave wiggle room before and after the flight).
• For foreigners who are used to the typical western airline service (i.e. expect it to take-off and land on-time and fly the route it says it would), the price justifies it and shuts people up from complaining, and instead people will have a laugh (or write a blog post!) about it.
• It doesn't call itself "Precision Airline" (the Tanzanian airline), and can afford to deviate a little. 8-)
  

P.S. If you are planning to visit Ethiopia, and intend to fly within the country, you may want to consider buying the tickets within the country rather than online. It is significantly cheaper due to price control. This is true as of spring 2011, so double check this before you travel.

YoungOR Conference 2011 - Talks in the Consultancy Stream

It's only 1 week away from the YoungOR conference in Nottingham, UK. I am looking forward to chairing the consultancy stream, so I finally get to meet the speakers I worked hard at recruiting.

It will be the busiest the consultancy stream has seen it! We have 2 keynote talks plus 5 titles lined up for the stream over the first 2 days of the conference. The conference schedule is packed, with 5-6 talks to choose from at any time (except for the plenary slots, of course). If you are young to OR, that is 10 years or less in Operational Research, come and check it out.

The Consultancy talks are as follows in chronological order:

• Keynote: OR Joining Analytics, by Russell Hodge, Capgemini Consulting
• Revenue Management At British Airways, by Peter Wilson, British Airways
• Pharmacy Service Cost Inquiry, by Nicholas Jones, PriceWaterhouseCoopers
• Roundtable Panel Discussion Consultancy on "OR and Enterprise 2.0", "can OR people be leaders or are we destined to be the brains in the back room", and "Who is the boardroom champion for OR". Serving on the panel is a host of talent from various OR consultancies plus an independent, who is also giving a plenary talk at the conference
• Keynote: An OR Professional On ‘The Apprentice'?, by Dave Buxton, dseConsulting
• OR Consultancy For The Emergency, by Guy Bickerton and Graham Holland, OR in Health (ORH)
• Scottish Rugby: Tackling Meaningful Statistics, by Ursula Mulholland, Capgemini Consulting
• Day In The Life Of An OR Consultant, by James Lally

Also, if you're an experienced conference chair, care to share some of your tips on what to do / not to do, etc.?

85% of Statistics Are Made Up On The Spot

I had a good chuckle the other day when I was caught by an example of numerical illiteracy on the part of at least two people: an author and an editor. I had to share.

I was flying with Air Asia from Banda Aceh, Indonesia to Kuala Lumpur, Malaysia. The in flight magazine isn't exactly high production value, as the airline is all about saving. Consider Air Asia to be the Ryan Air of the East. Anyways, here's the tasty treat now:

I can take no issue with the first section on young billionaires as it was actually quite interesting. In the second section, I am entertained by the translation of $122.1k GDP per capita to an average income of about $120,000 per year. Taking the crown though, was the gem at the bottom.

"72% of the 14.5 million population in Mali, Western Africa, earn about $0.003 a day with the average worker's salary of only US$1,500 per year!" Now what is that supposed to mean?

Before you reach for your calculator I can tell you that $0.003/day = $1.10/year.
Also I can tell you that 72% of 14.5 million = 10.44 million.
And that (10.44 million people * $1.10 per person per year ) / $1,500 per worker per year = 7656 workers.
And that 7656/10.44 million = 0.07% employment.

Curiously I can't quite determine what I think they were going for. Anything I try to explain the numbers I see gets destroyed anyway by the strange "72% of 14.5 million". According to Wikipedia, only 43.51 million out of the 81.76 million people in Germany are employed. I suppose I could say that 53% of Germans earn about $0 per day. By adding a dash of real workers I could make that figure $0.003.

Please comment and speculate.

Faking It On Your Wedding Day

Earlier this month we wrote about our love of podcasts and just last week I was listening to Japan: A Friend In Need from the BBC Documentaries Archive. Here I was in the month of love, listening to a podcast on the subject and I found math in an unexpected place.

The documentary is about an agency in Japan that supplies fake people, or actors I suppose. In particular, this agency will supply people to fill out your side of a wedding. In the given example, we met a young man whose parents were deceased and his siblings were astranged, such that he only had two friends to attend his wedding. So as to keep up appearances, unbeknownest to the bride, he hired parents, friends and relatives. All told, 30 people at his wedding were fake, costing him something like £3,000, equal to his recent redundancy compensation.

The agency claims never to have been caught, and they say that they "research their assignments assiduously", but it got me wondering just how long you could operate such a service without getting caught. How many weddings could you do before a repeat guest noticed that they had seen one of your actors at a wedding before?

The first wedding is simple, and guaranteed to go off without a hitch, but what about the second? Suppose every wedding has on average 30 guests from each family. In the second wedding we need all 30 people to not be from the 30 in the previous wedding. Still pretty easy in a country of 127 million. But what about the 30th wedding when there are 900 previous guests out there in the population? Things are still looking pretty good, but the probabilities are starting to pile up in a similar way to the phenomenon that means that in a group of 23 people there's a 50% chance that two will have the same birthday.

So given a constant wedding size of 60, 30 real and 30 fake, what is the probability that this is the wedding that breaks us? This is the same as the probability that one or more of today's guests attended a previous wedding. This is the same as one minus the probability that none of today's guests attended a previous wedding. For wedding n and a population p:
Assuming 127 million people in Japan...

• For wedding 1, it's a sure bet as nobody has attended a previous wedding.
• For wedding 2, we face only a 0.0011% chance of getting caught.
• Even for wedding 100 our risk is only a 0.11% chance. No problem!

But wait, the above probabilities are conditional probabilites. Our chance of getting caught at wedding 100 given that we got to wedding 99 is 0.11%. What is our chance of getting to wedding 99? This is the the probability that we didn't get caught in one or more of the previous weddings, the probability of a perfect record. Mathematically our chance of getting to and past wedding n is:

• For wedding 1, it's a sure bet.
• For wedding 2, it's 99.99%
• For wedding 100, it's 94.58%.
• For wedding 500, it's 24.57%.

Even though by the time we get to wedding 500, ony 15,000 people in Japan have been to weddings with our staff, we would be lucky to have made it that far.

If we started this agency today, on average how long can we expect to go before we get caught? Now I'm not going to bother expressing that mathematically, but hacking at it with Excel numerically, I can tell you that it comes to roughly 374. If we were to start such an agency today under such conditions and such assumptions, we would on average expect to do 374 weddings before getting caught.

So I think the moral of the story is, if you're looking to hire fake people for your wedding, you're doing alright, but if you're looking to run a business doing it, you might want to reconsider. Then again, if we're looking for morals in this story, honesty might come first.

I heart smartphones and podcast favourites

I heart smartphones. It is the symbol of the new world, where the world is at your finger tips, and, in your pocket! There is so much information out there, digesting it is a big quest. I'd love to have the time to sit down and browse the net for a couple hours every day to catch up on all the news and events, but now I can do all this while on the move.



I am an owner of an HTC Hero on Android. It is the only digital device I carry in my hand bag (other than my obligatory work phone). Living in a busy city like London means I spend a fair amount of time in transit. If you are a Google fan like me, then Google Reader and Google Listen would be your good friends. My favourite activity during transit when I'm not walking about, is to catch up on the news and my favourite blogs through the RSS reader. My favourite activity during transit when I am walking about, is to plug into one of the following podcasts, which keeps me informed and entertained. If this is not optimising your time, then I don't know what would. I guess the next step is to jog to work while listening to podcasts: information downloading and calorie offloading all at once!

• LSE lecture and events: London School of Economist half hour to hour long lectures or guest speakers plus Q&A session (frequent publishing of events)


• The Economist: I like the magazine, but there is so much content to digest. The podcasts do a great job summarising the highlights (weekly publishing or more frequent ones available too)


• NPR News: short bursts of news that keeps me informed of the North American highlights (hourly publishing)


• Science of Better: Operations Research podcasts/interviews by INFORMS (monthly publishing)


• More or Less: BBC radio programme making sense or debunking the numbers behind the news


• Freakonomics: spin off by the authors of the ever so popular Freakonomics book/movie/blog/etc.

What are some of your favourite podcasts?

Aside from being my RSS reader and podcast player, my smartphone is also my:
- phone (first and foremost)
- email
- calendar
- access to the internet
- Skype to call anyone around the world
- instant messaging to keep in touch with friends
- handy document storage
- camera / video cam
- GPS and compass
- maps (offline maps too)
- ebook reader
- notebook (takes my hand scribbling too)
- news reader
- scanner
- games when I'm bored waiting in a queue somewhere
- MP3 player
- all the other things that come with a phone (alarm clock, calculator, voice recorder, etc.)
- and thousands of other applications available for download (often for free) that keep my life organised and what not