How to collect and use project metrics

I was collecting metrics for an IT infrastructure development project recently and once again saw how metrics, rewards related to metrics, team learning and constant improvement  produce unintended results. In this occasion the results are fairly positive, so no disasters waiting.

The project is run on three week sprints by two vendors located on three continents. Agile principles are followed as much as they can in such setting - as you can imagine, no daily scrums with about 10 timezones to cover.

There is contractual aspect to the metrics. The project invoicing is made dependent on sprint completeness reaching a certain limit for each sprint and both vendors separately. That makes metrics that much more interesting, as there is quite a lot of interest on that specific metric. 

Such a metric also affects team  dynamics. One would hope for a unified crew pulling together, solving  problems and addressing issues. In practice the incentive and goal is to make sure the invoicing approval is secured, and that is clearly visible. In this regard, completeness as an invoicing criteria fails the project and team effort unless the goal and the metric is shared between the whole crew. 

This obviously is different from most situations you'd see on a sailing vessel.

Completeness is defined as "deliverables produced in a sprint / deliverables planned in a sprint", and the produced refers specifically to what was planned. This makes good sense on paper, and is a fairly good start. 

Team's development is visible in other metrics much better than completeness. A few illustrations about six consecutive sample sprints may be helpful. First, completeness and planned velocity:

Left axis: Agile points. Right axis: Completeness, %.

As you recall, completeness is based on the tasks and deliveries included at the start of sprint. Anything else is excluded. This results in emphasis on  predictability and also sets an upper bound for team's metrics - the "best" one can do is 100%. 

That 100%, the perfect plan,  is subject to a continuous discussion about how hard should one try. 100% completion, everything is done,  looks good on a report, and is a sensible goal for a certain sized system. However, 100% also implies that the envelope was not pushed, that the IT system is flawless at release, and could also be taken as lack of ambition. 

The team in this project has worked together for more than a year, and is improving at a reasonable rate. This is not visible at all in the completeness metric, as can be seen in the grey line above (note the axis on the right). Please note the  growing trend of the orange line, axis on the left. 

Looking at planned and actual velocity, the picture changes somewhat:

Unit: Agile points. Can't be compared between teams or projects, but are consistent for one team in subsequent sprints.

There is a slow start, followed by a rapid increase and general upward trend of how much is actually delivered. For the most part the actual delivery is larger than the planned, i.e. the project team produced more than was directly asked. This happened as the actual work  was indeed agile and tasks and objectives were identified, agreed  on  and executed during the  three week sprints. The team was pushing, and so was the product owner and that combination is often and also in this case a very effective setup.

There is one more view, which is accuracy. It would be  great to know how much work there is, and when do we arrive - the infamous "Gimme a Latest Estimate"  (read this about that folly). Most interesting - this happens:

Accuracy. Positive indicates larger delivery than planned, negative smaller.

As the speed and ambition level rises, accuracy goes from substantial over delivery to small miss. In absolute terms such a graph indicates an accuracy of about 10% for any given sprint, without taking into account that all items are not equally relevant for business value earned metrics.

These and many other metrics can  be collected and analyzed. In this case the biggest  business emphasis is on completeness, and there is  a conscious effort to keep that within parameters. Other aspects - improvement, speed, ambition - are side effects of how the project operates, and  are of interest to analysts and skippers like yours truly.

Does this entry have a nautical aspect? Perhaps if one considers e.g. the constant improvement required to win events from entry level dinghy races to the America's Cup. Combining that with IT project metrics is a stretch, and best left alone.

Sense of urgency, funding, preparing and in battle

Sense of urgency is essential for a startup. It is the driving force to move the startup forward, and it needs to strike an effective balance so that the right issues and opportunities are addressed at the right  moment. The Entrepreneur Middle East writes very well about the subject from a startup point of view.

A pirate ship had the faced the  same  challenge. A prospective entrepreneur needed to form a plan, get it funded, find a ship, recruit a crew, provision, set sail and find and  address opportunities. Sense  of urgency comes naturally during a crisis, but  the other stages are much more difficult.

The early stages, funding, planning and networking to get the  venture under  way have less to do with seamanship. Once there is a ship, and  a skeleton crew, and the  ship  needs to be prepared for  sea, the  challenge of a captain to keep the crew, suppliers, merchants, and  other stakeholders engaged. For a seaman, there is always the  tavern  by the  pier, and similar each participant  can too easily get distracted with something else. The result of losing that sense  of urgency is leaving the docks unprepared, or late, or both. The consequences can be fatal for a voyage that starts with poor preparation.

Captains had considerable authority to keep their crew working, but   much less so for outside help. If provisions didn't arrive, the captain needed to address it and press the importance of a timely delivery. That is an all too familiar issue for a modern entrepreneur: Goods and services have been ordered, but  the supplier is late, as there are other pressing concerns. The captain (or the  entrepreneur) has not  much else to but  to keep asking and  ultimately complaining.

Once provisions - food,  water, cordage, sails,  ammunition, spares etc - are on board, the  captain is  in a  much better position. The crew has a much bigger stake and interest in making  the  voyage possible, especially if the  compensation is "no purchase no pay". However, as long as the ship is attached to land, and not underway, each  crew member has the opportunity to desert,  to leave the   venture, and that they can do if the preparations do not proceed. The  captain can command and ask and direct and guide, and ultimately needs to articulate the priorities and  essential goals that need to be met before the  ship can sail.
For a startup that is called "MVP", the Minimum Viable Product. Once that is ready, the  startup can sell and is in operative mode.
For a ship it  means seaworthiness. Modern definition is probably different that the language used by Mr Morgan and other privateers, but the essence is the same.  The ship must be ready to face the sea, to sail to its destination, to fight the battles it can and  should expect and return safely to harbor after a lucrative and profitable cruise.

Making the preparations is perhaps the  hardest time from the sense of urgency point of view. The ship and the crew and preparing  for events that will take place, but  they are in a safe harbor. The captain and the officers of the ship can command  and inspect, and  remind  the crew without end - but the most effective source of urgency is old sea salts.
Crew members, able seamen  that have tasted storms, hunger, thirst, and battle personally are the most effective messengers about the  urgency to address hull integrity, masts, rigging, provisions, guns, cutlasses, steering and any of the myriad little details that need to be in sufficiently good shape for a cruise.
The  wise captain and the  lucky captain has a crew with sufficient  experience on board. So in reality for a ship the cultivation of sense of urgency starts at crew selection. Experience, motivation, capability, desire to set sail for a lucrative cruise are key elements, and if a captain fails to find a suitable crew, there is not much that can be done  after the fact.

A startup, a company, has a similar but not the same  constraint. A captain and a crew are fully committed after they leave the harbor; a startup can, at least in theory, change some members. Substantial changes - say key founders leaving - rarely work, so in that sense the startup has the same challenge as privateering ship. The crew must bee just about right  for the voyage, for the venture, and if not, there is  only so much a captain can do about it.

Once underway the situation changes. There is a ship to sail, watches to keep, lookout to be kept and  eventually battles to be fought. While most of that is fairly tedious, it is at the  same time familiar and the purpose of all activities - stay afloat, make progress, find riches - is  more or less clear to everyone onboard. Captain can delegate much of the sence of urgency to the natural order of seamanship.
Similarly, once the  startup is under way and has something resembling the MVP, a great external force called "The  Customers" comes in and provides natural pressure on the crew. Sense of urgency comes without asking, and the captain needs to worry about speed, priorities and  direction.

What if customers don't arrive? Or, what if the ship is becalmed? Similar situations, and often with dire results. Roaring forties is the similar to a ship as is a "Inside the tornado" for a startup and that is more manageable that horse latitudes and slow death by thirst - or lack of customers.

So what did we learn? Pick your crew, and make sure you find the wind - or customers. If you don't - your ship will be in trouble, and you could die, and similarly without customers your startup will fail.

The Latest Estimate

My favorite phrase - something breaks, there is a bug, system is not working, and the one thing most managers ask in the IT world is "when". Not what, can you fix it, do you need help, spare parts but "what is your latest estimate to fix this".

The same event out on the sea is equipment failure - not unexpected, as all equipment will fail, which is why we carry spare parts for everything, and life rafts. Undesired, and the timing is always a surprise, but not unexpected.

Here's an example - this boat's battery and electric system is shot. Small thing, and the skipper (your's truly, sad to say) is furious for breaking it. And would like to go back sailing, and would like to know when - but does not know that until the spare part is there, properly installed and the next problem (blew also an alternator) is identified and fixed.

The latest estimate is also a key question for IT projects and shipping schedules. In shipping, say a container ship route between Rotterdam and Baltimore is pretty well defined, has been traversed thousands of times and in general the ships have been tested and the crews have done the same trip in different conditions in the past. All this means cumulative, collective experience which makes it possible for shipping companies to post schedules and have a pretty accurate view on the expected arrival of a ship in any port.

IT projects, in comparison, rarely repeat the same task and deliverable. Be it an upgrade, new system deployment, new system development, replacement system setup, there is always something new in the project.

New implies "uncharted waters" in nautical speak, which brings us to the point. The phrase "Latest Estimate" turns mushy and meaningless, to wishful thinking, when one applies that to a new voyage. If the ship and it's capabilities are unknown, if the  destination is not charted, if the weather patterns are not known, if the crew does not know the ship or the seas, then the expected arrival date is pretty much just speculation.

Latest estimate (LE) for a workaround or fix is understandable as business does depend on technology and outages cause direct and indirect costs. In many cases the LE question in fairly sensible. Equipment failures for example and in general are straightforward to address.


The difficulty arises if and when the change, new feature or repair is more complex one. The main question is still too often the same, and most IT organizations try to answer that, giving educated guesses. Reliable information may be available, but not in the form of time but content.

On this regard maritime best practices beat IT easily. Fixing things at sea is often unpredictable, and the focus is on the task at hand, ways to address that and their pros and cons. Usually the entire crew is aware of the implications, and there are processes for all possible outcomes, including the dreaded "abandon ship".

A captain of a ship understands most systems, structures and processes on the vessel. The same does not seem to apply to business managers. This may have an impact  - if there is no or limited understanding about how the business (ship) works and how the technology relates and interacts with its progress, how can one ask but "when".

It would be much more productive and would make a much better working environment to deal with the likes of sea captains who keep their cool heads even in the face of a disaster than your average board member. If you happen to know about such a place in need of professional geeks, please let me know.

Until then - LE on that new network segment is "once we get the parts and they work", and for making features A, B and C to work is "once we figure it out and the pilot customer says OK".

No purchase, no pay - profit sharing in the past and present

How much is a capable, talented and productive IT professional worth? Should the compensation be base salary, shares, profit sharing or something else? How do you count and distribute bonuses? What, if anything, is the connection and effect of compensation and productivity? How do you measure productivity in the first place?

Traditional Prize Money

 Somewhat simplistically one can start with No Purchase, no pay from Henry Morgan's times. A raiding trip - a venture - is funded by merchants, richer patrons, bankers etc. The crew, including the captain, is compensated with a share of the profit, hence "no purchase, no pay".

This model of profit sharing was very effective and used not only in Caribbean waters but elsewhere, including Sir Walter Raleigh and the Golden Hinde.

Different privateers and pirates and national navies used the system for prize money sharing between the officers and crew of a ship. Royal Navy used that system until 19th century - see 1808 change in crew shares.

Navies, privateers and often also outright pirates have other, sometimes conflicting goals than just filling the pockets of the crew, so the practice has been abandoned in Navies, privateers are no longer commissioned, and I do not know how present day pirate share their booty in Strait of Malacca and elsewhere.

Commercial fishing and profit share

Whalers followed a similar profit sharing scheme, as anyone who has read Moby Dick remembers. A more concrete example about whaling ship Milton is explained on New Bedford Whaling Museum pages.
Whaling, like privateers and prize money, is pretty much gone (with small exceptions). Fishing vessels used - and still do - a similar profit sharing system.

The long history and experience collected about profit sharing in fishing suggests that it is actually beneficial to the industry and to the participants. Newsweek  reported that also Thomas Jefferson found evidence for profit sharing - quote:

"Research commissioned by Thomas Jefferson found that, when fishermen bargained for their pay in advance and shared in the profits, the operations were highly efficient."

 IT is different from privateering, fishing or any other seagoing activity

IT projects and companies are different than fishing, let alone privateering.
Even the longest whaling trip is seldom 5 years, and the single goal of filling the vessel with as much catch as possible is very clear, fully shared and easy to understand.  Crew members don't leave the ship (except overboard!), new crew members don't join and the accounts are straightforward to settle. The ship returns to port, catch is sold, expenses are calculated and shares can be done right there. There is no recurring revenue, no complexities about vesting, questions about  residual value etc.

An IT project, service, company, has time span that is much more different. There is initial investment, which can be followed by maintenance, additional innovation, changes of direction, services launched and services discontinued. Teams start small, grow, evolve, and eventually shrink or move to other projects.
How would one apply the useful principle of profit sharing?

An obvious answer is to issue company shares. That is slightly different that the fishermans share, as the Shares in the company imply ownership in "the vessel", not just fair share of the income.
Profit shares like practiced at Southwestern Airlines looks like it works in a focused service industry (see Huffington Post about it). Interestingly, owner's share is pretty similar for the airline as it is for a fishing vessel.

A capital intensive service provider, e.g. hosting services provider, could follow a similar model. A knowledge intensive company, whose valuation is based on intellectual property, customer base (followers, brand) would be different. Tangible capital assets are few, and the bulk of the value generated depends on the crew. Yet, the founders, the  capitalists would not want to forfeit the reward for the risks they have taken early on.

In such circumstance a share that would be smaller for those who joined late could work. Everyone can have a share, but not all shares are equal even if the jobs are similar.

Profit share example

Vietnam Van Thuy Tu profit shares are explained in a paper by Kenneth Rudle and  Luong Thanh Son. Please note the fairly detailed rules and specific fishing gear and catch explained in table 4.1 in the paper.

For a hypothethical IT operation one could argue the following:
- Revenue 105MEUR
- Profit 5MEUR
- Labor costs 50MEUR

If such a company runs into slowdown, and revenue is reduced to 100, all profit is gone. If, however, a part of labor costs was based on profit sharing, in bad times the costs would be flexible, and the company would have an easier time to adjust to slowdowns. The other  side is that in good times the labor costs would be higher, but as Thomas Jefferson observed, that tends to increase the efficiency - productivity - of the operation over salaried alternative.

References and pointers

 Joseph Blasi discusses profit sharing in his blog entry for the Huffington Post

The Argument for Profit Sharing 
Blasi's profile is worth checking before you buy his books.

Reference for Business opens up aspects of profit sharing in their article.

-- Note about Golden Hind:

Venture capitalist Queen Elisabeth I netted more than the then-current national deficit from the spoils of that circumnavigation. Unusually, the crew had agreed to sail for wages, and so the £8000 they received was a token of good will by their Captain.
State power was misused as Queen Elisabeth confiscated most of the loot right at Plymouth Harbor. Lesson learnt - if you run a business, protect your assets from state intervention and make sure you pay your  crew first.

Good Crew Members

On a small ship a good crew member is easy to recognize.



They are those persons who do their jobs, and a bit more, looking after the ship, fixing what needs to be fixed right away. Small things such as putting away the used coffee mug you forgot on the cabin table. Going out in the night for a sail change without extra griping (and mind you, there is the "right" amount of griping and complaints to keep up morale).



They are those persons who show up for their watch just before without asking, perhaps with something hot for the ones coming off watch, and who leave for their off-watch only after the next one is solidly on deck.

They are those who are easy to get along with, and know when to keep up a happy banter with the others, and when to leave someone alone.

There is not much to say about physical, technical or other abilities as such - those can be learned, and will be learned if one takes a passage. But the basics of looking after the ship, after the crewmates, after everyone's well-being, that attitude is the essential foundation of a Good Crew Member.

Occasionally you'd wish we'd have more Good Crew Members in corporate life. People who would notice things that need fixing, and quietly just fix them. People that wouldn't complain when there is a little extra to be done. People that get along with everyone else.

That would be great. Sometimes you meet those people, and end up realizing that what works on a ship with a known destination and a voyage of limited duration does not work in a modern firm, excluding perhaps small companies.

So you have a good buddy is a Good Crew Member (GCM) in your company. She finds things to be improved or repaired and goes ahead and does it. There may be some that complain that it was wrongly done, the execution should have been different. Still, there are others that notice - this person gets things done. And so it starts, the GCM is asked to do more and more, and as long as she does that, the more she will be asked.

All the better, more responsibility, more visibility, proceeding up the ladder, thinks the Crew Member, and takes on more. And they pile it on - why not, this guy is getting things done.

Eventually and by default even the best of the best can handle everything, as unlike a ship - a closed community - a corporation is virtually boundless. Good Crew Members get to work more and more, and rarely are they given the necessary resources - surely they can pull that additional project off, too?

And so the structure which is looking for growth often ends up stressing the most valuable assets, Good Crew Members to a breaking point. What happens next is crucial.

In some cases the GCM is left alone, struggles for some time, perhaps suffers a person burn-out.

In some cases the GCM if promoted, moved to another position or otherwise disturbed so that the effective state is disturbed.

In some, very lucky cases, the GCM is driven to a point, but not beyond that, the GCM is not broken, but offered help, additional resources, whatever to make more happen.

And those lucky cases, those occasionally end out being success stories.
Great stories for all of the participants to tell tales about!

Working together does not work over e-mail

Over the past few months I've seen numerous examples of how things start happening once people get together to talking distance, face to face in the datacenter (engine room, this is a "nautically biased" blog).
The latest was the funniest. Someone wanted to move functionality (an instance) from amazon to a private cloud and someone else was helping. The e-mail dialogue was fascinating - first party in India, second in Europe:
India: "i'd like to move an aws instance to cloud xyz"
Europe: "Do x y and z"
I: "did not work,"
E: "check /dev/doohickey"
I: "I only see /dev/whatchmacallit"
[ repeat with variations]
Manager of I: "escalate"
-- at this point a call was agreed on and I and E were talking to each other.
Joint discovery " the aws instance is an old model and x y and z do not apply".
I'm picking on this anecdote (the tall, excitable bearded one - please note how carefully I'm removing all traces if identity) as it us so usual and actually pretty funny.
Two things:
- Face to face is best, phone and screen sharing is good, email chat is marginable and ticketed request response systems are effectively non-functional for troubleshooting.
- Remove managers and other non-essential parties from the discussion! Non professional advice is bad, and is worse if it is coming from a manager - bosses need to be listened to, obeyed abd respected if they are to function, and a boss interfering with troubleshooting is very counterproductive.
Is there a maritime equivalent! Yes, there is to the boss part (face to face is not much of an issue on  a ship).  Captains or officers who don't have the competence to let their crew to do the job seldom run happy, effective and safe ships.

State of the art multivendor outsourcing and innovation

This time I don't have a sailing anecdote or context to match the subject, and I believe there is a reason for that. Sailors would never be as silly as us IT folks, that is!

I'm writing about outsourcing and especially innovative approaches to that, which is splitting the service components into solution cards, obtaining the best price points per category by aggressive global sourcing policy, and resulting in nimble, multivendor outsources service and solution teams.

In other words, crews that no sane sailor could ever even imagine. And if you could imagine that - which some people do - you could not implement. Why not, you may say? Well, because unlike an (outsourced) IT organization and service components for that, a crew is literally on the same boat.
 If one has a problem, everyone has a problem. A good crew member is paying attention to problems on your boat - and her boat - and not on something else.
So this time all I can say is that good seamanship is out and all I can talk about is Information Technology. Oh how I wish I was sailing....

In Information Technology tasks are atomic. You ask this party to open your firewall, you ask the other party to assign the IPs to your servers, and you ask for a third party to install your application. And if you have other topics - EMC filer shares, LUNs to set up and iscsi shares to configure - you need to worry about those, too.

This surely is not a problem, if you are able to plan ahead without flaw.

If you happen to try and improve and innovate - what then?

Innovation, as everyone since Thomas Alva Edison knows, is 1% inspiration and 99% perspiration. In IT language -. your original configuration is WRONG.

You'll need to change the Firewall. And you'll want to redo you management IPs. Your application and your SAN and Storage will be, err, WRONG.

Now, every development step is a ticketing process, and if your client did the fashionable thing, these tickets will go to separate groups, and you will definitely not be able to just access the DC and do the job.
No, and that also makes sense, since DC is filled with systems that are under SLA, and ultimately the DC production network is quaranteed by SLA to stay up, and they can't let you touch that.

So the customer gets what the customer ordered. If the customer wanted rapid development and innovation - if the customer needed that! - Well, that is not happening.

This is something I've seen now repeatadly. I've found one cure, and that is elastic cloud computing, which eliminates several manual steps in the process above. If you have experienced good ways to evolve systems rapidly in production data centers - please let me know.

Oh, and if you wanted to change your data center and computing infrastructure to something nimbler - I'd like to hear about your approach there, too.