How to collect and use project metrics

I was collecting metrics for an IT infrastructure development project recently and once again saw how metrics, rewards related to metrics, team learning and constant improvement  produce unintended results. In this occasion the results are fairly positive, so no disasters waiting.

The project is run on three week sprints by two vendors located on three continents. Agile principles are followed as much as they can in such setting - as you can imagine, no daily scrums with about 10 timezones to cover.

There is contractual aspect to the metrics. The project invoicing is made dependent on sprint completeness reaching a certain limit for each sprint and both vendors separately. That makes metrics that much more interesting, as there is quite a lot of interest on that specific metric. 

Such a metric also affects team  dynamics. One would hope for a unified crew pulling together, solving  problems and addressing issues. In practice the incentive and goal is to make sure the invoicing approval is secured, and that is clearly visible. In this regard, completeness as an invoicing criteria fails the project and team effort unless the goal and the metric is shared between the whole crew. 

This obviously is different from most situations you'd see on a sailing vessel.

Completeness is defined as "deliverables produced in a sprint / deliverables planned in a sprint", and the produced refers specifically to what was planned. This makes good sense on paper, and is a fairly good start. 

Team's development is visible in other metrics much better than completeness. A few illustrations about six consecutive sample sprints may be helpful. First, completeness and planned velocity:

Left axis: Agile points. Right axis: Completeness, %.

As you recall, completeness is based on the tasks and deliveries included at the start of sprint. Anything else is excluded. This results in emphasis on  predictability and also sets an upper bound for team's metrics - the "best" one can do is 100%. 

That 100%, the perfect plan,  is subject to a continuous discussion about how hard should one try. 100% completion, everything is done,  looks good on a report, and is a sensible goal for a certain sized system. However, 100% also implies that the envelope was not pushed, that the IT system is flawless at release, and could also be taken as lack of ambition. 

The team in this project has worked together for more than a year, and is improving at a reasonable rate. This is not visible at all in the completeness metric, as can be seen in the grey line above (note the axis on the right). Please note the  growing trend of the orange line, axis on the left. 

Looking at planned and actual velocity, the picture changes somewhat:

Unit: Agile points. Can't be compared between teams or projects, but are consistent for one team in subsequent sprints.

There is a slow start, followed by a rapid increase and general upward trend of how much is actually delivered. For the most part the actual delivery is larger than the planned, i.e. the project team produced more than was directly asked. This happened as the actual work  was indeed agile and tasks and objectives were identified, agreed  on  and executed during the  three week sprints. The team was pushing, and so was the product owner and that combination is often and also in this case a very effective setup.

There is one more view, which is accuracy. It would be  great to know how much work there is, and when do we arrive - the infamous "Gimme a Latest Estimate"  (read this about that folly). Most interesting - this happens:

Accuracy. Positive indicates larger delivery than planned, negative smaller.

As the speed and ambition level rises, accuracy goes from substantial over delivery to small miss. In absolute terms such a graph indicates an accuracy of about 10% for any given sprint, without taking into account that all items are not equally relevant for business value earned metrics.

These and many other metrics can  be collected and analyzed. In this case the biggest  business emphasis is on completeness, and there is  a conscious effort to keep that within parameters. Other aspects - improvement, speed, ambition - are side effects of how the project operates, and  are of interest to analysts and skippers like yours truly.

Does this entry have a nautical aspect? Perhaps if one considers e.g. the constant improvement required to win events from entry level dinghy races to the America's Cup. Combining that with IT project metrics is a stretch, and best left alone.

Sense of urgency, funding, preparing and in battle

Sense of urgency is essential for a startup. It is the driving force to move the startup forward, and it needs to strike an effective balance so that the right issues and opportunities are addressed at the right  moment. The Entrepreneur Middle East writes very well about the subject from a startup point of view.

A pirate ship had the faced the  same  challenge. A prospective entrepreneur needed to form a plan, get it funded, find a ship, recruit a crew, provision, set sail and find and  address opportunities. Sense  of urgency comes naturally during a crisis, but  the other stages are much more difficult.

The early stages, funding, planning and networking to get the  venture under  way have less to do with seamanship. Once there is a ship, and  a skeleton crew, and the  ship  needs to be prepared for  sea, the  challenge of a captain to keep the crew, suppliers, merchants, and  other stakeholders engaged. For a seaman, there is always the  tavern  by the  pier, and similar each participant  can too easily get distracted with something else. The result of losing that sense  of urgency is leaving the docks unprepared, or late, or both. The consequences can be fatal for a voyage that starts with poor preparation.

Captains had considerable authority to keep their crew working, but   much less so for outside help. If provisions didn't arrive, the captain needed to address it and press the importance of a timely delivery. That is an all too familiar issue for a modern entrepreneur: Goods and services have been ordered, but  the supplier is late, as there are other pressing concerns. The captain (or the  entrepreneur) has not  much else to but  to keep asking and  ultimately complaining.

Once provisions - food,  water, cordage, sails,  ammunition, spares etc - are on board, the  captain is  in a  much better position. The crew has a much bigger stake and interest in making  the  voyage possible, especially if the  compensation is "no purchase no pay". However, as long as the ship is attached to land, and not underway, each  crew member has the opportunity to desert,  to leave the   venture, and that they can do if the preparations do not proceed. The  captain can command and ask and direct and guide, and ultimately needs to articulate the priorities and  essential goals that need to be met before the  ship can sail.
For a startup that is called "MVP", the Minimum Viable Product. Once that is ready, the  startup can sell and is in operative mode.
For a ship it  means seaworthiness. Modern definition is probably different that the language used by Mr Morgan and other privateers, but the essence is the same.  The ship must be ready to face the sea, to sail to its destination, to fight the battles it can and  should expect and return safely to harbor after a lucrative and profitable cruise.

Making the preparations is perhaps the  hardest time from the sense of urgency point of view. The ship and the crew and preparing  for events that will take place, but  they are in a safe harbor. The captain and the officers of the ship can command  and inspect, and  remind  the crew without end - but the most effective source of urgency is old sea salts.
Crew members, able seamen  that have tasted storms, hunger, thirst, and battle personally are the most effective messengers about the  urgency to address hull integrity, masts, rigging, provisions, guns, cutlasses, steering and any of the myriad little details that need to be in sufficiently good shape for a cruise.
The  wise captain and the  lucky captain has a crew with sufficient  experience on board. So in reality for a ship the cultivation of sense of urgency starts at crew selection. Experience, motivation, capability, desire to set sail for a lucrative cruise are key elements, and if a captain fails to find a suitable crew, there is not much that can be done  after the fact.

A startup, a company, has a similar but not the same  constraint. A captain and a crew are fully committed after they leave the harbor; a startup can, at least in theory, change some members. Substantial changes - say key founders leaving - rarely work, so in that sense the startup has the same challenge as privateering ship. The crew must bee just about right  for the voyage, for the venture, and if not, there is  only so much a captain can do about it.

Once underway the situation changes. There is a ship to sail, watches to keep, lookout to be kept and  eventually battles to be fought. While most of that is fairly tedious, it is at the  same time familiar and the purpose of all activities - stay afloat, make progress, find riches - is  more or less clear to everyone onboard. Captain can delegate much of the sence of urgency to the natural order of seamanship.
Similarly, once the  startup is under way and has something resembling the MVP, a great external force called "The  Customers" comes in and provides natural pressure on the crew. Sense of urgency comes without asking, and the captain needs to worry about speed, priorities and  direction.

What if customers don't arrive? Or, what if the ship is becalmed? Similar situations, and often with dire results. Roaring forties is the similar to a ship as is a "Inside the tornado" for a startup and that is more manageable that horse latitudes and slow death by thirst - or lack of customers.

So what did we learn? Pick your crew, and make sure you find the wind - or customers. If you don't - your ship will be in trouble, and you could die, and similarly without customers your startup will fail.