Division of roles sinks the Armada - or why ITIL is bad for your IT health

King Filip's plan was great on paper, but execution was somewhat less impressive. The pride of Spain, the Great Armada left for England with precise plans and a massive invasion force to crush the heretics there.

Noblemen to lead sailors

The Spanish of the time were following the traditional naval model  where an aristocrat led soldiers and the sailing master was more or less  a taxi driver, with the clout to match. Accordingly, the sea was not accounted for in decision making as it should have been. This model had worked fairly well since the Roman times (see below) so why change?

 Follow the instructions to the letter, right?

 As plans are the first casualty of war, that should be no surprise to anyone. The weather in the Channel is not easy to predict and definitely does not conform to the  wishes and commands of anyone, including Kings and nobility. The storms and tides are difficult to handle, and can drive the best of ships to shore, which makes sticking to rigid plans a massive liability.

 Different experience, different way of sailing

 The English had by late 1588 moved into a more integrated process. Single captain, well-trained crews (comparative and at the time), clear focus on fighting naval battles instead of being landing craft for armies allowed the English fleet to adapt to the weather and attack in ways not possible for the Spanish. The same sea, the same elements but different crews and ships and the outcome is history.

Nowadays it is evident that any watery adventure needs to take weather and other shipping into account, and in naval war, the enemy. To make that happen, vessels and navies have spent considerable efforts to create clear lines of command, control and communication, leave a good degree of flexibility - "seamanship" - and rely on trained, skilled crews. Make a note - trained, skilled crews and remember that it includes drills and training together. The "Annapolis Book of Seamanship" is relevant, useful, but not necessary, and definitely not a guarantee of  survival, nor success. At sea, your "information library" is worth very little, it is your vessel, your crew and their ability that counts.

What is the lesson from 1588 for IT today?

In IT we have ITIL. It extensive, it is comprehensive, it synonym with bureacracy, division of responsibilities and relies on checks, controls, hand-offs and documentation.

We also have agile development, devops, peer (or pairwise) programming. We have realized that the  waterfall software development model was a dud (I hope we have, though quite a few commercial discussions seems to assume something different) and there are signs that some parts of the industry rely on professionals more than on a low daily rate.

So looking at the Armada and the different ways the fleets operated -- you need to have skilled crews with clear and capable decision making structure in place to win.
ITIL encourages documentations and committees, agile and devops steers towards a model "seagoing captaing and a trained crew".
ITIL encourages formal processes, which in  theory work but in practice are poorly suited to any fast changing situation.
An agile crew in a devops situation may not give an sla, but they are much more likelyto respond to a changing situation than a well documented ITIL corporation.

The  point here is - ITIL and complex layers look a lot like the Spanish Armada, and it would be foolish for us IT folks not to learn from it. After all, people have dealt  with changes and surprises and changed the operational models for centuries - why not take the best from history?


So happy sailing, good luck with your next project, and if you can - try to avoid a King Filip and Duke of Media Sidonia in leadership and precise instructions to a changing situation.
Cause if you do, you may end up in the receiving end of Sir Francis Drake and an inconveniently timed Atlantic gale!

Rigging and cabling

Rigging of a modern sailing ship is a pretty intricate matter - just look at the rig of Cutty Sark (see miniature model). One would hope not to have to maintain that, but that is exactly what they did. Not only is the rig durable (can take hurricane force winds), not only does it do the job to propel the ship, but it also can be maintained. Imagine that - a rope breaks, and the crew can splice a new one on its place, another one is worn out and they take it off and replace with a new one.

They can do that to running rigging, and also to standing rigging. This is possible to fairly extreme situations - sometimes  it is possibly to rebuild the rig if one or more masts are down after a really bad storm or battle.  Wonderful craftmanship, and really brilliant engineering based on centuries of experience: Whatever your build and send out to sea, the sailor must be able to maintain!


Cabling, on the other hand, is an art form in data centers. If you have something really large like Google does, you can have bespoke servers and cabling can be taken into account in the design. You can see yourself if you  "Take a walk through a Google data center" . There is also a Youtube introductory video - just freeze that when they show the details of server cabling.

In the smaller ones where the rest of us operate you have generally available hardware, and here's where it may get interesting. Here is a photo of an unnamed brand factory integrated rack cabling:

Net cabling, tied well and color coded.

This is really pretty and everyone who sees that is delighted, especially since some results of manual work are, erm, somewhat less tidy. Ooh and aahh, this is great, let's get more of that!
There's been a little note every now and then asking "so how do we replace a broken server if need be". No worries, and so far so good. Until....
It turned out that certain cables need to be replaced. Ouch. Now, to do that, the whole bundle containing data and power cables needs to be opened so that you can sort out what is what. After that the Data Center technician is supposed to put it back together. Unsurprisingly, there are few volunteers for such a task.

The thing that should have happened was to take thing to pieces while it was still 30-odd servers and to check the maintenance capability - that would have been seamanship. Now we are out there, it is blowing near gale, and we just realized the backstays are about to give (so no problem so far) - and we don't know how to put new ones in (baaaaad).

I am sure this anecdote will finish well. In the meanwhile - remember that there will be maintenance and be prepared to take apart everything you put in your server room. That's what sailors do!

Devops and watch system - where are the limits?

Everyone knows what the watch system is, right? You have the port watch and the starboard watch, the dog watch and all hands on deck and what not.
Every IT person interested in the "Devops" line of  work should be aware of how the watches work.

First of all - if you are raiding, fishing is good, trying to survive a storm, the crew is in "all hands on deck" mode, and so is the captain. The focus is on the task at hand, and it relentlessly there until the job is done, be that cargo hold full of fish, a conquered (enemy) vessel, or rounding The Horn and finding calmer waters.

What is less commonly known are the "idlers", which on larger ships are the people with special skills. Ships carpenter, sail maker, machinist and others can at best build a new ship, though not while under way. In general the specialists don't participate in the watch system, but they are there every day to inspect, fix, improve the condition of the vessel so that the "operative" side - the watches - have all systems shipshape, if not necessarily Bristol fashion.

So what we in IT consider a great step forward - Devops - is actually something where we could learn from the sailors. You can develop your ship, your crew, your equipment while under way. You can build more, you can set masts, fix engines, sow new sails, repair old ones, fix leaks, install pumps, heaters, thrusters, and whatever - but the ship needs to have the skills (idlers), the tools (a carpenter will not fix a leak without nails, oakum, hammer etc) and the time to do so.

What you can not do on your ship is to redo the hull, not while underway.

Now the devops idea is pretty similar. There are the people who run your system - the watch on watch crew. You have the specialist who can change, improve, modify and repair even substantial damage to the system.

How about a brand new system, something different? Can you do that by launching something and then tweaking and improving?
I don't know, but looking at the historical evidence in shipping, in maritime best traditions, I'd say it sounds a lot like trying to build a new vessel while sailing.  Your hull, your masts, your engine, your collection of spare parts (maintenance agreements in IT;-) are your architectural limitations and if you want to change those, you better do as the Captain would:

Sail home, and visit a naval (IT) architect and design and build a new ship (system) from the ground up.



http://www.cio.com/article/738544/Find_Out_What_Agility_Really_Means

Crew or project staff - what's the difference and why would you care?

Have you ever wondered about the motivation and work ethic of people in your IT project, or the next one? I have, it is fun and with a bit of perspective, also educational. You perhaps realize that for many professionals your project is one among many, that while they want to do a good and professional job, they don't see your project as the Next Big Thing.

Have you ever wondered why the focus may or may not be there for everyone onboard? You may be lucky in having a few guys who are deeply interested in the subject matter (I have, and such people are great in projects).

Take a little perspective and consider a fishing vessel. When they leave port, the captain can be pretty sure that the crew is committed but to what?
At the early phase they all share the same wish and goal - find fish, catch it, make some money, and get safely back home.

That works, but what happens if the Captain does not find the catch? Commitment to get home gets stronger and stronger, and the motivation and incentives deviate. For a fisherman - or project member - a trip in unproductive waters means loss of income for a few weeks. For the skipper and boat owner it means a dent in reputation and lost investment - putting out a ship is not cheap.

Similarly, your project staff is surely excited and fully committed at the start.
The difficult bit comes if and when a project runs into difficulties, and then things start to resemble failing fishing trips.
Bad fishing trips do not automatically mean mutiny. Seasoned crews that have been together for a long time can tolerate a lot of failure if the trust between the Captain and the crew is in place. Reality TV offers us interesting windows into Bering Sea crab boats, and how different things can be.
Junior captains that don't find the catch are in hot water immediately, and not necessarily because they are young, but definitely because they have not worked long enough with their crews.
Senior captains with a new crew suffer also if things go wrong, but much less. I'd guess two things are in play. The trust is deeper in place as they have worked together for longer, and - perhaps more importantly - they have been through rough times before.


So how does this help us IT project managers?
Projects are not fishing trips and we work with highly skilled IT professionals, right? Well, perhaps, but they all look like humans and the behaviors match occasionally.
A death march project equals a fishing trip or pirate vessel that is unable to find their catch. The crew / staff will stay on grimly - until you reach a safe haven or reasonable milestone so that they can leave with a good reputation.

Highly successful projects are easy, and death marches hopefully very rare. They average project lies somewhere in between, and it will be good for the project manager to pause and reflect what is in the project for staff members, and what do they value. That can make the difference if things get tough - or at least it seems to help at sea.

How do you know what they like? Well -- experience. If possibly find a crew that tolerates you and you can work with, and stick with them.

That will make life and work much better, and eventually you will sail home with the hold full of dubloons. Or Alaskan King Crab.


-- Disclaimer: Some claims have been made based on the show "Deadliest Catch". Sig Hansen, Eliot Neese, Wild Bill Wichrowski, Keith Colburn, and their crews. Thanks to Discovery Channel and my best wishes to all these guys down south!

Small trip musings

This I heard other day. All details are changed and the story is not about any specific IT related  undertaking.

So Jane wanted to sail across the Channel and bring some wine to Penzance. That should make a tidy profit and benefit everyone.
Unfortunately, Janedid not know that much about sailing, but no worries, her friend Adam said he'd find a small ship and get them across.

Now,D-day is coming but the boatis nowhere to be seen. Turns out that Adam forgot to pay for it, and that the owner has gone fishing. Oh rat's and what do we do now?

Sounds silly, doesn't it? Well, how about this: you are setting up a charity event, rely heavily on your online presence,and your friend Adam (all names changed) forgets to pay the hosting company, fails to act on notices and your web plus all contents disappear just days before the event. Bummer.

Now, that happens, right, and what's the big deal. I don't know about you, but the casual approach to basic best practices - make sure your business critical service is up and make sure your have essential data safe - are very casually neglected, and sometimes it can be highly amusing to bystaders!

Oh, about Jane...? She is fictional so let's say the skipper had a good catch of cod and that they sailed the next day with the tide. Adam, otoh, forgot to take care of water, but fortunately that was up to the skipper to do.

Goals, missons and crusades - what is that in IT parlance?

The previous analogy of a small time fishing vessel or open boat pirate crew is about the same as a "Change request" in IT project scale. A "Definition of Done" is required, there needs to be a reason for doing the job, and the people doing it need adequate tools and resources, but beyond that the work is done in professional and seamanship like way without further guidance or structure. A change and an small sea trip are pretty analogous, and from the project manager point of view that is not of exceptional interest.

Unless, if the change is being done as an operation to help your larger project to proceed. In nautical setting this would mean that a small boat is sent out to survey an unclear passage, and the ship's launch is expected to come back - just like the change your requested in the Data Center is supposed to finish.

If the boat does not return, the Captain of the ship either sends another boat (perhaps the first team is in trouble and need help), or abandons them. The latter is not that popular as it tends to send a fairly negative message about being expendable to the rest of the crew.
If the change does not finish, the IT project manager can ask again, can escalate, wait for longer, or try to identify a workaround. Most of the time the workaround is fine, unless the change and the issue is viewed as dear as a crewmate. This can happen when an essential piece of technology needs to be set up and it is not, and the cause can be as trivial as wrongly ordered optical transciever.

From IT project point of view it is more interestingto look at longer voyages. It is sometimes frustrating to try to define the outcome of  larger sw development or even an infrastructure deployment project. But let's compare with a 19th century whaling voyage.

So you have a little exerience in the business and you own a ship, a regular New Bedford made whaler. You know a great captain and a reasonably good first mate who want to spend the next 18-36 months somewhere in the far corners of the earth. How do you get the ship out to sea and what can you expect in return.
Whaling sounds straightforward - sail out, kill whales, create other environmental mayhem, return to port, find a pub, get obnoxiously drunk. Yep - let's start at the beginning. Sail out - with who? Where do you get the crew? If they are worth their salt, they'll want to know if the trip will pay or not. That depends on your plan, on your ship, on your captain and on the luck of the day. Your crew to be hired (except the occasional shanghaid case) needs to hear what you have in mind.

So off you go to the tavern and tell your story - Captain Ahab, who will always find the Great Fish, is looking for Good Seamen and brave Whalers to join him on a trip to the South Sea (that's the refreshing bit of water just around Antarctica, btw). He'll be preparing my stout ship the Matchstick for a three-year voyage to hunt for spermiceti whale.

And there you are - you have a plan and a Goal and the goal is NOT to fill the boat, it is to hunt whales. You, Cpn Ahab and the entire crew want to come back alive and rich, but that you can't plan.

Compare with the IT project. We think we will reach cost reduction, flexibility, increased operational and business effectivity but in reality we set up our projects to go ahead and do useful stuff and hope for the results. Sure, it is good to list the expectations, but the mire tightly you set the goals the fwer degrees of freedom there is for the captain and the crew to make smart decisions.

Now we have the  ship and crew, and provisions. Right after the lines are cast the voyage is very different from an IT project since now they are on their own, self-sufficient and there is no way you can help them - or get status reports, discuss strategy, impact of new data etc. You can run the IT project much more closely, be agile, change your plans and goals -- and yet the IT projects try to articulate the end result.

The whaler has now left and there are but a few goals. Return to port. Keep your crew alive. Find right whales, kill and proces them and fill your ship. Oh, and about the first goal - yep, return, but do not return empty handed if you ever want to command a ship again.

All this you must think about before you start looking for the crew. To where (any ocean!), for how long,what if they don't come back (your only ship? Better sail along, mate!) -i.e you have goals, expectations and raw plans and that sets the direction of the whole thing, from resourcing, schedules, risks, rewards, management structures and so on. Do a low risk short tripo North Atlantic and you'll have the ship back in a few months and a neat little profit. Want something bigger? Send them to Bering Sea, and if the come back, you'll have a windfall.

Set up an IT project with relatively modest goals and in a known environment, and you'll get people that enjoy that and produce. If you want the jackpot, the project needs to be ready to sail over the edge of the world, and your crew will match that degree of ambition, desparation, raw energy, and drive.

Be careful what you wish for - it has a great impact on what happens next.

Planning - goals and missions

On the surface it may look like a modern pirate vessel or a fishing boat just crashes out to the sea and grabs whatever happens to be in sight. I am sure some of them may operate that way, but not for long - as any voyager knows, there must be a goal and a plan of sorts. This is not that different from other projects, but the possible outcomes are so clearly different in scale that it is useful to consider event he smaller teams.

Imagine you are a small-time bandit somewhere on the coasts without effective regimes, be it corsairs in North Africa before crusades, Christians a little later, or possibly a team in Somalia.To go to sea, you must have a boat, and a crew willing to follow you - and a crew wants to have an idea of what is to gain, and what are the risks. Notably, everyone wants to come back to dry land, so you need to have a solid boat, and your team better know how seamanship. A successful venture requires seasoned hands, and only so many greenhorns can be accommodated.

The gain - the mission, the goal of your project - can be expressed in various terms. If you happen to know a specific target - good for you. If not, you better have some insight about the average traffic on the sea, or have a convincing story about where the fish are. That story, the target of the trip, becomes the mission, the goal of your project.

Target setting is dead serious. Most small time pirates set the targets to reasonable risk - reasonable reward level. This they can gauge even at the last moment - if they thought they were attacking a commercial vessel, but instead find a Navy frigate, the plan is quickly altered, and the resources (time, fuel, food etc) are a "lost investment".

IT projects have seldom the same luxury and clarity of their goals and capabilities. Small teams and experienced professionals are the preference, and greenhorns should not form the majority of the project team, so in theory it should be possible for a project to easily realize that instead of a minor change they are embarking on a significant system upgrade.
If the project behaved like pirates, or profit-share fishermen, all parties, down to the guy filling in the bait, shared an interest in the end result. Due to that, everyone excepts the Captain to gauge the situation intelligently, and if the Cap'n does not do that, she will hear about it.

For an IT project team and the manager the setup is often different. Members, and possibly the project manager, are running a portfolio of projects, so risk is spread over projects. The only really committed persons are the business owners, and this creates a discrepancy at the goal setting time.

Committed project teams is not a new invention, and and happily it is known to work pretty well in very demanding circumstances. Commitment starts at setting the goals and getting your crew to accept that - and to take the goal as their own.


Next - but not today - What about really challenging goals? Naval and general history has ample examples, so we can look into more organized private navies, whaling and exploration. In those the potential gains perceived by the crews have been astronomical and beyond, and the captains and crews have been equally open to risk. That story will follow once I get back here.

!!!

What the heck - pirates and projects?

Set sail with Captain Morgan? Take a trip to The Pirates of the Caribbean with Jack Sparrow? Great, but what does all this to do with project planning and management?!?

It does, more in some ways than others. I manage IT projects, and have had my share of project management buzzwords thrown around  - towards me and by me;-).
I am also a free-time naval history and sailing enthusiastic, and over time some similarities seem striking. So I wanted to put the vague thoughts into writing and that is why this is here.

So what are the similarities? Differences are easy to point to - Pirates were murderous thieves that behaved, judging on today's standards, worse than many of the worst war criminals (the term "war criminal" really applies to post WWII). They risked everything, including their lives, for a stake of great treasure, and worked hard and very focused to reach their goals. The best of them became wealthy or possibly stinking rich, but the failure rate was high.
Hang on - besides the environment and killing people - pirates, whalers, commissioned privateers, explorers all needed to have a plan, get it funded, find the resources, ships, sailors and commitment to execute their plans, and they took calculated risks, often on very incomplete information. Sounds like project planning to me, and more follows if you pay closer attention to what happens under way.

I'll be looking at "DevOps" and compare that with a ships crew, the sailors, officers and idlers, I'll be musing on Agility with on-watch, off-watch, with the easy days on the Trades and the all out effort to reach a safe port in a storm. Commitment and focus on shared goals is amply illustrated in various single vessel trips, be they pirates, whalers, fishermen, explorers or traders on the seas.
Waterfall project model compares nicely with some characteristics of Naval traditions, while inspired and well trained crews and ships demonstrate the immense value of nurturing the specialist tradespeople you have on your roster.

Some of the failures on seven seas are very instructive, and can add to the tool chest of a Project Manager. The nice thing about the sea is that like war it boils sound (and unsound) practices to their essence, and single ship voyages compare fairly well with medium sized IT projects.

So Godspeed, and stay on the horizon, and I'll be hoisting the Blue Peter for you to join this journey of comparing Pirates with Project management!

  &