Beware the Hero

The downside of coming down with a bad cold over a weekend is that I don't have the energy to work on my side projects.  The positive side of being forced to rest and drink copious amounts of tea with honey is the sugar induced haze which allows me to reflect on things.  This weekend it was about failed projects, software estimation, and technical debt.

Over the years as I've participated in (and led) software projects, I've come to a conclusion about which archetype is the most damaging member of your staff.  It is what we like to call the "Hero Coder".  The Hero Coder is that individual (or team) that frequently expends huge amounts of energy, often in the 11th hour, to overcome the shortcomings in process, scheduling, or requirements that put the project success at severe risk.

Oftentimes, the non-technical executives of the business LOVE the hero coder.  Despite all the challenges that have been placed in the way, the hero coder jumps in at the last minute and yells "I CAN DO IT!" before embarking on a caffeine supported massive overtime binge.  What the business often doesn't realize is the exceptional amount of technical debt that is created during this process.  Over time, the hero coder(s) go to bat again and again, and the clarity of the system degrades to the point where it becomes the big ball of mud.  As wikipedia states (emphasis mine):

A Big Ball of Mud is a haphazardly structured, sprawling, sloppy, duct-tape-and-baling-wire, spaghetti-code jungle. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.

Oftentimes an environment led by a hero coder is one where people are too busy bailing water to patch the leak in the boat.  Over time seemingly simple enhancements start to take longer and longer, small changes ripple through the system causing errors in what appear to be unrelated sections.  The system becomes nearly impossible to test.  Because documentation is neglected in 11th hour scenarios oftentimes the only way to understand what a function is supposed to do is to trace the code.  Ultimately, most rational developers at this point are afraid to touch it for fear of what else may break.

So at the end of the day, the damage to the business drags on over the years.  It's not just the technical damage that we should be concerned about either, it's the culture damage.  The hero coder has created a situation where the expectations of management are that proper planning and execution really aren't all that important, because at the end of the day the feature will ship.  The hero coder will make it so, come hell or high water.  Any future developers who come into the environment will face fierce resistance to attempting to bring some order and sanity to the "process", because the stakeholders have been trained to believe it isn't necessary.

Unfortunately it tends to take a series of absolute train wrecks of projects before management starts to realize the gravity of the situation.  By then, unfortunately, it is likely their shop has been getting a bad reputation in the area due to high turnover among otherwise competent staff.

In a future blog post, I'll go into some tips on agile estimation practices and how you can build a culture that sets reasonable expectations for creating quality code.

When does a vacancy start hurting you?

One of the reasons I've gotten into IT recruiting as a service offering, besides the obvious benefit of being paid to match employers to employees is to help educate my clients on their hiring process.  One of the most interesting things that I find about companies I speak to is that they generally have no idea about how much having an open position actually costs them.

Employers typically have very good macro information about the costs of their operations in general.  That is to say if I ask a small parts manufacturer how much they are spending in labor, they can give me that down to the penny.  If I ask them what the unit cost of a specific part is, they can typically break it down to labor, materials, and processing costs... again, down to the penny.  However, if you try to break this down to an individual position, employers typically have no idea which specific factors of a position we can attribute to a revenue activity.

So what happens on an IT team when vacancies are unfilled?  Well, for starters it can appear that your operation is more profitable because your costs are less than what you have budgeted.  While the books see a short term blip in profitability due to this, obviously it would be absurd if you were to reduce most or all of the headcount because eventually the wheels would come of the machine.  My point in this is that the pressure to fill a vacancy on your IT team rarely comes finance.  Almost certainly the need comes from the team (allieviating an overburden of work/burnout) or from the customers, internal or external who have valued work in the queue that they want to get done.  Getting a position open is often difficult in this respect because it's very easy to quantify cost which creates a management bias against hiring... precisely because the benefits of filling a job is rarely tracked.

Tracking the benefits of your IT positions does take some effort, but it is often worth the exercise.  Let's take a theoretical insurance company for example with nice round numbers.  The claims department has claims processors that handle claims submissions, and the company likes to complete a claim within 30 days of submission, since this is the sweet spot for regulatory and customer satisfaction.  They have a book of business of 100,000 policies and process 10,000 claims per month.  Each claims processor can process 500 claims per month, so in order to meet demand, the company at minimum needs to employ 20 claims processors who bear an average cost of salary and benefits of $50,000 per employee per year.

The business, particularily the marketing department has decided to embark on a big push to get new customers, and aniticipates that they will double the number of customers this year, so that means that to maintain the existing service levels they will have to hire another 20 claims processors at a cost of a cool million dollars per year.

Meanwhile, the claims department has been working with the I.T. on some ideas for automation and process improvements.  They think that if some changes are made to the claims processing applications that they could increase processor productivity by 20%, such that the number of claims processed per month would increase from 500 to 600.  By taking the time to do the math, we quickly learn that if marketing is successful in growing the book of business, these productivity enhancements would allow the company to only have to hire 14 additional processors instead of 20, at a cost savings of $300,000 per year.

Now that we have that number, we can quite easily look at the marketing projections and put a monthly valuation against those claims enhancements.  Of course, in large organizations there are typically far more requests for work than there are staff to handle them.  So lets assume that there are other priorities ahead of this enhancement that are taking up all the time of your existing staff, so the company decides to open a vacancy.  At what point does the vacancy really start hurting the company?  As soon as they hire the 14th processor.  Depending on the rate of growth that the book is facing, the claims processors will have to be hired in advance to maintain service levels, so by using the company projection we can find a reasonable inflection point on when that 14th person will be hired and start taking costs out from there.

The interesting thing about cost exercises like this is how quickly I get managers and executives eyes to bug out when they realize that since IT talent takes time to find (3 to 6 months on average at the time of this writing) that if they drag their feet on creating the vacancy and filling the vacancy that after the inflection point is reached the opportunity costs of having not filled the vacancy start to mount pretty substantially.  So what then is the value of that position?  At minimum $300,000, but the longer it drags on, the more value is lost by the company.  Point being, if you have access to the numbers, it's very easy to create a visual aid like a graph.  If the reason a vacancy lingers open is because of unrealistic requirements or unwillingness to raise salary rates, you have a very compelling arguement to change hiring strategies the closer you get to that inflection point.

Another, much scarier scenario is that of burnout.  When vacancies are open because the team is overburdened, the team's overall productivity starts to drop as morale starts to decline.  As your baseline begins to fall the added value of the additional staff starts to increase.  In the worst case, employees get tired of the burnout and extra hours and leave the firm, which represents a massive productivity hit to the team and increases the burden proportionally on the remaining staff.  In IT this should be a huge concern, because turnover is exceptionally expensive when you add up productive time lost, time spent onboarding new workers, recruitment fees, etc.  Studies have put the cost of employee turnover in IT at 150% of annual salary, so burnout is something I encourage my clients to be acutely aware of.

Unfortunately, although my IT brethren tend to be very gifted at math, I very rarely see them try to break down the value of their positions like this in spite of fielding many complaints about being treated like a cost center.  My advice?  Jump at the opportunity to get your hands dirty with the numbers and strive to understand the costs that drive your customer's departments.  It will make you a better partner, more persuasive in the board room, and drive the profitability of your business.  So ask yourself, what steps can I take to better understand the valuation of my IT staff?