Beware the Hero

The downside of coming down with a bad cold over a weekend is that I don't have the energy to work on my side projects.  The positive side of being forced to rest and drink copious amounts of tea with honey is the sugar induced haze which allows me to reflect on things.  This weekend it was about failed projects, software estimation, and technical debt.

Over the years as I've participated in (and led) software projects, I've come to a conclusion about which archetype is the most damaging member of your staff.  It is what we like to call the "Hero Coder".  The Hero Coder is that individual (or team) that frequently expends huge amounts of energy, often in the 11th hour, to overcome the shortcomings in process, scheduling, or requirements that put the project success at severe risk.

Oftentimes, the non-technical executives of the business LOVE the hero coder.  Despite all the challenges that have been placed in the way, the hero coder jumps in at the last minute and yells "I CAN DO IT!" before embarking on a caffeine supported massive overtime binge.  What the business often doesn't realize is the exceptional amount of technical debt that is created during this process.  Over time, the hero coder(s) go to bat again and again, and the clarity of the system degrades to the point where it becomes the big ball of mud.  As wikipedia states (emphasis mine):

A Big Ball of Mud is a haphazardly structured, sprawling, sloppy, duct-tape-and-baling-wire, spaghetti-code jungle. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.

Oftentimes an environment led by a hero coder is one where people are too busy bailing water to patch the leak in the boat.  Over time seemingly simple enhancements start to take longer and longer, small changes ripple through the system causing errors in what appear to be unrelated sections.  The system becomes nearly impossible to test.  Because documentation is neglected in 11th hour scenarios oftentimes the only way to understand what a function is supposed to do is to trace the code.  Ultimately, most rational developers at this point are afraid to touch it for fear of what else may break.

So at the end of the day, the damage to the business drags on over the years.  It's not just the technical damage that we should be concerned about either, it's the culture damage.  The hero coder has created a situation where the expectations of management are that proper planning and execution really aren't all that important, because at the end of the day the feature will ship.  The hero coder will make it so, come hell or high water.  Any future developers who come into the environment will face fierce resistance to attempting to bring some order and sanity to the "process", because the stakeholders have been trained to believe it isn't necessary.

Unfortunately it tends to take a series of absolute train wrecks of projects before management starts to realize the gravity of the situation.  By then, unfortunately, it is likely their shop has been getting a bad reputation in the area due to high turnover among otherwise competent staff.

In a future blog post, I'll go into some tips on agile estimation practices and how you can build a culture that sets reasonable expectations for creating quality code.

When does a vacancy start hurting you?

One of the reasons I've gotten into IT recruiting as a service offering, besides the obvious benefit of being paid to match employers to employees is to help educate my clients on their hiring process.  One of the most interesting things that I find about companies I speak to is that they generally have no idea about how much having an open position actually costs them.

Employers typically have very good macro information about the costs of their operations in general.  That is to say if I ask a small parts manufacturer how much they are spending in labor, they can give me that down to the penny.  If I ask them what the unit cost of a specific part is, they can typically break it down to labor, materials, and processing costs... again, down to the penny.  However, if you try to break this down to an individual position, employers typically have no idea which specific factors of a position we can attribute to a revenue activity.

So what happens on an IT team when vacancies are unfilled?  Well, for starters it can appear that your operation is more profitable because your costs are less than what you have budgeted.  While the books see a short term blip in profitability due to this, obviously it would be absurd if you were to reduce most or all of the headcount because eventually the wheels would come of the machine.  My point in this is that the pressure to fill a vacancy on your IT team rarely comes finance.  Almost certainly the need comes from the team (allieviating an overburden of work/burnout) or from the customers, internal or external who have valued work in the queue that they want to get done.  Getting a position open is often difficult in this respect because it's very easy to quantify cost which creates a management bias against hiring... precisely because the benefits of filling a job is rarely tracked.

Tracking the benefits of your IT positions does take some effort, but it is often worth the exercise.  Let's take a theoretical insurance company for example with nice round numbers.  The claims department has claims processors that handle claims submissions, and the company likes to complete a claim within 30 days of submission, since this is the sweet spot for regulatory and customer satisfaction.  They have a book of business of 100,000 policies and process 10,000 claims per month.  Each claims processor can process 500 claims per month, so in order to meet demand, the company at minimum needs to employ 20 claims processors who bear an average cost of salary and benefits of $50,000 per employee per year.

The business, particularily the marketing department has decided to embark on a big push to get new customers, and aniticipates that they will double the number of customers this year, so that means that to maintain the existing service levels they will have to hire another 20 claims processors at a cost of a cool million dollars per year.

Meanwhile, the claims department has been working with the I.T. on some ideas for automation and process improvements.  They think that if some changes are made to the claims processing applications that they could increase processor productivity by 20%, such that the number of claims processed per month would increase from 500 to 600.  By taking the time to do the math, we quickly learn that if marketing is successful in growing the book of business, these productivity enhancements would allow the company to only have to hire 14 additional processors instead of 20, at a cost savings of $300,000 per year.

Now that we have that number, we can quite easily look at the marketing projections and put a monthly valuation against those claims enhancements.  Of course, in large organizations there are typically far more requests for work than there are staff to handle them.  So lets assume that there are other priorities ahead of this enhancement that are taking up all the time of your existing staff, so the company decides to open a vacancy.  At what point does the vacancy really start hurting the company?  As soon as they hire the 14th processor.  Depending on the rate of growth that the book is facing, the claims processors will have to be hired in advance to maintain service levels, so by using the company projection we can find a reasonable inflection point on when that 14th person will be hired and start taking costs out from there.

The interesting thing about cost exercises like this is how quickly I get managers and executives eyes to bug out when they realize that since IT talent takes time to find (3 to 6 months on average at the time of this writing) that if they drag their feet on creating the vacancy and filling the vacancy that after the inflection point is reached the opportunity costs of having not filled the vacancy start to mount pretty substantially.  So what then is the value of that position?  At minimum $300,000, but the longer it drags on, the more value is lost by the company.  Point being, if you have access to the numbers, it's very easy to create a visual aid like a graph.  If the reason a vacancy lingers open is because of unrealistic requirements or unwillingness to raise salary rates, you have a very compelling arguement to change hiring strategies the closer you get to that inflection point.

Another, much scarier scenario is that of burnout.  When vacancies are open because the team is overburdened, the team's overall productivity starts to drop as morale starts to decline.  As your baseline begins to fall the added value of the additional staff starts to increase.  In the worst case, employees get tired of the burnout and extra hours and leave the firm, which represents a massive productivity hit to the team and increases the burden proportionally on the remaining staff.  In IT this should be a huge concern, because turnover is exceptionally expensive when you add up productive time lost, time spent onboarding new workers, recruitment fees, etc.  Studies have put the cost of employee turnover in IT at 150% of annual salary, so burnout is something I encourage my clients to be acutely aware of.

Unfortunately, although my IT brethren tend to be very gifted at math, I very rarely see them try to break down the value of their positions like this in spite of fielding many complaints about being treated like a cost center.  My advice?  Jump at the opportunity to get your hands dirty with the numbers and strive to understand the costs that drive your customer's departments.  It will make you a better partner, more persuasive in the board room, and drive the profitability of your business.  So ask yourself, what steps can I take to better understand the valuation of my IT staff?

Onboarding Technical Staff

You've done it!  After months of searching, interviewing, and wooing, you have finally located a candidate to fill your open position.  At many companies, this means it's time to pat yourself on the back, set up a work area, and turn them loose on the team, maybe even going so far as to mention that if they have any questions they can ask Bob or Mary in the other cubes for help.

Good knowledge workers are hard to find.  Good knowledge workers that aren't productive are expensive. And yet so many companies drop the ball early in the employee experience.  In the absolute worst case scenario, you haven't properly prepared a space.  The employee shows up for their first day after all the positive spin you put on things during the recruiting process and they find that their accounts aren't set up, they're using a second-hand machine because theirs is on order, and everyone else on the team is too busy focusing on their deadlines to do much except point them towards some dusty, out of date documentation.  By the end of the day the employee is rather bored and wonders when things will really get going.  Congratulations, you've demotivated your employee on day 1!

Onboarding is what good companies focus on for at least the first 90 days.  For technical teams that have shops of any sort of complexity I like to think that good onboarding lasts 6 months to a year.  Every technical manager should have a well defined plan for every new employee that joins their team, one that is tailored to their individual goals and expectations, and gets frequently reviewed.  These first days at the company are going to set the stage for retention of employees.  Modern knowledge workers really don't have a lot of loyalty to one employer anymore, they are looking for companies that show appreciation for their skills, can help them grow to higher levels of mastery in those skills, and provide them clear goals and paths for obtaining them.

Onboarding is also something that managers typically don't get trained on, and us IT folk aren't exactly known for our soft skills, so here are some tips of things that I have found effective when onboarding new recruits to my IT staff.

Day One: You only get one chance to make a first impression.  As I mentioned above, make sure that their work area is set up, all their materials are in place, that they have accounts on all the relevant systems they need to be productive.  We want the new hire to feel welcome.  This means meeting them at the door when they show up, it means actually scheduling sit downs with all their important peers and contacts and facilitating conversations.

When it comes time to get into the systems, make sure that one of their new coworkers actually gives them a guided tour through the relevant systems and is able to answer questions.  At the end of the day, schedule some time to recap the first day experience with the employee, give them a chance to ask any additional questions and recap the names and duties of the coworkers or customers who will be most important to them.

This is also the day you should introduce your 6 month assessment and growth plan.  These plans are individualized for each employee and are structured such that if they ever find themselves in a position where they don't know what to do next, instead of spinning their wheels or sitting idle they can grab the cheat sheet and go to work on the next item.  My typical plan generally had the first month or two spent familiarizing themselves with systems, learning any tools that are new to them, and working on small, introductory units of work.  I would set up time every week to go over progress  on these objectives, and give the employee a forum to ask questions.  On each item there would also be a suggestion of one of their coworkers who is familiar with the objective and can assist with questions.  Additionally, since you front load the familiarization and easier tasks, it gives the new employee a sense of accomplishment and contribution quickly, which I believe leads to higher job satisfaction (and on the other side of the coin, gives you a nice checklist that if not met alerts you very early to a potential problem)

After those first few months, the plan starts digging deeper into the systems and processes and generally would include handling the job responsibilities relatively solo.  Several opportunities should be given to demonstrate mastery of things the position is expected to accomplish as well as starting to explore cross-training into some other, closely related areas.

Seeing is believing though, so let me show you a sample few months for a mid-level developer:

Week 1

• Welcome to the team!  Verify the following:
• Domain login, email configured, and all critical applications are installed and functional (visual studio, sql management studio, dev server / source control access, and ticketing system).  If anything isn't configured correctly see me right away so we can get you set for success!
• Spend time acclimating yourself to the company wiki, here are some helpful pages
• PTO request form, Sick Day policy, Call-down list for emergencies, etc
• I have asked Mary, one of our senior developers to meet with you and assist you in setting up a clean build of our software applications.  She will also review source control policies and familiarize you with the workings of Git, since I am aware you've never used it before.  Additionally, here is a link to some helpful documentation on it: <LINK>
• I have also asked Joe, one of our QA specialists to meet with you and show you around the ticketing system.  We have also identified some open ticket requests for some small enhancements to the interface of our CRM application that will help familiarize you with it.  Go ahead and work on these.  For technical questions I will ask Bill to follow up with you, since he works a lot in that application, and Bob will show you how to push changes to the QA server.

Month 1

• Congratulations on making it through the first week!  From here on out our goal is to start getting you more involved in the day to day workings of the team, start empowering you over the things in your domain, and start growing your skills in the areas that will help you and the team going forward.  
• One of the areas we discussed during the hiring process is that you are relatively inexperienced in T-SQL.  We have several skilled practitioners (Jack and Jill) on staff who will be delegating some tasks to you and pair programming with you.  At the end of the month, we will sit down together and do a review and write some queries covering the following items:
• SELECT query basics: SELECT, FROM, WHERE, GROUP BY, ORDER BY, HAVING, INNER/OUTER/CROSS joins.
• UPDATE, INSERT, and DELETE statements
• Calling stored procedures
• Creating, committing, and rolling back transactions
• Continue working on coding tasks.  As you've discovered, we assign cards for iterations.  Work with Mary to proactively identify some cards across applications to continue to expose our systems, but I would like you to keep the amount of production work to around half the work-week so that you have time to develop your SQL skills.

Month 2

• You should be well on your way to feeling comfortable in SQL now.  The plan for this month is to leverage that learning and dive into Reporting Services.  Our report writer, David, has prepared a half day presentation that he gives to new hires on basic report writing concepts.  Supplement this presentation with the following online guides we have purchased (LINK).  By the end of the month, you should be able to demonstrate aptitude in creating reports of low to medium complexity.  I will have asked David to pair program with you on some of his reports, and then give you some tasks to do on your own.
• Continue working on coding tasks.  I have instructed Mary to start focusing you in on Project X, this is a major initiative for the company and you are going to be a key part of its success.  At this point we should be in the planning stages so we will be sure to get you involved in the planning process and included in meetings with the business analysts.  I feel it is critical to quality software for our development team to understand the business processes as well as the analysts do, so do not hesitate to ask questions and as a fresh pair of eyes to challenge our assumptions.
• Be mindful that your skill growth is just as important to us as the coding tasks, if you feel like your workload is such that you don't have time to explore the system and accomplish the learning objectives set before you, let me know early, so we can fix it!

So what I've tried to convey here is a few concepts:

1. I started the employee off slow, but I gave them a sort of checklist for the first week.  At our end of week followup meeting I can have them quickly demonstrate that they've learned everything they have checked off.  This is a low bar, but it gives a sense of accomplishment to check things off in the first week!
2. I continuously reiterated my dedication to growing the skills of the employee and getting them involved  with other team members both in areas of strength (in this case, coding), and also in areas of weakness.  My hope with all my teams is to encourage workers to assist their peers and cross train.  This hopefully gains respect, camaraderie, and appreciation for what the person across the way does. Facilitating open communication is one of the best ways to drive process improvement and innovation.  My continued interest and dedication to growth is a key point of satisfaction for employees.
3. As I've moved on to new tasks, I've recognized them for doing well.  In my followup meetings when they demonstrate mastery in tasks I've assigned I will be quick to openly praise their efforts.  If there is criticism or correction needed it will be where it belongs, behind closed doors.
4. I've also recognized that the new employee is eager to please, but re-iterated my desire for them to grow and produce at a pace which is comfortable to them.  Fatigued and stressed out knowledge workers produce less with lower quality, so I strive to keep on top of that.

So what happens at months 3 and 6?  For one thing we do a formal written review and sit-down.  I collect feedback from their peers and make any corrections that need to be made while being mindful to praise and recognize the achievements.  Because we have this plan, I have a handy list of all the accomplishments and I like to jot down some areas where they've exceeded expectations if that occurred.  Just the idea that the boss notices those extra things is highly satisfying to knowledge workers.  Then... we set up the plan for the next year!  The employee gets feedback into their goals and what they want to learn mixed in with what I want them to work on.  Now that they're onboarded though, the detail level can be less but we still have to review and show progress towards these goals.  In the end, the process should never stop, because if there's one thing about knowledge workers and retention that I have learned it is that the day you stop recognizing and growing them is the day they start shopping their resume!

Doomed Before You Start

I've had some blog posts floating around in my head for a while about various things I come across in environments that are good indicators that down the road in a development project you are going to run into some severe problems.  By severe I mean failure.  By failure I mean way over cost / exceedingly late.  I have spent quite a bit of time over the years working with executive teams to understand why their IT initiatives fail.

The project is late... again, and over budget... again.  Releases are filled with bugs, the users have lost faith, morale on the development team is low.  Management is furious and dismayed, they don't understand how all these highly paid "professionals" can put out such a low quality work product, especially when they started practicing that hot new "Agile Development".  Hopefully at this point they reach out for help.

 "So how many Agile projects have you completed?", is usually the first question I ask.  Usually the answer to that is "well this is our first one".  "So this project you're so concerned about, it's not really that important?"  At this point I tend to get looks of disbelief, "Of course it's important, it's key to X".

The Virginal Methodology

Here's big huge mistake number one.  You decided to change your process at the outset of a critical, time sensitive project.  Implementing any process takes time, the team has to grow accustomed to it, and there is always a period of inefficiency as they feel their way through it.  By implementing a new process on a critical, time sensitive project, you have already sacrificed a good chunk of your productivity.  So we shouldn't be surprised when the schedule begins to slip.

Loss of Trust

Big mistake number two is continuing onward, without stopping, doing a retrospective, and repairing the damage that has been done.  Feeling pressure to meet the deadlines, the team forges onward even though every member knows in their heart that things are not going well.  Unfortunately, over time this leads to an even bigger organizational problem, the various team members begin to stop trusting each other.

Trust is a critical component of any methodology.  When trust starts to fail, the signs are pretty easy to see.  Teams will begin circling the wagons, scapegoating other team members, former team members, just about anything to deflect blame.  Oftentimes as a defensive measure, teams will start insisting on unreasonable levels of detailed specification before work begins.  The ad-hoc/hallway conversations that so often spread domain knowledge far more effectively than formal meetings cease, as team members turn to email communications so that they have a "documentation trail" in case someone tries to throw them under the bus.

Bottom line, if the team has lost trust, your organization is critically ill, and drastic measures must be taken.  To build trust, you must create an environment where the following behaviors are present:

1. Absolute Honesty- As much as we feel pressure to please, we must be clear about what we are capable of and within what time frame.  It is your responsibility at the end of the day to get your tasks completed with sufficient quality, and hamstringing yourself before you begin is foolish.  This also means that your peers accept that you know your capabilities better than they do and support you in your assessments.  If the time estimated is unacceptable, it is up to the team to find a create solution as a collective, not tear down the person and pressure them into setting unreasonable expectations.
2. All communication is safe- Building on #1, the environment must be safe.  Everyone involved in the project should feel comfortable asking questions and voicing concerns without fear of retaliation.
3. Empowerment & Accountability- As a manager or executive, you need to trust your team.  If they need resources and can make the case for them, you should strive to provide them.  This also means that you hold them accountable for achieving the goals.  It is vital to any human resources decisions (hiring and firing) that you ensure that you have removed as many organizational and cultural challenges that you possibly can from the team.
4. Remove Single Function Groups- Having a Development Team, a QA team, a Business Analyst team, etc all separated off into their own worlds builds barriers to communication and trust.  If you are in the same geographic location there almost no excuse to not create cross functional teams and communication zones.  
5. Avoid the Silo- Jack does Jack's tasks and Jill does Jill's tasks.  Here you are creating silos of domain knowledge and making the blame game easier.  Don't see a software task as separate components, see it as a whole.  A developer and a QA analyst and a Business Analyst should be responsible for taking feature X from start to finish.  This helps you combat that situation where decisions by non-technical staff have little value into the actual development and the situation where decisions by developers aren't aligned with organizational interest.

You Assumed Process = Quality

Processes and methodology are tools, and tools can be misused.  Big mistake number three is assuming that just because you implement a process that your quality will improve.  "Well, we moved to short iterations, but we're still falling behind schedule", "We're making the developers write tests, but the code quality is still poor".

For a process to be successful it must be embraced by the team and practiced.  The testing quote above is something I see a lot in the wild, the organization says "the code quality isn't good enough, we need more tests!".  It's very appealing to blame a process for failure, since generally nobody will be held accountable for it.

The truth of the matter is that it is your people who create quality.  I'm sorry to say that most struggling organizations I come across have the wrong people organized the wrong way.  Going back to Agile there is something I must point out:

Implementing Agile on a weak team makes them WORSE

Software teams are comprised of knowledge workers and the ones who are skilled an passionate are going to succeed provided that the organizational culture supports them.  When a weak team is presented with Agile, it usually means that whatever sense of discipline and craftsmanship may have been there will quickly go out the window.

Your Organization Misunderstands That Software is a Craft

This is where you really need a champion in the organization to really communicate with all levels and stakeholders.  Software is a craft and as such it has far more in common with Artisan skills than Science (it's a mix of both, but the Artisan part is more important in day to day development).  

To become an Artisan requires a lifetime of learning and discipline

Some organizations don't believe in dedicating funds to training.  Other organization provide funds to training, but they assume that just because they sent a developer to a week of "Technology X boot camp" that they will return and be able to perform at a good level.  Have you ever heard of anyone getting piano lessons for a week and then becoming a concert pianist?  We learn from applying our skills in repetition.  Patterns emerge and confidence builds over time.

The Artisan system has been around for thousands of years, and it always involves shared learning and apprenticeship.  This is the biggest reason why it is absolutely crucial to foster communication and also why Paired Programming is increasingly adopted in organizations.  You will learn far more through continued interactions with your peers than you ever will from training or technical manuals.  Training and manuals are absolutely great for getting your feet wet, but excellence will be achieved far more effectively when working with peers.

Building an artisan team also requires that executives let go of the myth of 100% utilization.  You simply can not produce a good product if your utilization is 100% because it leaves no time for personal growth, mentoring, or improving existing code based upon what you know now versus what you knew then.  I often tell developers that I mentor that if you don't look back at code you wrote 6 months ago and hate it, you probably aren't learning anything.

If you look back at the points above, you can see why I left the craftsmanship portion for last on the list.  Building and retaining a team of Artisan software professionals requires the organizational culture to be in the right place before it has any chance of occurring.

The Moral of the Story

The vast majority of the time I am asked to assess and assist with a struggling organization when it comes to IT, the biggest problems start with the culture.  Once you've embraced a culture of communication, trust, and accountability then you can begin building that team of craftsmen who can take your shop to the next level.

MSDB tip, clean up your mail

Just a quick tip since I encountered this recently.  Came across a MSDB database in my journey that was exceptionally large, did some delving and found that they were using database mail to send attachments frequently and no one ever bothered to clean up the history.

By default, MSDB will store all those attachments you send, so you should clean them up or it will grow... well forever.  So Microsoft provides a system stored procedure that allows you to purge ones older than a specified date.  So for example if you wanted to purge everything older than 30 days, you could do something like this:

 declare @purgeDate datetime  
 SET @purgeDate = dateadd(dd, -30, getdate())  
 EXECUTE msdb.dbo.sysmail_delete_mailitems_sp   
   @sent_before = @purgeDate;  

T-SQL : Use the right temp holders

Just a quick blog here.  Working on some queries for a client, and they are performing pretty poorly.  This is in a SQL 2005 instance and I notice that the developer who created these reporting queries is using table variables.

I love table variables, they're quick and easy, they reduce logging and lock contention.  They have limited scope so my .NET developer colleagues who reach for them tend to do so because they don't like mucking about with understanding global temp tables vs local ones.

However, and let me be clear about this, table variables with large amounts of data perform... poorly.




Case in point, this report.  It was taking about 90 seconds to render using table variables, upon examining the query I noticed that it was processing well over 10,000 rows in these table variables and then doing some joining between them.  Couple things to note about table variables:

1. You can't add indexes to them
2. If you do data modifications (insert, update, delete) SQL server won't generate a parallel execution plan
3. They don't have any statistics

Long story short, simply by converting the table variables to temporary tables, the processing time of the query went from 90 seconds to... 5 seconds.

I polish my Super DBA badge, and walk away happy.

Don't Repeat Yourself

Want to be a better developer?  By far the easiest place to start is by implementing the DRY principle of software development.

Don't
Repeat
Yourself

Duplicated code is bad.  It's a code smell, it makes you look like an amateur, and worst of all, it makes the next developer who has to maintain your code want to drag you out into the parking lot and slap you around for a while.

Learning to recognize DRY violations is a core foundation to writing cleaner code, but I'll be the first to tell you that recognizing and squashing repetition violations is a discipline.    It requires you to slow down, really think over your code, look for patterns and refactor them.

Here's the thing though: making your entire code-base purely DRY is pretty difficult, and runs the risk of reducing the readability and the flow of the application from a maintenance perspective (excessive abstraction is a code smell as well, known as contrived complexity, and a fun fact is that it almost annoys me more than DRY violations because the people who create that code smell are more likely to be arrogant/inept where DRY violating people are more like children running with scissors).  So in my mind the goal of DRY is not to purge every possible ounce of duplication out of your system, but the closer you get to the business logic the more DRY your code should become.

To put this in perspective, let's consider an enterprise solution with the following layers (from outside in):

• Presentation- Various UI clients, executables, etc.
• Service- Feeds presentation layer
• Business/Interactors/Rules- Actually does the work
• Persistance- Stores and retrieves data

Presentation layer?  Sure, you might have some duplicate service calls on various forms.  This in the grand scheme of things is not all that big of a deal, because your presentation layer should not be performing business logic.  Thus, even if you duplicate some calls and structures, those calls shouldn't need to change for a behavior update.  It's nice to make this layer reasonably DRY, but there's a line to be crossed where it seems like developers start cramming unrelated methods together into the large class or "god object", which is another code smell.  You need to weigh the risk of change and readability and make an educated decision.  However, even our designer brethren understand DRY, what do you think CSS does?  Reduces duplication, provides a single source of truth!

A well designed service layer just takes requests and returns responses.  Again, due to minimal business logic, duplication here really should be avoided, but it probably won't kill you if some similarities pop in, particularly in cases where similar services are exposing to different clients.  I definitely take a harder look at this layer than the presentation layer.

Business layer is where we really lock things down.  For any business rule, logical sequence, calculation, etc there must be a single source of truth.  One of the main reasons the architecture forefathers created multiple layers was to purge duplication of these very important facts within software systems.  Until you experience the nightmare of having to update a calculation that is spread throughout multiple layers and duplicated everywhere you can't really appreciate this.  It's a game of coding whack-a-mole that you are almost guaranteed to lose by introducing bugs into production.  

Cool story bro, I once had to work on a system where an important fact was in a middle tier, in stored procedures, also hidden in expressions in reports, and additionally was even duplicated in JQuery on the presentation layer, what was literally a 2 line change took about a week by the time all the duplication spots were located and shaken out.  The hellish thing about duplication like that in distributed systems is that you can't do any cross platform searches easily to find all the buggers.  As to be expected with such shoddy code structure there weren't any tests to run either.

In the persistence layer, particularly in the realm of relational databases any database architect worth their salt will take care of DRY for you.  The whole concept primary keys and unique constraints is to make your data DRY.  Duplicated code is bad enough, but let me assure you that unintentionally duplicated data is a nightmare in its own right!

In the end, DRY is a key to making your code reusable and more maintainable.  Having mentored a lot of developers over my career, I've also found that the pursuit of the DRY principle actually is what leads many new developers to discover architecture patterns in more progressive and natural way than handing them a gang of four book and telling them to go away.

Sometime soon we'll talk about the Single Responsibility Principle (SRP).  Mixing SRP violations with DRY violations is a great way to punish future maintainers if you're a sadist, but that's a topic for another day!