How the Hacker puts your system into debt

In the past I’ve posted on some of the differences between the hacker school of coding and the non-hacker school of coding.  I recently encountered another anecdote along the lines of this thread.

The Hacker needs to provide users with a way to switch between multiple identical windows in a form.  It’d be nice to minimize use of vertical space since monitors tend to have less of it available and this application is image oriented.

Sounds like a job for a ToolBar/ToolStrip right?  It’s got all sorts of built in functionality for adding and removing buttons, responding to user events, customizing the appearance, OS theme support, designer support, etc…  It’ll even support runtime repositioning in case there are users out there that prefer to dock it against a different edge of the form.

Given such an easy way to provide the required functionality (and then some) why doesn’t the Hacker use it?

1. He doesn’t know that it exists.  One thing I’ve noticed is that coders from the Hacker school tend to have average to below-average recall.
2. He thinks he can write a better ToolBar.  Part of being a Hacker is not having a good grasp of the big picture.  So his analysis of better tends to ignore things like designer support, accessibility, OS themes, System Preferences, System Events, Display Resolution, etc…
3. It didn’t occur to him to use a built-in widget.  Hackers don’t think this way.  Oftentimes they were trained in an era where there weren’t very many built-in widgets.  Being Hackers they wouldn’t have used them even if they were available. 

Each of these in some respects arises from the natural desire and concomitant tendency to “subvert the system” that is a hallmark of most Hackers.

Why does this matter?

By writing his own toolbar the Hacker has just raised the technical debt both for himself and for any other team members that have to work with what he has built.

One effect of technical debt is malfunction.  Situations where the Hacker’s homemade toolbar are likely to break: multiple monitors, large DPI fonts, different version of the Common Controls library, OS upgrades.  The homemade toolbar no longer functions as intended and/or no longer matches the native environment.  This debt costs time and money to track down and repair.  Often the Hacker is long gone or, if he’s still there, has completely forgotten the mechanics of his homemade toolbar.

Another effect of technical debt is interface mismatch.  Put another way, Hackers tend to produce software that doesn’t interface with other software well.  So not only does the Hacker make it harder for the team to do its job, he makes it harder for the company to take advantage of the work of others.

In any complex project there are going to be many instances where design decisions need to be made.  Some of these will be obvious, some less-so.  These will often provide a moment at which technical debt can be increased or avoided entirely.  Hackers, because of the reasons outlined in this and earlier posts, will almost always tend to choose the path that increases technical debt.

Do your system and your company a favor; protect it from Hackers.

A Shortcut a Day Keeps Efficiency Away

What follows is a totally fictional dialog.  It’s the kind of conversation that might happen several times a week on a software team as a release nears.

Hacker, speaking to co-worker, announces: “There’s no way to make these buttons invisible.  They’re initialized by an array that gets set before the screen is displayed. … So I’m just going to disable them instead.”

Co-worker: “But isn’t there a way to remove the buttons?”

Hacker: “There’s a way to add buttons.  … I think it’ll just be easier to disable them.”

In walks Subject Matter Expert (SME) who, after overhearing this conversation, points out: “There’s another screen that does that, it’s probably a good example to use.”

We have reached an inflection point: Does the Hacker go with the technique he figured out after spending quite some time tracking down the issue?  Or does he take the SME’s advice and use the approach used by other screens?

There are many things to be gleaned from the Hacker’s decision that directly bear on how well the software he builds will stand the test of time.

Abandoning an approach that you’ve invested in does not come naturally.  In fact, in other areas of life it’s what you’re not supposed to do.  It’s hard for us even to detect how attached we are to an approach and how that attachment influences our judgment or willingness to abandon it.  But when it comes to software design, like any other exploratory activity, being able to recognize a dead end quickly is crucial to delivering quality on time.

Even though it doesn’t come naturally I’ve found that it does become easier with practice.  Once you’ve done it a few times and, more importantly, reaped the rewards of a system that either functions better or is more easily maintained (or both), those benefits make it easier to swallow the next time.

Hackers, Sculptors, Instincts and Design

Although there are many things that a Hacker does well design is not one of them.

The key question is, why not?

Partly it's because design doesn't lend itself to the Hacker style of working. For one, design is explicitly creational. The Hacker, on the other hand, enjoys probing an existing system for faults. The goal of design is to create systems that produce the desired output without those faults. Since the job is creating the system there's no system to probe (yet), so the Hacker's favorite activity has nothing on which to operate.

For another, design often involves choosing what to avoid. Put another way, sometimes good design is about choosing what to not do. Is it worth optimizing a given process or is it better to go with the naive implementation? The Hackers instincts run so strongly counter to choosing to not do something that he will often not recognize that such a choice is at hand (let alone make that choice). The consequences of his "hack" live on in infamy until it's undone.

During the design process you inevitably have to make choices. Do you optimize for space or speed? Do you design for caller simplicity or callee simplicity?

When the Hacker encounters such a decision his instincts work against making the design tradeoff in exchange for a more coherent system. The Hacker specializes in discovering system faults and working around them no matter how much the workaround is at odds with the manifest intent in the design of the system. The choice, to the Hacker, appears to be a problem in need of fixing and he's got tried and true methods for fixing problems.

The Sculptor is less likely to see the choice as a problem requiring a workaround. To him it's a chance to make the right tradeoff for the particular problem the system is intended to solve. To the Sculptor it goes without saying that all systems have limitations. He's more interested in having the system do whatever it was intended to do well than in having it do anything that can be done.

For the Sculptor the discovery of a system fault is somewhat less pleasant than it is to the Hacker. The system fault might mean somewhere along the line he made a bad design choice. Many design choices, once made, are very hard (or impossible) to undo once a system is in use. The discovery of a bad design choice is also unpleasant to the Sculptor because he enjoys making good design choices.

Sometimes it isn't obvious which tradeoff is the correct one. This is often a clue that the Sculptor doesn't understand some aspect of the problem domain. Ideally he'll be able to get a better understanding; either by talking with the client, referring to existing documentation, Wikipedia, reference books, etc... Sometimes this isn't possible - the client may not be available, there may be no existing documentation, nothing else to go on. The Sculptor has to make a call. However, because the Sculptor has gotten into the habit of getting to know more about the underlying problem domain, there's a good chance that he's run across something in the problem domain that points to the right decision.

Sometimes the Sculptor has a pretty good idea of the relative costs of the choice but doesn't know how significant either decision is to the client. A choice that's many times more expensive can often be discarded because it turns out that the client is more than happy with either result. The longer the Sculptor has been sculpting the better he gets at recognizing these "decision moments".

For all these reasons, the Hacker's instincts work against good design. Where good design is found there isn't as much need for hacks. Changes to existing systems becomes less about probing, trial and error or making the system work in ways it wasn't intended and more about finding the right pieces in the existing system to assemble or extend just-so to add the needed functionality.

As a code base increases in size good design can mean the difference between shipping a new, nearly bug free feature in 6 weeks instead of 6 months. It can mean the difference between a new employee becoming productive in a few weeks instead of a few months. The systems tend to survive and thrive long after the designer is no longer working on them.

Systems designed by Hackers, on the other hand, are plagued by bugs. Some easily repeatable, some seemingly non-deterministic. They're fixed quickly but seem to crop up again a few months later. The Hacker isn't (usually) intentionally creating a buggy system so that he has something to do; it's just that he isn't as good at design. He doesn't enjoy it nearly as much as he enjoys hacking and so puts commensurately less effort into design.

Systems designed by Hackers tend to accumulate hacks at several levels. At the level of the user interface there may be major discrepancies both internally and with respect to other, similar products. This is the manifestation of the Hacker's instinct to hack the system. He may have come across a user interface need that wasn't directly met by the available user interface widgets. So he baked something entirely from scratch. Ironically, the desire to hack the system often acts as a disincentive to taking advantage of the built-in functionality in a given system; why bother to figure out what's available when all you need is enough to start and you'll hack out the rest?

At the level of coding constructs systems designed by Hackers will often have a lot of copy/paste re-use. Somewhat more experienced hackers may make use of libraries but they'll often have strange quirks. Maybe they throw lots of exceptions during execution. There will be lots of exception handling, even in places where it doesn't make sense. The Hacker may think of this as "defensive programming" but in reality it's another manifestation of the Hacker instinct: try a bunch of things and hope something works. Oddly enough, the Hacker is often proud that he doesn't really understand exactly why his fix works as long as it appears to work (most of the time).

Hacker vs Sculptor Redux

In the previous post on the dichotomy between the 2 dominant modes of creating software I labeled these modes "The Hacker" vs "The Professional". After thinking about this it strikes me that it unintentionally implies that Hackers can't be professional. So the nomenclature needs some refactoring.

Hackers can be, and often are, professionals. For example, security researchers, those that discover (and sometimes fix) exploits in existing systems, must spend a lot of their time hacking to be successful in their job.

Quality Assurance is another profession that lends itself to the Hacker mindset. QA Engineers have to probe a system in expected and unexpected ways to find out whether it performs as intended.

As a final piece of evidence that I believe that the Hacker mindset is compatible with professionalism, articles such as "Under the Hood: Apple iPhone 3G exposed" clearly require a significant degree of technical expertise, discipline, knowledge of standards and standard techniques all of which are the hallmarks of a profession.

The Hacker vs Sculptor split isn't limited to Software Development. If it, as I believe, is truly an expression of fundamental personality preferences, then it isn't suprising that other professions find practitioners aligning along similar boundaries.

Take the music business. Many years ago a friend of mine asked, "So are you a technician that has an ear for music or a musician that knows how to push a few buttons?" He was an aspiring music producer who placed himself squarely in the former camp.

Back in the domain of software, the founder of PlentyOfFish.com evidences both his own hacker tendencies and the existence of the dichotomy when he says, "I was able to go in and see where errors were and fix them, which no one else could do. Everyone else could program, but didn't particularly know how to fix things. I was not particularly great at building stuff but I could fix things and make them work really, really well."

During a recent blog episode the commentators (Joel Spolsky and Jeff Atwood) discuss another expression of this dichotomy when talking about the differences between a good tester and a good programmer. Perhaps more importantly Joel alludes to the significance of the issue: being a good programmer probably means that you're not going to be a very good tester.

In the same vein, being a good Hacker probably means you will not be a good Sculptor. By Sculptor I mean designer of libraries, interfaces, systems and the like. The Hacker's instincts are very similar to the QA Engineer's instincts in that he is far more excited by and interested in how, where and when the system breaks.

The Hacker will gravitate towards this "quest for system faults" even when it doesn't directly bear on the task he finds before him. He genuinely enjoys it. He basically can't help it. Give a Hacker a system that isn't working and he is thrilled at the chance to get it to work, by hook or by crook.

Next time we'll focus on the Sculptor and what it is that makes him tick.

Hacker vs Professional

Even with the luxury of highly detailed specifications there arise moments where the developer has to make choices about how to implement a feature. The decisions made during these moments provide a pretty good indication of whether or not the developer is of the hacker variety or of the professional variety.

Let me provide, in lieu of a formal definition of a hacker, a few examples that illustrate the hacker mindset.

The user needs a way to enter a password with the text masked as it's entered. The framework you're using, e.g., Windows Forms, doesn't have a built in way to do this. How do you go about meeting this need?

The Hacker's instincts will involve one or more of the following steps:

• Using the debugger to identify the internal structure of the textbox.
• Using Reflector to decompile the framework and reverse engineer its logic.
• Using Spy++ (or ManagedSpy) to reverse engineer the communication patterns.
• Hooking directly into the underlying API on which the framework is based and, using the knowledge acquired in the previous steps, finding a way to intercept keystrokes then replace their display with asterisks.

The Hacker may not settle on this approach but those are his instincts. To him there is an inestimable joy in unraveling the framework, peeling back its layers, identifying what its inventors overlooked and plugging into the framework in a clever way likely not envisioned by its creators. The fact that its creators apparently didn't think of this obviously important feature is evidence of their mere mortality. And since the Hacker is also a mere mortal, he's got as much reason as anyone to fix the shortcomings in the framework. He might even get a little kick out of subverting the confines of the framework.

The Professional is also a mere mortal and like other mortals also has instincts. His instincts will typically involve one or more of the following:

• Using the framework's documentation to identify any events exposed by the framework that can be used.
• Mining the samples that ship with the framework for examples of solutions to similar or related problems.
• Mining the guidance/overviews that ship with the framework for ways to plug-in and/or extend the textbox.
• Lobby for an alternative solution (e.g., a popup that quickly disappears) that can be built with stock components included in the framework.

The Professional, like the Hacker, may also end up settling on a different solution but his instincts are radically different than the Hacker's instincts. To him the framework, shortcomings and all, represents a genuinely better way to do things. He remembers the bad old days of MFC and the worse old days of Win32 and doesn't miss the oodles of lines of plumbing necessary just to perform tasks the framework allows him to do without writing a single line of code. He'll go "hack into" the framework if he absolutely has to but would much rather find a way to do things the "framework" way.

The Hacker and the Professional have a lot in common. They're both writing software. They'll often use many of the same tools (e.g., debuggers, profilers, tracers, IDEs, etc...) but their instincts will drive them to use them at different times and in different ways. The Professional will tend to think of the debugger as a way to identify what's wrong with his own code. The Hacker will tend to think of the debugger as a way to figure out what the framework is doing "under the hood"; a way to find the secret spot that he can use to plug-in and do things the way he thinks they should have been done in the first place.

Say what you will about Hackers a few things are, in my experience, pretty universal to them:

• They're very smart. Sometimes they're even geniuses.
• They like to write code.
• They have a strong sense of curiosity.

The big problem with the Hacker is that he tends to write code that is very difficult to maintain. Because the knowledge the Hacker gains by reverse engineering is usually very incomplete, often in ways germane to the way he intends to solve the problem, his solutions are plagued by seemingly inexplicable errors. His solutions work well 90% of the time then, for no known reason, crash or hang the other 10%. He might have been able to piece together what was going on "under the hood" at the time he was looking but is usually not able to reverse engineer all of the assumptions in effect.

Over time, the Hacker's hacks accumulate. He forgets the insight he had while stepping through the decompiled framework method. The next version of the framework changes whatever internal peculiarity allowed his hack to work causing it to stop working long after anyone even remembers the hack. As these hacks accumulate they start to impact the rate and quality of development. The rate slows down because making changes requires accommodating hacks that must first be deciphered. The quality goes down because the accumulated hacks interfere with each other.

Given this indictment of the Hacker, does the Professional do things any better? I'll cover that angle in the next "Hacker vs Professional" post.