Surveys say that despite the record unemployment in the U.S. and much of Europe, jobs are still going unfilled in software development. Hiring managers cite a shortage of qualified candidates. This makes it more important than ever for development organizations to hang on to the workers they’ve already got.

We know, however, that in general, developers tend not to stay in one place for long. Employee retention plans aren’t working.

If you’re in charge of a development team, you want to hang onto the workers you rely upon. So it’s important to understand why they leave. I found a good analysis of the factors leading to worker defection in an article at forbes.com. Author Eric Jackson lists 10 reasons large companies lose their top talent. I think most of the reasons apply to small and medium-size companies too.

First on Jackson’s list is bureaucracy. No one likes rules that must be followed just because they are rules. Jackson says that when an employee cites meaningless rules as the reason for their departure, what they really mean is that they had no voice in creating the rules. Nothing creates worker alienation faster than being shut out of decisions about how they do their work.

The second reason workers leave, Jackson says, is that they are assigned to projects that don’t interest them. We don’t often ask our employees which projects they’d like to work on. If we did, there’s a better chance they’d remain excited about the work and stay on the job.

Rushed, superficial performance reviews are third on Jackson’s list. When supervisors don’t take the time to do a thorough review of a worker’s performance, the worker can rightfully conclude that the company isn’t invested in helping him do a better job and advance. Rushed reviews show a real lack of regard for the employee’s future.

The employee’s future is key to Jackson’s fourth factor as well. Too few companies, Jackson says, take the trouble to engage employees in discussions about career advancement. To keep key employees, take the initiative in discussing career development. Outline how they can gain more authority, autonomy, and compensation with your company.

Fifth on Jackson’s list is shifting whims. When the company changes its course frequently, and is always pushing employees to commit to a new set of strategic priorities, workers understandably feel jerked around. Employees are happiest when the company has and maintains a long-term vision.

Next: feedback. Workers, even top workers, need to hear what they could be doing better. If nothing else, it demonstrates that managers are aware of who workers are and what they are doing.

Number seven: Get rid of the deadwood. Competent developers want to work with other competent developers, and the best developers enjoy the challenge of working with other superprogrammers. It’s hard to let underperforming employees go, but their continued presence takes a toll on the rest of the team. Hang on to the bad performers and after a while they may be all you have left.

In position number eight is vision. This should be obvious. Developers want to work in an organization that is pursuing a worthy vision, and they want to feel they have a role in defining the vision. Duh.

Jackson’s ninth factor is close-mindedness. Everyone wants to feel that decision-makers will listen to their suggestions. If you don’t listen, then the very workers who have the most to contribute will get tired of bumping their head against the wall and leave for greener pastures. You’ll be left with a department full of yes-men. Good luck with that.

Finally, Jackson says, some people leave because of bad middle managers. It’s not uncommon to find that certain managers lose team members at an accelerated rate. It could well be because they are bad managers. You can replace the manager or try to fix him, but you’ve got to do something or you’ll have a permanent talent drain in the manager’s department.

You can read Jackson’s article here. I thought it was a worthwhile list, and if nothing else, it reminded me that we all have to do a better job of hanging on to good workers.

Web recommendation: Could programmers make it possible to crowd-source government? That seems to be the central question behind this interesting CNN article and the TED presentation behind it. It’s an intriguing and unlikely thought. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He lost a couple hours to National Geographic videos of dangerous predators today. Those guys really know how to make nature interesting.

I got hooked on creative programming during the brief few years I sullied the field of software development by working as a programmer. I had a clear sense of myself as an explorer, taking the Apple II places no one had ever taken a computer before. The industry’s relatively few books on algorithms and program structure were unknown to me, and I wouldn’t have understood them if I had blundered into them. So I learned to program through trial and error, intuition and experimentation. When I got a program to run – and I’m still proud of the apps I created in those days – my triumph was both practical and artistic. For me, programming was a creative endeavor.

I know I’m not alone. I’ve talked to many programmers over the years, and the chance to exercise creativity is one of the things they like best about software development. Many of the developers I talk to these days are preoccupied with mundane tasks like extracting data from mainframe tables, formatting information for output, and patching up apps with new layers of data-security scaffolding. It doesn’t look like creative work to me, but I’ve learned that creativity refers to the way you approach a problem, not the problem itself. Even the most mundane task can be accomplished in a creative way (without compromising the quality of the app or missing the project deadline, of course) by practitioners who see themselves as creative.

I hope you see your work as an outlet for creativity. For me, that was what made programming satisfying. Yes, I liked solving users’ problems. I loved dismantling complex challenges and finding the inner simplicity. But mostly, I saw programming as a means of personal expression.

Everyone who works in a creative discipline draws a blank sometimes. It’s hard to be creative on demand, especially when facing tight deadlines.

Of course, little we do as developers is truly original. For me, at least, a creative triumph was when I arranged well-known, frequently used programming tropes in novel ways that met my application’s needs. You might call this “combinatorial creativity.”

I was reminded of this the other day when I happened upon a letter that Mark Twain once wrote to Helen Keller. It seems that Keller had been accused of plagiarism. The charge was dismissed, but Twain still had more than a little to say about it, and about creativity itself:

Oh, dear me, how unspeakably funny and owlishly idiotic and grotesque was that “plagiarism” farce! As if there was much of anything in any human utterance, oral or written, except plagiarism! The kernel, the soul – let us go further and say the substance, the bulk, the actual and valuable material of all human utterances – is plagiarism. For substantially all ideas are second-hand, consciously and unconsciously drawn from a million outside sources, and daily use by the garnerer with a pride and satisfaction born of the superstition that he originated them; whereas there is not a rag of originality about them anywhere except the little discoloration they get from his mental and moral calibre and his temperament, and which is revealed in characteristics of phrasing. When a great orator makes a great speech you are listening to ten centuries and ten thousand men – but we call it his speech, and really some exceedingly small portion of it is his. But not enough to signify. It is merely a Waterloo. It is Wellington's battle, in some degree, and we call it his; but there are others that contributed. It takes a thousand men to invent a telegraph, or a steam engine, or a phonograph, or a telephone or any other important thing – and the last man gets the credit and we forget the others. He added his little mite – that is all he did. These object lessons should teach us that ninety-nine parts of all things that proceed from the intellect are plagiarisms, pure and simple; and the lesson ought to make us modest. But nothing can do that.

Web recommendation: I know too much about how computers work to consider robots a threat to my life or my job. But this series of articles got me thinking. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He enjoys a shot of Scotch now and then.

The world is full of people eager to sell you training and specialized information. Authors and publishers push books. Consultants sell insights by the hour. Speakers at seminars and conferences promise to revolutionize your thinking.

I’m the last person to disparage these conduits to learning. Although it is commonplace to dismiss these teachers as principle-free cynics who craft their messages to the marketplace, my experience is that authors, consultants, and conference speakers believe in their messages, very often passionately. In general, the money you spend on this kind of information is money well-spent.

But when I say “become a better developer,” I’m not talking about picking up the basics of a new programming language, or learning how to manipulate a new set of APIs, or grokking how to craft code for a new platform, or seeing how to translate the principles of a development method into practice. Those are all examples of skill acquisition, and skill acquisition is a fine thing. But it doesn’t transform you. If you were a mediocre programmer before a course in Dart or F#, then at the end of the course you’ll be able to code in a mediocre fashion using a new language. You’ve acquired new skills, but you’re the same developer you always were, not a better one.

If you really want to be a better developer, you’ve got to look inward. Understanding lambda expressions may help you write better code in certain situations, but being a better developer is more likely to come from a higher level of commitment to quality, a greater level of enthusiasm, enhanced patience, and more willingness to communicate clearly, frequently, and honestly with colleagues.

That’s the secret behind McCarthy BootCamp, which I wrote about recently. Jim and Michele McCarthy don’t teach any programming tricks during their training sessions (what they teach, mostly, is personal integrity – but you didn’t hear that from me). Nonetheless, they may just help you become a better developer.

Because when it comes right down to it, there’s just one way to be a better software developer: Become a better person, and then develop software in whatever way seems most natural.

Web recommendation: This is an amusing slide show that does a pretty good job of displacing a belief that is held by no one I know. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He’s done some simple home repairs today and is feeling uncommonly proto-masculine.

While I wasn’t looking, the discipline of user-interface design was rechristened UX – short, I am led to believe, for “user experience.” That seems a rather breathless, wide-eyed way of referring to the shopworn set of UI widgets that let users interact with applications, but so be it. The user-interface world has always suffered from stylish overstatement.

UX is like the weather. You know – everybody talks about it, but no one does anything about it. Every programmer I meet is a self-appointed expert in how user interfaces ought to work. Everyone seems to know what makes an application “user-friendly” or “compelling” or whatever the current buzzword is.

At the same time, our industry perpetuates the myth that it supports a specialized cadre of user-interface designers, UX engineers with deep insights into user expectations and how to meet or exceed them. But tell me: Have you ever met a UX engineer? I haven’t. I think there may be some at Microsoft, behind the scenes of the Windows 8 team. Everywhere else, UX is understood to be part of the software developer’s expertise.

There seems to be a conflict between best practices in user-interface design and the incremental development processes we call Agile. UI designers count pixels and specify detailed behavior of program elements before they are coded. The entire discipline of UX depends upon a waterfallish development model in which the UI takes emerges as vapor from the analysis phase and solidifies into the spec during design. Remember analysis and design?

Bloggers across the Internet have attempted to downplay the incompatibility between traditional UI design and modern development models. You can read some of these articles here: InfoDesign, Interfaith Marriage: Experience Design Meets Agile Development (by my old friend Larry Constantine), The Integration of User Experience into Software Development, Great User Experiences Require Great Front-End Development, Twelve Emerging Best Practices for Adding UX Work to Agile Development, Agile + UX: Six Strategies for More Agile User Experience, Adapting Usability Investigations for Agile User-Centered Design, Bringing User-Centered Design to the Agile Environment, Design Chunking with Scrum…well, you get the idea. There’s a lot out there.

What strikes me about all these articles is that the programmers and writers and designers who created them are so earnest and eager to stand on one foot, and squint, and somehow make a discrete UI-design phase seem like a natural part of the Agile development model, not something that is awkwardly grafted on.

UI design has always been a black art. If you’re old enough, you may remember the Babel of UIs every user had to deal with before Microsoft and Apple imposed UI standards on developers. If you think things are bad now, you’ve got a short memory. UIs were really dreadful, then. And while there is always room for improvement, standard UI widgets and guidelines have done a lot to make the user experience – er, the UX, I guess – significantly better.

Web recommendation: Intel introduced the 2012 version of its SDK for OpenCL a few days ago. If you’re looking to write parallel code that takes advantage of the GPU for compute-intensive tasks, this is a significant release. You can learn more about the SDK – and download a copy free – here. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. His lovely wife is uncharacteristically impatient for him to finish work today.

Let’s start with the basics, shall we?

We are not born with the ability to write software. Though most agree that individuals have different potential for success in the coding craft, the undeniable truth is that we all start at square zero (or square one in Cobol, Fortran, RP/G, and Smalltalk). What I’m trying to say is this: Programming is an acquired skill. It’s something we learn.

The pioneers who were the industry’s first programmers were self-taught. They had to be – there was no one to teach them how programming ought to be done except, perhaps, Ada Lovelace. So we have all these computers, and none of them are a bit of good without software. To get software, we must have programmers. And to have programmers, we must teach programming.

That’s where it gets tricky. As I have observed here, here, here, here, here, here, and here, and as the Editorial Board of SD Times has somewhat belatedly agreed here, there is a fundamental mismatch between the skillset embodied in the typical university computer-science program and the actual skills required by actual programmers doing actual jobs. The widely acknowledged mismatch between the university curriculum and the real-world needs of IT departments has been the source of a great deal of anguished hand-wringing, as manifested in the hyperlinks I listed a couple sentences back.

Let’s back up a step. Our lamentation over the skills gap is based on an unexamined premise: the notion that the mission of colleges and universities is to prepare students for employment. This premise is not universally accepted. Among the places it is most critically rejected is within colleges and universities. The idealism at the core of the American educational system is that colleges and universities exist in order to imbue students with educations. They set up elaborate programs of required subjects (music appreciation for those seeking degrees in electrical engineering, for example) and struggle to ensure that students are well-grounded in the history of ideas and developments both outside and within their majors. The unquestioned premise at the core of the university system is that knowledge is valuable in and of itself, without regard to whether it makes the student more employable.

The gap between the computer science curriculum and the needs of IT is therefore a manifestation of a fundamental question. Are universities primarily intended to prepare graduates to make a living, or do they have the luxury of preparing graduates to live more fulfilling lives by exposing them to our culture’s history of important ideas and events?

As public funding for education falls, as unemployment rates rise, universities are coming under increasing pressure to serve as trade schools – though, as they point out, there is no shortage of for-profit two- and four-year trade schools for those who value employability over wisdom.

Our field has always had an ambiguous relationship with academia. Most employers don’t require a degree. Our best practitioners sometimes lack academic credentials. A substantial list of accomplishments and a presence on GitHub are more likely to impress enlightened HR departments than a university degree. And rightly so.

When we say that universities are not doing a good job of preparing graduates for software development’s on-the-job challenges, we are echoing practitioners in many other fields. The point is, preparing graduates for work is not what universities are for.

Web recommendation: This blog post is fascinating reading. It is, of course, just one developer’s point of view. But it’s a coherent, nuanced view, and I think the writer has some valid insights. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. At this very moment he is sitting in an outdoor café, listening to a young woman hum “Yellow Submarine” as she rocks a baby in her lap.

Have you read the 501 Manifesto? It’s a short document – it’s posted online here – that is obviously based on the Agile Manifesto. It’s a statement of principles asserting that developers should have lives aside from coding. Here’s the core of it:

We are software developers who take pride in our work but choose not to be wholly defined by it.

As such, we are proud to say that we value:

• Our families over the commercial goals of business owners
• Free time over free snacks
• Living our lives over maintaining our personal brands
• Sustainable pace over muscle-man heroics
• Our personal creative projects over commercial products the world doesn't need
• Having money for nice clothes over getting free t-shirts from Microsoft
• Playing fußball in the pub with our friends over playing fußball in the office with our team leader
• Not being a dick over being a rockstar

That is to say, we value the things on the left more than we value the things on the right. And some of the things on the right aren't even on our radar.

The manifesto is named, it turns out, for those programmers who save their work, log off, and leave work at 5:01. They go home to spend time with their families while their colleagues – you know, the ones described in decades of stereotypes – keep cutting code far into the night.

The 501 Manifesto is mostly just good old common sense. We work to make the rest of our lives possible. If we’re lucky, we are also able to wrest some enjoyment out of our work. Yes, extra hours are sometimes required, and sometimes we get caught up in hack-attacks and the hours fly by until we look up and find we are still hacking past midnight. But yeah, we get it. We work to support our real lives.

Nonetheless, the manifesto has generated substantial controversy on the Internet. Some programmers resent being described, albeit indirectly, in terms of stereotypes. Others object to individual words or statements, usually found in the text framing the manifesto proper. And of course, the Internet being the Internet, some folks object just because they’re in the mood to be ornery.

I’m sympathetic to the 501 Manifesto. It’s too bad, I think, that the authors felt the need to spell out such commonplace facts of life.

The 501 Coder blog is worth visiting, if you’re interested in an overview of the arguments for and against the manifesto. There’s some interesting stuff there, including this insight into the state of nerdiness: “Here’s the thing: If you are truly a nerd, being pitied by ‘normals’ is part of the deal. It’s not contempt, don’t mistake it for that. It’s a kind of rather-you-than-me bewilderment at your current life choices. That’s all.”

Web recommendation: The Internet keeps spawning new ways of representing information. The Newton Tour was a new one for me. I like the idea behind this project, and I foresee a day when such data might be collated on-demand, for arbitrary individuals, based on Web-scouring strategies. Why not? J.D. says check ’em out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is looking forward to seeing a six-year-old take a look through binoculars for the first time.

I can’t deny it, I’m officially skeptical about the Agile movement. I’ve read a lot about the Agile philosophy over the past year, and the more I read, the less impressed I am. The principles at the heart of the Agile movement are self-evidently true, but they haven’t translated into a useful set of methods for software developers. As I wrote in a recent post: “This is no way to achieve reliable, repeatable results. It’s de-evolution in action, a return to the days of late-night hack attacks and reliance upon the heroic contributions of uncommonly talented superprogrammers.”

I admit that my criticism is disarmed somewhat by the first statement of values listed in the Agile Manifesto itself: “Individuals and interactions over processes and tools.” So Agile hasn’t translated into a heavyweight set of processes and tools? Duh. It was never intended to.

This morning I stumbled across a blog post titled “Agile is a Sham.” It turned out to be an invitation to an entire subculture of Agile skeptics. I thought for a moment I’d found some likeminded thinkers to spend time with. But I couldn’t have been more wrong.

I followed a trail of hyperlinks that led me through a rising tide of dissatisfaction with Agile. Not because Agile hasn’t translated into a useful set of tools and methods, but because Agile imposes too many tools and methods! This anti-Agile community is contemptuous of consultants and trainers who are peddling Agile as the new cure for everything that ails software development. They despise Scrum and Kanban as snake oil that managers buy in an effort to get superior results out of inferior programmers. The blogger summed up his Agile skepticism like this: “[T]he success of the coding part of a project is dependent on the caliber of the engineers doing that coding and not the process they follow.”

Maybe you recognize this line of thought. Our industry is vulnerable to a very unattractive elitism. There are lots of programmers who believe the best way to ship working software on time is to get rid of onerous development processes, sweep out the below-average programmers, and leave the competent programmers – the real engineers – alone to do their thing. Just get out of their way, and they’ll make the code work.

Members of this elitist clan argue that project management and formal development processes are just obstacles to their brilliance: “If you need to follow a prescribed process in order to be in any way an effective coder then you are mediocre at best and so is your work and your project.  If you are a good programmer, you know yourself how to communicate and create your ‘process’ as you go along.”

I find myself totally at odds with this community of Agile skeptics. Their cynicism about Agile consultants and trainers is probably well-founded, I admit. But their insistence that raw competence makes methods obsolete – and that reliance on methods is an admission of mediocrity – sets my teeth on edge. Their elitist views are demonstrably incorrect and patently offensive.

So yes, I’m an Agile skeptic. But I’m the other kind – the kind that would like to see Agile principles instantiated in more discipline, more tools, and more methods. That’s how you translate good ideas into working processes. Our field needs more discipline, not less.

Web recommendation: The open-source Go language has achieved a milestone: Version 1.0 is now available. I like what I read about this lightweight language. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is hopeful that a certain video clip he saw today, featuring himself, never reaches the public eye.

If you haven’t noticed, software development faces some severe challenges right now. Serious problems face us – not in some hypothetical time-frame, but right now. And I am sorry to say that the tool vendors and thought leaders we count on are letting us down. They’re focusing on other matters entirely.

I don’t mean to pick on Java, but that’s the example that comes to mind. A year from now Java will support lambda expressions. The big brains directing the evolution of what is arguably the industry’s most important language surveyed the computing terrain and decided the best thing they could do for developers was add syntax for lambda expressions.

Are you freaking kidding me?

To be fair, the Java team is also grafting modularity-control features onto Java via the Jigsaw project, and these features could be of real benefit to Java programmers. But still…lambda expressions?

Here are the things we developers ought to be focusing on.

Security

Yes, I know. I’m a broken record on this issue. But we don’t face a bigger challenge, and the days of shrugging off security as an operations concern are over. If you have a wireless router in your company, a bad guy in your parking lot can have access to your network in two or three hours using off-the-shelf tools and a $350 laptop.

You think your firewalls and strong passwords are sufficient protection? You’re dreaming. The U.S. military spends billions defending its servers, and last week it told the Senate Armed Services Committee that these security measures have failed. The military now assumes that hostile forces have network access, and it is shifting its focus from controlling access to protecting data. “[W]e have to go to a model where we assume that the adversary is in our networks,” said Dr. James Peery, director of the Information Systems Analysis Center at Sandia National Laboratories. “It’s on our machines, and we’ve got to operate anyway.” Anonymous has demonstrated it can compromise pretty much anyone it targets. If your network hasn’t been compromised yet, it’s because the bad guys haven’t selected you as a target yet. When they do, your security measures will fail.

This isn’t just an operations problem. It’s everyone’s problem.

Development processes

The Agile movement is popular – and why not? It’s a feel-good set of aesthetic principles unencumbered by a development process. XP, Scrum, and Kanban let us throw off the chains of heavyweight development methods and get back to coding.

This is no way to achieve reliable, repeatable results. It’s de-evolution in action, a return to the days of late-night hack attacks and reliance upon the heroic contributions of uncommonly talented superprogrammers. Too many companies are betting their futures on this family of untested, unproven non-methods.

CASE tools and formal methods were no fun – I get that. They sacrificed flexibility and improvisation and even personal fulfillment for reliable, repeatable results. They weren’t the fastest way or the most enjoyable to get from Point A to Point B, but they did guarantee you’d get there. You can’t say that about Scrum.

Platform fragmentation

It was a big deal when we went from building Windows apps to building net-enabled apps that split program logic along the well-established seam between lightweight clients and back-end servers. But that was nothing. In the very near future, we’ll be asked to deliver apps that run properly on arbitrary hardware with dramatically varying specs, all running different operating systems. It’s an unprecedented challenge for the software development community.

The traditional approach has been for IT to set up a list of approved hardware and software platforms, and thereby to limit the demands on application developers. But that discipline has broken down. You can’t keep your company’s workforce from bringing in their new tablets and smartphones, and from demanding that these devices be given access to corporate apps. The security concerns alone are daunting – how do you keep your network secure when the CEO misplaces his iPad in an airport lounge on another continent?

And don’t get me started on cloud computing. The security implications alone should give you serious pause. Rearchitecting your apps may not take as long as you fear, but the split between your resources and your cloud vendor’s servers will remain brittle. I lived in San Francisco long enough to know you don’t build something important on a fault line.

Inadequate tools

If I read the surveys correctly, you probably don’t remember the transition from DOS programming to Windows. I remember it well – I was at the heart of it. The programming tools and languages that had served us well in the single-tasking, character-mode environment were inadequate to the demands of GUI programming. The industry responded with visual programming environments, object-oriented programming languages, application frameworks, and plug-in reusable modules. Eventually these tools allowed us to cut the challenges of Windows programming down to size.

What tools and languages are addressing today’s challenges? Python? Ruby? C#? Honestly, they all seem to be addressing niche problems. It seems to me we’re being sent into this battle empty-handed. Or am I missing something?

Yes, these state-of-the industry rants are supposed to be posted in December or January. I’ve broken one of the unwritten laws of tech bloggers, and the authorities will no doubt crack down on me. But I had to get this off my chest.

Am I the only one who has noticed that we’re in deep, deep trouble?

Web recommendation: The evocative phrase “Internet of Things” always catches my attention. Here’s a rare substantive discussion of what the term refers to, by Google’s Vinton G. Cerf, a U.S. Medal of Technology recipient, ACM Turing Award winner, Japan Award winner, etc., etc. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He has rediscovered the joy of peanut butter.

The ancient Greeks had no computers, but their mythology includes one figure who embodies an unchanging reality of programming. I am speaking, of course, of Tantalus. You may remember the story.

Tantalus was a king of Phrygia, in what we now call Turkey. He was a friend of the Greek gods until he offended them. Then he received a distinctive punishment. In the underworld, Tantalus was tied to a fruit tree that stood in a river. When he reached for a fruit, the branches would rise until the intended meal was just beyond his grasp. When he leaned down to get a drink, the water receded. Tantalus was sentenced to eternal hunger and thirst, and the objects of his longing were forever just out of reach.

So it is with programmers. But instead of fruit and water, we hunger for reusability.

You don't have to write code very long to realize that almost all of what we do has been done before. We are constantly solving problems that have been solved over and over again in the past. We write boilerplate code and reinstantiate well-known algorithms.

We would be more productive – and our days more enjoyable – if we weren't always reinventing the wheel. What if we could bundle code for reuse? Then we could incorporate algorithms and boilerplate code by reference instead of tediously rewriting it with each new app.

This dream has been the motivation behind many of the technologies that support our industry. Off the top of my head, the list includes these:

• Unix command-line utilities

• system libraries

• function libraries

• class libraries

• dynamic link libraries

• subroutines

• object-oriented programming

• package-oriented languages like Ada and Modula-2

• code generators and CASE tools

• component-based development

• visual programming

• use cases and design patterns

• application frameworks

• service-oriented architectures

It's a rare project that doesn't make use of several of these technologies. But the software we write can still be characterized as ad hoc. We continue to reinvent the wheel with each new project.

I don't have a solution to offer – the technologies on the list, plus, no doubt, others that escaped my attention – should be sufficient. The benefits of reuse are obvious. Yet software reuse remains maddeningly – tantalizingly – out of reach.

Web recommendation: Imitation is a form of reuse, I guess. And they say (do they still say?) it is the sincerest form of flattery. If so, the good folks at Apple must feel awfully flattered these days, due to this effect. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is grateful for the days Internet service, water delivery, and electricity are all uninterrupted.

It was the French writer and philosopher Voltaire who observed, “The perfect is the enemy of the good.”

(Actually, since Voltaire was French, what he really observed was, “Le mieux est l'ennemi du bien.” It's a line from a 1772 poem called “La Bégueule.” He apparently meant “the better” or “the best,” not “the perfect,” but the aphorism survives in English as “the perfect is the enemy of the good.” People quote the line all the time without thinking too hard, I imagine, about its literal meaning.)

In the software development world, we quote Voltaire when we warn of the dangers of analysis paralysis: the pernicious state of spending so much time and energy on preparations that we never actually begin coding. It arises, I think, from the fear of making a wrong move, which is a fine fear to have, in moderation.

The aphorism also applies to the malady of elegance creep or gold-plating. Blogger Rick Kotermanski says that's what you get when you blow deadlines by addressing implied or unwritten nonfunctional requirements. “I have frequently found cases where programmers have built exotic and academically intriguing solutions to problems that don't really exist,” Kotermanski writes, “usually under the guise of addressing an anticipated but never realized performance or extensibility problem.” To which I can say only, “Um...guilty as charged.”

A thoughtful fellow named Vesa Palmu has observed, as many have, that an unbridled quest for perfection can lead to endless costs and delays. He's summarized the problem with a little chart that shows the relationship between value and development cost, and identifies an optimal zone in which costs are incurred most efficiently:

Obviously, the solution to perfectionism is to stop working when the code is good enough. But that's devilishly hard to do – coding isn't a “good enough” profession, in my experience. Palmu says, “I'm not trying to say it's good to lower your standards, not care about the visuals, or to cut back on quality. I'm just saying that you should know what is really important and pay attention to what your definition of done is.”

(A good many discussions of the perils of perfection wind up citing the Pareto Principle, the so-called 80-20 rule. The rule suggests that achieving the last 20 percent of perfection consumes 80 percent of your overall effort and budget. This insight sounds as if it ought to be true, and is constantly cited as if it were established fact, but to my knowledge it has never been tested in software development.)

The Buddhist notion of dukkha is helpful in any analysis of perfection. Buddhists accept as a core belief the idea that because the world is constantly changing, it cannot be in a state of perfection. As I understand it, dukkha is both the essential imperfection of life and the soul's inevitable dissatisfaction with the state of the universe.

I think the Japanese Zen concept of wabi-sabi, which is built upon a Buddhist foundation that includes dukkha, is likewise worth exploring. Wabi-sabi is an aesthetic principle. Wikipedia (which I am not above quoting when I am on deadline) says wabi-sabi ideals include “asymmetry, asperity, simplicity, economy, modesty, intimacy, and appreciation of the ingenuous integrity of natural objects and processes.” The site quotes Richard R. Powell, author of a 2004 book on wabi-sabi, who says the aesthetic acknowledges three simple realities: “nothing lasts, nothing is finished, and nothing is perfect.” It seems the Zen masters anticipated the Agile movement by several centuries.

The tendency toward perfectionism is natural in human beings and, I think, pretty much universal in programmers. Compilers reinforce our natural tendency to pay attention to the tiniest of details.

Ward Cunningham coined the term “technical debt” to refer to imperfections in source code. Speaking at OOPSLA in 1992, Cunningham said: “Shipping first-time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite...Every minute spent on not-quite-right code counts as interest on that debt.”

Others have expanded on Cunningham's acute observation. Many note that just as businesses justifiably and deliberately take on debt to meet major goals, so a development team might reasonably incur technical debt in the service of a sufficiently high priority. This trade-off is implicit in Agile development.

At this point, I should be offering an assessment to help you identify whether you have succumbed to perfectionism on the job, plus a few tips on overcoming the malady. That's what the architecture of the magazine article requires. (Despite making hundreds of blog posts over the years, I don't understand the structure of the blog post, and I keep writing magazine articles. TL;DR, as they say in the online communities.)

But after hours of online research, I have no advice to offer. Articles that promise to help readers overcome perfectionism are packed with touchy-feely nonsense. Tech sites advise developers to weigh the costs of shipping a few bugs versus the cost of fixing them – advice that is simultaneously impossible to follow in the real world, wrong-headed, and almost certain to transform the developer's satisfying career into an unrewarding hell on earth.

Even the fabulously smart Joel Spolsky, whose Joel on Software I crib from when the rest of the Internet lets me down, backed down when he addressed this topic. After building a rigorous demonstration of why it makes economic sense to let certain imperfections remain in a code base, he closed with these weasel words: “With all that said, I'm optimistic at heart, and I believe that there is a lot of hidden value to producing very high quality products...So I tend to err on the side of quality (indeed, we fixed every known bug in FogBUGZ, not just the big bang ones) and take pride in that...”

Given an overwhelming absence of useful advice on this subject, I'll hazard one original thought. I think perfectionism is natural, inevitable, and desirable in software developers. Coders should always include perfectionism as part of their motivation for learning, exploring alternatives, tinkering, debugging, and rethinking. The tendency to pursue perfection makes developers more valuable.

Projects go astray, however, when that tendency is given free rein. For that reason, managers – not developers – must cultivate a big-picture perspective so they can call a halt to endless development cycles that no longer justify their cost.

I realize I am merely shifting the burden of combating perfectionism to someone else, not solving it. But the bottom line is that this is precisely the sort of task management was invented to perform. We count on managers to monitor the big picture and ship the code when it's time. Evaluating the trade-off between costs and benefits is what managers are good for.

Web recommendation: Larry O'Brien and I continue to poke holes in the persistent notion that our industry has, and needs, superprogrammers. My experience is that many excellent coders suffer from personality traits – elitism, dismissal of others' talents, endless self-congratulatory back-patting – that would make me avoid working with them even if they were as good as they thought they were. Development-industry traditions make the situation worse, perpetuating the myths and lionizing the arrogant. One harmful bit of intellectual scaffolding that supports this persistent notion is the Dreyfus Model of Skill Acquisition. The Dreyfus Model doesn't explain skill acquisition at all. Its purpose is to classify learners into five slots, from Novice to Expert. According to the model, once you've attained the Expert level, rules no longer apply to you. You're a master of improvisation and your hunches are more valuable than weeks of earnest coding by those who follow the rules. Blah blah blah. This stuff is both offensive and wrong-headed. As it turns out, the text in which the Dreyfus brothers summarize their research is mostly supportable. It seems the model has been misused since then to excuse self-aggrandizement. The original Dreyfus paper is worth reading. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He spent much longer writing this post than he expected to.