We are often called upon to estimate how much time or money it will take to develop an application, add a feature, or eradicate a bug. This is such a common and important occurrence in software development that researchers have attempted to create methods, guidelines, and tools to help us render accurate estimates. Perhaps you’ve used COCOMO, planning poker, or story points.

The sad truth is that none of these methods consistently yield accurate estimates. It turns out that the best estimates come when you turn to the most experienced coder in the room and ask what he thinks. An IEEE research review concludes that the expert will underestimate the required time and effort in 60 to 80 percent of cases, and his estimate will likely be about 30 percent optimistic…but none of the formal methods reliably deliver better estimates.

Estimation is hard because we have to combine information from all four quadrants of the knowledge/ignorance graph.

First is the information we know that we know. Then there’s the information that we know we don’t know. Then there’s information we don’t know that we know. And finally, there’s information that we don’t know we don’t know.

Each of these kinds of information contributes to software cost, time, and effort estimates in a deliciously different way.

It’s no wonder our estimates are so reliably inaccurate.

Web recommendation: It was with a strong blow to the gut that I learned today of the death of poet Adrienne Rich. It was engagement with her work that first opened poetry to me. I drew upon that illumination all through my years at Columbia as I pursued my degree in English and comparative literature. Here are three articles that will, I hope, incite you to search out the work of this incomparable writer. 1, 2, 3. J.D. says check ’em out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He spent an idyllic hour sitting in the sun outside a café this afternoon.

The best conference presentation I ever attended was at the Software Development Conference in 1989. I was editor of Computer Language magazine, the sponsor of the conference, and I had major responsibility for signing up speakers and assigning topics for the keynotes and technical sessions.

Larry Constantine spoke at our conferences, and Grady Booch. Bjarne Stroustrup and P.J. Plauger and Adele Goldberg. Bill Gates spoke several times. Ed Yourdon was a regular. Our conference attracted the industry’s best-known thinkers.

The session I’m talking about wasn’t delivered by one of software development’s legendary figures, but by a technical manager from The Whitewater Group, a small company that published an object-oriented programming language called Actor. Jim McCarthy had proposed a seminar on project management and slipped deadlines that he called “Slipping without Falling.”

Jim’s talk was both extremely entertaining and extraordinarily insightful. Drawing on his experiences in project and process management, he delivered a series of memorable, colorful aphorisms, each of which expressed in shorthand a profound truth about the messy process of shipping software. Jim focused on techniques for building healthy teams, a black art that is still almost universally mismanaged.

One of Jim’s most memorable bits of advice was this: “Beware of the guy in a room.” Many projects, he explained, have at their core a challenging bit of code that is assigned to the development team’s acknowledged superstar. This programmer takes custody of the crucial module and retreats to his office. The team spends the next weeks tiptoeing past his door so they don’t interrupt his intense focus. No one asks for forecasts, bug-tracking data, or progress reports. The team knows that its success hinges on this single programmer and his brilliance. His code will be ready when it’s ready.

This dynamic is common and pathological. It is anathema to teamwork. The pattern reinforces harmful stereotypes about the relative value of programmers and superprogrammers. It reduces meeting deadlines to a game of chance. As McCarthy said, “the results are uniformly fatal to the professional development organization.”

McCarthy’s observation hit home. I had seen organizations held hostage by a guy in a room. Worse, I recognized that on a few occasions, I had been the guy in a room. I had helped erode healthy teamwork in companies where I had taken sole custody of creative challenges and everyone else just waited for me to deliver.

Jim’s insights into team leadership led him to Microsoft, where he led the team that developed Visual C++ – a team that is legendary for its productivity and innovations. He expanded his Software Development Conference talk into a book called Dynamics of Software Development, which I commend to you as essential reading. Jim now runs The McCarthy Show, which offers leadership training. If you visit the Web site, which you should, be sure to check out the Episodes page, which currently offers 115 podcasts on elements of McCarthy’s leadership principles.

Curiously, Jim’s warning about the danger of “a guy in a room” is not unique in our industry’s literature. In 1971, Gerald Weinberg wrote an extraordinary book called The Psychology of Computer Programming. Like McCarthy, Weinberg focused on effective teams. His book included a short section called “The Ten Commandments of Egoless Programming,” which listed rules for functional teams. Item number nine on the list was this: “Don't be ‘the guy in the room.’ Don't be the guy coding in the dark office emerging only to buy cola. The guy in the room is out of touch, out of sight, and out of control and has no place in an open, collaborative environment.”

I don’t know if Jim picked up the “guy in a room” concept from Weinberg or it occurred to him independently, and I don’t really care. It’s an important insight with wide applicability.

Does your team rely on the efforts of a guy in a room? Are you the guy?

Web recommendation: McCarthy’s still got it! You can catch a video of his Agile 2008 presentation, “11 Commitments for a Shared Vision,” by clicking here. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He had such a nice day today.

The ancient Greeks had no computers, but their mythology includes one figure who embodies an unchanging reality of programming. I am speaking, of course, of Tantalus. You may remember the story.

Tantalus was a king of Phrygia, in what we now call Turkey. He was a friend of the Greek gods until he offended them. Then he received a distinctive punishment. In the underworld, Tantalus was tied to a fruit tree that stood in a river. When he reached for a fruit, the branches would rise until the intended meal was just beyond his grasp. When he leaned down to get a drink, the water receded. Tantalus was sentenced to eternal hunger and thirst, and the objects of his longing were forever just out of reach.

So it is with programmers. But instead of fruit and water, we hunger for reusability.

You don't have to write code very long to realize that almost all of what we do has been done before. We are constantly solving problems that have been solved over and over again in the past. We write boilerplate code and reinstantiate well-known algorithms.

We would be more productive – and our days more enjoyable – if we weren't always reinventing the wheel. What if we could bundle code for reuse? Then we could incorporate algorithms and boilerplate code by reference instead of tediously rewriting it with each new app.

This dream has been the motivation behind many of the technologies that support our industry. Off the top of my head, the list includes these:

• Unix command-line utilities

• system libraries

• function libraries

• class libraries

• dynamic link libraries

• subroutines

• object-oriented programming

• package-oriented languages like Ada and Modula-2

• code generators and CASE tools

• component-based development

• visual programming

• use cases and design patterns

• application frameworks

• service-oriented architectures

It's a rare project that doesn't make use of several of these technologies. But the software we write can still be characterized as ad hoc. We continue to reinvent the wheel with each new project.

I don't have a solution to offer – the technologies on the list, plus, no doubt, others that escaped my attention – should be sufficient. The benefits of reuse are obvious. Yet software reuse remains maddeningly – tantalizingly – out of reach.

Web recommendation: Imitation is a form of reuse, I guess. And they say (do they still say?) it is the sincerest form of flattery. If so, the good folks at Apple must feel awfully flattered these days, due to this effect. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is grateful for the days Internet service, water delivery, and electricity are all uninterrupted.

We all know that women are a minority in software development. This bothers us because it's not consistent with how we view the development world. In the ideal profession that exists in our minds:

• All coders have an equal opportunity to be hired – all that matters is the quality of their work.

• Once on the job, developers are treated fairly regardless of gender, race, and other factors – all that matters is the quality of their work.

• Compensation and advancement are awarded fairly in proportion to developers' contributions – all that matters is the quality of their work.

In other words, we view the world of software development as an open meritocracy.

The truth is more complicated. White men are hired disproportionately over other candidates with equal qualifications. Women programmers are widely subjected to discrimination and hostile work environments. And women are under-compensated and under-promoted compared to equally qualified men. We don't want to believe these things, but study after study confirms them. These are the facts.

According to the Labor Department's Bureau of Labor Statistics, women make up 58 percent of the workforce, but only 25 percent of the computing workforce. Between 2000 and 2009, the the percentage of women in the computing workforce dropped 13 percent, while the percentage of men in computing increased 11 percent.

A study by the London Business School revealed that software development teams are most innovative and efficient when they are composed of 50 percent men and 50 percent women. Any other ratio yields lower innovation and lower efficiency. Yet women remain under-represented.

Most of us are uncomfortable with these truths. We'd like to alter the reality to make it conform with our ideals.

Initiatives to open our field to full participation by women have largely been of two kinds.

Some address the “pipeline problem.” The idea is that women are under-represented in high-tech careers because our colleges and universities prepare too few qualified female candidates. Proposed solutions to the pipeline problem can be very inventive. Some activists start with research showing that boys and girls accept gender roles as toddlers, and advocate a wholesale revolution in the way society regards femininity. More commonly, activists attack the pipeline problem by creating incentives for young women to choose technical majors in college.

The second major focus has been networking. Women in tech have formed myriad organizations to provide information and support to other women. These peer-support organizations have attracted high-profile advocates, women who have beaten the odds and ascended to executive positions in high-tech firms.

Neither of these approaches can eliminate the systematic exclusion of women from an equal role in software development. To welcome women into our field – and make our teams more innovative and efficient as a happy result – we must start by accepting the truth about the barriers that currently exist. Then we must change the world. By changing ourselves.

Web recommendation: Unlikely, creative, outstanding nerdy projects! I love this site. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He woke up unaccountably happy this morning. Don't you love it when that happens?

Job burnout is very common and more serious than you probably think. Psychologists say most people suffer from at least a mild form of burnout at some point in their lives. It can have severe consequences...you shouldn't take it lightly.

How do you know you're burnt out? The Mayo Clinic says you should ask yourself these questions:

• Have you become cynical or critical at work?

• Do you drag yourself to work and have trouble getting started once you arrive?

• Have you become irritable or impatient with co-workers, customers or clients?

• Do you lack the energy to be consistently productive?

• Do you lack satisfaction from your achievements?

• Do you feel disillusioned about your job?

• Are you using food, drugs or alcohol to feel better or to simply not feel?

• Have your sleep habits or appetite changed?

• Are you troubled by unexplained headaches, backaches or other physical complaints?

You don't have to answer affirmatively to all of these questions, or even to most. Answering yes to even one symptom could be a sign of on-the-job burnout. (It could also point to depression, or even a thyroid disorder. So you should probably see a doctor.)

The factors that cause burnout are familiar to most programmers:

Excessive workload: You are overwhelmed by the excessive amounts of work you must manage on a daily basis without sufficient downtime.

Lack of personal control: If you don't have the authority to decide what needs to be done and how to do it, you're at risk. Everyone needs some flexibility to do their work the way they want to.

Lack of recognition: If your work isn't adequately valued or acknowledged, you're at risk of burnout. There are far too many managers who give only negative feedback.

Role ambiguity: Managers in our field don't have the best reputation regarding clear expectations. Many just wave their hands and expect you to figure out what needs doing and how to do it. If it's not clear what is expected of you, you may be at risk of burnout.

Limited opportunities for advancement: No one wants to feel that he'll spend his whole life on the same treadmill.

Lack of teamwork: If your team doesn't work together well, that may contribute to burnout. Is the load shared unevenly or unfairly? Do you always get the boring part of the job? Are you undermined by your colleagues, or criticized unfairly?

Uninteresting work: Are you interested in the work you do? Many of us get into the field because we love the challenge of solving algorithmic puzzles. No doubt you have noticed by now that this is a very small part of the job. Are the tools you use and the applications you write a good fit for you? If not, you may be at risk of burnout.

Extreme pace: Does your team always work at a breakneck pace? Or is it a sleepy environment with too few real challenges? Either one can contribute to burnout.

Poor corporate values: Do you believe your company is essentially unethical? If your job is to contribute to the success of a company that isn't aligned with your values, you may be vulnerable to burnout.

If you're burned out, you may suffer from insomnia, stress, deteriorating personal relationships, anxiety, fatigue, depression, and alcohol or substance abuse.

What's to be done? Well, you've got two options. You can endure the burnout and hope it goes away, or you can address the issues. If you decide to take action, you've got to change the environment or change yourself – probably a little of both. The Mayo Clinic offers this advice:

First, evaluate your situation and identify the factors that are contributing to burnout. You can't solve the problem until you define it – every programmer knows this. (And every programmer I've ever known forgets it regularly, but that's a story for another day.)

Next, figure out if your job can be restructured to make it more engaging and rewarding. Your manager may be a resource. You might choose to work at home part-time, to switch to more flexible hours, to shift your responsibilities, to take night classes, or to act as a mentor to a newcomer. Be creative – you may come up with a solution that benefits the company while making your work life more tolerable.

Don't forget to do some work on your attitude. Remind yourself why you got into programming in the first place. Reward yourself with short breaks. Spend time talking with colleagues. Combat cynicism.

Reach out for support. Family members, colleagues, and friends are likely to be supportive if you tell them what's going on. Your doctor may be able to hook you up with a support group for burnout victims. Your company may offer confidential counseling services. Don't hesitate to tap these resources.

Finally, do a serious self-inventory. Are you in the right job? Are you even in the right field? Maybe you want to work part-time while working toward a law degree. Or turn down overtime so you can finally finish that novel.

If you need to make a change, make it. You only get one life, so you've got to make the best of it while you can.

Good luck.

Web recommendation: This video made me laugh out loud! J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is no stranger to burnout, but just now he's doing fine.

Statistics tell me that I get lots of extra readers when I write about Agile development. And why not? The Agile movement is the most interesting trend in software development right now.

I would argue that coding for multicore processors is at least as important as making a move to Agile. So are testing methods and security, security,security. But these are technical problems, not philosophical paradigm shifts.

The problem with Agile is that it isn't really a development method, or even a philosophy. It's an aesthetic. It's a system of values. Heck, the document that started it all, the Agile Manifesto, is written as a statement of values. The Agile movement is little more than a statement of values. The values are intended to serve as a touchstone for developers as they make decisions about how to build teams and applications.

So the modern approach to software development is based on a the value system outlined in the Agile Manifesto. So if we're to evaluate the state-of-the-art in software development, we should start by evaluating those values.

There's just one problem. As many others have pointed out, the values are both vague and obvious. There's nothing revolutionary about any of the four values espoused in the manifesto. They're simple common sense.

Of course our development projects should be agile. The opposite of agile is slow and clumsy. Who would choose slowness and clumsiness over agility?

Translating an aesthetic statement into a guideline for real-world team structures, workflows, and development models is a nontrivial task. The values turn out to be the easy part. Developing software is still hard. Pair programming and unit testing and daily builds are consistent with the values espoused in the Agile Manifesto, but they are separate innovations in their own right. They would be good ideas with our without the manifesto.

When you describe your team or your values or your project-management philosophy as agile, all you're really saying is that it's good. Based on straight thinking about priorities. You're not saying anything technical and you sure aren't cutting software development's many challenges down to size. You're just expressing agreement with a handful of uncontroversial tautologies. Sure, we like Agile processes. We also prefer code that works. And projects that are finished on time. And teams that come in under budget.

Commitment to Agile isn't enough anymore. What are you really doing to improve your process?

Web recommendation: Today marks the 200th anniversary of the birth of Charles Dickens. The writer's contributions to literature are too easily overlooked these days. Certainly, many of his novels and stories now appear quaint, needlessly complicated, wordy, and baroque. But Dickens made lasting contributions to characterization, plot, and fiction's role as social commentary. Check out his life and accomplishments at Dickens 2012, a Web site created by the Charles Dickens Museum and Film London in association with The Dickens Fellowship.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is currently rereading Salman Rushdie's masterful Midnight's Children, which offers no small debt to Dickens's innovations.

I keep meaning to write about RedCritter. The innovators at this Dallas-based company are really thinking out of the box. I'm not sure if the result is awesome or awful, but it certainly has captured my attention.

RedCritter is the publisher of Tracker, a project-management tool for Agile development. It's a lightweight tool, but that's OK – Agile is a lightweight approach to development. In fact, most teams manage their Scrum and Kanban processes without recourse to a project-management tool. Using a tool seems...un-Agile, somehow.

But Tracker is well-suited to Agile methods. The folks at RedCritter built the product especially to support Agile development (though they believe Tracker is suited to project-management tasks in fields besides software development as well). And they added a twist: the software treats development like a game.

Tracker allows members of development teams to earn points and badges by fulfilling project goals. The badges – the system allows program managers to define as many as 50 of them – can be displayed on developers' profile pages, much like the badges gamers earn by acquiring skills in computer games like World of Warcraft. Points can be accumulated and traded for incentives in a company rewards store. A live feed, sort of like an in-house Twitter, lets developers track each other's progress toward incentives and gives managers a view into the development process.

The idea is that treating development like a game makes the whole project more fun. Letting developers compete for badges and points isn't just fun, it's motivating. At least, that's what RedCritter's customers have found.

RedCritter makes Tracker available on a 30-day free trial basis. So why not give it a try? It sounds like it could make development more fun. And isn't that what everyone wants?

Web recommendation: You have probably heard that Google is killing off its Code Search feature. Stepping into the void is search[code], an innovative site run by hacker Ben Boynter in Australia. According to Boynter, the site is “an attempt to put all programming documentation in one searchable place.” That's a big job for one guy, but it's pretty impressive what Boynter has done so far. You can see for yourself by clicking here. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. Today's lunch was spaghetti with an excellent homemade bolognese sauce.

I ran across an interesting Web page today: Agile @ 10: Ten Authors of The Agile Manifesto Celebrate its Tenth Anniversary. The editors at The Pragmatic Bookshelf contacted the 17 signers of the original Agile Manifesto and asked them to contribute their thoughts about developments in the Agile world over the succeeding 10 years. Ten of the 17 signers contributed publishable responses, which are collected in the article. The contributors – Andy Hunt, Kent Beck, Ron Jeffries, Jon Kern, Ken Schwaber, James Grenning, Arie van Bennekum, Stephen J. Mellor, Ward Cunningham, and Dave Thomas – shared reflections on the extent to which their bold statement changed the development world.

The article led me to read lots more about Agile, including the Agile at 10 series of articles here at SD Times.

For the most part, these articles stick to a predictable formula. With varying degrees of humor and self-effacement, the signers express parental pride in the path the Agile philosophy has traversed from offbeat mission statement to mainstream acceptance.

But as I read more closely, I noticed something else. At one point or another, in one article or another, almost every one of the signers tacitly admitted that the Agile movement has not lived up to his hopes. This quote from Andy Hunt is typical:

Last summer, I had the good fortune to visit a “very advanced” Agile shop. These folks really did embrace agile methods with a discipline, completeness, and a zealous fervor that would be hard to match. In many ways, they could have been a poster child for Agile methods...But these weren’t productive developers freed from mindless process dogma. They were Agile slaves. The dogma they followed was ours, and they followed it well. And as with many organizations in a similar position, they saw some promising results. Continuous integration, refactoring, unit tests, pair programming—all these techniques yielded some benefit. But they weren’t thinking, they weren’t reacting, they weren’t being agile. When problems came up, they addressed them with all the grace and elegance of a deer caught in the terrifying blaze of alien headlights. They knew how to do Agile; they didn’t know how to be agile.

Ron Jeffries told The Pragmatic Bookshelf, “I had imagined more. I had hoped that many people would adopt these ideas, and I had imagined a significant step upward in project success among those who did adopt them. I had imagined the industry really moving up a notch. What happened is that many have adopted the ideas, at least in name, but that few of them have attained anything like the benefit that is possible.”

In the same article, James Grenning said, “Many developers are still unaware of Agile, or only know the misconceptions. The idea that more upfront work is needed is deeply ingrained in the software development mindset. Code bases are a mess, dragging teams and products to a standstill.” He concluded, “Now we have organizations that have a few Scrum Masters and proclaim themselves Agile, but that continue to spend months in analysis and design, and similar amounts of effort in test and fix. They have stories and iterations, but ignore relative effort estimates and velocity. Code is deteriorating. Tests are not written. And they wonder why Agile is not working for them.”

I find Hunt's wistful observation insightful and moving. At its heart, the Agile Manifesto was a declaration of independence from the shackles of traditional development methods. The signers advocated not new processes, but new values and a new attitude.

The sad irony is that instead of overthrowing slavish commitment to orthodoxy, Agile has spawned a new orthodoxy – complete with certified tools, recommended texts, and a thriving culture of consultants to descend upon your organization and tell you how to do Agile right.

More and more shops now think of themselves as Agile. They're investing in Agile training and attempting to adopt Agile methods such as Scrum and Kanban. But they've missed the point. Instead of becoming enlightened freethinkers, they've drunk the Kool-Aid and joined the Cult of Agile. Dogma and orthodoxy are alive and well. At the cutting edge of our profession are the people Andy Hunt calls “Agile slaves.”

As Dave Thomas told The Pragmatic Bookshelf, “There’s this growing tendency to treat the word 'Agile' as a noun. People say, 'we’re doing Agile.' But friends, Agile is not a noun. It’s an adjective, meaning to be able to move quickly and easily.”

Is your organization agile? Or is it just “doing Agile”?

Web recommendation: The Supreme Court of the United States has agreed to hear a case in which police tracked a suspect's location by placing a GPS device on his car and querying its location every 10 seconds for 28 consecutive days – without a warrant. The question, of course, is whether the government should have the power to use technology in this way. The stakes are high and the right answer should be obvious to all of us. (Isn't it?) The case is summarized well in this GigaOM article. J.D. say check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He likes raisins and walnuts in his oatmeal cookies.

I admit it. I love software development methods.

Development methods (most people say “methodologies,” but that error has always bothered me) are born when we take a step back from debugging and deadlines, when we stop to take a look at the big picture. It was methodologists who first observed that development projects may be broken down into analysis, design, code generation, testing, and maintenance tasks. Successive generations of methodologists have spawned an array of development methods for making these steps deliver results more efficiently and reliably.

I'll provide a taxonomy of methods – including my thoughts about Agile methods and Lean Development and Scrum and all of that – in a future post. Meanwhile, I want to bring DevOps to your attention.

DevOps isn't really a development method, though vendors are working to expand it into one so they can sell books and consulting services. It's an observation about how IT organizations work.

The founders of the DevOps movement observed that IT organizations consist of two parts: development and operations. Many of the problems and inefficiencies that plague the IT business come from lack of communication and understanding between these two parts. DevOps was born when someone asked: What if we instituted increased communication and sharing between development and operations?

It's a worthy goal, there's no doubt. The DevOps people (they're easy to find via Google or, I suppose, Bing) have made one of those emperor's-new-clothes observations that look obvious in retrospect. I like the insights that come from considering the IT operations group, not the end-users at their desks, the real user base for applications.

It's hard to argue against the DevOps mantra. They're right, obviously. Development and operations should work together more closely, and should have more appreciations of each other's challenges.

Beyond that...well, DevOps doesn't go anywhere. It doesn't scale up into a development method, a management blueprint, or a toolset. The observation seems as if it ought to be valuable, but as near as I can tell, it runs out of steam very quickly.

Yes, I know that development methods must be very lightweight to be popular these days. But DevOps is too lightweight to be useful. I wouldn't initiate expensive cross-training and reorganization without the promise of tangible benefits to replace the feel-good handwaving offered by most DevOps devotees.

What do you think? Am I missing something?

Web recommendation: For shame! The man called “the guru of Web page usability” by The New York Times is the author of the latest in my growing list of design-hostile Web sites. For years, I've believed Jakob Nielsen, author of several books on Web design, was a genius. But his site – oh my! Take a look, right here. I think you'll agree that it's more than a little depressing. J.D. say check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He recently relocated to a small town outside Belgrade – stop by if your travels take you through Serbia.

In a recent post, I argued that putting software developers in cubicles costs employers more than they save. An investment in private office space quickly pays for itself with improved productivity. This is not merely my opinion. It is the invariable result of everyone who has ever conducted a serious study of workplace productivity.

I promised, at the end of that little rant, to follow up with a theory as to why offices support higher levels of productivity. With this post, I endeavor to keep that promise.

I'll begin by sharing a memory from my early days as a programmer. I had inherited an Apple II application – written in Applesoft BASIC – that managed the membership database for the nonprofit agency where I worked. I was working late one night, trying to optimize the code that let us retrieve a member's record by name. The search routine simply took too long.

I polished each line of the sequential search until it shone, and still the search was too slow. In desperation, I decided to implement a hack. I would compare the search string to the middle element of the index array, and confine the subsequent sequential search to either the first half or the second half of the list.

My hack cut the average search time in half.

I pondered the performance improvement as I took a pizza break. I read and reread the code – I'm sure I memorized great portions of it. I doodled a bit on the printout. And then I returned to work.

It was then that inspiration struck. What if, after cutting the list in half by comparing the search string with the middle element, I cut the list in half again by comparing the string with the middle element of the new, shorter list? And couldn't I cut the list in half again with another comparison? And again and again, until my array had just one element?

You're way ahead of me, I'm sure. That night, I “invented” the binary search. My new algorithm cut the average number of compares in a 1,500-element array from 750 to 11. The performance improvement was incredible.

If I'd had a semester of computer science under my belt, I wouldn't have had to invent this algorithm, of course. My point in relating this story is not to assert that I am some kind of superprogrammer. I've been lucky enough to befriend some superprogrammers over the years, and knowing them has only served to persuade me more seriously that I am not among their ranks.

I share the story to highlight one aspect of my experience: the enhanced mental state that allowed me to achieve the insight at the heart of the new algorithm. I wasn't working in my usual distracted state at that moment. I was operating in a hyperfocused, highly effective mental mode. I was in flow.

You have experienced flow too, I'm sure. We all have moments when our minds flip into high gear, when we become one with the problems we are working on. We do real work at these moments, but it doesn't feel like work. We lose the sense of time in the joy of ultra-concentration. It's a great way to work. I wish more of my work life resembled those in-flow moments.

What if we could find a way to trigger and sustain in-flow moments? What if we engineered our work environments to support more in-flow periods? What if we trained people to enter flow more readily?

These are among the questions posed – and answered – by psychologist Mihaly Csikszentmihalyi in a series of remarkable books. (You can also check out his ideas, including TED Conference presentations, at YouTube.)

I love Csikszentmihalyi's work. Browsing through his books I am always struck anew by an obvious question: Instead of optimizing our workspace or our tools, why not invest our energy in optimizing ourselves?

Flow, of course, is the missing element in cubicle culture. If you're subject to constant interruptions and distractions, you can't get in the zone. You can't work in flow. That's why private workspaces so reliably yield higher productivity. It's simple, really. It's all a matter of flow.

Web recommendation: I will admit to finding steampunk a compelling genre. Steampunk is a movement in art and literature arising from a simple question: What if we knew everything we know now about how the world works, but our technology was at the 19th century level? What would such a world look like? Well, I guess it would look like the photos in this wonderful gallery: Steampunk pictures gallery. J.D. say check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He recently relocated to a small town outside Belgrade – stop by if your travels take you through Serbia.