In a recent post, I argued that putting software developers in cubicles costs employers more than they save. An investment in private office space quickly pays for itself with improved productivity. This is not merely my opinion. It is the invariable result of everyone who has ever conducted a serious study of workplace productivity.

I promised, at the end of that little rant, to follow up with a theory as to why offices support higher levels of productivity. With this post, I endeavor to keep that promise.

I'll begin by sharing a memory from my early days as a programmer. I had inherited an Apple II application – written in Applesoft BASIC – that managed the membership database for the nonprofit agency where I worked. I was working late one night, trying to optimize the code that let us retrieve a member's record by name. The search routine simply took too long.

I polished each line of the sequential search until it shone, and still the search was too slow. In desperation, I decided to implement a hack. I would compare the search string to the middle element of the index array, and confine the subsequent sequential search to either the first half or the second half of the list.

My hack cut the average search time in half.

I pondered the performance improvement as I took a pizza break. I read and reread the code – I'm sure I memorized great portions of it. I doodled a bit on the printout. And then I returned to work.

It was then that inspiration struck. What if, after cutting the list in half by comparing the search string with the middle element, I cut the list in half again by comparing the string with the middle element of the new, shorter list? And couldn't I cut the list in half again with another comparison? And again and again, until my array had just one element?

You're way ahead of me, I'm sure. That night, I “invented” the binary search. My new algorithm cut the average number of compares in a 1,500-element array from 750 to 11. The performance improvement was incredible.

If I'd had a semester of computer science under my belt, I wouldn't have had to invent this algorithm, of course. My point in relating this story is not to assert that I am some kind of superprogrammer. I've been lucky enough to befriend some superprogrammers over the years, and knowing them has only served to persuade me more seriously that I am not among their ranks.

I share the story to highlight one aspect of my experience: the enhanced mental state that allowed me to achieve the insight at the heart of the new algorithm. I wasn't working in my usual distracted state at that moment. I was operating in a hyperfocused, highly effective mental mode. I was in flow.

You have experienced flow too, I'm sure. We all have moments when our minds flip into high gear, when we become one with the problems we are working on. We do real work at these moments, but it doesn't feel like work. We lose the sense of time in the joy of ultra-concentration. It's a great way to work. I wish more of my work life resembled those in-flow moments.

What if we could find a way to trigger and sustain in-flow moments? What if we engineered our work environments to support more in-flow periods? What if we trained people to enter flow more readily?

These are among the questions posed – and answered – by psychologist Mihaly Csikszentmihalyi in a series of remarkable books. (You can also check out his ideas, including TED Conference presentations, at YouTube.)

I love Csikszentmihalyi's work. Browsing through his books I am always struck anew by an obvious question: Instead of optimizing our workspace or our tools, why not invest our energy in optimizing ourselves?

Flow, of course, is the missing element in cubicle culture. If you're subject to constant interruptions and distractions, you can't get in the zone. You can't work in flow. That's why private workspaces so reliably yield higher productivity. It's simple, really. It's all a matter of flow.

Web recommendation: I will admit to finding steampunk a compelling genre. Steampunk is a movement in art and literature arising from a simple question: What if we knew everything we know now about how the world works, but our technology was at the 19th century level? What would such a world look like? Well, I guess it would look like the photos in this wonderful gallery: Steampunk pictures gallery. J.D. say check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He recently relocated to a small town outside Belgrade – stop by if your travels take you through Serbia.