I encountered a fascinating Web site today. The care and love behind its construction are evident. The personal messages from the site’s managers reflect their commitment to exceeding users’ expectations. The idea is novel and the implementation, from what I’ve seen, is first-rate.

So what’s the problem? The problem is that the site fills a need that doesn’t exist, as far as I can tell.

The site is compilr.com. It’s an online IDE and compiler for Python, node.js, PHP, JavaScript, HTML, C, C++, Ruby, Java, C# and VB.NET. Think of it as Google Docs for developers.

compilr.com instantiates a bunch of hot buzzwords. It’s cloud-based. It’s virtual. It’s SaaS. It supports collaboration. All good, right?

Yeah, it’s all good. But I’m left with one question. Who needs it?

If you’re a professional, you want your development environment on your machine, or at least on a local server where you can control the installation. You want to know which patches and revisions have been loaded, and which versions of the libraries are installed. At compilr.com, you don’t know what compiler is running in the back end. If the installed libraries are documented, I can’t find the details. These details matter.

Also lacking are version control and facilities for testing and debugging. Judging from the site’s user forum, these features have been on the back burner for a couple years.

I want to like compilr.com, but I think it’s a solution in search of a problem. It’s easy enough for students and casual developers to download free development environments from the Web. Professionals know where to get their power tools. A Web-based development environment sounds like it ought to be useful, but once you think about it for a few minutes, the usefulness evaporates.

Too bad.

Web recommendation: Here is another alarming article, this one from the good folks at Wired. Little by little, piece by piece, a larger picture is becoming clear, and it’s a disturbing one. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He wishes Aaron Sorkin still had a series on the air.

One of the most appealing aspects of the Agile movement is the breakdown of development efforts into themes, epics, and stories. I have long thought that the real problems in software development have less to do with algorithmic complexity and more to do with the difficulty of getting everyone on the same page. The theme-epic-story structure helps everyone see his place as part of a tale that has a beginning, middle, and end. We know how stories work.

All of this resonates powerfully with the work I did at Columbia in pursuit of my degree in English and comparative literature. Stories, I have come to believe, are how we make life make sense. Life is, in essence, a series of moments. The events that fill the moments have no intrinsic meaning. It is only our sense of story that gives each moment significance. This is, loosely speaking, the essence of existentialism.

Computers don’t have any sense of their existence as art of a story. To a computer running conventional software, facts have no meaning and events are not judged against values. A word-processing program spell-checks a ransom note as happily as a poem.

But what if computers comprehended the world, understood the relationships among the things whose verbal stand-ins they manipulate so fluently for us?

That’s the goal of NELL, the Never Ending Language Learning system at Carnegie Mellon University. Researchers have given NELL a rudimentary set of categories and verbal relationships, and turned it loose on the Internet. The system, which runs on supercomputers donated by Yahoo! with financial support from Google and DARPA, uses the Internet to check its hunches about relationships – what books an author has written, for instance, or whether a particular phrase refers to an item on a menu or a bill under congressional consideration. NELL’s learning isn’t supervised. The system constructs facts and relationships by assembling webs of information sources and evaluating their credibility. NELL scans and rescans millions of Internet pages and from the assembles a view of the world.

The results are impressive. NELL has composed more than 15 million beliefs about the world, structured as simple statements like playsInstrument(George_Harrison, guitar). Every day, NELL learns to read a bit more efficiently and accurately. It hones previous statements of belief and compiles new proposed beliefs. The researchers don’t see an end-point for the project. They intend to let NELL keep learning and learning indefinitely, and see what happens.

NELL doesn’t always get it right, of course. For a few days, before researchers stepped it, it believed Internet cookies were baked goods. The difference between “She bought the bread with the money” and “she bought the bread with the poppy seeds” still flummoxes Nell. For now.

If any of this sounds interesting, check out the NELL project page at Carnegie Mellon. You can download NELL’s knowledgebase if you want. You can even follow NELL on Twitter, where you’ll receive a continuing stream of the system’s new conclusions about how the world works.

Researchers say that NELL is an attempt to simulate the way human beings learn – cumulatively, gradually, with tentative assertions gradually becoming more certain. It’s true that NELL represents a new approach in that regard. But I don’t think NELL matches the human model. Human learning relies upon an extremely rich broadband stream of feedback. We learn that things fall to the ground by repeatedly dropping them and seeing it happen. NELL evaluates and weighs inputs, but it doesn’t seem to test assertions against reality. It’s unreasonable to think that NELL could perform such tests before its hardware includes robotic extensions. But until that day, I don’t think it’s fair to say that NELL’s belief-construction process mirrors human learning.

As for the differences between NELL’s beliefs, which are factual assertions that the system drops when they are disproved, and human beliefs, which members of our species cling to in the face of overwhelming contradictory evidence – well, that’s a topic for another day.

Web recommendation: This story is both important and chilling. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. Reading about NELL made him itch to do some AI coding.

Have you read the 501 Manifesto? It’s a short document – it’s posted online here – that is obviously based on the Agile Manifesto. It’s a statement of principles asserting that developers should have lives aside from coding. Here’s the core of it:

We are software developers who take pride in our work but choose not to be wholly defined by it.

As such, we are proud to say that we value:

• Our families over the commercial goals of business owners
• Free time over free snacks
• Living our lives over maintaining our personal brands
• Sustainable pace over muscle-man heroics
• Our personal creative projects over commercial products the world doesn't need
• Having money for nice clothes over getting free t-shirts from Microsoft
• Playing fußball in the pub with our friends over playing fußball in the office with our team leader
• Not being a dick over being a rockstar

That is to say, we value the things on the left more than we value the things on the right. And some of the things on the right aren't even on our radar.

The manifesto is named, it turns out, for those programmers who save their work, log off, and leave work at 5:01. They go home to spend time with their families while their colleagues – you know, the ones described in decades of stereotypes – keep cutting code far into the night.

The 501 Manifesto is mostly just good old common sense. We work to make the rest of our lives possible. If we’re lucky, we are also able to wrest some enjoyment out of our work. Yes, extra hours are sometimes required, and sometimes we get caught up in hack-attacks and the hours fly by until we look up and find we are still hacking past midnight. But yeah, we get it. We work to support our real lives.

Nonetheless, the manifesto has generated substantial controversy on the Internet. Some programmers resent being described, albeit indirectly, in terms of stereotypes. Others object to individual words or statements, usually found in the text framing the manifesto proper. And of course, the Internet being the Internet, some folks object just because they’re in the mood to be ornery.

I’m sympathetic to the 501 Manifesto. It’s too bad, I think, that the authors felt the need to spell out such commonplace facts of life.

The 501 Coder blog is worth visiting, if you’re interested in an overview of the arguments for and against the manifesto. There’s some interesting stuff there, including this insight into the state of nerdiness: “Here’s the thing: If you are truly a nerd, being pitied by ‘normals’ is part of the deal. It’s not contempt, don’t mistake it for that. It’s a kind of rather-you-than-me bewilderment at your current life choices. That’s all.”

Web recommendation: The Internet keeps spawning new ways of representing information. The Newton Tour was a new one for me. I like the idea behind this project, and I foresee a day when such data might be collated on-demand, for arbitrary individuals, based on Web-scouring strategies. Why not? J.D. says check ’em out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is looking forward to seeing a six-year-old take a look through binoculars for the first time.

We are often called upon to estimate how much time or money it will take to develop an application, add a feature, or eradicate a bug. This is such a common and important occurrence in software development that researchers have attempted to create methods, guidelines, and tools to help us render accurate estimates. Perhaps you’ve used COCOMO, planning poker, or story points.

The sad truth is that none of these methods consistently yield accurate estimates. It turns out that the best estimates come when you turn to the most experienced coder in the room and ask what he thinks. An IEEE research review concludes that the expert will underestimate the required time and effort in 60 to 80 percent of cases, and his estimate will likely be about 30 percent optimistic…but none of the formal methods reliably deliver better estimates.

Estimation is hard because we have to combine information from all four quadrants of the knowledge/ignorance graph.

First is the information we know that we know. Then there’s the information that we know we don’t know. Then there’s information we don’t know that we know. And finally, there’s information that we don’t know we don’t know.

Each of these kinds of information contributes to software cost, time, and effort estimates in a deliciously different way.

It’s no wonder our estimates are so reliably inaccurate.

Web recommendation: It was with a strong blow to the gut that I learned today of the death of poet Adrienne Rich. It was engagement with her work that first opened poetry to me. I drew upon that illumination all through my years at Columbia as I pursued my degree in English and comparative literature. Here are three articles that will, I hope, incite you to search out the work of this incomparable writer. 1, 2, 3. J.D. says check ’em out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He spent an idyllic hour sitting in the sun outside a café this afternoon.

Again and again, over the past months, I’ve returned to the topic of security. Events have shown that our networks, data, and applications are extremely vulnerable. The potential attackers are malicious kids, criminals who hack for profit, other companies, and even governments (foreign and, alas, domestic). The U.S. Defense Department, which has substantial resources, including classified technologies, to throw at the problem, has concluded that it is safest to assume all of its networks have been compromised. As the director of the Information Systems Analysis Center at Sandia National Labs says: “[Malware] is on our machines, and we’ve got to operate anyway.”

I hope by now that I’ve convinced you there is a problem. Obviously, there is. And it’s a big one.

The next question is, what do we do about it?

The traditional answer, for developers, is that we do nothing. Security is an operations issue, not a development issue. We implement password protection where it belongs, we use encryption libraries if users specify that level of protection, and that’s it. Keeping the computers and data secure is someone else’s problem.

That answer doesn’t suffice any longer. The security problem can’t be solved without the active engagement of developers. Our contributions are necessary.

So…how do we begin?

This question has been asked hundreds of times on online programming forums. “Never trust user input” is the most common response. You can spend a few hours reading itemized lists of advice, and much of it makes sense. But it’s hard to translate all of those imperatives into working code and changed work habits.

I think it’s better to start with a comprehensive overview of potential vulnerabilities. You know – a systematic approach. Take a look at the Common Weakness Enumeration site. CWE is a software assurance initiative sponsored by the National Cyber Security Division of the U.S. Department of Homeland Security. It’s a sort of community-based dictionary of security vulnerabilities. The hours you spend browsing this database will be time well-spent. You can translate the site’s taxonomy of vulnerabilities into an action plan for design, coding, and testing

If you can’t solve all your security problems at once, you need to prioritize. Start with this list of the 25 most dangerous software errors.

I really like this series of security articles for Java programmers.

I’m curious about the Microsoft Security Lifecycle system. If you have experience with it, or with comparable offerings from other vendors, get in touch with me via the comments.

I’ll come back to this topic in future posts.

Web recommendation: I came to the Oracle v. Google patent-infringement lawsuit with no preconceptions, and I haven’t heard Google’s side of the story. But I do find this pretty convincing.

J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He has recently become acquainted with the wonderful work of Studio Ghibli.

I can’t deny it, I’m officially skeptical about the Agile movement. I’ve read a lot about the Agile philosophy over the past year, and the more I read, the less impressed I am. The principles at the heart of the Agile movement are self-evidently true, but they haven’t translated into a useful set of methods for software developers. As I wrote in a recent post: “This is no way to achieve reliable, repeatable results. It’s de-evolution in action, a return to the days of late-night hack attacks and reliance upon the heroic contributions of uncommonly talented superprogrammers.”

I admit that my criticism is disarmed somewhat by the first statement of values listed in the Agile Manifesto itself: “Individuals and interactions over processes and tools.” So Agile hasn’t translated into a heavyweight set of processes and tools? Duh. It was never intended to.

This morning I stumbled across a blog post titled “Agile is a Sham.” It turned out to be an invitation to an entire subculture of Agile skeptics. I thought for a moment I’d found some likeminded thinkers to spend time with. But I couldn’t have been more wrong.

I followed a trail of hyperlinks that led me through a rising tide of dissatisfaction with Agile. Not because Agile hasn’t translated into a useful set of tools and methods, but because Agile imposes too many tools and methods! This anti-Agile community is contemptuous of consultants and trainers who are peddling Agile as the new cure for everything that ails software development. They despise Scrum and Kanban as snake oil that managers buy in an effort to get superior results out of inferior programmers. The blogger summed up his Agile skepticism like this: “[T]he success of the coding part of a project is dependent on the caliber of the engineers doing that coding and not the process they follow.”

Maybe you recognize this line of thought. Our industry is vulnerable to a very unattractive elitism. There are lots of programmers who believe the best way to ship working software on time is to get rid of onerous development processes, sweep out the below-average programmers, and leave the competent programmers – the real engineers – alone to do their thing. Just get out of their way, and they’ll make the code work.

Members of this elitist clan argue that project management and formal development processes are just obstacles to their brilliance: “If you need to follow a prescribed process in order to be in any way an effective coder then you are mediocre at best and so is your work and your project.  If you are a good programmer, you know yourself how to communicate and create your ‘process’ as you go along.”

I find myself totally at odds with this community of Agile skeptics. Their cynicism about Agile consultants and trainers is probably well-founded, I admit. But their insistence that raw competence makes methods obsolete – and that reliance on methods is an admission of mediocrity – sets my teeth on edge. Their elitist views are demonstrably incorrect and patently offensive.

So yes, I’m an Agile skeptic. But I’m the other kind – the kind that would like to see Agile principles instantiated in more discipline, more tools, and more methods. That’s how you translate good ideas into working processes. Our field needs more discipline, not less.

Web recommendation: The open-source Go language has achieved a milestone: Version 1.0 is now available. I like what I read about this lightweight language. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is hopeful that a certain video clip he saw today, featuring himself, never reaches the public eye.

The best conference presentation I ever attended was at the Software Development Conference in 1989. I was editor of Computer Language magazine, the sponsor of the conference, and I had major responsibility for signing up speakers and assigning topics for the keynotes and technical sessions.

Larry Constantine spoke at our conferences, and Grady Booch. Bjarne Stroustrup and P.J. Plauger and Adele Goldberg. Bill Gates spoke several times. Ed Yourdon was a regular. Our conference attracted the industry’s best-known thinkers.

The session I’m talking about wasn’t delivered by one of software development’s legendary figures, but by a technical manager from The Whitewater Group, a small company that published an object-oriented programming language called Actor. Jim McCarthy had proposed a seminar on project management and slipped deadlines that he called “Slipping without Falling.”

Jim’s talk was both extremely entertaining and extraordinarily insightful. Drawing on his experiences in project and process management, he delivered a series of memorable, colorful aphorisms, each of which expressed in shorthand a profound truth about the messy process of shipping software. Jim focused on techniques for building healthy teams, a black art that is still almost universally mismanaged.

One of Jim’s most memorable bits of advice was this: “Beware of the guy in a room.” Many projects, he explained, have at their core a challenging bit of code that is assigned to the development team’s acknowledged superstar. This programmer takes custody of the crucial module and retreats to his office. The team spends the next weeks tiptoeing past his door so they don’t interrupt his intense focus. No one asks for forecasts, bug-tracking data, or progress reports. The team knows that its success hinges on this single programmer and his brilliance. His code will be ready when it’s ready.

This dynamic is common and pathological. It is anathema to teamwork. The pattern reinforces harmful stereotypes about the relative value of programmers and superprogrammers. It reduces meeting deadlines to a game of chance. As McCarthy said, “the results are uniformly fatal to the professional development organization.”

McCarthy’s observation hit home. I had seen organizations held hostage by a guy in a room. Worse, I recognized that on a few occasions, I had been the guy in a room. I had helped erode healthy teamwork in companies where I had taken sole custody of creative challenges and everyone else just waited for me to deliver.

Jim’s insights into team leadership led him to Microsoft, where he led the team that developed Visual C++ – a team that is legendary for its productivity and innovations. He expanded his Software Development Conference talk into a book called Dynamics of Software Development, which I commend to you as essential reading. Jim now runs The McCarthy Show, which offers leadership training. If you visit the Web site, which you should, be sure to check out the Episodes page, which currently offers 115 podcasts on elements of McCarthy’s leadership principles.

Curiously, Jim’s warning about the danger of “a guy in a room” is not unique in our industry’s literature. In 1971, Gerald Weinberg wrote an extraordinary book called The Psychology of Computer Programming. Like McCarthy, Weinberg focused on effective teams. His book included a short section called “The Ten Commandments of Egoless Programming,” which listed rules for functional teams. Item number nine on the list was this: “Don't be ‘the guy in the room.’ Don't be the guy coding in the dark office emerging only to buy cola. The guy in the room is out of touch, out of sight, and out of control and has no place in an open, collaborative environment.”

I don’t know if Jim picked up the “guy in a room” concept from Weinberg or it occurred to him independently, and I don’t really care. It’s an important insight with wide applicability.

Does your team rely on the efforts of a guy in a room? Are you the guy?

Web recommendation: McCarthy’s still got it! You can catch a video of his Agile 2008 presentation, “11 Commitments for a Shared Vision,” by clicking here. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He had such a nice day today.

I’m writing this post on March 11, 2012. If you’re in the United States, set your clocks forward one hour. And if you’re reading this on the Internet, pause a moment with me to celebrate an anniversary.

It was 20 years ago today, on March 11 1992, that researcher Nathaniel Borenstein emailed the first MIME attachment. The Multipurpose Internet Mail Extensions standard has proved to be a most flexible specification, accommodating every rich data type the emailing world has thrown at it. It is also, of course, the information coding strategy for the Web. Borenstein recently told The Register that he thinks about a trillion MIME attachments are exchanged every day.

Check the header on a random email message and you’ll discover a curious fact about MIME – it’s still at version 1.0. The standard has turned out to be extensible enough to accommodate every kind of digital data. Also curious is the fact that despite its universal use, MIME isn’t an IETF standard, but still a draft standard. It may not be adopted as a full standard before it becomes obsolete.

Edit: For a couple weeks, the second paragraph of this article defined MIME as “Multipurpose Internet Male Extensions.” Is it possible no one noticed this delightful typo?

Web recommendation: The phenomenal Grady Booch has embarked upon an inspiring new project. He has gathered up a bunch of his buddies in the software and arts worlds, and set out to create “Computing: The Human Experience.” The group’s initial funding appeals on kickstarter.com were successful, so we can look forward to seeing content – including an 11-episode television series – start rolling out. In the meantime, we can get previews of what Booch has in mind on the project’s YouTube channel. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is pretty sure he’s running a fever, but due to the oddity of Celsius-standard thermometers, he can’t really say how sick he is.

If you’re not doing parallel programming yet, just wait. Multicore processors and on-chip GPUs are already on the desktop. It would be criminal to let all those MIPs go to waste.

You may have started your parallel programming career with multiprocessor systems. Clusters and symmetric multiprocessing systems are commonly used as servers, and high-level task partitioning has wrung a lot of performance out of them.

The next wave of parallelism arrived with multicore processors. All the major CPU vendors are now delivering multiple cores on single chips. Your app won’t perform any better unless you use special libraries and algorithmic techniques to rewrite your code to take advantage of multiple cores. Tools to ease this process are now routinely included in vendors’ development systems.

A variant of multicore processing is now being implemented on GPGPU systems. GPGPU stands for “general-purpose computing on graphics processing units.” Now that hardware vendors are building chips that contain both general-purpose CPUs and GPUs on the same silicon, you can farm out certain kinds of tasks to the GPU. Researchers are finding that a wide variety of programming problems are vulnerable to the GPGPU approach, but you may not recognize your code once you’ve adapted it to the new architecture. Say goodbye to loops, and hello to arrays and vectors.

Parallel-processing features are creeping into even high-level languages (in which, thankfully, the compiler does most of the heavy lifting). So when it comes to parallel programming, the question isn’t if, but when.

Web recommendation: A blogger known as Numbergrinder has written a succinct critique of a common hiring practice. If you’re involved in enlisting developers to join your team, I’m an Engineer, not a Compiler is worth a look. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He’s coming down with a cold.

The ancient Greeks had no computers, but their mythology includes one figure who embodies an unchanging reality of programming. I am speaking, of course, of Tantalus. You may remember the story.

Tantalus was a king of Phrygia, in what we now call Turkey. He was a friend of the Greek gods until he offended them. Then he received a distinctive punishment. In the underworld, Tantalus was tied to a fruit tree that stood in a river. When he reached for a fruit, the branches would rise until the intended meal was just beyond his grasp. When he leaned down to get a drink, the water receded. Tantalus was sentenced to eternal hunger and thirst, and the objects of his longing were forever just out of reach.

So it is with programmers. But instead of fruit and water, we hunger for reusability.

You don't have to write code very long to realize that almost all of what we do has been done before. We are constantly solving problems that have been solved over and over again in the past. We write boilerplate code and reinstantiate well-known algorithms.

We would be more productive – and our days more enjoyable – if we weren't always reinventing the wheel. What if we could bundle code for reuse? Then we could incorporate algorithms and boilerplate code by reference instead of tediously rewriting it with each new app.

This dream has been the motivation behind many of the technologies that support our industry. Off the top of my head, the list includes these:

• Unix command-line utilities

• system libraries

• function libraries

• class libraries

• dynamic link libraries

• subroutines

• object-oriented programming

• package-oriented languages like Ada and Modula-2

• code generators and CASE tools

• component-based development

• visual programming

• use cases and design patterns

• application frameworks

• service-oriented architectures

It's a rare project that doesn't make use of several of these technologies. But the software we write can still be characterized as ad hoc. We continue to reinvent the wheel with each new project.

I don't have a solution to offer – the technologies on the list, plus, no doubt, others that escaped my attention – should be sufficient. The benefits of reuse are obvious. Yet software reuse remains maddeningly – tantalizingly – out of reach.

Web recommendation: Imitation is a form of reuse, I guess. And they say (do they still say?) it is the sincerest form of flattery. If so, the good folks at Apple must feel awfully flattered these days, due to this effect. J.D. says check it out.

J.D. Hildebrand has written hundreds of articles for dozens of publications and online communities dedicated to software development. He is grateful for the days Internet service, water delivery, and electricity are all uninterrupted.