For an industry that prides itself on its analytical ability and abstract mental processing, we often don't do a great job applying that mental skill to the most important element of the programmer's tool chest - that is, ourselves.

In 2007, Mark Zuckerburg, the CEO of Facebook (and the subject of much controversy and no small number of conspiracy theories), stated that anyone over the age of 30 is a “little slow:”

“I want to stress the importance of being young and technical. If you want to found a successful company, you should only hire young people with technical expertise...Young people are just smarter. Why are most chess masters under 30? I don't know. Young people just have simpler lives. We may not own a car. We may not have family...Simplicity in life allows you to focus on what's important.” (quoted from “Startup advice for entrepreneurs from Y Combinator”, venturebeat.com, Mar 26, 2007)

In so doing, he not only won over the hearts of every college graduate-turned-entrepreneur, but also cemented the idea of ageism in the programming industry, even justifying it, and left a whole generation of developers wondering what they did wrong and/or whether they were missing something.

At the very least, it brought out onto the table this question of experience; Given an industry in which technology is introduced, gets popular, reaches its zenith, becomes old hat and finally moves into the legacy category all within a decade (at most!), do traditional thoughts around experience still hold water? Or is that just a way for old guys like me to justify a larger salary than those simpler young people?

Tropes

A few years ago, I wrote about the experience trope in the column entitled “Years and Experience”; it's the classic old story of recruiters trying to find developers with 10 years' experience in a technology that's only two years old.

Humor aside, it's actually pretty easy to see what they were trying to do: They were looking for senior-level candidates, developers with some seasoning under their belts, as opposed to new college graduates or other recent entrants into the professional programming ranks through other means. And in fairness, the recruiting industry has learned (a little); now, instead of demanding a set number of years of experience, they instead classify jobs as Senior Developer, leaving it a little vague (and open to interpretation) as to what exactly defines a developer as “senior.” But the trope still comes to pass, sometimes: “Senior Swift Developer” is one of my favorites these days. (Any day now, somebody is going to email me looking for an experienced C# 7 developer, with at least two years of experience with Roslyn.)

And, to be honest, at the start of my career, I was in a bit of a reverse situation - I had a ton of technical knowledge, but no years of experience. I was exactly what Zuckerberg suggested people should hire: young, technically intelligent, and under the age of 30. But in one of my earliest jobs, I got (figuratively) strung up the wrong way about it. It's now one of my fondest memories.

Memories

I was contracting for a medium-sized company, working for Mike Cohn (of Mountain Goat Software, the agile author and speaker) at a local company in Sacramento. This was 1995, long before either of us was out in the speaking or writing game. (My favorite line is that I worked for Mike before either of us was famous.) When it came time to renew my contract (or, his preference, hire me full-time), Mike listened to my salary demands and promptly shot me down. When I pointed out that I knew more C++ and Windows than any two people on my team combined, which was only partly an exaggeration (and a small one at that), Mike pointed out that I lacked experience: “You have such great technical skill that people assume that you're a far more senior developer than you really are.”

Experience? Pshaw. He just didn't want to pay for a good developer, I thought. I challenged him as to what “experience” really meant, and he said, “Look, experience is like driving a car. When a bus accidentally veers into your lane a mile away, the experienced driver sees it and notes it, and takes steps early to avoid the bus if the bus doesn't get out of the way; the new driver is so focused on so many other things, they don't see the bus until the last minute, and only gets out of the way violently and with much screeching of tires, if they can get out of the way of it at all.”

I was convinced Mike was an idiot. The truth is that, of the two of us in the room that day, he wasn't the idiot.

Experience Curve

The world of business and management has wrestled with this, too.

In 1963, a young Harvard Business School dropout took up a job to set up a consulting arm for the Boston Safe Deposit & Trust Company, and created the Boston Consulting Group. Three years, 18 consultants, and an office in Tokyo later, that company, headed by that dropout, named Bruce Henderson, created the “experience curve” theory. (The following year, they put forward a game-theory view of business strategy, something that wouldn't really be picked up for another 30 years.)

The heart of the experience curve is this: the more you do something, the less it costs you to do it.

That's it. Really. That's the sum total of the theory.

But this ridiculously simple theory has some really important ramifications. If your costs are lower than your competitors', you have a distinct market advantage over your competitors. You can either underbid your competition (and use that to gain market share, acquire key customers, or potentially drive your competitors under when they try to match you), or else yield more profit per unit than your competitors do (which in turn gives you the opportunity to pay your people more, attracting better and/or more talent away from your competitors, or re-invest into your infrastructure or employees, which can in turn drive down costs further, and so on).

Or, for those who don't like business-speak: If you can write a mobile app in less time than your competitors, you can either make more profit on the fixed-bid project, or get it done sooner (assuming a time-and-materials contract) and save your client money. Either one gives you a huge competitive advantage over your competitors.

Outside the software world, this theory has been pretty solidly proven over and over again. T.P. Wright (who posited the theory of the learning curve, which is the ancestor of the experience curve) observed in the 1930s that every time cumulative aircraft production (the total number made over time) doubled, the man-hours required to make each one fell off by a constant percentage (his study noted 10-15%).

Now, arguably, Wright was studying a process that was highly labor-intensive yet entirely unoriginal - each aircraft had to follow precise specifications and were all expected to be exactly the same aircraft when the process was finished. Software is, despite all our attempts to suggest otherwise, not that. Each time developers “go to the line,” so to speak, we produce something that might be similar to what we produced before, but isn't exactly the same creature.

And yet, we know that there's something to this idea: the first time a developer works with a framework or language or tool, there's a period of time during which the code he writes isn't the same as the code he writes after using it for a year or two. We look back at the code we wrote five years ago and cringe. Or immediately spot the bug that we couldn't find. Or see a number of ways to refactor and improve the code. Or see ways to more easily unit test it. And so on.

All of these things - those little things that you don't pick up until after you've used a language or a tool or a framework for a while - add up over time; they are what distinguish the experienced developer from the inexperienced. There are simple things, like knowing how long to wrestle with a problem before giving up and asking the assistance of a colleague. Experience espouses simple ideas, like knowing that asking a colleague for help isn't really “giving up,” but actually a chance to engage two powerful mechanisms at once: your colleague, to be sure, but also the speech centers of your brain, which often help spot the critical assumption or flaw in your thinking that you hadn't even realized that you were making. Even when that colleague is a stuffed animal that you keep in your office for just such an occasion.

In some cases, too, the experience plays itself out in a “been there, done that” kind of way: When a crisis of some form strikes the team (an outage in production, a concurrency Heisenbug that manifests itself only in production, data suddenly appearing in production databases as corrupted, and so on), the experienced developer has often seen problems like this before, knows what the symptoms of various causes looks like, and is quicker to discard lines of thought that don't match the evidence before the team. Sometimes, it's even just realizing that taking a step back and thinking accomplishes more than rushed and hurried action, no matter how loud management is screaming.

Summary

There's an old story, told around the conference room table when manufacturing executives get together, about that day when the most important computer on the floor suddenly stopped working. No obvious reason for its shutdown was apparent and all the traditional fixes had failed. The company was completely at a halt, so an expert was brought in, at great cost, to figure out what was wrong. The expert showed up, walked over to the computer, poked and prodded, opening a few panels to peer inside, then closed all the panels, stepped over to grab a hammer, and WHAM! He hit the side of the computer. It immediately fired up again. When his invoice for $100,000 for the five minutes-worth of work came in, the company was livid and demanded an itemized accounting. His response: “Hitting computer: $5. Knowing where to hit: $99,995.”

This story isn't limited to manufacturing; A company was once having a performance problem with their software (written in Smalltalk, although that's not relevant to the point itself):

In a last-ditch effort to figure out whether their work could be salvaged, Dave Leibs, Smalltalk performance guru from ParcPlace Systems, Inc, was called in for a day of consulting. The customer quickly described an overview of what they were trying to do. Nothing stood out to Dave as anything that would bring Smalltalk to its knees. So he said, “Where did the profiler indicate most of the time was being spent?”

“What's the profiler?”

He ran the profiler on the section of code in question and found that it was spending more than 98% of its time in [the Smalltalk equivalent of toString()]. ...He noticed [an object creation expression] inside of a loop. So he chose an experiment and pulled the expression out of the loop, storing the value in a temporary variable that he then used inside the loop. Ten minutes after running the profiler for the first time, he ran the code again. This critical section of code that was central to the client's program now took less than a second.

Dave then turned to his hosts and said, “So, what are we going to do for the rest of the day?”

Lifting a variable out of a loop: $5. Knowing how to use the profiler to find the right variable to lift out of the loop: Priceless.

Will I, with this one article, manage to reverse the ageism in the industry? Probably not; that would take far greater power than anything a few words of mine could accomplish. Eventually, the industry will either learn how to tap into the collective wisdom of its elders in order to avoid learning the same lessons over and over again, or it will continue to waste millions (if not billions) learning each of those lessons the hard way. Senior software developers aren't the ones who have spent twenty years writing code in one particular language - they're the ones who have escaped some knotty situations (or not - sometimes the best teacher is failure, another thing Mike taught me) and know how to apply that knowledge to current situations.

And that's not something that you can get out of a YouTube video or screencast.

(Profiler story paraphrased from an article on the “Lazy Optimization” pattern from Kent Beck and Ken Auer, which I quoted in “Effective Enterprise Java”, p 70. The original online attribution is no longer available, sadly.)