I hate that expression, “It's complicated.” When people tell me it's complicated, it almost always isn't. What they're really saying is that they don't want to talk about it. But when we're talking about software development, it really IS complicated. Software development is notoriously difficult and it's getting steadily harder. And we need to talk about it.
Imagine a world where a developer who understands the basics of programming (things like variables, branching, and looping) can spend a weekend reading a manual cover to cover and become productive in a new environment by Monday morning. An environment that includes built-in UI, database and reporting tools, its own IDE, integration paths for third-party systems, and everything else they might need to build and maintain a system. A world where the tool is only updated about once every two years or so and where reading a couple of articles or attending a conference for a few days gets them up to speed on ALL the new stuff. A world where developers get very skilled at using the tools and become masters of their craft.
That world existed between 20 and 30 years ago. Since then, the developer's world has accelerated and expanded at an increasing pace. Today, just keeping up with what's available to us is like drinking from a fire hose. Where do we even find time to write code?
Before you go thinking this article is about nostalgia, it isn't.
I believe in the original definition of nostalgia, as a medical diagnosis for grave illness, one that couldn't be cured, even when the patient was lucky enough to be able to try to relive those memories. I believe we can't - and shouldn't - go back, only forward. But all hope is not lost. Let's start a conversation about complexity and begin to deal with it.
Where is complexity coming from? I believe it stems from the plethora of platforms and toolsets, higher expectations, larger systems, bigger teams, faster delivery of new ideas, and an ever-expanding set of opinions. Today, a full-stack developer must have a good working knowledge of multiple user interface tools, databases, cloud offerings, and business logic tools. Even picking just one UI platform, say HTML and JavaScript, there are hundreds of frameworks to choose from and they need to be mixed and matched to get a good result. New frameworks and major updates to existing frameworks are coming out almost daily. No one can make a perfect decision.
That's just for development; I didn't include new Web assembly-centric UIs like Blazor. I also ignored desktop and native mobile applications and more, and included choosing and learning an IDE to develop in. The number of platforms and toolsets available today is astounding and the pendulum is now swinging back for many companies who previously touted giving teams complete autonomy over their platforms and toolsets, to choosing and standardizing on a smaller subset. These companies are coping with too many choices, which leads to both decision paralysis and an inability to get everyone on the same page. Constraining the number of choices is one way to combat complexity.
Too many choices leads to both decision paralysis and an inability to get everyone on the same page.
Higher expectations, larger systems, and larger teams are other drivers of complexity. Development is no longer just getting the code to function and doing some performance tuning. It also has to be secure, scalable, a great user experience, accessible from a variety of devices, cloud-deployable, tested, maintainable, kept up to date, and continuously improved. Also, at one time, most apps targeted internal users; now most apps are targeted at consumers, which ups the bar considerably.
The move toward both more systems and larger systems requires more teams and more developers, which leads to difficulties getting everyone rowing the boat in the same direction. When I say teams in this article, I don't just mean a development team within a company. I also mean the team developing the toolsets and platforms and those consuming what the team is building. We must work with teams both inside and outside our walls. This complexity can be mitigated by better collaboration.
The collaboration problem isn't unique to the software industry, and we're getting better at meeting these challenges, but the trends aren't going to slow down or reverse themselves.
The biggest drivers of complexity, in my experience, and those we can do the most about, are last two I mentioned. The faster delivery of new ideas and the ever-expanding set of opinions have, ironically, sprung from our own attempts to solve complexity issues and to make things easier. Today, everyone who solves an obstacle to coding productivity can publish and promote that solution. The solutions are often quite good and are easy to find on the Internet by anyone having a similar problem. However, sometimes the solution becomes widely accepted and is thought of as the “right way” to code.
What our industry lacks is a good way to tell if a particular solution even applies to a project. We sometimes adopt things like SOLID principles, microservices, containers, Domain Driven Design, and container orchestration because they're considered “good” or they're thought of as a way to future-proof our solution, when in fact, they are often solutions to problems we don't have. In those cases, we're only adding complexity by adopting them.
How, then, can we be better at using the right tools at the right time? There's no magic solution, only hard work, careful reflection, and creativity. Let's examine the problem.
How to Solve the Problem of Complexity
When we think about how to solve the problem of complexity, the answer invariably comes down to breaking down a complex system into a series of smaller, more discrete, less complex systems that aren't too complex standing alone. This makes the new modules more approachable and the goals easier to achieve. But nothing comes for free and the catch is that we now have to communicate and coordinate among these smaller, more discreet systems, which often means new tools and new patterns, and that actually ADDS complexity to the system as a whole. It results in a new (but usually manageable) set of problems and more code to write, test, and maintain. Because of this, we have to be very careful in choosing how we approach each problem. It's time we stopped living by the dictum that every problem can be solved by adding another layer of abstraction and start asking if it should be solved that way.
What can we do about the complexity that comes from the steady gushing of new ideas? I believe that we, as an industry, should be more demanding of ourselves and of others. Instead of looking only at all of the good that comes from something new, we should demand to also know the downsides and demand help in determining when and where these ideas can be used to our advantage. Often, when looking at a new technology, the material is presented by diving in and showing how it works. There isn't even a mention of the tradeoffs or even which problems are being solved.
New technologies should be approached with litmus tests and rules of thumb, even if we have to develop them ourselves. Is this new idea “good” or “bad”? It really depends on the problem you're trying to solve. What we need are better, more available ways to make that determination. Wouldn't it be refreshing to do some research into a new technology and read about why it was created, which things it does really well, and which things it isn't intended for, instead of reading about how sliced bread has a new challenger? I see this kind of documentation occasionally, but not nearly enough. It does us all a disservice to become such a fanboy of a technology that we don't present information that allows our peers to make good decisions. That information is at least as valuable as the sunshine we spread. Often more.
Wouldn't it be refreshing to read about a new technology and why it was created, which things it does really well, and which things it isn't intended for instead of reading about how sliced bread has a new challenger?
When thinking about the rapid growth of complexity and information in development, I often think about industries that have gone through this kind of evolution ahead of us. The airliner industry is a good example. The early airliner industry had a lot of competitors and the challenges were mainly engineering problems. Building commercial airplanes had been figured out and the challenge was in improving the airplanes. As the industry matured, the planes became increasingly larger and more complex and companies handled the complexity by adding more draftsmen, more engineers, more workstations, more bodies. They next handled the increased complexity by breaking the process down into smaller, more discreet processes. Some employees only worked on cabin interiors, some only worked on aerodynamics, some only on landing gear. All this segregation and specialization added complexities in other areas, like coordination among the teams and common goals. Hundreds of draftsmen with pencils were generating millions of drawings to be presented and argued over in endless meetings. New designs started to take years, sometimes decades, to come to fruition. Companies failed or were acquired by other companies.
The process had become too complex. This is where our industry is now. Developers are specializing more and we have lots of bodies all headed in different directions.
Companies like Boeing and Airbus innovated and began using software and collaboration tools that replaced much of the manual engineering, drafting, and meetings, and things began to speed up again, but not for the reason you may think. Software wasn't a magic bullet. All of that innovation also led to new capabilities, offsetting or even increasing complexity instead of reducing it. Today, airliners are more complex than ever and the next generation will be even more complex.
What turned things around for the airliner manufacturers was that the teams and their software were getting better at working together; they created and adhered to standards and, most critically, they simplified the work other teams need to do to make use of what they'd built. The individual components were trending toward becoming black boxes that either did or didn't fit the requirements of a particular project and were relatively easy to make decisions about, whether that decision was to modify the black box or to build a new one. It wasn't the tooling or the innovation that made the defining difference. It was the ethos that each team was part of a larger team and that how those teams interacted was as important as what that team actually produced.
Each team is part of a larger team and how those teams interact is as important as what that team produces.
As the software industry continues to evolve, the unfettered gushing of new ideas, tools, technologies, and platforms must become more stringent. Developers aren't keeping up. All of the wonderful toys have become as much a burden on the industry as a blessing. Software projects still fail at an astoundingly high rate and it's almost always because either the requirements are too complex or the chosen implementation is too complex.
Complexity is killing us. Our industry needs to develop the airliner ethos that each team (in or out of your walls) is part of a larger team and that how those teams interact is as important as what the team produces. Our efforts must not be only to build newer, shinier, better, faster things, but to trend toward building black boxes that either do or don't fit the requirements of a particular project and are easy to reason about. We must allow others to make good, informed decisions about what we build.
How teams interact is as important as what the team produces.
This is where the next big innovation will happen. Evolving technologies, tools, platforms, ideas, and design patterns are good things, but they're only useful if they can be discovered, understood, and leveraged. I believe we need a self-curating system beyond the plethora of take-it-or-leave-it open-source projects and the commercially driven resources we have now. Our current situation has taken shape and that shape is a pile. And it's a very large and unruly pile that's overwhelming and doesn't serve our needs. We must organize that pile if we are to evolve.
Conclusion
There are things we can do to keep complexity from killing us. Currently, each of us is faced with a gigantic pile of new ideas, each labeled with a catchy name, some industry buzzwords and some marketing blurbs. The pile is too big and too messy. We need to change that so that each idea in the pile is labeled with its categorizations and specifications, so we can organize them and make rational and appropriate decisions about them. We need to specialize so that we can reduce the sheer number of ideas that we need to worry about and increase our understanding of and improve our discourse about those we do care about. We need to create better ways to collaborate with other specialists and their parts of the pile. We need to pick reasonable, curated defaults, so that not every specialist has to decide between every option in their part of the pile, every time. And finally, we need to develop a professional approach to these ideas so we can train the next generation, not just on looping and branching and technical basics, but also on how to make good use of the organized, categorized, curated, and well-described pile of ideas we're creating.
It is complicated. But there's hope.
