Mastering Digital: The State of Software Engineering Report 2018

“It’s not just about software — the Enterprise is the System”

The “Mastering Digital: Software Engineering Report” (2018) provides analysis of detailed feedback from Architects and Engineers about their work on software projects for organisations of all types and sizes.

The report looks at the following:

• Demographics of the respondents (Company Types, Team Size, Project Spend, Role Types, etc)
• Performance of current software engineering capabilities & outcomes
• The challenges & opportunities with the technology we use (Including: Server Side, Front End, Database and DevOps & SysOps Technologies)
• The challenges & opportunities with the delivery of software projects (Including: Technology & Design, Organisation & Behaviour, Knowledge & Skills, Requirements & Planning)
• A summary of the key pain points and root causes
• Some key takeaways

If you want to get straight into it, you can read the “Mastering Digital: Software Engineering Report” here.

Otherwise, let’s jump into a bit of commentary on the findings…

#1: Complexity is Killing Us

“Technology Spaghetti: Building enterprise systems is increasingly complicated. The technology and architecture choices we must make are increasing, and less obvious. This means that just making the tools work together, so that you can build your application, is a major investment that complicates life throughout the entire lifecycle.”

By far the number 1 challenge across the board for the survey respondents can be classified as “complexity”.

Complexity is an issue experienced everywhere. It is experienced when integrating different layers of the stack, using different protocols, languages and data constructs. It is experienced in the sheer volume of choices we have to make regarding our languages, platforms, tools, libraries, frameworks (yes, Javascript, your name did come up), as well as protocols, patterns, techniques, and the like… It is experienced in the integration of our software applications — some we control, some we don’t. It is also experienced as systems change over time — version upgrades, new features, deprecated components, dependencies, data model upgrades, etc.

For some this creates a considerable barrier to entry, but even seasoned developers are finding it challenging to build complete end to end solutions without running into complexity problems. “Full Stack Developers” were the most frequent respondents to the survey, and usually the most critical of the complexity and integration issues across the stack. Many simply wanted a single standard solution — one language, one protocol, one framework, one data construct, etc. They were frustrated with spending so much of their time solving the impedance mismatch between the layers and components to create a coherent and integrated solution. Selecting “just the right” combination of tools, libraries, patterns, and frameworks can be a daunting task. Respondents felt that once this had been learned the pain lessened, but as technology, patterns, and best practices change so frequently, they felt they were continually going up and down the ‘learning curve’.

Dealing with this complexity distracts engineers from producing outward facing value (like user features). Handling technical debt, legacy compatibility issues, and tooling complexities as a pre-requisite to delivering a new feature creates longer lead times and jeopardises the predictability of the planning. Given the common lack of investment in solving these issues upfront, Engineers are forced to deal with these difficulties under the pressure of delivery. This has led to a frustrating development experience for many of the respondents, and usually leads to increased technical debt as a result.

What engineers really want is more simplicity, easier integration, more standardisation, more technology ready ‘out-of-the-box’, technology that is easier to understand (just having accurate, up to date, and easy to understand documentation would be a start), and of course much easier to change over time.

When we focus on solving the problem right in front of us, it is easy for our paths to meander, become lost, and ultimately trapped in a world of complexity. We need to acknowledge that the world always changes, and that enterprise systems are more than just ‘a bunch of applications’ sitting on some hardware — a role that good architecture, especially enterprise architecture, should be performing, but was found lacking in one way or another by most respondents.

To understand this problem space better, and where each component you build fits into this bigger and more complicated picture, we need a good architecture — a big picture view of the enterprise, the ecosystem the enterprise exists in, and the alignment of goals, requirements, designs and solutions into a coherent model that engineers can build against. This is as true for single product startups as it is for large enterprises.

But before we can focus on solving that, we need to stop spending our time tinkering with the plumbing.

“Simplicity & Change: By reducing this complexity, and introducing more standardised and simple to use development models and technologies, we can increase the efficiency of our software projects. We need to understand and build into everything we do the reality that everything changes — data, processes, interfaces, designs.”

#2: Save $$ Now, Pay 10x Later

“Feature Frenzy: Our focus is often on getting features shipped, rather than building core foundations and a strong scalable architecture to help with long term quality and ease of management. This means we continually battle technical debt, deficient tooling, complications with dependencies and an ever growing legacy.”

Respondents complained about the focus on short term user features over long term architecture. This shows in the respondents scoring of their ability to build user-pleasing software vs their ability to build easily changed and managed software — the latter suffering greatly due to a primary focus on the former. For some, this felt like they were continually paying for today with money needed for tomorrow — or that they were paying off one credit card with another. It is only a matter of time before the music stops playing and we find ourselves without a chair. But as this doesn’t necessarily happen on every project, we are happy to chalk it up to a freak one-off issue, and to fix each specific problem that time round, and move on.

It is understandable that we want to create the outward facing value as a priority, though sometimes the cost of doing this without the proper preparation or investment is high in the long term. Few seem to measure that long term performance, other than comparing each short term view to the previous — which reinforces the motivation to always be thinking of the here and now, and not the future.

Respondents felt the lack of funding and time investment into building solid platforms, frameworks, methodologies, tooling, etc was a major reason they struggled with performance on delivery projects — partly due to technical debt, but also due to solving core problems for the first time during that project. Most didn’t see any investment in these areas, and those that did felt it was far too little, and often attempted as part of the delivery projects for new user features/applications, or as a hurried “Sprint 0”, which meant it was rushed and usually specific to that one project’s needs.

Technical Debt generally comes from the things that you skipped or rushed in a previous project. When you are facing technology & integration issues for the first time, estimates can become unpredictable, leaving projects under delivery pressure to make difficult decisions about what to leave out, what to leave broken, and what to sticky tape up temporarily. Many of these issues could be resolved by pre-integrating and pre-building the platform/framework, so each new feature implementation is more like cutting cookies, than it is creating Michelangelo’s David.

“Less is More: If we invested more in creating a solid platform, that truly supported the entire lifecycle for enterprise systems, we would find we need to do less work on each project to release new or changed features. This upfront investment is hard to attain in most organisations. The goal should always be to reduce the work needed across the board.”

#3: Agile is not a silver bullet

“Rigid Agile: Many engineers are struggling with the same issues of communication challenges, requirements issues, planning problems and the like that Agile is intended to fix. If Agile is used without proper understanding, especially of the wider organisation’s processes, and without compromise or adaptation, it soon becomes a nuisance.”

If we were to believe the rhetoric we often hear about Agile, we would expect software development projects to become magical places to work, with happy customers and developers alike, and a continuous stream of high quality outputs that only get better over time. We would expect that things get done faster, with less hassle and complication.

The reality is a little different.

Many of the respondents are unhappy with how their teams work together (in almost all cases they’re using SCRUM based Agile). They are still struggling with poor communications both inside the team and with external parties. The decision process leaves them feeling powerless and reactive, and they struggle with the lack of collaboration across different teams that they depend on — a problem that amplifies greatly at scale. In many cases, respondents felt that they simply didn’t have a working methodology — and certainly not one that connected all the teams and processes together in a useful way.

“Projects often overrun because of technical problems, but the biggest challenge found here is handling continual change.”

One of the biggest complaints respondents have, is with requirements — the collection, clarity, prioritisation, and management of requirements fell short of the mark for most. This together with the poor communications, processes and complexity challenges, often results in a failure of estimation and planning that leaves the team in a reactive and sometimes firefighting mode of behaviour — focused only on the tasks they have in front of them.

Respondents believe that a lot of these issues come from a lack of understanding about what agile is and isn’t. Organisations are quick to brand themselves as “Agile”, but fail to actually be “agile”. It is easy to perform ceremonies — it is hard to change mindsets. Being agile requires you to embrace the essence of the philosophy: be flexible, move incrementally, learn quickly, be open & collaborative, get things done. “Doing Agile” is more about standups, retrospectives, burndowns, stories, and kanban boards, without necessarily the mindset that adopted those concepts in the first place.

Additionally, it seems for many organisations that the business and IT teams are still quite separate and often work in a customer/supplier style relationship that reinforces this delineation, allowing each side of the relationship to retreat into their own domain and avoid the concerns of the other. It is interesting to see that quite a few of the respondents, whilst being technical, were very keen to see greater integration and collaboration across this ‘virtual void’, with a much greater involvement from the business in the software engineering process, and a more involved technical team in the business problem they are solving.

Whilst respondents are critical of some of their own team’s technical skills and abilities, and want more emphasis on education and self learning, they are also critical of non-technical team members for not understanding enough about the technology side of things. They believed that a lot of the challenges being faced by the team are due to isolated decision making by people who don’t really understand the consequences of their decisions on the technology or software engineering process.

In today’s world, it is simply not acceptable to throw your hands up in the air and proclaim you don’t really get all this fancy computer-y stuff, and to just leave that to the ‘geeks’. The enterprise is a system made of people, processes, information, and technology — and as software systems increasingly embody all of these things, you simply cannot separate business and technology the way it has been in the past — it is ‘one team’ and ‘one goal’ creating ‘one system’: the ‘enterprise system’. On the other hand, we should be mindful of the tendency to blame things we don’t understand — there are sometimes very good reasons why disruptive decisions are made; what we need to do better is ensure the entire team (business & technical, managers & technicians) understands them, and ideally, participates in some way to make them.

“One Team: The business and technology teams still see themselves as separate, often in a customer/supplier style relationship. This leads to many of the problems with requirements, planning & decision making. The enterprise is the system. We need to encourage this thinking to break down barriers, and bridge the gaps holding us back.”

From my own experience working on technology projects over the last 20 years — from small startups to $billion transformation programmes in some of the largest organisations in the world — I have seen both waterfall style methodologies and Agile style methodologies succeed and fail. So, instead of judging organisations based on whatever process they say they are following, I’ve found that there are better indicators of likely success:

• People — the individuals involved really matter : with the right people, anything is possible. With the wrong people, even the best teams can be poisoned from within. Teams need a balance of people —from the big thinkers, to the detailers, the artists, to the engineers, to the optimists and even the (constructive) pessimists. Everyone needs to be open to change, and everyone needs to be engaged, and everyone needs to be honest — with themselves and each other.
• Culture — the organisation, and its mindset, matter more : people shape the culture, and the culture shapes the people. Entrepreneurial spirit, friendly competition, being open and flexible, motivated by purpose, striving for deep quality, a friendly & supportive atmosphere, a safe environment: to share ideas, to try things out, to get things wrong. But most of all, treating people as humans — you’d think this was obvious, and natural. Sadly not.
• Vision — ignorance is bliss until the future hits : long term strategies aren’t prophecies — they are directions and way points for when the fog closes in. Organisations that lack this tend to work tactically, on short term problems, that more often than not materialise from long term failures. Delivery without vision & strategy is like a ship without a map & compass — you’ll go places, you just might not like where you end up.
• Practicality — just get things done without all the fuss : sometimes, you just need to get on with things — and have the permission to do so, and to potentially fail. You really don’t need that pre-meeting for a meeting to make a decision about taking a decision to a decision board to make a decision on it. This really happens. It is a waste of time. It stops things getting done.
• Politics —if you curry favour by playing games, go play elsewhere : too often, individual or team politics cause problems in getting the right things done. It causes stress. It affects lives. It can literally waste billions. See Culture and People.

You don’t need to “Do Agile” to get those right. But you do need to “be agile” in your thinking and behaviour if you’re to stand a chance. There is plenty to take from agile that benefits our thinking, and Agile done well can be damn effective, but it is essential to properly understand why things are done the way they are, not just blindly implement bullet point behaviours. Agile is not a silver bullet that will fix everything else.

“There is no quick fix short cut to the top — doing hard things is inherently hard. We either pay now, or pay 10x later.”

It is clear that we still have a way to go to solve the problem of efficiently operating the human processes involved in turning ideas into operational software, and that simply labelling yourself with the latest buzzword-du-jour doesn’t really solve the problem, it simply masks it for a while. So don’t just say you’re “Doing Agile”, try to really become agile.

What next?

We have a great opportunity to build better systems that help us create the Digital Enterprises of the future. Let’s not waste our time repeating the same age old mistakes, or chasing the latest buzzwords — let’s focus our effort on building solid, simple, and effective foundations that let us spend our time on creating outward facing value and highly automated digital enterprises of the future.

You can download the report in full here: https://enterprisecore.io/mastering-digital-software-engineering-report

This study and report were done as part of my research for an up-coming book: Mastering Digital: how technology leaders, architects, and engineers build the Digital Enterprises of the Future