Puppet’s 2024 State of DevOps report was subtitled “the evolution of platform engineering”. Platform engineering, it said, can be a “barrier against the chaos of tools, tasks and information” assailing developers – a state of overload, which some would attribute to the lack of discipline in DevOps.
Or, as Paula Kennedy, co-founder of platform consultancy Syntasso, says, it is the practice of taking “an intentional, focused” approach to curating the combination of infrastructure, services, capabilities and tooling needed to make software available.
The aim, she adds, is to ensure “developers get a better experience, that software is delivered faster, safer, more securely, more reliably”.
One key element is the IDP. Depending on who you listen to, that stands for “internal developer platform” or “internal developer portal”.
But let’s stick with the latter. At its most basic, this is the central self-service platform developers use to access the tools, services and other resources, such as documentation, they need. Or which their organisation decides they need.
Backstage, the open source platform developed by Spotify, is the flagship portal for platform engineering, and the undoubted market leader. But there are a number of alternative products offering platform engineering capabilities.
A survey of 180 companies by DX found that over two-thirds used Backstage. The State of platform engineering report cited 55% penetration, with Port garnering 8% of the market and Cortex gaining 5%.
Harness, the continuous integration and continuous delivery (CI/CD) platform, also plays into this space. Kennedy says there are also a few “platform-in-a-box” products, such as Red Hat’s Developer Hub, which is based on Backstage, and some of the big software development firms will have their own products.
For developers who already use team collaboration in Atlassian, there’s Atlassian Compass. As Matt Saunders, vice-president of DevOps at solutions provider Adapavist, points out, it’s an Atlassian tool and the Australian company wants to own the whole stack. It does integrate with other tools, he says, “but the reality is that Compass integrates best with the Atlassian stack”.
More than the portal
By contrast, Saunders says: “When you run Backstage, you have to do the manual work to get it to integrate with all your tools.”
He adds: “[The fact] there is competition in the IDP world is obviously an indicator that businesses are succeeding in doing IDP right … It’s a validation. These ideas are correct.”
And a portal is not the be-all and end-all – or necessarily essential. Stack Overflow had a legacy monolithic system supporting its public-facing sites. Its newer software-as-a-service (SaaS) offering, Stack Overflow for Teams, has been cloud-native from the outset. It has recently consolidated all of this into the cloud.
Right now, explains Stack Overflow’s chief product and technology officer, Jody Bailey, it’s managing things internally while deciding between Backstage and another open source project, Kratix.
“They’re both open source tools, but kind of have different purposes,” he explains. “Kratix allows us to help with the actual orchestration and the work being done by platform teams, whereas Backstage is more about surfacing, making it visible and useful.”
Bailey says committing to a specific portal also means “you have to dedicate somebody to managing it, keeping it up to date, connecting all the dots”.
But the portal is just the start, adds Saunders. Getting the underlying tools in place is critical. “Setting up things like proper CI/CD, proper source code management, build environments in cloud providers, and, more importantly, putting the right sort of guardrails around that,” he says.
Or as James Sturrock, director of systems engineering at Nutanix, puts it, ensuring “developers are liberated from infrastructure chores such as provisioning, cloud configuration, networking, or setting security policies, and they get to focus purely on code”.
This becomes a bit of a Goldilocks job. Part of the rationale for platform engineering is to address the tool sprawl that DevOps unleashed.
“The big problem we see is where people think that a platform is a feature factory and it should just add more and more and more stuff,” says Kennedy.
At the same time, being over-prescriptive can be counter-productive, to the extent of encouraging developers to adopt shadow IT. And then we’re back in the early 2010s.
“One of the challenges that we see often with platform engineers is that they are engineers,” says Kennedy. “So they think to themselves, ‘I don’t need to ask what the developers need, because I am an engineer, so I know what I’m going to build’.”
Describing where he’s seen platform strategies fail in the past, Bailey says: “You have a team that’s focused on building out services for the platform, and say, ‘This is how you’re going to do things, you have to do this’.”
That tends not to work with developers and technologists, he adds. “They want to do whatever is easiest, that golden path, to be successful.”
Enter the agents
Rather says Kennedy, the aim should be the right mix of tools – off-the-shelf and bespoke – together with the right balance of ease of use. In effect, a bespoke system for a given organisation. “That’s really the end goal – your platform should be a force enabler,” she says.
This cannot be a one-off job and requires constant dialogue with the teams that will be using the platform. Which is also a reminder that, as Kennedy says, leaders have to think beyond the development team.
“We try to think about it a bit more holistically, because you might have security folks who are using the platform. You might have legal teams going into the platform. Typically, yes, it’s developers, but your platform can be bigger than that,”
This means ensuring there is a “modularity or composability” that allows direct inputs from security, compliance, finance, and anyone else who has responsibility for critical business requirements.
Once that perfect platform is in place, Saunders continues, this can then be a wake-up call for an organisation, because it exposes what’s really going on. “How do our developers actually get stuff done? Is it meetings all the time, twiddling their thumbs, waiting for an opportunity to actually write some code? The answer in a lot of organisations is ‘yes’, and things like the IDP start to surface that.”
And before long, it won’t just he human developers that platform teams need to worry about.
“Your platform itself needs to be able to adopt and bring in new capabilities and new technologies like AI [artificial intelligence] quickly. Your platform can’t be brittle or static,” says Kennedy.
This affects platform engineering efforts in multiple ways, according to Jeffrey Sica, head of projects at the Cloud Native Computing Foundation (CNCF).
“Backstage is a great example of the kind of work that’s going to be needed to make AI agents even more effective,” he says.
After all, he adds, we are already creating all these integration points to create this single funnel for the developer, and agents are going to need the same thing.
It’s a truism that AI and agents will take over much of the grunt work of developing. Code generation is part of that. But Sica explains: “Imagine that through VS code, you can ask [Microsoft] Copilot to spin up a new development environment. Well, at that point, what’s happening is the Copilot agent can communicate with Backstage’s MCP [Model Context Protocol] server to spin that up for you.”
Having the agent go to Backstage “creates this very, very solid way for developers to query and consume Backstage, but without necessarily using the front end”, he adds.
Again, compliance and guardrails are necessary. Sica says these should already be in place up and down the stack. “You should not get access to the production cluster. You should not be able to ask, ‘Hey, Copilot, delete this prod instance of a database’.”
Because, after all, you wouldn’t let a human developer do that. Would you?
