Some things about the way that companies run are old-school for a reason. Take corporate boards. When you're invited to sit as a director of a company's board, it's usually because you've earned some admiration for your expertise and experience in a given field.
It used to be that large companies wanted well-known names on their boards, but since the Sarbanes-Oxley law of 2002, it has become a much more tightly regulated and scrutinized affair.
It's also a lot of work. Directors must work their way through onerous briefing documents and examine things like due diligence reports prepared ahead of M&A deals. For decades all of that required a lot of paper, prepared for each and every director. But since the late 1990s, the software company Diligent has specialized in making that process digital, first with a Web-based 'board pack' service, and more recently apps that directors can run on their iPads.
As luck would have it, I wrote a profile of Diligent for Recode in 2015, and so I was interested to attend vice president of operations Tricia Burke's keynote at PuppetConf 2017 to see how things have been going since then.
Many corporate boards still use email to communicate and do board business. And that's not ideal for a lot of the same reasons that software and IT teams have adopted tools like Slack and Hipchat. It's not secure, it's not efficient. 'As a company we're dedicated to providing a secure and effective way to allow directors to communicate,' and providing tools that all directors want to use, Burke said.
But Diligent had fallen a bit behind in how it delivered and updated its products. 'Ironically communication [across our teams] was our biggest problem.'
With teams spread around the globe, Diligent was pushing out an average of four monolithic releases every year. 'The developers would write the code and QA would test it. They would create a build and they would throw it over the wall to the operations team.'
And that's where the communication breakdown would occur. 'Sometimes the operations team would get a release and they wouldn't know what to do with it. Sometimes they would know what to do with a release, and it wouldn't work.'
The result was a lot of tension between the teams. The response was to schedule meetings. 'We talked a lot,' Burke said. 'We talked about releases that were coming. We talked about things that weren't working in production and what we were going to do to fix them. And this helped. But it wasn't enough.'
What was missing, Burke said, referring back to Thorsten Biel's session on Porsche's transformation, was trust. 'We brought everyone together and got them hanging out together in order to build rapport.' The hope was that building relationships between members of each team would make for a smoother process of solving problems and answering questions when they would occur. That helped too, she said.
But it wasn't enough.
'What we realized was that we were missing something. We needed a team that was just focused on the releases. So we created a release engineering team,' she said. This team was familiar with the development process, but they were experienced operations people, too. Their role was to bridge the gap between development and operations teams.
They were also dedicated to the idea of DevOps. 'It took a lot of communication to help everyone understand that this new team was here to help them produce higher quality software and get it out to the market faster.'
It was also the right time to introduce Puppet to Diligent's release process, Burke said.
The first step was to unravel some custom configuration scripts that had been in place for years using a home-grown configuration tool. 'We had to run that tool in every development and test environment,' plus eight production data centers, she said. 'And if something didn't work we were combing through logs and looking at servers and trying to identify the problems.'
So a few components of the old tool were re-written as Puppet code. The process took two months - two weeks to write the code and the rest of the time convincing the dev and ops teams that it was okay to use it.
Despite buy-in from the CTO and upper management, there was still a lot of institutional resistance to making the changes across the company. 'The reality was when we got to the point that we were ready to make the change, people weren't ready for it,' Burke said. 'They were really worried that the changes were going to impact our releases. And in the end when you work for a software company delivering your software is the main reason you're there. If you can't do that then you really have a problem.'
It was about that time that Burke and her team started pushing the idea of continuous delivery of the software. 'We talked about how all companies who built software were having the same kinds of problems that we were having and how DevOps was the de facto way of solving them,' she said.
And while it's one thing to say it, it's another to show it in action. As has been the case for a lot of Puppet's customers. It took a greenfield project - one lacking any links or constraints to prior or ongoing work - to finally convince everyone, including the holdouts.
'We started doing releases every two weeks and suddenly everyone realized it was going to work.'
Fast-forward about 18 months to the present and Diligent's main product line has been 'puppetized' and three additional greenfield projects are also underway using Puppet. 'We ended up maxing out our whole release engineering team.'
Now the priority is figuring out how to best organize that team, and decisions will have to be made fast. Diligent has acquired five different companies. And each of those has its own set of tools, processes and communications challenges.
Some of them already have great pipelines and are already embracing DevOps. Others are still doing monolithic releases. 'When I tell these folks that we're doing 50 releases a year they practically fall out of their chairs.'
Burke says she advises those teams to 'take it slow,' at first.
'Use the DevOps approach that we used, start to get your team moving forward with automation, and try to double the number of releases you're doing every year until you can release at will,' she said.
Once you've got your first win and have buy-in from the top, she has one last bit of advice: 'Brace yourselves because a whole lot of work is going to be flowing your way.'
Arik Hesseldahl is veteran technology journalist and independent analyst. He was a founding editor at Recode, has written for The Wall Street Journal, Bloomberg BusinessWeek and Forbes, and has contributed to CNBC and NPR.