Podcast: Cloud Native Computing Foundation with Dan Kohn

Dan Kohn is the Executive Director of the Cloud Native Computing Foundation. In this podcast, he discusses the goals of the CNCF and the reason why Kubernetes is under the CNCF's umbrella--plus his take on serverless computing. In addition to Kubernetes, the CNCF also hosts Prometheus, Fluentd, OpenTracing, and Linkerd.

In the links below, check out the cloud-native landscape in particular, which catalogs the broad set of projects playing within this technology area. As Dan puts it: "Kubernetes is the cornerstone of a containerization and orchestration solution but is not a complete solution."

• Cloud-native landscape on github
• Blog post about cloud-native landscape
• Cloud Native Computing Foundation
• CloudNativeCon Berlin, March 29-30

Audio:

Link to MP3 (0:22:25)
Link to OGG (0:22:25)

Transcript:

Gordon Haff:  Hello everyone. Welcome to another edition of the "Cloudy Chat" podcast. This is Gordon Haff, technology evangelists with Red Hat, and I'm sitting here, at the Open Source Leadership Summit in lovely Lake Tahoe, with Dan Kohn, who is the Executive Director of the Cloud Native Computing Foundation, which is under the Linux Foundation. Welcome Dan.

Dan Kohn:  Thank you very much. Glad to be here.

Gordon:  Dan, first of all, could you give us a little bit of background about yourself?

Dan:  Sure. I actually used to be the Chief Operating Officer of the Linux Foundation a decade ago, when it was a much smaller organization, back when there was just a few of us. I helped Jim, the Executive Director merge together the two predecessor organizations. I then went off and worked on a few startups, one of my own, one of another.

Then as the Linux Foundation has grown and brought in new organizations under it, and it has become a foundation of foundations. Jim has been recruiting in different folks to run the different sub‑foundations and pulled me back into run the Cloud Native Computing Foundation.

Gordon:  Cloud Native Computing Foundation is probably best known as the home of Kubernetes. That’s a very well known container orchestration platform. Maybe we'll start off talking about Kubernetes, and how you see the role of the CNCF with respect to Kubernetes is? How you see things are going? Then maybe about what are some the next steps you see happening are?

Dan:  Sure. We definitely are incredibly proud to be the host for Kubernetes. It's one of the most exciting software projects on the Internet today. It's also one of the highest velocity projects by almost any metric of number of commits per day, number of companies participating, number of developers participating, total volume of issues, pull requests. It's actually, probably just second or third behind the Linux itself in terms of velocity that it's been able to keep up.

Then even more than that, it's just the fact that it's out there solving real problems for users, for enterprises, for startups, all kinds of companies today, both in the public cloud and bare metal and private clouds where a containerization is this trend that's taking over the world to allow people to run all kinds of different applications in a variety of different environments.

When they do that they need an orchestration solution in order to keep track of all of those containers and schedule them and orchestrate them. Kubernetes is an increasingly popular way to do that.

Gordon:  Before it joined the CNCF or really formed the core of the CNCF, Kubernetes was already becoming pretty popular. Although there was contributions from a number of companies including Red Hat, certainly Google, which contributed in the first place. What was the genesis of the CNCF in the context of Kubernetes? Why was it really needed?

Dan:  The origin of Kubernetes was three folks at Google, and it was really built on the intellectual foundations of Borg, which comes from 15 years of Google experience with containerization.

As you said, they built that out and then they recruited folks from Red Hat, from Huawei, from a number of different places in the community and they said, "Hey, this project has a huge amount of potential. What will it really take for it to reach that potential?"

One of the things they realized very early on is that a project with a neutral home is always going to be able to achieve a higher level of collaboration. They really wanted to find a home for it where a number of different companies could participate.

A huge piece of that is the intellectual property framework where the idea is that Kubernetes operates under the Apache license what we think of as an intellectual property, no‑fly zone. Everyone contributes, there's no patents that the companies will file against each other.

Generally that the trademark, the rules are neutral between all of the different participants, all of the different users that, there's a trusted neutral body that that they can look at. Those early users went to the Linux Foundation and said, "You're one of the best‑known folks in open source. Can you help us work with this?" That's why they set up the Cloud Native Computing Foundation.

I think that's a good segue to say that, they also said that, "They were not just interested in creating a Kubernetes Foundation." They saw Kubernetes as a cornerstone of containerization and orchestration solution. As a critical piece of it but not as a complete solution and then seeing that there should be a number of other projects that were really very important to their stack.

Since I've joined over the last nine months, we've begun the process of bringing in new projects into CNCF. Prometheus is a very popular, well‑respected monitoring application that interestingly originally came out of Soundcloud.

Not a services company like Google is, but just they were initially scratching an internal itch, but it's since used by hundreds of different companies around the world and commercialized and are very popular. Now there's three new projects behind that one, Fluentd, OpenTracing and the newest one, just as of a couple of weeks ago is Linkerd.

We have several new ones in the pipeline. What we're trying to do is, over time, build a comprehensive open source stack of software that provides all the solutions that companies and enterprises and start‑ups and individuals need in order to deploy Cloud Native solutions.

Gordon:  Now, there's a lot of activity, obviously, going on in the broadly speaking cloud native space, there could probably be hundreds of projects in the CNCF, it casts a wide‑enough net. What are your criteria? What do you see the walls being around what you want to accept? What would cause you not to be interested in a project? Give us a little color about all that.

Dan:  One piece that I would recommend ‑‑ and I'm sure you can link to this from the show notes ‑‑ is that we're publishing a Cloud Native landscape, which is an open‑source document. It's on GitHub. It's trying to track basically all of the projects in the space. As you would imagine, there's a ton of them.

We're also tracking closed‑source startups and companies' offerings. It's a project, so if you see something on there that we're missing ‑‑ your company or your startup ‑‑ please open an issue, and we'll try and get it in there in the next version. That's a good way of tracking the progress and how we think of the space and the landscape. It's available at github.com/cncf/landscape, but we can include a link to it.

One of the interesting things about the way that CNCF is set up is that I can't actually bring in any new project. No one from the CNCF staff can and no one from the governing board, which are the vendors who provide most of the funding for our foundation, can't bring in projects, either.

Instead, we have a group of technical architects, experts in the field, folks like Bryan Cantrill from Joyent, and Alexis Richardson from Weaveworks, Brian Grant from Google, Solomon Hykes from Docker, who are our technical oversight committee, nine folks. It takes a supermajority vote of that group to bring in any new project. That's really a technical gatekeeping function, that they have very high standards for the kind of projects that can come in.

Now interestingly, when we first got started, with Kubernetes being such a successful, high‑velocity, exciting project, and then Prometheus as well setting very a high standard, we had a little bit of anxiety that the hurdle was going to be so high, say, "Hey, you know, there's not that many projects out there that already have hundreds of users, or, or thousands of developers."

The TOC recently just approved a new graduation guidelines that include a lower tier of project that we call, "Inception level." This is a little bit more of an experimental level to say, "Hey, this level of project isn't quite as mature as the other one, but it's very promising. We think it's really worth taking a look at, and we're optimistic that it will get there."

Then, it requires the TOC every 12 months to come back and essentially renew its status, either at inception level, or move it up to incubating, or just to have it exit the foundation.

Gordon:  What do you provide in terms of resources for those projects which are not up to the highest level yet?

Dan:  We actually try to provide the same resources for all of our projects, and so we love all of our children equally, or do our best to. There's a whole set of resources that by far the most important one is what I said before, which is that a neutral home for a project increases collaboration. That's the biggest piece of it.

If you say, "Hey, you're providing a CLA bot," which is a little robot hooked into GitHub which keeps track of whether the user has signed a contributor licensing agreement. Yes, we provide that, but are our engineers really better at Google than Google's at doing that? Probably not.

It's not that we're able to do something uniquely that Google couldn't, or Sound Cloud couldn't, or some of the other homes couldn't, but it's the neutrality is the huge value. That's the most important one, but we do have a whole set of foundation services, starting with the fact that I'm a full‑time employee, and we have several others.

We're all dedicated to promoting our projects and trying to help them succeed. We have a press and an analyst relations team, we have a really fantastic events team. We do two big events per year. CloudNativeCon, KubeCon, coming up in Berlin at the end of March and then in Austin's at the beginning of December.

We also help our projects with smaller events if they want to do it. For instance, Prometheus is going to be running PromCon in Europe this summer, that we are helping them organize. They're going to still do it as their own developer event, and not that's going to be a smaller scale one.

Then one of the really amazing, extraordinary resources that we have, it was made available to us by Intel, is a $20 million, 1,000 server cluster, which is housed by another one of our members SUPERNAP, at their switch facility in Las Vegas.

This is available for priority access to our five projects, but it is actually available to any open source project that's interested in demonstrating or experimenting with or working with a Cloud Native technology. Anyone can go to github.com/cncf/cluster, open an issue and file a request for an allocation from that cluster for as little as 20 machines and soon up to 1,000.

Gordon:  You obviously have a lot of experience in foundations, the Linux Foundation. There's a whole lot of experience, but things are always changing, you're always learning, environments change. What have you learned?

Dan:  That's an excellent one. I'd say by far the biggest part for me has really come down to a respect for the developer. I've been involved in open source in different ways in 20 years, but I really see the developer as ‑‑ I'm not quite sure, calling them the plankton of the ecosystem is the most pleasant metaphor, but I actually mean it in a very nice way ‑‑ in that every other aspect of what we do, absolutely depends on them.

When you look at a healthy project it has a ton of developers that are incredibly excited about it, that are contributing. Certainly the core maintainers that are actively involved in and hopefully they're being paid by companies to work on it or as consultants or such, and then probably a number of others who are doing it more part‑time or as a hobby or making specific bugs fixes or filing issues.

When you have those developers that feel like their contributions are valued and taken seriously, then there's a whole ecosystem that forms around them, of companies that are interested in offering services to them, employing them, that want to make these services available to other folks. Then a foundation like ours can come up and help make those services available. I really think that, that developer focus is the key thing to keep in mind.

Gordon:  At the risk of being a little inside baseball. I want to ask you about relationship between a couple of the projects under Linux Foundation. OCI, the Open Container Initiative is separate from the Cloud Native Computing Foundation. What is the reason for that? What do you see is the relationship there being?

Dan:  It's a very close relationship because the head of the OCI, Chris Aniszczyk is also the COO of CNCF. Whatever you want to say like the Chinese wall, we don't share it. It's definitely not that. There's a incredibly high overlap of 80 or 90 percent of our membership and otherwise. The other reality of helping foundations work is that we're pragmatic.

When we set this up, a lot of the core folks wanted to have standards organization, that was standardizing the technology behind containers, and so the Linux Foundation was willing and eager to help make that happen. At the same time there was an interest in having a foundation that was hosting some of the core technologies here, and we have that as well.

It's a little bit confusing. Sometimes it seems that we have too many panels of, what's the relationship between OCI and CNCF, but hopefully it's becoming clearer over time.

Gordon:  You've talked a little bit about the wide range of projects that do or can fit under the CNCF. What some your criteria are for membership or for becoming a project under the CNCF. If we look forward, let's say 12 months ‑‑ I'm not sure we can look any forward than 12 months in this industry, particularly in the Cloud Native computing space ‑‑ what changes might we expect to see?

Dan:  It's a great question. Even 12 months is a little challenging for me. My hope is that you'll see just more projects in CNCF that people are excited about, they see as complementary to the ones that we already have. I do want to emphasize that when we add in a new project, we're never saying to existing users of Kubernetes or Prometheus. "Oh, you must use OpenTracing, Oh, you must use Linkerd."

We're saying, "Hey, these are projects that we think are a complementary. We're investing continuous integration resources to try and make sure that these projects work well together. We're setting a signal that we think that these different projects attain the same level of quality."

My hope is that you would see another half dozen or more projects along the same lines that have been added into CNCF. I also hope that you would see all of our existing projects have continued to grow and thrive.

Then particularly these projects are happy with their home at CNCF that they feel like they're getting value from us, that our members feel were providing value to them. The bigger picture is, this is my comment to Jim Zemlin about the brilliance of the Linux Foundation where he said, go back 10 years.

You say, "Oh, you know, what was it about how it was set up? Was it that the technical advisory board of Kernel Developers was separate from the Linux Foundation Board?' Which it was and was very valuable and we replicated that model and I don't mean to downplay that was useful. Fundamentally, the reason the Linux Foundation was successful is because Linux was successful.

Similarly, I think one of the reasons that CNCF has a very good shot at being successful is because Cloud Native computing is an incredibly exciting trend that just has a huge amount of momentum behind it, in the public cloud, in private and bare metal computing and hybrid cloud computing. Kubernetes is one of the most exciting projects around there as is our number of our other projects. The future is definitely looking very bright.

Gordon:  What are seeing about serverless computing on the horizon? Certainly what you have now ‑‑ I think it's pretty fair to say its container‑centric. Do you see serverless computing as under CNCF purview and where do you see that going?

Dan:  I definitely do. We're talking to a number of serverless options that work with Kubernetes in different ways and there's different folks who are solving different pieces of that. I think AWS Lambda is incredibly exciting piece of technology.

There's a blog post that I just love of a YCombinator startup up called Benchling and they had an intern rewrite their application from running on a bunch of EC2 machines to working on Lambda and it's dropped the average time for all their users dramatically because of they didn't have to spin up the same number of machine that didn't have contention. They dropped their hosting bill from thousands of dollars a month to $60 a month.

It just been a compelling enough story. I mean it was a uniquely good used case for it, to say, "Hey, there's really something here." It's definitely an area that we're looking at, but what we would love to do is find some open source solutions for it that don't lock you into a single cloud provider, that can work across different cloud providers.

Gordon:  I think you touched on this may be a couple minutes ago but obviously the OCI for example in terms of container standardization has been a pretty significant benefit in keeping away a level of fragmentation, which might very well probably would have slowed adoption.

In Kubernetes being under the CNCF there's obviously other orchestration projects, but there does seem to be a certain role in terms of not picking winners necessarily but maybe, showing a preference for stronger projects and projects with larger communities. How strong is the CNCF in terms of showing preferences for certain technology choices?

Dan:  Our vision is that we're trying to promote Cloud Native computing, and we think that, that entails an open source stack for each of the different functions. We think that, as an example Mesos, Docker Swarm and Nomad are all perfectly valid orchestration platforms that are alternatives to Kubernetes.

Now, we host Kubernetes and so we obviously love that and promote it and are excited about it, but we also think that companies choose different projects. For example, Prometheus works great with all of those other orchestration platforms, as do a number of our other projects.

Certainly our expectation is that most of our or all of our projects should continue to work with all of their competitors. There's not any sort of and just in general terms of how the open source world works. That serve our hope of how things will continue.

Gordon:  Great. Thank you there's obviously been a vast amount of interest in projects under the CNCF. Your last cloud day of KubeCon, sold out. I don't think I could get in there in Seattle and you have another event, Berlin coming up, which as I understand you've allocated a whole lot more space for.

Dan:  To be honest with you we tripled the capacity from London up to 1,500 but we are on track to sell out in Berlin as well. Folks who are thinking about coming, we would really encourage you to sign up now. It's March 29 and 30th. We think it's really going to be a fantastic event. The schedule is up there now. This would also be a great time to begin thinking about Austin, this December 5th and 6th.

Podcast: Blockchain and Hyperledger with Brian Behlendorf

I sat down with Brian Behlendorf, the executive director of the Hyperledger project, while I was out at Lake Tahoe for the Open Source Leadership Summit last week. The Hyperledger project is an open source collaborative effort created to advance cross-industry blockchain technologies. It works as essentially an umbrella project on top of projects such as Fabric and Sawtooth Lake.

Brian was a primary developer of the Apache Web Server and a founding member of the Apache Software Foundation. In our conversation, he covered topics such as balancing innovation and standardization, the use of blockchains for transactions within consortiums of competing organizations, technical challenges, and early examples of blockchains use that go beyond cryptocurrencies.

• Hyperledger project
• Use of blockchain to stop spread of conflict diamonds
• Current Hyperledger projects
• Video: State of Blockchain (OSLS Keynote), Christopher Ferris, IBM

Audio:

Link to MP3 (0:14:32)
Link to OGG (0:14:32)

Transcript:

Gordon Haff:  Brian Behlendorf's the Executive Director of the Hyperledger Project, which is basically a blockchain project. Brian, you were just giving a talk, and someone said that was the most laid‑back talk about blockchain they had ever heard.

I think you're probably a great person to give a real quick summary of what blockchain means in the context of Hyperledger without the hype.

Brian Behlendorf:  I thought that was a nice way of him saying that my voice put him to sleep. Hopefully, I won't do the same to your listeners.

Now, we take a very pragmatic view of what's being built here, and what the world has needed, really, in this space.

Think of it as a decentralized database where the heads of that database, the nodes on that network, are potentially competitive, potentially rivalrous, and yet, they all have business that they need to transact together. They all need a system of record that keeps the order and the data of what they've transacted consistent and clear.

On top of that, you can build a lot of interesting things. You can have validation logic at each of these nodes so that you can do things like keep somebody from spending the same asset twice, or doing the same transaction twice for two different people. This is why you can build a cryptocurrency on top.

Cryptocurrencies are one kind of application, but there are lots of other types of assets that you would track in the system, lots of other kinds of data that you might log. The temperature here in Lake Tahoe might be a bit of data we want to record for permanence, because maybe I have an insurance contract that depends upon that data being accurately recorded, and never being able to be deleted.

We also take the point of view that these networks don't have to be the size of the whole wide world, that there are interesting systems of record built by consortia, built by collections of organizations, companies, government agencies, those sorts of things, where you don't need to have the physics of a cryptocurrency to be able to allow for these interesting applications to be built.

Hopefully, that dumbs it down a bit. On top of that, you can build amazing things ‑‑ smart contracts, that sort of thing, but at its core, that's what it is, and that's what we're shipping product today around.

Gordon:  From what you just said, I think maybe the best segue into, "What does blockchain give us in a private or semi‑private consortia, I think is the term you used, context that a more traditional distributed database, which obviously has certain technical advantages from the point of view, perhaps, of transaction rates and so forth?"

Brian:  The big difference is more political than technical. The big difference is most of the way that these distributed database systems have historically been built, they've presumed that they've all worked for the same person.

Therefore, their profile of attack doesn't presume that one node wants to corrupt the data that's sitting on another node, or gain an unfair advantage when that node wants to write entries and keep the other side from being able to write entries as well.

Within all of these systems, there is a consensus mechanism that establishes a fair process for each of these parties to be able to write into this network, and then cryptography at every layer, implementing what are called Merkle trees, simply ways of taking hashes so that you can guarantee that the data is kept in order.

You have your copy of the database, I have my copy, Ray has his, other people have theirs, but we can guarantee by comparing these hashes that our copies of the database are all in sync, have the same exact order to them, and this is something traditional databases have not provided any guarantees around.

Gordon:  From a technical standpoint, what, from your view, are the biggest technical challenges that have to be overcome to use Hyperledger, to use blockchain, with smart contracts, supply chain, financial transactions, and so forth?

Brian:  I'd say initially, it'll be speed, and in the long term, it'll be confidentiality. In the short term, we're going to have numbers that are not nearly so impressive as in a traditional database when it comes to transaction rates.

Right now, we see, with Hyperledger Fabric, for example, on a well‑tuned system, a couple of hundred transactions a second. We know we need to get that up to a couple of thousand, maybe over 10,000, and we have pathways to get there, but talk to any database admin, and that's a trivially low amount, especially when you have tools like sharding and other things.

That's because what we're trying to do with a blockchain is very difficult. Every proposed transaction is broadcast to everybody else, and some threshold number of those have to confirm it before everyone locks it in and says, "Yes, that's actually the next link in the chain, the next block in the chain."

That's a much more sophisticated thing, let alone if you have any validation logic you're trying to apply to prevent double spend, that sort of thing, or smart contracts that you want to run on top. That's the first challenge.

The second is the simplest blockchain is one where all the data's unencrypted, and all the data is shared with everybody. There are use cases where that's valid, where everybody wants to know the same data, say if we were building a database of certificates for TLS. Everybody wants to know the public signatures for TLS certificates.

But if we're doing something anywhere more meaningful, like recording transactions between two parties, we want the rest of the world to know about, but maybe not the price that we're paying for that transaction, maybe not some part of that transaction, and the fact is, right now, like with Bitcoin, you have exact information about who's spending what money on what.

Confidentiality is all about the set of techniques that you might apply to providing greater privacy for some of those transactions while still getting the benefit of logging into a chain, and having the consortia know that that transaction happened, and perhaps even know something about it.

This is where zero knowledge proofs, homomorphic encryption, there's all these emerging fields that are still fairly young by computer science standards, that will come into play to help give us confidential transactions. That's the horizon.

Gordon:  From what's going on right now, what are some of the specific interesting use cases that you're seeing?

Brian:  The world of finance is full of ledgers, full of systems that keep track between two parties, between thousands of parties. "Who has wired what money to who, who owns which shares of stock of a different company? Where is the underlying paperwork behind this or that?"

There's lots of drop‑in use cases for this technology there, where you don't have to talk about different governance models. What you're doing is simply dropping out a central authority that might have already existed in that system, and say, "Here's a peer‑to‑peer way to do the same thing."

Much less expensive, much more rapid for confirmation. This way, wires between two banks might clear in five minutes rather than three days.

Those aren't the use cases that get me up in the morning. The ones that get me up in the morning have more to do with helping people secure the ownership of their house in a database that can be a public database or semi‑public database. Land title databases are an active field of conversation.

More concretely, there's a project already in pilot that's moving to production ‑‑ that is the diamond industry ‑‑ reinventing their provenance tracking system that's used to keep conflict diamonds out of the market and help you know when you buy a diamond what part of the world that came from, what mine that came from.

That is, today, implemented as paper‑driven, centralized, bureaucratic kind of process with very little transparency. That's being reinvented as a blockchain, partly for greater transparency, but partly, as well, for greater audit ability. It's already in pilot. It's already caught millions of dollars in fraud, and when they move to production, it'll become an airtight system.

That's what gets me excited is these positive social impacts that at the same time, are also potentially helping solve structural problems for the business sector. I haven't seen that kind of synergy, that kind of combination of value from these two different things since the early days of the Internet.

Gordon:  Let's switch gears to talk about Hyperledger, specifically. You've described Hyperledger as, essentially, an umbrella to a number of different pieces that fit together. Can you tell our audience a little more about that?

Brian:  This space is still pretty young. You could say it was kicked off with Satoshi Nakamoto's paper in 2008, and it even taps into distributed database design going back to the '80s.

Understanding how to map this landscape and the diversity of different consensus models, different ways that we might write smart contracts, that's still pretty young for most people. We realized it wasn't going to work if we came out and said, "We've got the one true architecture, the one true consensus model, the one true framework."

We bootstrapped the project just about exactly a year ago with two different projects, one called Fabric, another called Sawtooth Lake. It started life as internal projects at two different companies, but now with an open source, both of which have blossomed, both of which now have a diverse contributor ecosystem behind them.

They solve the same problem in two very different ways. This is a challenge, because explaining this to people can be about as challenging as explaining why they would want to use Debian versus Red Hat versus Ubuntu.

We know there's real differences there, but explaining those to the outside world is part of our challenge. Now we've added other projects like Iroha. One's coming in called Corda. These are different distributed ledger technologies. There's other technologies built above that that we will bringing in, too.

We feel right now it's important to map the landscape to encourage R&D efforts across that landscape, to decentralize, in a way, and then let some combination of market suitability and Darwin basically play out to see which of these emerge as the ones that people tend to gravitate to.

The others, either they merge, or they find some interesting niche, like Darwin's finches on Galapagos Island, to, "Here's a specific kind of prey, a specific kind of use case that a certain technology becomes appropriate to."

All of that happening within Hyperledger means we get, hopefully, at least, a culture of remix, a culture of creativity, a culture of borrowing from each other, and of helping each other, looking for places to integrate with each other and combine efforts so that this makes sense to the outside world.

Gordon:  It's interesting, as I was talking to Chris Aniszczyk earlier about OCI. Their charter is really around standardization. It sounds like you sort of see your charter at this point in time as really focusing on innovation, and let the standardization, maybe some of the ease of use, come with time.

Brian:  I think so. I think that's the way that innovation has tended to work best, is for standardization to follow implementation. When you have implementations under an open source license then all of the traditional concern about closed de facto standards starts to dissipate.

"Here's this. Anybody can use it. There's no rivalry around an interface implemented by open source software." While we're developing this code, we're kind of developing open standards at the same time. What everyone wants is stability.

What everyone wants is to know, "When is it time for me, if I've got a $20 million project to bring over to this that requires me orchestrating this with six different business partners, I want to move to something that's not going to be thrown away in a year by the core developers, and forgotten about."

Again, part of our challenge is going to be making this activity make sense to the outside world, sending a signal, "Fabric will hit 1.0 in the next few months," which will send a big signal that this is now a platform that is solid enough that we feel people can run this in production and start their serious migration towards.

The other projects will start to hit that same kind of milestone this year as well, so that's a focus for us, for sure.

Gordon:  Before closing out, maybe I get to ask you one last question. There are a number of blockchain‑related projects out there that aren't directly related to Hyperledger, like Etherium, for example. What do you see as Hyperledger's relationship with those?

Brian:  I think the cryptocurrency use cases are part of the landscape. We have started from a different place, which was more of the consortium chains and tracking non‑currency types of things, partly because we know how hard it is to run a public chain.

There's serious challenges that those communities, I feel, are tackling fairly well, and there may be opportunities to find projects that span those worlds and our world, especially as you go up the stack. There'll be a lot of demand, I think, to have transactions that talk both on ‑‑ in some cases, on a consortium chain, in some cases, on a public chain.

I think that'll happen naturally, that kind of convergence and collaboration between these communities. We don't have any antagonism towards them. We don't see them as competition. We're sharing ideas, and we're learning from them, too. We're at a point now where at least the world is expanding fast enough that we can all be friends.

When blockchain winter comes, maybe it'll be tougher ‑‑ I'm smiling as I say this ‑‑ but it's a pretty healthy collaboration now, and I respect the deep technical work going on that side of the world, but they are solving fundamentally different problems.

Gordon:  Thanks a lot, Brian, for your time. Is there anything you'd like to leave our audience with?

Brian:  No, I just want to thank the companies that have been committed to open source software for a long time, like Red Hat. It really has helped serve as a model for explaining to others how you can build a business on top of open IP.

That keeps coming up as a recurring question in the blockchain space. "How do all these venture capital‑backed startups and these big companies make money from this?"

The good news is, we're now a couple of generations in to this open source transformation, and so I think Red Hat has been a stellar example of that, so, really, thank you.

Podcast: Why sysadmins hate containers with Mark Lamourine

In this podcast, Red Hat's Mark Lamourine brings the perspective of a former sysadmin to explain why containers can seem like a lot of work and potential risk without corresponding benefit. We also discuss the OpenShift Container Platform as well as a couple of Fedora projects aimed at removing barriers to sysadmin container adoption.

Show notes:

• Containers overview
• Podcast on configuration management with Mark
• OpenShift
• Fedora Modularity
• Fedora Layered Docker Image Build Service 

Audio:

Link to MP3 (0:29:20)

Link to OGG (0:29:20)

[Transcript]

Gordon Haff:  Hi, everyone. Welcome to another edition of the "Cloudy Chat" podcast. I have my former partner in crime, Mark Lamourine, back with me today. He's been off doing some other things.

Mark came to me with a fantastic title for a podcast, and I knew I just had to sit down with him, "Why Do Sysadmins Hate Containers?"

Mark, you've been a sysadmin.

Mark Lamourine:  That's my background. My background is a system administrator. I have computer science degree, but I've spent most of my time either as a system administrator or as a developer advocating for system administrators who have to manage the software that people I'm working with produce.

Gordon:  I go to an event, like Amazon re:Invent, for example, and there are a lot of sysadmins there, maybe a little more new‑age system admins, DevOps, whatever the popular term is this week. They seem to love containers. Where do you make that statement from?

Mark:  There's actually two. What brought this up to me was I was at the LISA Conference, LISA16, in Boston this fall. I noticed that there were only a couple of talks, one tutorial, and a couple of books on containers. There was a lot of the other traditional sysadmin things. There's new tools. There's people learning different areas.

I was there because I assumed that sysadmins were going to still think that containers are growing and that this would be a big thing coming. There was some of that, but I got an awful lot of, "Yeah, we don't do containers. We tried containers. It didn't work. That's old hat." There were a whole bunch of things which ranged from disinterest to disdain for containers among that group of people.

The difference between that group and a group at re:Invent is that re:Invent is specifically aimed at the technology. It's aimed at that company. It's aimed at Amazon. All the people who come are self‑selecting interested in cloud, in Amazon, in their products, and in their tools.

At LISA, the self‑selection is I am a professional system administrator without regard to the technology I use. There were a bunch of people there who use Amazon. They use virtual machines. They use cloud. They didn't find containers to be a compelling thing to follow.

Gordon:  Why don't they find containers a compelling thing to follow when everyone says they're so great?

Mark:  There were a number of different reasons that I heard. Some of them were just misinformation. There were people who said, "Yeah, we knew about that with jails." In 1970s, BSD had jails in its chroot. I'm not going to go into it, but there's an answer to that. Containers are not that. That was a very old thing.

I liken that to saying, "Well, this guy, Jameson," or, whatever his name was, in France, "discovered inoculation back in the 1800s. Why do we need flu vaccines and monoclonal antibodies?"

Gordon:  It's like, "Oh, what's this cloud thing? We had time‑sharing." "Oh, virtualization. That was invented by IBM in 1960. What do we need this new?" It's this idea of, "Oh, everything's been done before."

Mark:  There are a number of things like that. There's a number of flavors. The, "Oh, Solaris had Zones. We know about that. See where that went." There were a number of responses like that. There was a number of, "Oh, it's hype," and those people aren't wrong.

It also is an incomplete answer. I agree that it's hype, but I also agree that it's important because while the hype maybe way out in front of reality, reality is way in front of what it was three or four years ago.

Gordon:  They just don't see any benefit for themselves?

Mark:  That's really the sense that I got. When I got past the people who were just naysayers for whatever reason, and I started bringing up, "Here are these tools. Here are these things I've used. Here's what I've done with it," the response was, "Well, but how does that help me?"

They're getting their developers, and they're getting their managers coming saying, "Oh, we need, well, cloud in some form." Some of them are OpenStack, some of them are Kubernetes, some of them are OpenShift, but their managers and their developers are saying, "Hey, there's this cool thing," and the sysadmins respond with the two kind of predictable responses.

One is, "Yeah, OK. I'm going to build a service for you that's work for me." The second one is, "This doesn't really help me. It gives me a lot more work. I've got to build new containers, I've got to build all of this stuff," or they would say, "Let's put our app into containers," and everyone's first response is, "Let's shove the entire application suite into one container and treat it like it's a virtual machine lite."

Everybody finds quickly that's not productive. It requires a lot more work to do refactoring. Somewhere in that process, many of them have said, "Our engineers got tired of it," or, "We got tired of it, and we just went back to the old way of doing things, because it doesn't buy us anything right now."

Gordon:  I'll get back to doing things the new way versus the old way in the moment, because I think it's an important point. There's something that those of us who promote technology often forget. Without saying these sysadmins are Luddites, they have a job to do. That job is to keep systems up, and the idea of, "Let's do this new stuff that's going to put me out on the bleeding edge, and probably get me on pager duty in the middle of the night when I'm trying to sleep." That just doesn't sound very appealing.

Mark:  As a sysadmin, I think sysadmins are a slightly different breed from many other geeks and technophiles. Sysadmins are, by their nature, conservative. They are probably the least bling attracted technophiles you'll find. In large part, they're the ones responsible for making sure that it works, and so they're going to tend to be conservative.

They'll explore a little bit, but their goal really is to make things work and to go home and not get paged.

Anything that you introduce to them that is both a lot of day work, and an opportunity to get paged, they're going to greet with a certain amount of skepticism.

Gordon:  That's absolutely fair. I don't think developers, and certainly not the move fast, and break things crowd, really appreciate that aspect of sysadmins. I think there's also an element though of, "This new stuff is going to abstract more. I already understand how the system works. It's going to create a new point of failure. It's going to complicate things. It is something else that I'm going to need to learn." That's not necessarily always the right point view either.

Mark:  No, but they're human. Their goals are to make their own lives easier. As a sysadmin, one of the other characteristics of sysadmins is that they will spend a lot of time avoiding doing a tedious task twice. They'll spend a lot of time creating a script to do something that only takes them 10 or 15 seconds to type. When they've typed it the 100th time, they get tired of it. Those things they know how to do.

When you impose something new, because it's a requirement for other things, they're going to be resistant to that until it helps them because they're inherently lazy people. I mean that in the best sense.

Gordon:  Actually, that was pretty much the topic from some Google presenter at a recent conference. I don't remember what the details of it were, or who it was exactly, but he went through the Google infrastructure, and how Google approached problems.

It was at Cloud Native Con/Kubecon actually, and they talked about how their approach was, "Oh. I've done this three times. It must be time to automate it."

Mark:  Sysadmins, I find it, and I know this is true of me, are pathologically lazy. Again, I use that in the nicest sense in that there are times when I have spent an hour, or more, understanding and encoding an automated solution to a problem that literally took me a minute a day.

It sounds like, "Well, that was a waste of an hour in a day," except that after a while, it saves me an hour.

Gordon:  We're doing a podcast right now and there's a lot of fairly repeatable manual processes associated with this. I'm putting some intro on, I put some outro on, I do some transcoding to different formats.

There's manual editing of course, and you can't really automate that. I spent a couple of days at some point, writing a Python script, and now, it's super quick and not nearly as error‑prone. "Oh, I forgot to make that file public on AWS."

Mark:  This is a characteristics of sysadmins that they do want to automate, but they are always going to use the tool they know first. Containers certainly present a real long learning curve before they start seeing the benefit. That's where a lot of the resistance really comes from.

Gordon:  It's probably worth contrasting containers with virtual machines in that regard, because the way virtual machines came in was, we were in the dot-bomb, dot-com type of era, nobody had any money, with these underutilized servers.

People just wanted to improve the utilization of those servers. They didn't want to change their operational procedures in major ways to do it. That's one of the reasons virtual machines became so popular as the approach at the time. They solved a specific problem, utilization of physical servers that no one had any money to buy, but without requiring a lot of changes.

Of course, virtual machines did evolve over time. Things like Live Migration and different things in terms of storage pooling and that kind of thing. But, fundamentally, virtual machines didn't present a big barrier to sysadmins from an operational point of view.

Mark:  All it really added was the virtual machine barrier. Once you got your virtual machine going, you could hand it off to another sysadmin or to a Puppet script and say, "That's a computer. Acts just like your other computers."

Containers are not VMs. They are processes with blinders on. Building those blinders is a new way. You'll see people who still try and treat them as VM lite. There are people who will try and stuff essentially a virtual machine inside.

They very quickly find that there's either no benefit, or it actually incurred some cost. Those people will often go back to using virtual machines because they don't want to change their paradigm. The argument of the people who actually are advocates of containers is that there is some benefit to this model.

It doesn't appear quickly and it requires a lot more retooling, both of the real tools and of the mindset of the administrators before they see the benefit.

Gordon:  We've gone into configuration management in a lot more detail in another podcast, but there are some analogs there. You can certainly do automation using scripts, but for example, as VMs came in and suddenly you're multiplying your number of "servers," by 10 or 20 or whatever the number is, doing the scripting didn't work so well any longer.

You really needed some of these newer types of tools that did things in more declarative ways, or that did things in other ways that were more suitable for very large and complex app configurations.

Mark:  That actually brings up a psychological or behavioral analog that I wanted to highlight, is that we've had situations like this before where either sysadmins or various other areas of the community were resistant to changes that some of us saw as important early.

I remember trying to convince people that they should use sudo. They're like, "What do you mean? I can't code without root. How can I possibly work as a non‑root user? Why should I use this sudo thing? I just always log in as root anyway."

It took about a decade from when I first saw sudo, to the point where it was just accepted common practice that everyone did. There were other analogs like that. Configuration management was one of them. I remember having conversations with people who would say, "I run a shop with 10 or 20 hosts. Do I need configuration management?"

Those of us in the room were going, "Yes, you absolutely do." They go, "But, I don't have time for that." I'm like, "That's why you need it." It took a while for that to catch on. I think virtual machines were actually a big influence in the adoption of configuration management in the small shops.

They went from having four or five machines to having 20, 30, or 50 VMs. Even though they had the same hardware, they saw the multiplication of the manageable units. Configuration management made sense to them. Again, that's become a common part of a sysadmin toolbox, is to know Puppet or Chef--not CF Engine so much anymore.

Ansible now is big one. You're talking about the familiar tools model. Ansible's largest appeal to a lot of people is that it looks like shell scripting, or it looks like these other scripting languages where Puppet and Chef were more traditional configuration management.

That's a bit of insight but the analog still holds. In containers, you've got people going, "Do I need that? Why do I need that?" Those of us who have worked with them a lot are going, "Yes. Yes, you need this." It's going to take something to induce them to see it.

Gordon:  You're seeing the cycle continuing with security around containers, for example. I was just reading yesterday, there was a minor possible exploit related to containers. Details don't really matter here, but there was a discussion going on in "Hacker News" that basically people were going, "Well, you could change this and that would fix this," and so forth.

Somebody wrote, "Or you can use SELinux and setting enforce=1, and this exploit can't happen."

Mark:  That's another tool that still is in the usage phase. There are still people who go, "SELinux is too hard to use." They'll disable it just as a matter of course. It still is hard to use, but most people don't need to interact with it that way most of the time.

Another barrier to entry for people creating containers is they have to think about resource usage that they didn't have to think about when everything was on the host. They just install a new thing on the host and the resource would be there.

Now, they have to make sure that all the research sources they need are inside the container. For a sysadmin who's just working on the box and trying to run some tool, the idea of putting that tool in a container really doesn't make sense for them, until they can treat the container the same way they treat the installed package.

Gordon:  Do sysadmins need to just suck it up, or is there something we can do for them?

Mark:  First thing is that, no, they don't have to suck it up. They are the ones doing the job. It's up to them to decide when to use this stuff. We can advocate and we can try and give them the tools, and we can try and make a point and help people understand.

We have the responsibility of understanding too. They are our users. If you present software to a user and it doesn't help the user, they are not going to use it, no matter who they are. We do have a responsibility within the container community to look at their usage, look at their needs, and find ways to help them.

There are a couple of projects I know about right now that are trying to address that. The Fedora Modules Project which is a project a guy here works on, Langdon White, is one of the leads on it.

I'm going to do a little aside here. One of the objections that people had to containers was they said, "Well, they're just packages. We've done this before. We know how packages work."

I would disagree that that is a sufficient answer because there are packages with other stuff on them that can work in different ways, but they had a certain point. If you're going to use containers to do sysadmin jobs, sysadmin tasks on a host, you need to be able to treat them like a package. The sysadmins need to be able to use a model similar to what they're used to.

Fedora Modules is an attempt to take packages which are commonly installed on hosts and put them into containers that can be used in a manner similar to packages. The best examples right now are things like web browsers and certain system services. The packages are trying to address early are things that sysadmins would use on a regular basis.

They would commonly install a web server, Nginx or Apache. They'd install it as an RPM, and then they'd configure a bunch of root files or a bunch of owned files. The Modules Project is trying to produce a container image which can be used the way an Nginx RPM would be used in that you say, "Container install this thing," and, "Container enable," instead of "Package install, package enable."

Now, you have an Nginx running on your box. It's running in a container instead of as a host process. The benefit, if you're using containers, is that you can run multiple Nginx on the same host without having to worry about separating the configuration files. They can run as independent things.

The Modules Project is just trying to create these containers which are analogues to host RPMs for things that are appropriate. Chrony was one I heard someone talk about, so NTP services, web services, other systems services like that that don't necessarily have to be bound, especially ones that are for users, like the Apache Server, or a MySQL Server.

There's no good reason for that to be installed on the host. This would allow a way to do it.

The second project that is trying to address it is working from the other side, from the developer side. That's the Fedora Layered Docker Image Build Service. The people in the Fedora Project have looked at the model, the build model for RPMs, and said, "Why don't we apply this to containers?"

They've created a container service, a container built service, which is an analogue to the RPM submission and build service so that instead of submitting an RPM spec which points to a GitHub repo somewhere for software, you submit a Docker container spec, and it might be a Docker file, but it might also be some other Docker build mechanism.

What you get is a professionally packaged container in a well‑known repo signed by the Fedora Project so that unlike Docker Hub where anyone can put anything out there, in the Fedora Project, it has to go through a vetting process. It has a set of standards in the same way that RPM specs have a standard. They have maintainers. They have certain kind of tracking.

These two projects, the Modules Project which works from the sysadmin's side and the container built project which work from the build developer, package developer side, those two projects together are working towards that middle where a sysadmin could choose instead of installing DNF install, or Yum install Nginx, they could do container install, or a module install Nginx.

For them, the behavior is very analogous.

Gordon:  Bring this home. I'm going to steal something that you mentioned to me a little bit ago, that we talk about having green‑field and brown‑field environments and having to deal with both of those, but from a skills level and from and perhaps more importantly from an experience level, there really is no green‑field as far as sysadmins are concerned.

You can't just assume that they're a blank slate and forget all about the processes and experience that they've acquired over decades in some cases.

Mark:  Yeah. When I said it, it was the first time I kind of thought that going, "Yeah, you can't treat them the same way."

In the container world, we have done an awful lot of work which assumed green‑field or which green‑field was imposed on us because like I said, people started trying to stuff whole applications with multiple processes into a container, and they very quickly found that that was a real problem.

They would try to tease apart the components of their application on existing application and refactor it, and they'd go, "Oh my God. This is such a pain, too. It's not really working," and they would inevitably either quit or go back to green‑field and say, "Look. Let's redevelop these as containers, as micro services from the start because of the lack of a way to migrate easily."

I think part of the reason that lack is there is that we're treating containers as the thing that runs. We're still treating containers as something we build, not something we use. As we learn the patterns, we've talked about this before, we're still in the position of learning the patterns of how containers are used well. As we learn those patterns, we're going to start eliminating variables.

We're going to start eliminating parameters. We're going to start adding defaults and assumptions, and we're going to start addressing these real world used cases in ways that they hide the minutiae that doesn't matter anymore. Someone who uses a package, who installs a package doesn't care about how it was built, doesn't care where that source code came from, doesn't care...

I mean, they can go find out, but as far as a user goes, they just Yum install a package, and they're done. That's all they should have to do, and I think we'll get there with containers. I think we'll get there with containers for sysadmins. It will be longer before we get there for container services.

I've heard somebody recently in a position of advocacy to say, "Well, Kubernetes, OpenShift, or whatever will be the new operating system." While I think some kind of container cloud infrastructure is in the offing in the long‑term, I think we have a ways to go before we get there, and think there are a whole lot of transitional states we're going to go through.

That's kind of where I work, is, "What works now?" I want to look ahead, but I need to remember that there are people doing the work now.

Gordon:  I'm actually giving a presentation at MonkiGras in a couple of weeks, and hopefully I get this podcast posted before then, about packaging, not really software packaging, but the grand history of packaging going back to pottery. One of the things I bring forward in that is that, yeah, a lot of early packaging was pretty much functional.

You put your wine in this clay jar of some sort, but really where we've progressed to with packaging is this much more almost experiential model of packaging where you make it easier to consume, easier to use, easier to have confidence in the elements that are part of that package, so many other things we're trying to do with OpenShift for both developers and operations, for example.

You're absolutely right. Containers are probably ultimately going to end up being one of those components that most people don't need to think about very much, whereas the actual packaging and the actual user experience, whether they're a sysadmin or developer, is at some higher level platform, for the most part.

Mark:  That comes back to something I said earlier with the people who object to containers, who brush aside containers on the grounds that, "They're just packages and we've been there and done that," are ignoring a very significant part of the advancement of containers. It's the contents, and it's some structure for the contents which an RPM would have.

It's got the spec for building it, and yeah, we've done those things before. What it has that traditional packages don't is that traditional packages are static. They have information about how to install themselves, which is an advance over tarballs, but they don't' have the metadata about how they're expected to be used. That's the significance in the container packaging.

We do not have semantics or we're developing container semantics for that packaging metadata which will say, "Here's how I expect this software to be used, and here's the inputs it expects. Here's the parameters it expects. Here are the other packages or the other containers it expects to interact with." I think that's what the people who think containers are just packages are overlooking.

It's an area of research that we still don't know all the answers to.

Optimizing the Ops in DevOps

This post is based on my recent presentation at DevOps Summit Silicon Valley in November 2016. You can see the entire presentation here. 

We call it DevOps but much of the time there’s a lot more discussion about the needs and concerns of developers than there is about other groups. 

There’s a focus on improved and less isolated developer workflows. There are many discussions around collaboration, continuous integration and delivery, issue tracking, source code control, code review, IDEs, and xPaaS—and all the tools that enable those things. Changes in developer practices may come up—such as developers taking ownership of code and pulling pager duty.

We also talk about culture a great deal in the context of developers and DevOps. About touchy-feely topics like empathy, trust, learning, cooperation, and responsibility. It can all be a bit kumbaya.

What about the Ops in DevOps? Or, really, about the other constituencies who should be part of the DevOps process, workflow, and even culture? Indeed, DevSecOps is gaining some traction as a term. DevOps purists may chafe at “DevSecOps" given that security and other important practices are supposed to already be an integral part of routine DevOps workflows. But the reality is that security often gets more lip service than thoughtful and systematic integration.

But what’s really going on here is that we need to get away from thinking about Ops-as-usual (or Security-as-usual) in the DevOps context at all. This is really what Adrian Cockcroft was getting at with the NoOps term; he didn’t coin it but his post about NoOps while he was at Netflix kicked off something of an online kerfuffle. Netflix is something of a special case because they are so all-in on Amazon Web Services, but Adrian was getting at something that’s more broadly applicable. Namely that, in evolved or mature DevOps, a lot of what Ops does is put core services in place and get out of the way.

Ironically, this runs somewhat counter to the initial image of breaking down the wall between Dev and Ops. Yes, DevOps does involve greater transparency, collaboration, and so forth to break down siloed behaviors but there’s perhaps an even stronger element of putting infrastructure, processes, and tools in place so that Devs doesn’t need to interact with Ops as much while being (even more) effective. One of the analogies I like to use is that I don’t want to streamline my interactions with a bank teller. For routine and even not so routine transactions, I just want to use an ATM or even my smartphone.

It’s up to Ops to build and operate the infrastructure supporting those streamlined transactions. Provide core services through a modern container platform. Enable effective automated developer workflows. Mitigate risk and automate security. But largely stay behind the scenes. Of course, you still want to have good communication flows between developers and operations teams; you just want to make those communications unnecessary much of the time.

(At the same time, it’s important for Dev and Ops teams to understand how they can mutually benefit by using appropriate container management and other toolchains. There’s still too much of a tendency by both groups to think of something as an “ops tool” or a “dev tool.” But that’s a topic for another day.)

Let’s look at each of those three areas.

Modern container platform

A DevOps approach can be applied just about anywhere. But optimizing the process and optimizing cloud-native applications is best done on a modern platform. Take it as read that most IT is going to be brownfield and DevOps may even be a good bridge between existing systems, applications, and development processes and new ones. But here I’m focusing on what’s optimized for new apps and new approaches.

You need scale-out architectures to meet highly elastic service requirements. Application designs with significant scale-up components simply aren’t able to accommodate shifting capacity needs. 

Everything is software-defined because software functions, such as network function virtualization and software-defined storage, are much more flexible than when the same functions are embedded in hardware.

The focus is on applications composed of loosely-coupled services because large monolithic applications can be fragile and can’t be updated quickly.

A modern container platform enables lightweight iterative software development and deployment in part because modern applications are often short-lived and require frequent refreshes and replacements. 

As I wrote about in The State of Platform-as-a-Service 2016, a PaaS like Red Hat’s OpenShift has evolved to be this modern container platform, embracing and integrating docker-format containers, kubernetes for orchestration and using Red Hat CloudForms (based on the ManageIQ upstream project) for open source hybrid cloud management.

Automated developer workflows

When thinking about the toolchain associated with DevOps, a good place to start is the automation of the continuous integration/continuous delivery (CICD) pipeline. The end goal is to make automation pervasive and consistent using a common language across both classic and cloud-native IT. For example, Ansible allows configurations to be expressed as “playbooks” in a data format that can be read by both humans and machines. This makes them easy to audit with other programs, and easy for nondevelopers to read and understand.

A typical automated workflow begins with operations provisioning a containerized development environment, whether a PaaS or more customized environment. This provides an example of how a mature DevOps process separates operations and developer concerns; by providing developers with a dynamic self-service environment, operations can focus on deploying and running stable, scalable infrastructure while developers focus on writing code.

Automation then ensures that the application can be repeatedly deployed. Many different types of tools are integrated into the DevOps workflow at this point. For example:

• Code repositories, like Git
• Container development tools to convert code in a repository into a portable containerized image that includes any required dependencies
• Vagrant for creating and configuring lightweight, reproducible, and portable development environments¥IDEs like Eclipse
• CICD software like Jenkins

Mature DevOps systems may even push the code directly to production once it has passed automated testing. But this isn’t about removing Ops from its role of ensuring stable and robust production systems. Rather, it’s about automating the processes to ensure that deployed code meets set criteria without ops needing to be directly involved with each deployment.

Mitigate risk and automate security

Ops is also ultimately chartered with protecting the business. This doesn’t mean eliminating all risk—which can’t be done. But it does mean mitigating risk which is accomplished in part by managing the software supply chain and by automating away sources of manual error. 

That’s not to say that security is purely an ops concern—hence the aforementioned DevSecOps term. Creating a mindset in which everyone is responsible for security is key as is the practice of building security into development processes. Security must change from a defensive to an offensive posture that is both automated and constant.

Among the practices to be followed in such a proactive environment are:

• Components built from source code using a secure, stable, reproducible build environment
• Careful selection, configuration, and security tracking of packages
• Automated analysis and enforcement of security practices
• Active participation in upstream and community involvement
• Thoroughly validated vulnerability management process

I think this quote from Gartner captures the required dynamic well: "Our goal as information security architects must be to automatically incorporate security controls without manual configuration throughout this cycle in a way that is as transparent as possible to DevOps teams and doesn't impede DevOps agility, but fulfills our legal and regulatory compliance requirements as well as manages risk.” (Gartner. DevSecOps: How to Seamlessly Integrate Security Into DevOps. September 2016. G00315283)

----

Credits

Dev: Nelson Pavlosky/flickr under CC http://www.flickr.com/photos/skyfaller/113796919/

Ops: Leonardo Rizzi/flickr under CC http://www.flickr.com/photos/stars6/4381851322/

Piggy bank: https://www.flickr.com/photos/marcmos/3644751092

Stop: https://www.flickr.com/photos/r_grandmorin/6922697037

DevOps wall: Cisco