When companies focus too much on risk

When we think about security in the context of DevSecOps, an important mantra is that we need to move from thinking about providing absolute security to thinking about managing risk in the context of business outcomes. Move from “Just say no” to saying yes to small risks if the tradeoffs appear to be worth it.

Let me illustrate this principle (in addition to a couple of other things) with an example that’s not drawn from the IT world. 

Right before the holidays, I took a last minute quick trip to speak at and attend a couple of events being held next to the airport outside San Francisco. Loaded the bags up and off I went. As I was being dropped off at the airport, I pull out my driver’s license so I won’t be fumbling around with my wallet, get out of the car, and head into the terminal.

Somehow, in the course of 50 feet, space aliens made off with my license. Call the limo company. Driver takes a look. No luck. I still have absolutely no idea what happened. 

Now, normally, frequent traveler me has a travel folio with passport, spare credit cards, cash, and other potentially useful travel backups. But because this was just a quick trip I figured I didn’t need it.

Lesson #1: You may not think you need a backup. Until you do.

(See also. It’s just a small code change. We don’t need to re-run the test suite.)

Crap. Visions of my trip mashed up with mushroom clouds seemed appropriate. But I wandered over to the security line anyway.

Much to my surprise, my missing license turned out not to be a particularly serious problem. Yes, I had other ID although nothing government issued. I had my boarding pass on my phone. I have TSA Pre. And they gave me a thorough pat down and they inspected and detected my luggage very carefully. I was both impressed and surprised that I was able to hop on my flight.

I thought I had dodged a bullet.

Land SFO. Take shuttle bus to hotel. I won’t name the hotel. Let’s just say it’s a lower end chain I wouldn’t normally stay at but, as I said, this was a very last minute trip and with my usual chains either sold out or going for $700 a night I figured I could put up with the relative dump for a couple of nights.

They have my reservation that I made online. Give them my credit card.

“ID please."

I tell my story. Consternation. “Umm, do you have a passport?"

Well, no. But I can show you any number of cards. Here’s my company badge with a photo. You can easily look me up online. 

Nope. It was starting to look as if I’d have to start dialing various friends in the Bay area to see if they had a spare couch I could use.

At this point, what I really wanted to say was: “Look. If I wanted to concoct some complicated scam for free hotel nights that somehow involved having 1.) an online reservation, 2.) a wallet full of cards including the credit card used to make the reservation, 3.) an official looking company ID, but 4.) no government-issued photo ID, I’m pretty sure it would be at an exotic resort and not an SFO fleabag."

To bring us back to the original topic, sure, you can always impose more hard and fast rules but you really need to think about whether inflexibly imposing those rules is the best approach for the business. 

Lesson #2: Think about whether potential risks justify the costs of eliminating them (which you can never fully do anyway)

In the end, I was able to check in. I didn’t say what I was thinking and we reached an agreement whereby I could pay cash, including a security deposit. (Fortunately, the dollar amount was small enough that I was able to withdraw what I needed from the ATM in the lobby.) Luckily, I did have my company ID with a photo; I don’t think they’d have let me stay with no photo ID at all—my face being all over the Web notwithstanding. 

So I do give some small amount of credit to the local manager for bending, however slightly, to what I have to assume are quite rigid corporate rules.

Lesson #3: Empower employees to do the right thing as much as possible

I was also pleasantly surprised how easy and relatively inexpensive ($25) it was to replace my driver’s license on the Massachusetts DMV site. Which brings us to our last lesson.

Lesson #4: If your policies and customer experience fail to meet the standards set by both the TSA and the Massachusetts DMV, I’m pretty sure you’re doing something wrong

 

DevOps Culture: continuous improvement for Digital Transformation

In contrast to even tightly-run enterprise software practices, the speed at which big Internet businesses such as Amazon and Netflix can enhance, update, and tune their customer-facing services can be eye opening. Yet a miniscule number of these deployments cause any kind of outage. These companies are different from more traditional businesses in many ways. Nonetheless they set benchmarks for what is possible. 

Enterprise IT organizations must do likewise if they’re to rapidly create and iterate on the new types of digital services needed to succeed in the marketplace today. Customers demand anywhere/anywhen self-service transactions and winning businesses meet those demands better than their competition. Operational decisions within organizations also must increasingly be informed by data and analytics, requiring another whole set of applications and data sets.

Amazon and Netflix got to where they are using DevOps. DevOps touches many different aspects of the software development, delivery, and operations process. But, at a high level, it can be thought of as applying open source principles and practices to automation, platform design, and culture. The goal is to make the overall process associated with software faster, more flexible, and incremental. Ideas like the continuous improvement based on metrics and data that have transformed manufacturing in many industries are at the heart of the DevOps concept.

Development tools and other technologies are certainly part of DevOps. 

Pervasive and consistent automation is often used as a way to jumpstart DevOps in an organization. Playbooks that encode complex multi-part tasks improve both speed and consistency. It can also improve security by reducing the number of error-prone manual processes. Even narrowly targeted uses of automation are a highly effective way for organizations to gain immediate value from DevOps.

Modern application platforms, such as those based on containers, can also enable more modular software architectures and provide a flexible foundation for implementing DevOps. At the organizational level, a container platform allows for appropriate ownership of the technology stack and processes, reducing hand-offs and the costly change coordination that comes with them. 

However, even with the best tools and platforms in place, DevOps initiatives will fail unless an organization develops the right kind of culture. One of the key transformational elements is developing trust among developers, operations, IT management, and business owners through openness and accountability. In addition to being a source of innovative tooling, open source serves as a great model for the iterative development, open collaboration, and transparent communities that DevOps requires to succeed.

Ultimately, DevOps becomes most effective when its principles pervade an organization rather than being limited to developer and IT operations roles. This includes putting the incentives in place to encourage experimentation and (fast) failure, transparency in decision-making, and reward systems that encourage trust and cooperation. The rich communication flows that characterize many distributed open source projects are likewise important to both DevOps initiatives and modern organizations more broadly.

Shifting culture is always challenging and often needs to be an evolution. For example, Target CIO Mike McNamara noted in a recent interview that “What you come up against is: ‘My area can’t be agile because…’ It’s a natural resistance to change – and in some mission-critical areas, the concerns are warranted. So in those areas, we started developing releases in an agile manner but still released in a controlled environment. As teams got more comfortable with the process and the tools that support continuous integration and continuous deployment, they just naturally started becoming more and more agile.”

At the same time, there’s an increasingly widespread recognition that IT must respond to the needs of and partner with the lines of business--and that DevOps is an integral part of that redefined IT role. As Robert Reeves, the CTO of Datical, puts it: “With DevOps, we now have proof that IT can and does impact market capitalization of the company. We should staff accordingly.”

------------------

Photo credit: http://marshmallowchallenge.com/Welcome.html

DevSecOps at Red Hat Summit 2017

We’re starting to hear “DevSecOps" mentioned a lot. The term causes some DevOps purists to roll their eyes and insist that security has always been part of DevOps. If you press hard enough, they may even pull out a well-thumbed copy of The Phoenix Project by Gene Kim et al. [1] and point to the many passages which discuss making security part of the process from the beginning rather than a big barrier at the end.

But the reality is that security is often something apart from DevOps even today. Even if DevOps should include continuously integrating and automating security at scale. It’s at least in part because security and compliance operated largely in their own world historically. At a DevOpsDays event last year, one senior security professional even told me that this was the first IT event that was not security-specific that he had ever attended.

With that context, I’d like to point you to a session that my colleague William Henry and I will be giving at Red Hat Summit on May 3. In DevSecOps the open source way we’ll discuss how the IT environment has changed across both development and operations. Think characteristics and technologies like microservices, component reuse, automation, pervasive access, immutability, flexible deploys, rapid tech churn, software-defined everything, a much faster pace, and containers.

Risk has to be managed across all of these. (Which is also a change. Historically, we tended to talk in terms of eliminating risk while today it’s more about managing risk in a business context.)

Doing so requires securing the software assets that get built and well as the machinery doing the building. It requires securing the development process from the source code through the rest of the software supply chain. It requires securing deployments and ongoing operations continuously and not just at a point in time. And it requires securing both the application and the container platform APIs.

We hope to see you at our talk. But whether or not you can make it to see us specifically, we hope that you can make it to Red Hat Summit in Boston from May 2-4. I’m also going to put in a plug for the OpenShift Commons Gathering on the day before (Monday, May 1).

-------------- 

[1] If you’re reading this, you’ve almost certainly heard of The Phoenix Project. But, if not, it’s a fable of sorts about making IT more flexible, effective, and agile. It’s widely cited as one of the source texts for the DevOps movement.

Podcast: Open source and cloud trends with IDC's Al Gillen

Al Gillen is responsible for open source research and oversees developer and DevOps research at IDC. Al gave a keynote at the Open Source Leadership Summit at which he provided some historical context for what's happening today in open source and presented recent research on digital transformation, commercial open source support requirements, and how organizations are thinking about cloud-native architecture adoption and deployment.

Listen to the podcast for the whole conversation but a few specific points that Al made were:

Digital transformation can be thought of as taking physically connected systems and logically connecting them, i.e. connecting the processes, the data, and the decision-making.

It's important to bridge new cloud-native systems to existing functionality. Organizations are not going to be rewriting old applications for the most part and those "legacy" systems still have a great deal of value.

Enterprises are asking for open source DevOps tools, but most are specifically asking for commercially-supported open source tools.

Audio:
Link to MP3 (00:15:46)
Link to OGG (00:15:46)

Transcript:

Gordon Haff:  Hi, everyone. Welcome to another edition of the "Cloudy Chat" podcast. I'm here at the Open Source Leadership Summit with Al Gillen of IDC, who gave one of the keynotes this morning. Welcome, Al. How about giving a little background about yourself?

Al Gillen:  Hey, Gordon, thanks a lot. Thanks, everybody for listening. This is Al Gillen. I'm a group vice president at IDC. I'm responsible for our open source research, and oversee our developer and DevOps research.

Gordon:  One of the things you went through in your keynote this morning was the historical perspective of how Linux is developed. Both of us have pretty much been following Linux from the beginning, certainly from its beginnings as something that was interesting commercially. Maybe you could recap that in a couple of minutes or so.

Al:  I actually went back to a presentation I delivered to the Linux Foundation, an event at the Collaboration Summit that was back in 2008. I pulled those slides up, because I was curious. "What can I learn from what we talked about back then, and how does that relate to what's going on in the industry today?"

I went back, and I pulled up the deck. I was looking at some of the things that I thought were really interesting. For example, I was looking at one of the first pieces of data, which compared perceptions of Linux from 1999 and 2001.

Remember what the time frame was there. Linux had only just begun to be commercially accepted in the '99‑2000/2001 time frame. One of the things that served as a significant accelerator for Linux at that time frame was the dot‑com bust.

What happened then is we had a big contraction in the stock market. Most large companies, what they did is they went and they started to cut costs. We all know that one of the places they first cut costs is IT.

Suddenly, the IT departments were charged with standing up new Web servers and new network‑infrastructure servers and so forth, and they had no budget to do it. What did they do?

They went and they got a free copy of Linux. They recycled a well‑equipped PC or x86 server that had been taken out of service, and they turned it into a Linux server.

When we look back at the data that we saw then, really, one of the big drivers for Linux was initial price. People said, "Yeah, it was great. The cost was really low." One of the things that was also amazing was the users back then rated the reliability of Linux as very, very high.

In fact, when you compare it to other operating systems, it compared very favorably to much more mature operating systems. That context was really fascinating, but when you think about it, that was just the beginning of a long gestation period for Linux.

Over the next, what, seven, eight, nine years, as Linux became a truly mature and a truly robust commercial operating system that had both the features, had the application portfolio, and had the customer base to use it, it took, basically a decade to get there.

Gordon:  You've been doing some more research recently. What are your numbers showing today?

Al:  A couple of things that I showed in the presentation today. One is we presented data on Linux operating system shipments. One of the things that's happened over the last few years is that Linux has continued to accelerate, in part because of the build‑out of cloud.

Most of the public cloud infrastructure, with the exception of the Microsoft Azure cloud, is almost all Linux. To the extent that Google continues to build out and Amazon continues to build out, and companies like that ‑‑ Facebook, Twitter, and so forth, it's primarily Linux being stood up.

That has driven the growth of non‑commercial Linux, meaning distributions that are not supported by a commercial company you might think of, rather are either they're CentOS, or they're Debian, or potentially Ubuntu, that's not supported, things like that, as well as Amazon Linux, and Google's own Linux and so forth.

That's been really where a lot of the growth is, but that's not to say that there hasn't been growth in the commercial side of the market. There's been growth there as well.

Gordon:  What are some of the drivers that you hear? I know you did some research for us. You also have some research here around commercially‑supported environments and maybe some of the reasons why people buy those.

Al:  That has been something which has been really consistent through the years. We find that large enterprise organizations have a tendency to prefer commercially‑supported software.

That has always been the case with Linux and yes, we find that there is [also] non‑commercial Linux. You'll talk to any enterprise ‑‑ and you could talk to a really big Red Hat shop or a big SUSE shop, and you ask them, "What is in your infrastructure?" They'll typically tell you that, "Yeah, we're 95 percent or 98 percent Red Hat, but we've also got some CentOS," or, "We've got some Debian," or, "We've got SUSE for this one application."

They generally have a mix of other things in there. The same thing, if you talk to a SUSE shop, where they'll say, "Yeah, we're mostly SUSE, but we've got some openSUSE," or again, "We've got some Ubuntu or CentOS, or something else in the mix."

The reason why is that these things get stood up for work loads that are considered not critical. Maybe they might be something simple like a DNS server, maybe something that's a print file server, or maybe a Web server which is providing some internal Web serving capabilities, something that's not critical if it disappears off the network suddenly. There's not going to be customers that are left hanging.

Gordon:  Let's switch gears and talk about digital transformation. This is one of those terms that I think is almost a cliche at this point, at least when you go to these types of events, because we hear it at every one of these events.

As somebody I was talking to recently said, just because it's cliche doesn't mean it's not an important trend. What are some of the things that IDC is seeing about digital transformation?

Al:  If we go out to, say 2025, and we look back, I believe that we're going to look back and say, "Yeah, the mid‑teen years were important years from a digital transformation perspective."

When we at IDC talk about digital transformation, what we're really talking about is we're talking about the interconnection of all of the systems that are in our environments.

When I say interconnection, we're not talking about getting them all on the same network. We've done that. It's been done for 20 years, already. What we're talking about is interconnecting the processes, interconnecting the data, interconnecting the decision‑making. In many cases, that's not done.

We've got systems that are physically connected, but are not connected from a logical sense. That's what's happening with digital transformation. I might add that the way we expect that that's going to happen is it's going to be a model where we're going to be building new applications that are going to essentially bridge the existing functionality that's on these servers.

We're not going to be rewriting those old applications in a cloud‑native format, for example. We're going to keep those applications. We may wrap them with some consistent API so we can get access to the logic and the data.

But at the end of the day, the business value that's in those applications that are in place today remains valuable, and frankly, it's going to mean that the applications themselves and the servers that they run on are workloads that are going to be around for the long term.

Gordon:  I think that's a really important point, because one of the things that I hear a lot when I talk to customers is the importance of this bridging of the older systems that may be modernizing, but as you say, you're not turning them into cloud‑native systems.

On the other hand, you have these cloud‑native infrastructures. I think probably in the industry, there's too much thinking of those as two disconnected islands, and not enough thinking of the bridges, the integrations, and so forth, between them.

Al:  There's a really good parallel here, and I like to bring this story up, especially when I'm in a room full of end users. I like to say, "You guys remember what you were doing in 1999?" You get a little bit of a quizzical look, and I say, "Were you remediating any applications that had ywo digit date codes?"

People start nodding their heads. The next question I say is, "What did you do? Did you fix them?" and the heads keep nodding. I say, "OK, can I see a show of hands? How many of you have gotten rid of those systems, or do you still have them in use?"

All those same people, they put their hands up, reluctantly, I might add, and say, "Yeah, we still have those systems in use." The point being is that the value of the systems does not go away. The value of the systems is in the data. It's in the processes and the business logic that are coded in those applications.

Going forward, we think the same thing's going to be true for the distributed computing environment. All of the Linux servers that you have in place, all of the Windows servers you have in place, have real important business value. The logic and the data there is really valuable to your business, which means that you're going to want to use that going forward.

I do agree with you, when we build that cloud‑native applications, they're going to help bridge these systems, but don't for a minute assume that those old systems have no value left.

Gordon:  As we talk about the new applications that are going to be required for this digital transformation, what are some of the...I think you even used the term, they were the "pivot point," this morning. Tell us a little bit more about that.

Al:  Again, taking a long view, so if we look out, say, if you go out to 2025, and you look back, I think that we'll be able to draw a line in history and say, "Somewhere between 2015 and 2017 or 2018, there's going to be this line where everything before that line will be considered a legacy application, and everything after that line is probably going to be a cloud‑native and a modern application."

Again, let's not associate the term legacy application with something that has no value. Let's assume that that's an architectural statement more than anything else.

When I think about it, I believe we're right in the midst of this transition where we begin to build all of our applications using a cloud‑native format, which means that our applications are built to run on‑prem in private cloud, or off‑prem in public cloud, which means that we have flexibility on where they run, how we want to run them, how we want to scale them.

The other thing I might add is remember that cloud‑native and cloud‑scale are not necessarily the same thing. There's lots and lots of applications that should be cloud‑native, but not all applications have to have cloud scale.

Take for example your average enterprise. You've got ‑‑ pick your number ‑‑ it's 1 thousand, 10 thousand, 100 thousand users, whatever the number is, that access your business applications. That number does not scale up to 1 million or 10 million overnight.

By comparison, somebody who's doing, say, business to consumer, where there could potentially be a consumer event that causes everybody to come in and access that application. There you'd need to have the ability to do cloud‑level scaling.

Gordon:  Al, going back to the commercial support still being important, you certainly see in the cloud an awful lot of this consumption of free software, but you mentioned earlier that enterprises by and large do want commercialized tools. We're not talking just operating systems here.

Al:  No, in fact, operating systems are probably one of the best understood pieces of open‑source software today. As we go up the stack, customers still see value associated with commercialization, so a company that will take your project and make it something that is consumable will provide the support.

The reason why that's so valuable is that then the company does not have to have the expertise on staff. You don't have to have a kernel guy, you don't have to have a guy that knows how to patch Xen, for example, or a KVM if there's a problem.

It's really important to have that taken care of by somebody if you're a commercial organization, which is in the business of selling widgets or manufacturing things, or providing health care. That's your primary business. Your primary business is not being in the business of IT.

We ran a survey earlier this year. I guess it was actually late in 2016. We were talking to people about their consumption of DevOps products. We asked a question which I think was really interesting.

The question was if they have a chance to buy a product, are they going to look at a product which is open‑source, or are they going to look at a product which is likely to be a closed‑source and/or a proprietary‑type product?

What we found is we asked people to rank their preference on these things. The answer came back as 45 percent of the people we talked to ranked their preference for an open‑source base product as their first choice over anything else.

If you asked them what their preferences were, for example, for a proprietary product, only 15 percent of the people said that was their first choice.

The reason why this is really interesting is that when we look at that, these companies are telling us that they want an open‑source base product, but they also told us that they wanted it to be commercially supported.

They could get the bits and run it as a project themselves. We asked that question as well. That's not what they're asking for. They're asking for it to be commercially supported. The reason why is if it breaks, you pick up the phone or you get on your computer and send an email, and you say, "Fix it."

Gordon:  That was true going back with Linux in the early days as well. I think one of the things that's happening today when you look at DevOps tools is there is an incredible amount of innovation and number of products out there. That's good news.

The bad news is there's an incredible amount of innovation, rapidly changing products, and a need to integrate all of those together.

Al:  You know what, Gordon? It's one of the challenges that we've had with fast‑moving markets like this. When I say markets, I'm referring to open‑source, collectively.

The problem we have is that the technology changes so fast that the people, the end‑user organizations, are not able to gain the skills fast enough to keep up with these technology changes.

Frankly, when we asked questions about Linux in the early 2000s, people said one of their top challenges was having the skill set to support Linux. Today, we find the same questions about things like container technology and using DevOps tools.

Gordon:  To net it all out and close this out, what are some of the recommendations, the guidance, that you give on probably pretty much a daily basis to your clients?

Al:  There's a few things. Number one, recognize that cloud‑native applications are going to be architected very differently than classic applications. That's pretty much a given, but when you think about it, it affects your choice of tools, it affects your choice of deployment scenarios, and it affects your skills that you need to have on staff.

Another thing is recognize that we've moved to an era of platform independence far beyond anything we ever had before. We always like to talk about platform independence, but we've never really had it.

Now, with container technology and the ability to produce a true cloud‑native application that's running on some kind of a framework which happens to be available on‑prem or in cloud, you suddenly have the ability to move that application on‑prem or off‑prem, or both ‑‑ run in both places at the same time if so you choose ‑‑ and be able to do that in a way that's been unprecedented in our industry.

Finally, just to reiterate the other point is recognize that the existing applications don't lose their value. They still have value. Yes, they may get bundled up in a VM, or maybe packaged up in container and dropped into somebody's IS cloud, but they're going to be around for the long term, and recognize that that's something you have to support.

Again, driving home that point I made earlier, recognize that all the new applications we build today are going to have to bridge the classic applications we've had, and the data that those applications support together with the modern things that we're going to be doing with our new applications.

Optimizing the Ops in DevOps

This post is based on my recent presentation at DevOps Summit Silicon Valley in November 2016. You can see the entire presentation here. 

We call it DevOps but much of the time there’s a lot more discussion about the needs and concerns of developers than there is about other groups. 

There’s a focus on improved and less isolated developer workflows. There are many discussions around collaboration, continuous integration and delivery, issue tracking, source code control, code review, IDEs, and xPaaS—and all the tools that enable those things. Changes in developer practices may come up—such as developers taking ownership of code and pulling pager duty.

We also talk about culture a great deal in the context of developers and DevOps. About touchy-feely topics like empathy, trust, learning, cooperation, and responsibility. It can all be a bit kumbaya.

What about the Ops in DevOps? Or, really, about the other constituencies who should be part of the DevOps process, workflow, and even culture? Indeed, DevSecOps is gaining some traction as a term. DevOps purists may chafe at “DevSecOps" given that security and other important practices are supposed to already be an integral part of routine DevOps workflows. But the reality is that security often gets more lip service than thoughtful and systematic integration.

But what’s really going on here is that we need to get away from thinking about Ops-as-usual (or Security-as-usual) in the DevOps context at all. This is really what Adrian Cockcroft was getting at with the NoOps term; he didn’t coin it but his post about NoOps while he was at Netflix kicked off something of an online kerfuffle. Netflix is something of a special case because they are so all-in on Amazon Web Services, but Adrian was getting at something that’s more broadly applicable. Namely that, in evolved or mature DevOps, a lot of what Ops does is put core services in place and get out of the way.

Ironically, this runs somewhat counter to the initial image of breaking down the wall between Dev and Ops. Yes, DevOps does involve greater transparency, collaboration, and so forth to break down siloed behaviors but there’s perhaps an even stronger element of putting infrastructure, processes, and tools in place so that Devs doesn’t need to interact with Ops as much while being (even more) effective. One of the analogies I like to use is that I don’t want to streamline my interactions with a bank teller. For routine and even not so routine transactions, I just want to use an ATM or even my smartphone.

It’s up to Ops to build and operate the infrastructure supporting those streamlined transactions. Provide core services through a modern container platform. Enable effective automated developer workflows. Mitigate risk and automate security. But largely stay behind the scenes. Of course, you still want to have good communication flows between developers and operations teams; you just want to make those communications unnecessary much of the time.

(At the same time, it’s important for Dev and Ops teams to understand how they can mutually benefit by using appropriate container management and other toolchains. There’s still too much of a tendency by both groups to think of something as an “ops tool” or a “dev tool.” But that’s a topic for another day.)

Let’s look at each of those three areas.

Modern container platform

A DevOps approach can be applied just about anywhere. But optimizing the process and optimizing cloud-native applications is best done on a modern platform. Take it as read that most IT is going to be brownfield and DevOps may even be a good bridge between existing systems, applications, and development processes and new ones. But here I’m focusing on what’s optimized for new apps and new approaches.

You need scale-out architectures to meet highly elastic service requirements. Application designs with significant scale-up components simply aren’t able to accommodate shifting capacity needs. 

Everything is software-defined because software functions, such as network function virtualization and software-defined storage, are much more flexible than when the same functions are embedded in hardware.

The focus is on applications composed of loosely-coupled services because large monolithic applications can be fragile and can’t be updated quickly.

A modern container platform enables lightweight iterative software development and deployment in part because modern applications are often short-lived and require frequent refreshes and replacements. 

As I wrote about in The State of Platform-as-a-Service 2016, a PaaS like Red Hat’s OpenShift has evolved to be this modern container platform, embracing and integrating docker-format containers, kubernetes for orchestration and using Red Hat CloudForms (based on the ManageIQ upstream project) for open source hybrid cloud management.

Automated developer workflows

When thinking about the toolchain associated with DevOps, a good place to start is the automation of the continuous integration/continuous delivery (CICD) pipeline. The end goal is to make automation pervasive and consistent using a common language across both classic and cloud-native IT. For example, Ansible allows configurations to be expressed as “playbooks” in a data format that can be read by both humans and machines. This makes them easy to audit with other programs, and easy for nondevelopers to read and understand.

A typical automated workflow begins with operations provisioning a containerized development environment, whether a PaaS or more customized environment. This provides an example of how a mature DevOps process separates operations and developer concerns; by providing developers with a dynamic self-service environment, operations can focus on deploying and running stable, scalable infrastructure while developers focus on writing code.

Automation then ensures that the application can be repeatedly deployed. Many different types of tools are integrated into the DevOps workflow at this point. For example:

• Code repositories, like Git
• Container development tools to convert code in a repository into a portable containerized image that includes any required dependencies
• Vagrant for creating and configuring lightweight, reproducible, and portable development environments¥IDEs like Eclipse
• CICD software like Jenkins

Mature DevOps systems may even push the code directly to production once it has passed automated testing. But this isn’t about removing Ops from its role of ensuring stable and robust production systems. Rather, it’s about automating the processes to ensure that deployed code meets set criteria without ops needing to be directly involved with each deployment.

Mitigate risk and automate security

Ops is also ultimately chartered with protecting the business. This doesn’t mean eliminating all risk—which can’t be done. But it does mean mitigating risk which is accomplished in part by managing the software supply chain and by automating away sources of manual error. 

That’s not to say that security is purely an ops concern—hence the aforementioned DevSecOps term. Creating a mindset in which everyone is responsible for security is key as is the practice of building security into development processes. Security must change from a defensive to an offensive posture that is both automated and constant.

Among the practices to be followed in such a proactive environment are:

• Components built from source code using a secure, stable, reproducible build environment
• Careful selection, configuration, and security tracking of packages
• Automated analysis and enforcement of security practices
• Active participation in upstream and community involvement
• Thoroughly validated vulnerability management process

I think this quote from Gartner captures the required dynamic well: "Our goal as information security architects must be to automatically incorporate security controls without manual configuration throughout this cycle in a way that is as transparent as possible to DevOps teams and doesn't impede DevOps agility, but fulfills our legal and regulatory compliance requirements as well as manages risk.” (Gartner. DevSecOps: How to Seamlessly Integrate Security Into DevOps. September 2016. G00315283)

----

Credits

Dev: Nelson Pavlosky/flickr under CC http://www.flickr.com/photos/skyfaller/113796919/

Ops: Leonardo Rizzi/flickr under CC http://www.flickr.com/photos/stars6/4381851322/

Piggy bank: https://www.flickr.com/photos/marcmos/3644751092

Stop: https://www.flickr.com/photos/r_grandmorin/6922697037

DevOps wall: Cisco

 

Data and DevOps with Splunk's Andi Mann

Andi Mann is Chief Technology Advocate at Splunk. In this podcast, he discusses some of the ways in which data plays an important role in DevOps. I’ve known Andi for ages since we were both IT industry analysts and we had a chance to sit down at CloudExpo/DevOps Summit/IoT Summit in Santa Clara where Andi was chairing a DevOps track and I was one of the speakers. (We also did a data and DevOps panel together on the main stage but that video doesn’t seem to be up yet. I’ll post once it is.)

Among the topics we tackle are choosing appropriate metrics that align with the business rather than just technical measures, creating feedback loops, using data to promote accountability, and DevSecOps.

Show notes:

• Andi’s blog
• Andi on Twitter

Listen to MP3 (22:13)

Listen to OGG (22:13)

[Transcript]

Gordon Haff:  I'm sitting down here with an old analyst mate of mine, Andi Mann. Also, formally of CA, also the author of some books, and now he is the chief technology advocate with Splunk. What we're going to talk about today is data in DevOps. Welcome, Andi.

Andi Mann:  A lot of the customers I talk to, who are doing DevOps in various versions using Splunk...It boils down to three key areas is that they really want to know about, the metrics that matter for them.

The first is really about how fast are they? What's their cycle time? How quick does it take for an idea to get in front of a customer? How long does it take someone in business to come up with something and then basically make money from it, or, in government, they service their citizens with it? That cycle time is really important, the velocity of delivery.

The second key area that people look at is around the quality of what they're delivering. Are they doing good? Are they delivering good applications? Are they creating downtime? Are they having availability issues? Is one release better than another?

The third area is really around what sort of impact do they have? Measuring real business goals, MBOs, things like revenue and customer sign‑up rates, and cart fulfillment, and cart abandonment. These sorts of things. Those are the metrics that my customers, the people I talk to are interested in for DevOps, closing those feedback loops in those three areas.

Gordon:  One of the things I find interesting, what you just said, Andi, is that you read these DevOps surveys, DevOps reports, and often the metrics, or at least what they're calling metrics, are framed in much more technical terms. How many releases do we have per year, or per week, per hour?

What's the failure rate? How quickly can we do builds? How quickly can we integrate? Which, I think to your point, are probably worth measuring, but they're really...The ultimate goal of DevOps is not to release software faster.

Andi:  Exactly. It's interesting because you do look at these metrics in isolation, and they matter. All this matters. 10 deploys a day, we all know that from 2009 in Velocity. That matters, but 10 deploys a day is no good if they're all bad deploys. You need to measure quality in that.

But even if it's a good quality deploy and you do it quickly, if it's not moving the needle on what your business wants you to be doing, then again, it doesn't matter. I think it's actually really important to connect these together so you really are getting metrics, correlating metrics, that matter across the whole range to really understand whether you're doing good or not.

Gordon:  One of my favorite Dilbert cartoons, I don't remember the exact wording but to the effect of...Pointy Hair goes, "We're now going to measure you on the number of lines of you code you write," and Wally says, "I'm going to off to write myself a new car today."

Andi:  [laughs] Yeah, exactly. That's one of the things that I actually do measure. We measure it internally. A bunch of our customers do actually measure code volume. There's a couple of interesting reasons for that. Especially in a DevOps and Agile mode, actually delivering too much code can be a signifier that you're doing things badly.

You're writing too much code, you're doing too much in one release rather than doing small, iterative releases. It can also signify that one person has too much of a workload. When you think about DevOps and the concepts around empathy and wanting to make sure that life doesn't suck for everyone, when one person is doing all the work, that sucks for them.

There are actually good things that come out of measuring code volume [laughs] but saying that more code equals better code, equals a bonus? That's a really bad thing. [laughs]

Gordon:   I think a lot of people tend to lump data metrics into this one big bucket. As we've had discussions before, there are these business metrics which have to be somehow connected to things.

It's not clear that overall company revenue is necessarily a good DevOps metric. Some of the other things you mentioned certainly are. In many cases, it does make sense to collect a lot of underlying data for data analytics and things like that. Then, you also have alerts.

Andi:  Yeah, the business stuff is really interesting. I know one of our customers releases the software-as-a-service. They're a SaaS company, cloud native and all that. Their developers actually do care about who uses specific features.

They'll implement a feature. They do canary releases. They'll implement feature on 10 out of a 1,000 servers, or whatever. Certain volume or percentage of their customers will get access to it. Then they'll measure using Splunk the way that those features are being used or not. They also measure the satisfaction of those customers.

They've got these nice smileys, and tick marks, and stuff that say, "Yes, I enjoyed using this feature." They can correlate that together, and it actually means the next day after doing a commit, after doing a release, they actually know whether the business use cases being satisfied, which is very cool.

I know a television company in the UK that we work with. They actually send reports on a weekly basis, I think it is, to their marketing department, based on whether users are using the website, what they're doing on the website, whether they're clicking through on competitions.

That's actually really important, but obviously mostly what people are doing in using data and the feedback...Closing the feedback loops is what I'm talking about here at DevOps Summit.

They're closing the feedback loop around those technical measurements.

Am I creating more bugs? Am I creating availability issues? Am I creating problems with uptime? Am I closing out the feature set that is in the story or in the epic that I was promising to do? Partially, it's also around this accountability to each other. Am I doing what I'd promised I'd do?

Gordon:  Talk a little more about accountability.

Andi:  Yeah, that's one of my soapboxes at the moment. I see a lot of the empowerment that DevOps gives developers to make decisions. I think that's great, especially in companies where you've got systems thinking and they understand their role in the organization and what it means to deliver good outputs for their customers.

You give them a lot of responsibility. Their manager is the leader. You give your developers, and your operations team, and those DevOps professionals a lot of responsibility and lot of empowerment to do the right thing.

Also, I think that there's a need for them to be accountable for doing the right thing as well. Especially as DevOps grows in larger organizations and there are more and more people involved. Also, with the concept that DevOps is about helping and making sure that each other is having a good experience at their life and their work.

As a developer, you're not making sure that operators are getting called out late at night, and all this sort of stuff. If DevOps is about helping to work with each other, to collaborate, to communicate better, to make sure each other's lives get better as Dev and Ops professionals, then I think you need to be accountable in two ways.

You need to be accountable to your business, which often means being accountable to your manager for doing the work that you're meant to do, and doing the work you promised you would, within the bounds of the responsibility you've been given.

It's also being accountable to each other, from doing good work, and doing the right work in ways that helps your whole team move forward, and makes everyone else's life positive. I think we talk a lot about empowerment and enablement. We don't really talk much about the flip side of that, which I think is that accountability.

Gordon:  I think the culture talk around DevOps, and we did have lots of discussions around culture and some of the ways that it can be overextended and over‑applied. Yeah, it can turn into this "don't fear failure," empathy, transparency, etc. Unicorns farting rainbows. This very touchy feely, everyone's happy and sings "Kumbaya," but you are, at the end of the day, being paid to produce business outcomes.

There does need to be some accountability there. If you crash the SQL server three weekends in a row, and call in Ops, somebody's going to have to talk with you, as they should.

Andi:  Exactly. Especially when you talk about the DevOps toolchain and the life cycle of software. It's a very complex and opaque theme to try to see what's going on at every stage, especially if you're a manager who's not necessarily fully fluent in specific tools. They can't dig into the specific tools to have a look at that.

I think reporting up to your management and reporting to each other and saying, "I introduced these bugs and I'm sorry for it. I won't do it again." By the same token, "I introduced these newest features, and they were really successful. We should all celebrate that as a team."

I think that accountability is actually really important. You'll see this in manufacturing as well where we get a lot of our examples from. You'll see that if one person makes the same mistake several times, then they'll get into a training program, or they'll get different mentoring.

Maybe they'll move into a different part of the line where they're better suited, and their skills are better suited. You don't know how to make your team better if you're not being accountable to each other, and to your management.

That's, I think, something we've got to step up to as DevOps professionals for want of a better term, is how do we be accountable to each other, and to the company that pays us as you said to do the job?

Gordon:  You just talked about manufacturing. You just mentioned quality, and I think that's a pretty good segue because we often think about DevOps primarily, well, through the lens of developer for one thing, but that's another topic for another day.

We also tend to view DevOps, first and foremost, through the lens of this velocity, business agility, and so forth, but there is a very important quality component there as well. What are some of the ways that data can help to surface that quality component?

Andi:  Absolutely. Some of the things we're looking at ‑‑ and our customers are doing a lot of this at the moment ‑‑ is looking at areas like code coverage and tests, number of defects, defect rates per release. Looking at the aggregating and correlating the quality metrics out of multiple test and scanning tools.

Doing static analysis and looking at the defect rates, doing dynamic analysis, and then also looking at the defect rates, as well as application performance and health scores. Looking at the performance in terms of resource utilization, response time, availability, execution failures, and so forth.

Comparing current release in production with next release just about to come forward, and being able to run that over time, so you can see whether you're making quality improvements over time.

If you're able to actually give your application a health score, and then you can measure that not just in production, but also in staging, or pre‑prod, whatever you want to call it, then you can start to make sure that you're getting better with every release. Your quality is going up with every release.

You can do with actual data, real measurements, so coming out of these testing tools, as well as coming out of actually running that in a stood up environment. There's lots of feedback loops you can close there.

Once you start to find problems as you find them especially in production, but also in pre‑prod and staging, feeding those things back into the test cycle so that you never find the same mistake twice, because the first time you find it, the next time you'd test for it.

Gordon:  This idea of doing things incrementally in stages, before they hit production, is really important from a security perspective as well. I was just having a conversation with one of my colleagues, or actually several of my colleagues, about this kind of tension between the traditional security guy who is sort of, "Stop. Stop. Don't push it out there," and this idea of whether you like the term or not, DevSecOps, where security gets baked in, and added incrementally.

What we were saying, and what was really coming out as we were having this discussion was that while the reason there's this tension or maybe disconnect is from the security guys' point of view, to a degree, the serious security flaws are pushed out into production.

Well, that is something that simply needs to be stopped to a degree that you can tolerate failures and errors in security that don't hit the actual production environment, because you found them through automated testing, or whatever. Then that makes more sense as this incremental, and sometimes breaking things sort of process.

Andi:  Yeah. Absolutely. This is actually something I've done a little bit of work, and most of the work is being done by someone that you probably know well, Ed Haletky of the TVP, @Texiwill on Twitter.

He's done a bunch of work, and I've put my two cents worth, and it's probably worth maybe one. Looking at security, and security testing, pin testing, code quality testing, so finding things like potential SQL injection, these sorts of things.

Also using some of those tools like Fortifywhich will do quality of code scanning for security purposes. You can start to shift left in that respect, but also continuing to get inputs from security testing even post release. There's no reason why security testing can't keep going even after you've released.

You can get to a certain coverage rate. This is where data helps. You get to a 90 percent, or a 92 percent, or a 95 percent coverage rate, or confidence level if you will. You go, "OK, I'm ready to release. I know that the remaining five percent is potentially low impact, or low risk. I'll put it out there anyway, but continuing to test."

There's some really interesting work out there that Ed's published about cloud, cloud promotion, and cloud delivery that actually really focuses on using these metrics from security testing, both pre and post release, which I think is actually really important.

Gordon:  We're going to be hearing a lot more about this whole security angle everywhere. This is partly an IoT show. We've heard a lot about security. I'm not sure we've heard a lot of solutions, but we've heard a lot about security.

Obviously, it is a big part of the DevOps discussion. It's a big, scary world out there, and it's pretty universally recognized that having an auditor sign off once a year, and then you don't think about security for that application for another six months or whatever, really doesn't work today.

Andi:  Yeah. It's not my joke. I saw someone post it the other day. "What did you get owned by? Your toaster or your fridge?" It's so true, especially in IoT, but in a DevOps perspective, or DevOps context, being able to do that continuous security testing, I think, is really important, and bring security a shift left.

We talk about a shift left in all sorts of other areas, and we're doing it with QA which I think is awesome. We need to start doing it more with security, I believe. At Splunk, we do have a whole security practice around incident event monitoring, or unused behavior analytics. Being able to start to apply some of that in the test, and pre‑prod and staging environment I think is really important.

Being able to do some automated audit reporting around what is happening, penetrations, security violations, or passwords, or PII exposure, potential hard coded passwords, stuff like that, there's a bunch of stuff that developers could be, and should be responsible for that actually make security pro's life easier. Not harder. I think there's a lot of work yet to be done on that.

Gordon:  Absolutely. I'd go back to DevSecOps. I think there's this school of thought that, well, if you read the Phoenix Project properly, you wouldn't be having to have this discussion. Know security was baked in. Meanwhile in the real world, security has tended to be this separate profession.

We were both at DevOpsDays London. I still remember security professional I guess in his 40s, standing up in an open space, and go, "I'm one of those security guys who's been getting in your way. You know, this is the first time I've ever been to an IT conference that wasn't purely a security conference."

I love that story. Certainly not to pick on that guy. That's quite brave of him, getting up like that. I think that's such a perfect illustration of how security has operated in his own world as this gatekeeper to releasing applications.

Andi:  Yeah. People joke about IT being a department of no. Security has that moniker for fear or not. Obviously, security teams are just looking out to protect the business. That's their job. Having them in the tent, I think, is a better option, and we started to bring other teams into the tent of DevOps.

I actually gave a presentation. You can find it online at the Splunk user conference, that was titled something along the lines of "Biz PMO Dev Sec QA Biz Ops," or something crazy like that, about broadening the tent of DevOps.

Security's got to come into this tent. Being a security pro into your team, into your scrum, that's got to be a good start, doesn't it?

Gordon:  Right, even if they're not in the meeting in the stand‑up every week, or every day, at least having them be as part of the team. Just like there used to be a business analyst who's a part of the team. Our product and technologies operations, their DevOps story.

I call it the "Banana Pickle Story," because they would get asked for a banana, and as Katrinka describes it, six months later, they deliver this pickle. Really, their DevOps story...Again, the business level, because it's what matters to me.

They used a lot of technology like OpenShift, Platform Service, and Ansible for automation , things like that. But again, they were really focused on the business story of how do we get the stakeholders iterating with us. "Oops, that banana's looking a little green. Let's dial that back to yellow and get on with the other things."

Andi:  Yeah, and this is the agile model for development, is getting someone from the business...You're creating an MVP and getting someone from the business to evaluate it, and continue to iterate with their advice.

You know that you're creating the right thing as you're creating it, rather than finding out in six‑months time, that you've created a pickle, instead of banana. [laughs] I love that analogy.

We should be doing that more and more with security. If security is saying no to you all the time, then maybe you're not inviting them to the party as much as you should, so that they can say yes iteratively, rather than one big no at the end.

Gordon:  Right. Just to cap off this podcast. In order to prove to security, much less external auditors and to prove to these other stakeholders, you need data.

Andi:  Absolutely. Exactly right. This is fundamental to what I believe. We cannot continue making decisions based on "I feel that this is the right thing to do. I think we're going to have good results here." We're living in a society that's driven by data and facts. Especially as developers or IT professionals, we need to have these feedback loops based on real data.

Not just people coming back and saying "I don't feel like you did the right thing. I don't think that this was good. I think our release worked and helped our customers." We need to come back and stop having these back‑and‑forths over opinions.

There's some very crude statements about, "Everyone's got opinions," right? I like to say, "In God we trust. All others bring data." That's how we get these real feedback loop in a system's mode, getting feedback from productions systems, from customer interaction, from the security violations and the passes that we do make.

From the coverage, to know if we are doing the right thing in terms of speed, in terms of quality, in terms of impacting our business, that's where data has a huge role to play. It’s those feedback loops that DevOps depends on.