Thursday, July 14, 2016

Presentation: Containers-Don't Skeu Them Up (Use Microservices Instead)

William Henry and I gave this presentation at LinuxCon Japan in 2016. (About 8 hours after getting a panicked 3am email to fill a no-show slot.) It's similar to what we gave at LinuxCon Dublin last year but there are a few updates.

Skeuomorphism usually means retaining existing design cues in something new that doesn't actually need them. But the basic idea is far broader. For example, containers aren't legacy virtualization with a new spin. They're part and parcel of a new platform for cloud apps including containerized operating systems like Project Atomic, container packaging systems like Docker, container orchestration like Kubernetes and Mesos, DevOps continuous integration and deployment practices, microservices architectures, "cattle" workloads, software-defined everything, management across hybrid infrastructures, and pervasive open source.

In this session, Red Hat's Gordon Haff and William Henry will discuss how containers can be most effectively deployed together with these new technologies and approaches -- including the resource management of large clusters with diverse workloads -- rather than mimicking legacy sever virtualization workflows and architectures.

Presentation: Fail fast, fail often

My colleague William Henry and I just gave this presentation at LinuxCon in Tokyo. It ties into central DevOps concepts such as experimentation, constant iteration, and having a culture that supports these types of activities.

Here was the abstract:

Software projects were historically managed on a bet the farm model. They succeeded or they failed. And when they failed (as big software projects often did), the consequences were typically dire for, not only organizations as a whole, but for many of the individuals involved. Today, by contrast, many software and the development projects have evolved toward a much more incremental, iterative, and experimental process that takes cues from the open source model which excuses (and even rewards) certain types of failure.

In this session, we’ll discuss how failure can be turned into a positive. This includes the organizational dynamics associated with tolerating uncertain outcomes, the need to define acceptable failure parameters, and the technical means by which experimentation can be automated in ways that amplify the positive while minimizing the effect of negative outcomes.

Tuesday, July 12, 2016

Gordon's HaffTime - Issue #7 is live

Issue #7 talks travels, Red Hat Summit, automation, and gives a shout-out to a really good game.

Become a subscriber.

Thursday, June 30, 2016

Red Hat Summit: Red Hat Identity and Access Management

Smite I Pal and Ellen Newlands talk identity management at Red Hat Summit.

In this session Red Hat’s Ellen Newlines and Dmitri Pal discussed Red Hat's identity management portfolio from a near-term perspective, and presented the long-term roadmap—along with some advice for implementing identity management.

These are some highlights and quotable moments from the talk.

Identity management is complex but it’s something you need to do to protect your environment and, ultimately, the assets of your organization. Red Hat is focusing on making IdM automated and cost effective so that customers can focus on their business. It’s Red Hat’s job to provide the expertise.

The areas of vision:


You need to be able to authenticate from different types of credentials including passwords, certificates, smart cards and OTP tokens. And use single sign-on using Kerberos, SAML, and OpenID connect. Weave together multiple operating systems, multiple credentials, multiple authentication schemes (including a trust relationship between IdM and Active Directory in a Microsoft environment). 

Managed security.

Consistent Delivery

If you have consistency in the identities and access to them, you can deliver to systems, service

s, and applications together with policies to control access and privilege escalation. The goal is to make the use of this environment relatively seamless (even given the complexity).

Managed Security

The challenge is the management of identities in a complex interoperable world. The keys, the certificates, the other secrets need to be automatically provisioned, tracked, and rotated on an as-needed basis.

DevOps Enablement

Developers need to have the tools to build the next generation of containerized and non-containerized applications with authentication and the consistent delivery of security. If the developer can’t do this, what they’re doing isn’t much use in a production environment.

Some guidance for identity management:

  • Single source of identities. Don’t copy pass words around! It also makes it much easier for audit when identities are in a single place. 
  • Single sign-on is good. You need to protect the keys to the kingdom, but once you’ve established, use it as much as possible.
  • Don’t put passwords into files. Instead use Kerberos or certificates, or fetch secrets on the fly. When you build applications and stitch things together, think about how they’re going to talk with each other. It requires a bit more effort but don’t be afraid to move forward.
  • Automate your operations. We are in an era where changes are happening in real-time. Continuous integration and deployment of applications are needed to meet these business needs. Adopt the tools you need to do things in a simple, repeatable way. (For example, Ansible.)
  • Integrate applications so that applications can interface with each other in the context of the user. These interactions need to be managed—which is where an IdM Fabric, as shown below, comes in.

Identity Management Fabric.

Wednesday, June 22, 2016

Getting Started with DevOps (for devs) at Red Hat Summit

This post shares some highlights from Decomposing DevOps: Understanding How to Get Started, a session that I gave at Red Hat Summit in San Francisco this week together with Dylan Silva of Ansible by Red Hat.

Screen Shot 2016 06 22 at 10 52 24 AM

Our general approach in this session was to look at DevOps on-ramps first from a primarily developer-centric perspective and then from a more ops-oriented one. These aren’t mutually exclusive especially given that DevOps inherently commingles dev and ops concerns to a certain degree. Nonetheless, thinking about these two different constituencies is one useful way to frame the discussion.

For the dev-centric point of view about DevOps (which is what I’ll cover in this post), I find that a manufacturing metaphor provides useful insights. Whether building a car or building an application, what are some of the important principles to follow?

The first is automation. Worker productivity in automotive manufacturing has doubled in something like the past 20 years. Automating repeatable processes has simultaneously improved predictability and quality. (I guess it’s an exception to the “Better, faster, cheaper: Pick two” rule.) Similarly, developer automation that leads to an iterative CI/CD pipeline with build/test/approve/deliver/deploy stages leads to both faster and higher-quality software delivery.

Screen Shot 2016 06 22 at 1 45 43 PM

W. Edwards Deming, one of the original champions of statistical quality control, once said that “Without data you’re just another person with an opinion.”

Measurement and metrics are likewise a key ingredient in an iterative software development pipeline. DevOps metrics were actually the topic of a separate birds of a feather session discussion that I led with my colleague William Henry. A full discussion is beyond the scope of this post, but when thinking about collecting data it’s useful to step back and consider your most important objectives and then design a plan from that starting point. 

The automotive industry, like others, has embraced the idea of modularity and reuse; about half of all cars are built on a modest number of platforms that share key components and design elements across models and across brands. We see this same modularity in the concept of microservices—small, autonomous, bounded context services that communicate through APIs. Even when microservices aren’t adopted in their purest form, automated DevOps pipelines work most effectively when deployments can be small, frequent, and generally decoupled from other deployments. 

If DevOps can be thought of as transitioning software development from craftwork to a more industrialized set of processes, then the place where new cloud-native apps run can be equally thought of as morphing from a workshop to a factory. This factory has characteristics such as the following:

  • Software-defined infrastructure and/or public cloud
  • Automation
  • Application lifecycle management
  • Developer experience including self-service
  • Application services (databases, messaging, integration, mobile)
  • Container ecosystem
  • Orchestration and resource control
  • Management

There’s enormous innovation happening in all these areas across a wide range of open source communities. However, making all this consumable for developers is certainly a challenge. That’s a big reason why, in a study of DevOps early adopters conducted last year for Red Hat, IDC found that 80 percent expected PaaS to play a crucial role in enabling DevOps because “Platform-as-a-Service (PaaS) cloud infrastructure, self-service developer platform and  tools, and lifecycle management with DevOps processes—speeding time to value for both developers and operations."

That’s exactly what OpenShift does by bringing together technologies such as Red Hat Enterprise Linux, docker-format containers, kubernetes orchestration, CICD pipelines, and developer tooling. Features include:

  • Self-service
  • Multiple interaction models
  • Polyglot, multi-language support
  • xPaaS services
  • Immutable, container-based platform
  • Automation for application builds, deployments, scaling, and health management
  • Persistent storage option

The result? Developers get up and running quickly and gain access to the platform and tools that they need to be productive without sweating the details of the underlying infrastructure.

Tuesday, June 21, 2016

What are the right metrics for DevOps?

5569561425 305949e779 z

If you’ll be at Red Hat Summit check out our BoF and other DevOps sessions.

Ask about metrics for DevOps and the natural reaction is to jump to familiar technology-focused measurements. Uptime. Code deploys per hour, day, or month. Deployment failure rate. Even lines of code.

Certainly, it’s important to have metrics.

DevOps works because continuous iteration and improvement is fundamentally a more rapid and flexible approach to software development than slow rigid project cycles. However, fully realizing this approach to developing and deploying software means putting in place the measurement systems, technologies, and metrics to present actionable insights that can then be acted on appropriately. Adding to the complexity is the need to present appropriate metrics for different audiences such as developers, operations, and business leaders.

Tracking narrow technical indicators can indeed be useful as part of tracking the success of your DevOps initiatives. Analysis may point to trend lines that are pointing in bad directions. An increased number of failures that lead to customer-facing outages can hardly be anything but a negative indicator. Conversely, continuous improvements in measurements such as time to deploy new services are a good indicator that a DevOps initiative is helping to produce positive outcomes.

However, one needs to be careful about overly focusing on easy-to-measure numbers that aren’t necessarily particularly correlated with business outcomes. It can be useful to step back. What are you really trying to accomplish? What’s important to you?

Are you most focused on the deployment velocity of new services? Is improved code quality or more rapid security updates the more pressing factor behind your DevOps? Or are you taking a broader organizational view that emphasizes cross-team collaboration?

These are some of the questions that we’ll be asking as part of what’s sure to be a spirited discussion at the How to Know if Your DevOps is Working birds of a feather session that I’ll be moderating along with Red Hat’s DevOps Strategy Lead, William Henry.

Also be sure to check out the many other Summit sessions related to DevOps.

I’ll be co-presenting with Ansible GM Todd Barr in Getting Started with DevOps. We’ll be talking about on-ramps from both primarily developer-centric perspectives (with a focus on OpenShift) as well as more ops-centric ones (with a focus on Ansible). Other DevOps sessions include:

There are also a variety of sessions that directly focus on how organizations are successfully making use of Red Hat products to implement DevOps today. These include:

Finally, I’d just note that a big reason that DevOps is such a hot topic right now is that it’s part and parcel of a host of interrelated technology, architectural, and business model changes that generally fall under the term digital transformation. Containers, container management, microservices, software defined infrastructure, mobile, and the internet of things are among the technologies that dovetail with DevOps to enable organizations to deliver new digital services quickly and cost effectively. There are lots of Summit sessions on those topics too. Check them out.

Photo credit: Fickr/CC

Thursday, June 16, 2016

Newsletter Issue #6 - Red Hat Summit, OpenShift and more

Read it and/or subscribe!

The end of cattle vs. pets

6830903723 eb2df17454 z

Metaphors and models have finite lifespans. 

This usually happens for one of two reasons.

The first is that metaphors and models simplify and abstract a messy real world down to especially relevant or important points. Over time, these simplifications can come to be seen as too simple or not adequately capturing essential aspects of reality. (This seems to be what’s going on with the increasing pushback on “bimodal IT.” But that’s a topic for another day.)

The other reason is that the world changes in such a way that it drifts away from the one that was modeled.

Or it can be a bit of both. That’s the case with the pets and cattle analogy as it’s been applied to virtualized enterprise infrastructure and private clouds. 

The “pets vs. cattle” metaphor is usually attributed to Bill Baker, then of Microsoft. The idea is that traditional workloads are pets. If a pet gets sick, you take it to the vet and try to make it better. New-style, cloud-native workloads, on the other hand are cattle. If the cow gets sick, well, you get a new cow.

Pets and cattle roughly corresponded to the Systems of Record and Systems of Engagement taxonomy proposed by consultant Geoffrey Moore (of Crossing the Chasm fame). The former were stateful, big, long-lived, scale-up, and managed/maintained at the individual machine level. The latter were assumed to be stateless, small, transitory, scale-out, and managed at the level of the entire application (with individual VM instances destroyed and recreated in the event of a problem).

As an initial pass at distinguishing between traditional transactional apps and those designed along more cloud-native lines, the metaphor isn’t a bad one. I've argued that “ants” is a better fit than “cattle” because it captures the idea that individual service instances are not only disposable but they work together cooperatively to perform tasks. However, the overall concept of long-running mutable instances vs. short-lived disposable ones would seem to capture an essential distinction.

It still does, but as we as an industry continue to evolve DevOps practices and services-oriented architectural patterns for software defined infrastructure and orchestrated pools of containers, the metaphor is breaking down for several reasons. 

State matters

Many of the components/instances of a cloud-native application should be designed so that they are stateless. That is, they should use ephemeral storage—which is to say storage and data that only sticks around for the life of the instance itself. However, no one’s claiming that the data doesn’t need to live somewhere. For example, in twelve factor app parlance, there’s the concept of a backing service which "is any service the app consumes over the network as part of its normal operation. Examples include datastores (such as MySQL or CouchDB), messaging/queueing systems (such as RabbitMQ or Beanstalkd), SMTP services for outbound email (such as Postfix), and caching systems (such as Memcached)."

As we move to containerized infrastructures, the stateful vs. stateless dichotomy becomes particularly important because containers are explicitly designed to be immutable. As Keith Tenzer describes in this post about OpenShift v3 persistent storage: "Docker images are immutable and it is not possible to simply store persistent data within containers. When applications write to the Docker union file system, that data is lost as soon as the container is stopped.”

However, it’s still possible to associate persistent data with containers so that an entire application can be containerized. Keith goes on to note that:

OpenShift v3 supports using persistent storage through Kubernetes storage plugins. Red Hat has contributed plugins for NFS, ISCSI, Ceph RBD and GlusterFS to Kubernetes. OpenShift v3 supports NFS, ISCSI, Ceph RBD or GlusterFS for persistent storage. Kubernetes deploys Docker containers within a pod and as such, is responsible for storage configuration. Details about the implementation of persistent storage in Kubernetes can be found here.

Kubernetes allows you to create a pool of persistent volumes. Each persistent volume is mapped to a external storage file system. When persistent storage is requested from a pod, Kubernetes will claim a persistent volume from the pool of available volumes. The Kubernetes scheduler decides where to deploy the pod. External storage is mounted on that node and presented to all containers within pod. If persistent storage is no longer needed, it can be reclaimed and made available to other pods.

Most cloud-native applications require there to be persistent storage somewhere. While one can assume that it’s provided through a service running somewhere else, a platform supporting the development of complete cloud applications needs to provide persistence mechanisms within that platform.

Virtualization and cloud/software defined infrastructure convergence

Software-defined infrastructure technologies, also made initial simplifying assumptions in other areas such as networking architectures and the maintenance of running instances. Some of this was a sincere, if sometimes naive, desire to dump legacy encumbrances. However, it was also about getting an MVP out the door in a rapidly changing world. 

We’re seeing today the reintroduction of virtualization features required for “enterprise use cases” into projects such as OpenStack. Thus, Neutron (OpenStack networking) isn’t just about flat networking architectures and current versions of OpenStack support live migration of instances whether using shared storage or block-based storage associated with a single image. The fact that many of the same technologies such as the KVM hypervisor in Linux bridge enterprise virtualization and cloud technologies simplifies the bridging of the two worlds. (Of course, it’s probably increased the complexity of OpenStack relative to what it would look like in a purist cloud-only world. Such a purist OpenStack would also likely not be very useful.)

The continued evolution of the new application platform

Perhaps most of all though, the metaphor is breaking down because the idea that there are two canonical application architectures seems increasingly simplistic. 

I’ve already covered how persistent storage is an important component of most modern cloud-native applications. 

It’s also the case that many new applications will indeed be decomposed into lightweight single-function microservices that expose public APIs. However, getting to that point will be an evolution. Martin Fowler, who helped popularize the microservices term, has even argued for a “Monolith First” strategy in some cases, including for projects that are big enough to justify a shift to microservices over time. As a result, blanket statements about horizontal app scalability and disposable services don’t apply universally—even for apps that are greenfield and reasonably considered cloud-native.

Applications also have substantially different patterns that relate to how instances are clustered together using technologies such as Kubernetes (which OpenShift uses as its orchestration layer). Some types of applications are batch oriented in the vein of traditional high performance computing/grid while others are composed from multiple layers of services communication through APIs. There’s also considerable variety not only in the absolute scale of application components being scheduled and orchestrated, but also in the variety of the components (large, small, frequency of scheduling, etc.) and requirements related to quality-of-service, latency sensitivity, and so forth.

In short, while there are certain patterns that we tend to associate with cloud-native applications, there’s also much variety and even divergence in key aspects. Furthermore, it turns out that some traditional enterprise application characteristics such as persistent state and tightly-coupled components continue to play a role even for greenfield cloud apps. 

It’s not cattle and pets out there. It’s a whole menagerie!

Photo credit:

Wednesday, June 08, 2016

Presentation: The New Platform-You Ain't Seen Nothing Yet

The now mainstream platform changes stemming from the first Internet boom brought many changes but didn’t really change the basic relationship between servers and the applications running on them. In fact, that was sort of the point. Today’s workloads require a new model and a new platform for development and execution. The platform must handle a wide range of recent developments, including containers and Docker, distributed resource management, and DevOps tool chains and processes. The resulting infrastructure and management framework must be optimized for distributed and scalable applications, take advantage of innovation stemming from a wide variety of open source projects, span hybrid environments, and be adaptable to equally fundamental changes happening in hardware and elsewhere in the stack.

From CloudExpo, New York City

Links for 06-08-2016

Hybrid and IoT themes from CloudExpo

Screen Shot 2016 06 08 at 11 46 32 AM

There are a couple of themes that I seem to keep running into and this week at CloudExpo was no exception. Neither is exactly new. In fact, the first is something that some of us have been saying for a very long time. But both seem to have crossed some threshold to become a widely-understood normal.

The first of these is the acknowledgement that computing is heterogeneous and hybrid. From my perspective, this is barely worth remarking upon at this point with so many companies flinging around the word “hybrid” with wild abandon—however newly they’ve come to this particular reality. 

At this point, let me mention that I wrote a piece for CNET titled “There is no Big Switch for Cloud Computing” in 2009 when I was still an analyst. The Big Switch in question being the title of Nick Carr’s book in which he laid out the argument for an electric grid-like utility for computing. And my now-employer, Red Hat, has likewise been talking about portability across hybrid physical, virtual, private, and public cloud environments for almost as long.

Nonetheless, IBM’s Phil Jackson felt the need to emphasize the hybrid theme in his CloudExpo keynote. By itself, this probably wouldn’t have caught my attention given how many vendors are now belatedly embracing hybrid environments in their pitches. 

However, by coincidence, I also got into a conversation with John Mark Troyer, formerly of VMware and a generally smart guy. It started with his comment about “multi cloud” being a driver of projects like Kubernetes and DCOS (Mesos). He added that he was mostly thinking about "how conventional wisdom has shifted quickly from AWS-only to multi-cloud” and that "despite recent Oracle-Google ruling, cloning a hostile API is still not for the faint of heart."

He’s right. It wasn’t that long ago that there was a significant school of thought that the AWS API was key to any cloud strategy. That was essentially the whole basis of Eucalyptus’ business plan—allow organizations to build an AWS-compatible cloud. (HP bought Eucalyptus in 2014.) 

There are a variety of reasons why API compatibility with AWS largely dropped off the cloud agenda. That discussion would deserve its own piece. Suffice it to say though that there’s effectively been an ongoing and steady movement away from the view of cloud as a homogenized commodity toward something that’s hybrid in place, hybrid in service type (IaaS, PaaS, and SaaS), and hybrid in the types of capabilities offered, and hybrid in the audience for which it’s designed.

The second is that IoT discussions can be maddeningly unfocused. It’s about so many largely orthogonal topics that it’s hard to talk about technologies, business models, ecosystems, etc. associated with IoT except in the most general sense. Sure, we can define it as the interface between the physical and the digital world or we can discuss how IoT uses data to gain insights and optimize actions. But that’s really broad. As my colleague Kurt Seyfried wrote me: "Saying IoT is like 'Shipping of Things,' aka every business using the post/delivery system... It's too generic to be useful."

As I wrote after the MIT Enterprise Forum’s Connected Things 2016 event: "IDC’s Vernon Turner admitted that "It is a bit of a wrestling brawl to get a definition.” (For those who don’t know IDC, they’re an analyst firm that is in the business of defining and sizing markets so the fact that IDC is still trying to come to grips with various aspects of defining IoT is telling.) 

We’ve had something of the same issue with cloud. (I also wrote “Just don’t call them private clouds” in 2009.) And I imagine that, at the end of the day, we’ll muddle through in more or less the same way. But it’s worth at least observing that consumer wearables and smart home devices have little in common with industrial IoT solutions. For that matter, it’s unclear the degree to which solutions associated with healthcare, retail, agriculture, “smart cities,” and logistics/transportation will have in common beyond some sensor technology. 

Wednesday, May 25, 2016

Issue #4 of my newsletter is live

This issue has links to an article I recently had published on public cloud security as well as to discussions around using Ansible with docker-compose and why it's important to orchestrate containers using tools such as Kubernetes.

Links for 05-25-2016

Thursday, May 19, 2016

Data, security, and IoT at MIT Sloan CIO Symposium 2016

As always, the MIT Sloan CIO Symposium covered a lot of ground. Going back through my notes, I think it’s worth highlighting a couple sessions in particular—in addition to the IoT birds of a feather that I led at lunchtime. They all end up relating to each other through data, data security, and trust.

Big Data 2.0: Next-Gen Privacy, Security, and Analytics moderated by Sandy Pentland of the MIT Media Lab

There were two major themes in this panel.

Sandy Pentland

The first was that it’s not about the size of the data but the insights you get from it. This is perhaps an obvious point but it’s fair to say that there’s probably been too much focus on how data gets stored and processed. These are important technical questions to be sure. But they’re technical details and not the end in itself.

I might be more forgiving had I not lived through the prior data warehousing enthusiasm of the mid- to late-1990s. As I wrote five years ago: "There are many reasons that traditional data warehousing and business intelligence has been, in the main, a disappointment. However, I'd argue that one big reason is that most companies never figured out what sort of answers would lead to actionable, valuable business results. After all, while there is a kernel of truth to the oft-repeated data warehousing fable about diapers and beer sales, that data never led to any shelves being rearranged."

However, the other theme is newer—or at least amplified. And that’s ensuring the security of data and the privacy of those whose data is being stored. One idea that Sandy Pentland discussed is the idea of sharing answers (especially aggregated answers) rather than raw data. See as an example of a system that's designed to make it possible for parties to use and maintain data without having full access to that data. Pentland also noted that because systems such as this make it possible to securely ask questions across jurisdictional boundaries, they could help address some of the often conflicting laws about the treatment of personally identifiable information.

Getting Value from IoT

At my luncheon BoF table, we had folks with a diverse set of IoT experiences including Ester Pescio and Andrea Ridi of Rulex Analytics, Nirmal Parikh of Digital Wavefront , and Ron Pepin, a consultant and former Otis Elevator CIO. The conversation kept coming back to value from data. What data can you gather? What can you learn from it? And, critically, can you do anything with that data to create business value?

Per my earlier comment about data warehouses, gathering the data is relatively straightforward. It may not be easy, especially when you’re dealing with sensors that aren’t on your own property and therefore need dedicated networks of some sort. But the problems are mostly understood. It’s “just" a case of engineering cost-effective solutions.

But what data and what questions? Ron Pepin shared his experiences from Otis. Maintenance is a big deal for elevators. It’s also the main revenue stream; the elevators themselves are often a loss leader. Yet proactive elevator maintenance mostly consists of preventative maintenance on a fixed schedule. 

Anders Brownworth, Principle Engineer Circle, on Blockchain panel

It seems like a problem tailor-made for IoT. Surely, one can measure some things and predict impending failures. But it’s not obvious what combination of events (if any) are reliable signals for needed maintenance. There’s a potential for more intelligent and efficient maintenance but this isn’t a case where you can cost effectively just instrument everything—someone else owns the building—and the right measurements aren’t obvious. Is it number of hours, number of elevator door reversals, temperature, load, particular patterns of use, something else, or none of the above?

The Blockchain

Given the level of hype around blockchain, perhaps the most interesting thing about this panel by Christian Catalini of MIT Sloan was the the lack of such hype.

Interest, yes. Catalini described how blockchain is an interesting intersection of computer science, economics & market design and law. He also argued that it can not only make things today more efficient (which could potentially redefine the boundary of firms by reducing transaction costs) but also create new types of platforms.

That said, there was considerable skepticism about how broadly applicable the technology is. Anders Brownworth of Circle (which has a peer-to-peer payment application making use of blockchain) said that the benefits of blockchain are broadly in the area of time-based transactions, with interoperability, and with many able to audit those transactions. However, with respect to private blockchains outside of finance, “we trust all the people around the table anyway” and, therefore, the audibility that’s inherent to blockchain doesn’t buy you much.

In the same vein, Simon Peffers of Intel agreed that it’s "hard to let thousands of users have the same view of data with a traditional database. But some blockchain use cases would fit with traditional database.” He added that "There is a space for smaller consortiums of organizations that know who the parties are with other requirements that can be implemented in a private blockchain. Maybe you know who everyone is but don't fully trust them."

To sum up the panel: You’re usually going to be giving up some features relative to a more traditional database if you use blockchain. If you’re not making use of blockchain features such as providing visibility to potentially untrusted users, it may not be a good fit.

Photos (from top to bottom):

Sandy Pentland, MIT Media Lab

Anders Brownworth, Principal Engineer, Circle

Tuesday, May 10, 2016

Links for 05-10-2016

My newsletter experiment

There’s a certain range of materials–curated links to comment upon, updates, and short fragments–that to me have never felt particularly comfortable as blog posts or on twitter. Tumblr never quite did it for me and I’ve little interest in shoving content into yet another walled garden anyway. I’ve been thinking about trying a newsletter for a while and, when Stephen O'Grady joined the newsletter brigade, I figured it was time to give it a run. We’ll see how it goes.

Here’s a link to the first issue:

It includes some DevOps related links and short commentary, links to a couple of new papers I’ve written on security and deploying to public clouds, and upcoming events including Red Hat Summit in San Francisco at the end of June. (Regcode INcrowd16 saves $500 on a full conference pass!)

You can also subscribe directly to this newsletter here.

The need for precise and accurate data

8266473782 fef433d94b k

Death by GPS (Ars Technica):

What happened to the Chretiens is so common in some places that it has a name. The park rangers at Death Valley National Park in California call it “death by GPS.” It describes what happens when your GPS fails you, not by being wrong, exactly, but often by being too right. It does such a good job of computing the most direct route from Point A to Point B that it takes you down roads which barely exist, or were used at one time and abandoned, or are not suitable for your car, or which require all kinds of local knowledge that would make you aware that making that turn is bad news.

It's a longish piece that's worth a read. However, it seems that a lot of these GPS horror stories--many from the US West--are as much about visitor expectations of what constitutes a "road" as anything else. It's both about the quality of the underlying data and its interpretation, things that apply to many automated systems. 

According to Hacker News commentator Doctor_Fegg:

This is clearly traceable to TIGER, the US Census data that most map providers use as the bedrock of their map data in the rural US, yet was never meant for automotive navigation.

TIGER classes pretty much any rural "road" uniformly - class A41, if you're interested. That might be a paved two-lane road, it might be a forest track. Just as often, it's a drainage ditch or a non-existent path or other such nonsense. It's wholly unreliable.

But lest you think data problems are in any way unique to electronic GPS systems, read this lengthy investigation into a 1990s Death Valley tragedy.

For what it’s worth, I did some cursory examination into what Google Maps would do if I tried to entice it into taking me on a “shortcut” through the Panamint Mountains in western Death Valley. My conclusion was that it seemed robust about not taking the bait; it kept me on relatively major roads. However, if I gave it a final destination that required taking sketchy roads to get there (e.g. driving to Skidoo), it would go ahead and map the route.)

After writing this, it occurs to me that for situations such as this, we need data that is both accurate (represents the current physical reality) and precise (describes that physical reality with sufficient precision to be able to make appropriate decisions).

Monday, May 09, 2016

Interop 2016: The New Distributed Application Infrastructure

The platform for developing and running modern workloads has changed. This new platform brings together the open source innovation being driven in containers and container packaging, in distributed resource management and orchestration, and in DevOps toolchains and processes to deploy infrastructure and management optimized for the new class of distributed application that is becoming the norm.

In this session, Red Hat's Gordon Haff discuses the key trends coming together to change IT infrastructure and the applications that will run on it. These include:

  • Container-based platforms designed for modern application development and deployment 
  • The ability to design microservices-based applications using modular and reusable parts 
  • The orchestration of distributed components 
  • Data integration with mobile and Internet-of-Things services 
  • Iterative development, testing, and deployment using Platform-as-a-Service and integrated continuous delivery systems

Tuesday, April 26, 2016

DevOpsDays London 2016

Devopsdayslondon 1

April London was cool. But DevOpsDays London was hot and happening, selling out its venue in the shadow of St. Paul’s Cathedral. In many respects, it was a fairly typical DevOpsDays event with a focus on organization, process, and culture over individual products and toolchains. 

In other respects, it reflected the evolution of DevOps from something most associated with Silicon Valley “unicorns” to a core set of principles, processes, and practices that are broadly applicable. Also reflecting a location not far from the City of London, Barclays was a major sponsor and both financial services firms and major system integrators were well-represented in the audience and in the booths. 

With that as preamble, here are some of the discussions and other topics that caught my eye in one way or another during the course of the two-day event.

Metrics matter

As Splunk’s Andi Mann  observed in an open spaces discussion, it’s nice to measure the things that you do—but it’s even better to measure what you actually accomplish. And better still is to measure accomplishments that closely map to business outcomes rather than IT outputs. 

One participant noted that “We had all these metrics. 1100 of them. We ran a report every month. But why do these metrics matter? Will it help someone make a decision on a daily basis?” Another wryfully observed that "shipping crap quicker isn't a metric anyone should want to measure."

This led to further discussion about the distinction between metrics, alerts, and logs—something that was also touched on in some of the presentations. Google’s Jeromy Carriere pointed out that, in contrast to logs that enable root cause investigation, "alerts need to be exciting. If they're boring, automate them."

Enterprise DevOps

As I wrote above, there was a significant enterprise, even conservative enterprise, angle to the event. For example, Claire Agutter talked about how to “Agile your ITIL.” (I suspect there are Silicon Valley companies lacking a developer who even knows how to spell ITIL.) 

Claire observed that “the reason companies look away from ITIL is it looks bureaucratic” even though "it's how IT gets done in many organizations.” She pointed out that the issue is that ITIL has been implemented as a slow-moving waterfall process in many organizations. However, it doesn’t need to be and, in fact, the best way to think about ITIL process is simply that it’s a consistent way of doing things. And what’s a great match for a consistent way of doing things? That would be automation (using a tool such as Ansible.)

Bimodal IT?

Arguments about definitions and appropriate models often seem a bit “how many angels can dance on the head of a pin”-ish to me. I mostly felt that way when I was an analyst (and analysts generally love creating definitions and models) and I certainly feel that way now. That said, it seems to have become sufficiently trendy to bash Gartner’s bimodal IT model (see e.g. Kris Saxton’s "Bimodal IT:  and other snakeoil” from this event) that I feel compelled to respond. 

Most of what I think is worth saying I have already and won’t repeat here. But, really, Kris largely made my general point in his talk when he said: "A lot of people take away the headlines. The details are largely sane but [bimodal is] most problematic as a vision statement communicated from the C level.” I guess I have trouble seeing the problem with a largely descriptive model for enterprise IT that will inevitably be upgraded and replaced in pieces and at different rates. And CIOs who don’t bother to read beyond the headlines and latch onto this (or any other model) to justify simply maintaining the status quo? Well, that organization has bigger problems than a Gartner model that’s possibly insufficiently nuanced or visionary.


I led an open spaces discussion on best practices for security in a DevOps world especially when there are compliance and regulatory issues to consider. We actually ended up having two back-to-back security discussions; the one prior to mine focused on what “tolerate failure” means in a security/risk context. In practice, the discussions flowed into each other. In any case, the only issue was that so many people wanted to participate that it was a bit hard for everyone to pack themselves in!

The shared experiences around security were generally consistent with what I’ve heard in other discussions of this type. For example, there was a lot of interest in automated vulnerability scanning using tools such as OpenSCAP. Also mentioned was using human and machine-readable formats such as Ansible Playbooks to document processes and ensure that they’re followed consistently. (Alas, also consistent with other discussions was the familiar refrain that a lot of auditors are still not prepared to move beyond whatever paper-based checklists they’re already familiar with.)

My “the times they are a changin’” moment came though when someone piped up that he was one of those security guys that are often described as roadblocks to rapidly releasing software. He went on to add that this was the first conference he had ever attended that was not an explicit security conference and he was going to go back to his company and recommend that the security team attend more events of this type. This really highlighted just how siloed security processes can be while providing a hopeful illustration that DevOps really is starting to create new opportunities for collaboration and communication.

This last point is crucial. I know folks who get a bit grumpy about the degree to which DevOpsDays majors on culture rather than the cool tool du jour. Tech is important both as a platform and a toolchain for DevOps certainly. However, so many of us operate in an environment where it’s so natural to fixate on the latest shininess that it’s useful to be regularly reminded about the degree to which culture and more open organizations are even more fundamental components of digital transformation.

Monday, April 11, 2016

Connected Things 2016 recap

Screen Shot 2016 04 11 at 3 32 50 PM

The Internet-of-Things (IoT) and DevOps seem to be in a race to win the “most conferences/events” race. The IoT corner notched a pair last week with the Linux Foundation’s new OpenIoT Summit in San Diego and Connected Things 2016 put on by the MIT Enterprise Forum at the Media Lab in Cambridge.

I haven’t looked at the contents from the OpenIoT Summit but I do have thoughts from Connected Things that mostly reinforced everything else I see going on in the space.

Everyone’s talking.

This 500 person or so event sold out. This is clearly a hot topic and there’s a sense that it must be important. As we’ll see, the whats, the hows, the whys, and the the wherefores are a lot fuzzier. I’ve been through plenty of these new technology froths and I’m not sure I’ve ever seen quite such a mismatch between the hype and today’s more modest reality. No, hype’s not even quite right. It’s almost more of a utopian optimism about potential. Cue keynoter Rose, the author of Enchanted Objects: Design, Human Desire, and the Internet of Things. This is about cityscapes and intelligent spaces and the automation of the physical world.

But what is it?

At a high level, I think the definition or definitions are pretty straightforward. There’s an element of interfacing the physical world to the digital one. And there’s a big role for data—probably coupled with machine learning, real-time control, and machine-to-machine (M2M) communications. 

But how should we think about the market and where’s the value? Things get a lot murkier. 

(As I was writing this, an email literally popped into my account that read in part: "That brand new car that comes preloaded with a bunch of apps? Internet of Things. Those smart home devices that let you control the thermostat and play music with a few words? Internet of Things. That fitness tracker on your wrist that lets you tell your friends and family how your exercise is going? You get the point.” My point is that we have to refine our thinking to have useful discussions.)

At Connected Things, IDC’s Vernon Turner admitted that "It is a bit of a wrestling brawl to get a definition.” (For those who don’t know IDC, they’re an analyst firm that is in the business of defining and sizing markets so the fact that IDC is still trying to come to grips with various aspects of defining IoT is telling.) 

In general, the event organizers did make a gallant attempt to keep the sessions focused on specific problem classes and practical use cases but you were still left with the distinct feeling that the topic was coiled and ready to start zinging all over the place.

Data data everywhere. What do we do with it?

Data is central to IoT. Returning to Vernon from IDC again, he said that “By 2020, 44 zettabytes of content will be created (though not necessarily stored). We’ve never seen anything that scales at this magnitude before.” He also said that there will be a need for an "IoT gateway operating system where you aggregate the sensors in some meaningful way before you get the outcome." (I’d add at this point that Red Hat, like others, agrees that this sort of 3-tier architecture--edge, gateway, and cloud/datacenter—is going to generally be a good architecture for IoT.)

What’s less clear is how effectively we’ll make use of it given that we don’t use data very effectively today. McKinsey’s Michael Chui, on the same panel, noted that "less that 1% of the data collected is used for business purposes—but I expect an expansion of value over time in analytics.” I do expect more effective use of data over time. It’s probably encouraging that retail is leading manufacturing in IoT according to Vernon—given that retail was not a particular success story during the c. 1990s “data warehouse” version of better selling through analytics. 

Security matters—but how?

I’m tempted to just cut and paste the observations about security I made at the MassTLC IoT conference last year because, really, I’m not sure much has changed.

MIT’s Sanjay Sarma was downright pessimistic: “We have a disaster on our hands. We'll see a couple power plants go down. Security cannot be an afterthought. I'm terrified of this."

No one seemed to have great answers—at least at the edge device level. The footprints are small. Updates may not happen. (Though I had an interesting discussion with someone—forget who—at Linux Collaboration Summit last week who argued that they’re network devices; why shouldn’t they be updated?) Security may to be instantiated in the platform itself using the silicon as the secret. (John Walsh, President, Sypris Electronics). There was also some resignation that maybe walled gardens will have to be the answer. But what then about privacy? What then about portability?

There’s a utopian side to IoT. But there’s a dystopian side too.

Sunday, April 10, 2016

Building a garage hoist for my canoe

IMG 1306

A couple of weeks ago, I finally got around to putting together a system that could 1.) Get my canoe into my garage in the winter when there are two vehicles there, 2.) Allow one person to lift it into position, and 3.) Fit it around existing structures, hardware, and other stored items. I’d been storing it on a rack outside but, especially with Royalex no longer being made, I wanted to treat it with a little more care.

The trickiest part, as you can see from the first photo, was that there’s a relatively small three-dimensional volume to fit the 17 foot canoe into. It had to go front to back, clear the garage door opener and garage door, ideally not force me to move the sea kayak, and have room for my small car to slide in underneath. It did all work, but barely, and it meant that I needed to cinch it up fairly precisely.

To start with, I just installed a couple of pulleys to lift the boat, but a Tripper with outfitting weighs over 80 pounds and it was just too heavy to readily lift up and then cinch into precise position. 

Now you can deal with the weight problem by adding additional pulleys so that you’re pulling more rope with less force. However, it can be hard to get the canoe to pull up evenly and I could never get this working in a way that positioned the boat as precisely as I needed it to be.

IMG 1307

I next considered an electric winch. I went so far as to buy one and I think it would have worked but I was having trouble finding an ideal place to mount it and it seemed like overkill.

The solution I ended up with was a manual 600 pound winch that cost under $20 from Amazon. As you can see, two lines go up to a pair of pulleys. (I have overhead beams for storage on this side of the garage so I ended up just tying the pulleys to the existing beams.) One of the lines then heads down over the swivel pulley and is clipped into one cradle holding one end of the canoe. The other line goes through its pulley, which changes its direction 90 degrees to run to the other end of the canoe where a final pulley drops it down to be clipped into the other cradle. 

Don’t read too much into the exact pulleys I used. I had a couple laying around and I bought another couple of “clothesline” pulleys at Home Depot. I could probably have mounted the canoe right side up with just a sling but I think I was able to get it up a little higher this way. (It’s a tight fit; I guess if I ever get a bigger car, I’ll have to revisit this. The canoe gets transported on an SUV.)

IMG 1308  1

I’ll probably add a couple more clips to the system just to make it a little easier to position the cradles. And, before next winter, I’ll put a backup safety sling of some sort in place. But, overall this system seems to work very well. It takes very little effort to hoist the canoe into place and, once all the rope lengths and slings are properly adjusted, it’s very repeatable and straightforward. The canoe hangs down a bit lower than is ideal but that’s pretty much dictated by the garage door layout.

Wednesday, April 06, 2016

Specialists may have their day

Irving Wladawsky-Berger, who among other accomplishments led IBM’s early Linux efforts, has a great post up regarding special report on Moore’s Law in The Economist. Among the highlights:

Tissues, organs and organ systems have evolved in living organisms to organize cells so that together they can better carry out  a variety of common biological functions.  In mammals, organ systems include the cardiovascular, digestive, nervous, respiratory and reproductive systems, each of which is composed of multiple organs.

General purpose computers have long included separate architectures for their input/output functions.  Supercomputers have long relied on vector architectures to significantly accelerate the performance of numerically-intensive calculations.  Graphic processing units (GPUs) are used today in a number of high performance PCs, servers, and game consoles.  Most smartphones include a number of specialized chips for dealing with multimedia content, user interactions, security and other functions.  Neural network architectures are increasingly found in advanced AI systems.  

As in evolution, innovations in special-purpose chips and architectures will be increasingly important as Moore’s Law fades away.

I agree with Irving. When I was an analyst I saw specialized architectures largely fail because "why bother when Moore's Law would get you there in a year or two anyway?" I'm not sure the implications of losing the CMOS scaling lever are as widely appreciated as they should be. (The former head of DARPA microelectronics peg them at about about a 3500X improvement over the past couple of decades; you don't lose a lever like that and just go on with business as usual.)

This will have a lot of implications for software certainly. I also wonder about the broader implications of smaller, lighter, cheaper, faster increasingly no longer being a given.

I wrote about this in more detail after the SDISummit in Santa Clara at the end of last year.

Wednesday, March 30, 2016

Links for 03-30-2016

Monday, March 28, 2016

The "laptops" in my bag

I took one of my Chromebooks to an event last week and a couple of people asked me about it. So I thought I’d take the opportunity to talk about my larger devices as an extension to my recent “What’s in my bag?” post. 

In general, I carry a laptop-like device and a tablet-like device. Laptop-like devices are really a lot better for typing on and tablet-like devices are better for reading or watching video on a plane. And I haven’t found anything that really switches between the two modes well. I’m happy to stipulate that the Microsoft Surface Pro 4 may be that device for some people but I’m really not in the Microsoft ecosystem and longer so that doesn’t work for me at this point. 

More on tablet-like devices in a later post but, usually, my laptop-like device for travel is my 2015 13” MacBook Pro. Because it’s a full laptop, it’s the most versatile thing to take with me—especially if I might not always be connected. It weighs about 3.5 pounds and is .71 inches high. For me, this is about the perfect compromise for working at a desk and traveling. The smaller MacBook models are just a bit too small or tradeoff things like a second USB port that keep me from wanting to use day-in and day-out. I do value compactness and lightweight when I’m traveling but I find that, by the time you add chargers and dongles and various adapters, another pound or so of laptop just isn’t a big deal. This is still a very svelte laptop by any historical standard.

How about Chromebooks?

First, let me share my thoughts on Chromebooks in general and then I’ll get to a couple specific models. 

Chromebook are pretty awesome for the right use. At around $250 for many models, they’re a great match for doing web-based tasks (browsing and online office suites or even many software development tasks). You even get a hidden Crosh shell that gives you utilities like ssh. You’re not totally dead in the water if you go offline—for example, Evernote switches back and forth between connected and non-connected modes pretty smoothly—but they’re definitely oriented toward situations where you have reliable WiFi. (On the one hand, this is increasingly common; tech conferences notwithstanding. On the other hand, it’s also hard to do many things disconnected even if you have a full laptop.)

For $250, you’re not going to get high-resolution screens, backlit keyboards, or things like that. But my 13” Dell Chromebook from 2014 sits on a table downstairs in my house where I often find it more convenient for doing a lot of searching than using a tablet. (Yes, I could go find my laptop but I find it being “just there” handy.)

A variety of higher-priced Chromebooks out there have more features. Personally I get a lot less interested in a Chromebook as it gets to a ~$500 price point and beyond given that it won’t replace a laptop for most people.

More interesting from a travel perspective is a device like the Asus Chromebook Flip. It’s a 10.1” laptop that weighs about 2 pounds. The touch-sensitive screen also flips into a tablet mode. In my experience, it also has pretty reliably “all day” battery life which is probably a couple hours longer than my MacBook. If I don’t need more than a Chromebook and want to go lightweight, this is what I carry.

A few caveats:

Unlike my 4GB Dell, I have the 2GB memory Asus model—mostly because that’s all they had at the Best Buy when I needed something during a trip when I accidentally left my MacBook at home. It does stutter every now and then if there are multiple tabs open, so go with 4GB. 

The keyboard is fine, but it is small. I have no issue with using this as a travel laptop but I wouldn’t want to type with a keyboard this size all the time.

The tablet mode is “OK.” By that I mean it feels a bit weird having the keypad under your fingers when you’re holding the folded laptop though it can be used as an ebook reader in a pinch. I also don’t normally get movies and TV from Google Play so I don’t have a simple source for video content. This isn’t a problem with the device so much as the fact that Google is yet another separate ecosystem for content that you may or may not already be using.

So. For most work trips today, my MacBook Pro still usually wins. It’s just more versatile and I have access to a lot of presentations and other materials even if I’m not online. But, it’s not hard for me to imagine smaller (perhaps convertible in some way) devices becoming a more practical travel option over time.