Trust, Enarx and TEEs, and open source security

Today, the Linux Foundation announced the intent to form the Confidential Computing Consortium, a community dedicated to defining and accelerating the adoption of confidential computing. As it so happens, I recorded this podcast at devconf.us last week with Red Hat security experts Mike Bursell and Nathaniel McCallum in which we discuss Red Hat Enarx, a project for providing hardware independence for securing applications using Trusted Execution Environments (TEE). It’s one of the projects that will be contributed to this consortium. We also cover broader issues of trust and open source security.

This episode is part of Innovate @Open, a new podcast that focuses on open source with a particular focus on how collaboration and openness are leading to new inventions and innovations.

Show notes:

• The Enarx project
• New Cross-Industry Effort to Advance Computational Trust and Security for Next-Generation Cloud and Edge Computing (Linux Foundation)
• Enarx for Everyone (a quest) by Mike Bursell

Listen to podcast:

• Trust, Enarx and TEEs, and the nature of open source security [15:51 MP3]

Transcript:

-->

Gordon Haff:  You're listening to "Innovate @Open." Stories from the cutting edge of technology innovation rooted in open‑source software and collaborative processes. I'm your host, Gordon Haff.

[music]

Gordon: What I have today is a podcast I recorded last week at devconf.us with Mike Bursell and Nathaniel McCallum. In that podcast we talked about the nature of trust and specifically about a new project called Enarx, which is an application deployment system that lets applications run within trusted execution environments.

This was particularly timely because today, August 21st, the Linux Foundation announced the intent to form the confidential computing consortium.

The basic idea here, is that as companies move their workloads to a bunch of different environments, hybrid computing environments, they need protection controls for sensitive IP and workload data and they're increasingly seeking greater assurances and more transparency of those controls.

The challenge is that current approaches in cloud computing address data at rest and in transit. Encrypting data in use is considered the third and possibly the most challenging step to providing a fully encrypted life cycle for sensitive data. Let's kick things off by having Mike tell us about trust.

Mike Bursell:  When you run any process, or you run any application, any program on a computer, it's an exercise in trust. You are trusting that all the layers below what you've written, assuming you've written it right in the first place, are things you can trust to do what they say they're going to do on the card.

I've got to trust my middleware, I've got to trust the firmware, I've got to trust the BIOS, I've got to trust the CPU or the hardware. The OS, the hypervisor, the kernel, all the different pieces of the software stack. I've got to trust them to do things like, not steal my data, not change my data. Not divert my data to somebody who shouldn't be seeing it. So that's a lot of pieces.

If you're looking at a standard stack of 10, 12 pieces, just think about all the different libraries you're using, all the different parts of the kernel, all those different bits. How can you trust that? That's a real difficulty.

It's the reason that people don't run sensitive workloads or keep really sensitive data on the public cloud, generally. Do you want to put your really sensitive data, your research, algorithms, programs on a public cloud service provider where they could look at it?

Or even, if you've got sysadmins, how much do you trust all of your sysadmins? Because a sysadmin, if they have root, could look at anything on any of those systems, even your internal systems.

Do you want your CEO’s payroll data to be in there? What about legal data about companies you are acquiring?

All of these things are difficult, and they are something that concerns a lot of people in the enterprise, in government, throughout the world.

We wanted to look at this. It just turns out there's some new set of technologies coming out right now called trusted execution environments. They are CPU and chipset technologies, which allow you to run programs, applications, in such a way that even the hypervisor, even root, even the kernel can't look into what you're doing.

That's great. Fantastic. They are publishing information from AMD, from Intel, from IBM, all the stuff coming out. But they're all different. They all handle the problem in a different way.

We, at Red Hat, started thinking about this and came up with some ideas. We decided that we wanted to make it easier for you to use these things.

Gordon:  That sounds really interesting. We are going into a little bit more about what this means, about what we have to trust, what we don't need to trust any longer. Nathaniel, could you walk our listeners through in a little more in detail, how this whole thing works?

Nathaniel McCallum: One of the things that we are concerned about is that a lot of our existing technologies require you, essentially, to write your application to the technology. It should be no surprise to the listeners here that Red Hat is very much against lock-in.

We want it to be possible for you to write your applications using the standard APIs that you already use, in the languages you already use, with the frameworks that you already use, and to be able to deploy these applications inside any hardware technology possible.

This is the goal of the Enarx project. One of the things we realized early on was that there's a new technology called WebAssembly which is being used in browsers all around the world. Literally, every single browser supports WebAssembly. It's being looked to very much as a sort of future to JavaScript.

The thing that's really interesting about WebAssembly to us, is that the capabilities that WebAssembly can deliver in conjunction with the WebAssembly system API. It is almost exactly the same set of functions that you can actually do inside these hardware environments.

It also means that you get to write an application in your own language with your own tooling. You can compile it to WebAssembly. Then Enarx will aid you in securely delivering that all the way into a cloud provider, and to be able to execute that remotely. The way that we do this, is we take your application as inputs, we perform an attestation process with the remote hardware.

We validate that the remote hardware is in fact, the hardware that it claims to be, using cryptographic techniques. The end result of that is not only an increased level of trust in the hardware that we're speaking to. It's also a session key, which we can then use to deliver encrypted code and data into this environment that we have just asked for cryptographic attestation on.

The end result is that you get to write your own application the way you want to write it. To get to deploy it in Enarx where you see fit, and you don't have to make your application depend upon specific hardware technologies.

Gordon:  In the show notes, I'm going to link to some information about this project. Could you take us through fairly a fairly high level how this works?

Nathaniel:  Basically, the way it works is that, once we've completed our attestation, we now have a session key that proves cryptographically that we are talking to our remote party. We can do so in a way that is encrypted using all of our standard cryptographic technologies.

We deliver the WebAssembly code that you have produced as part of your application, directly to our secure execution environment on the remote host. At that point, it is then just in time compiled for the actual CPU that you are going to run on. Then everything will be executing in that environment on the native processor.

We're also going to take care to enforce additional security measures. For example, if you persist any data, you'll be able to at some point, we don't currently implement this, but at some point will be available to read and write to a file system. The host will only see encrypted block devices.

The same thing is going to happen for networking. We're not going to allow unencrypted networking, but we will allow you to do TLS, for example, to communicate out. The end result is you just get to write your application. Then when you deploy it in this way, you have very strong assurances that there's a whole class of attacks against your application, that won't be able to get off the ground.

Mike:  What we're doing is we're basically...Remember, I talked to the beginning about how you've all got all of these layers you need to trust. We're removing the need for you to trust most of those layers, because the only things you need to trust are the chip vendor, and the firmware they provided. Which is all cryptographically signed, you can check that.

The Enarx code and the application you've written yourself and of course the Enarx code is going to be open. It's one of the kind of weird things about security. In order to be really confidential and to be closed to everyone else, you need to do your actual implementation and your coding and your design in the open. It's generally accepted these days that open source provides for better security overall.

We at Red Hat, of course, want everything to be open source. Enarx is completely open, will always be open. We're using open source technologies all over the place. We're using Rust as the main language. It's very well regarded for security, and for knowledge of what happens when things go wrong.

If you have faults in in your application, you know what's going to happen. It's not just going to start spilling digital as a place that could be used by a malicious host to work out what you're doing, for instance. We're developing in the open, so that you can do stuff in a closed way, with your sense of data, which you should control data and algorithms.

Gordon:  Take one of the things that we've seen over the last number of years and you being in security, Michael and Nathaniel, is that. I think so many people came to open source or looked at open source from the perspective of, "Oh, you never published the schematics for your alarm system if you're a bank."

No matter how good you think your security is, bring over that analog into the open‑source world and marry into the cryptography world, of course, that doesn't really apply.

Mike:  It's difficult because people assume that… That will be good analogy, that you've got your schematic of your bank vault, and a key. Cryptography is very different from that. You absolutely should never be using a cryptographic algorithm which is not known, and open, and peer‑reviewed. A really good cryptographic algorithm, the only thing that needs to be secret is the keys you're using.

They say that any fool can create a cryptographic algorithm that they can't break. I've certainly created cryptographic algorithms that I couldn't break and other people showed me how I'd gone wrong. It's one of these little things you learn to do as your apprenticeship in moving into security is doing this, so you understand how it goes wrong.

Cryptography is very much not like that. You need peer review, you need academics. Security is different, in some ways, to other parts of open source, in that there's this well‑known dictum that with enough eyes, all bugs are shallow, which is a great dictum of course.

But it's not quite that easy for security, because the number of people who have expertise in security, it's small. You need to ensure that their eyes are being applied. It's not good enough to have lots of un‑expert eyes looking at security. You need expert eyes, looking at security.

That's one of the reasons that companies like Red Hat ‑‑ but there are many others, Microsoft these days, Intel, IBM ‑‑ are spending a lot of time getting their security experts looking at cryptography and open‑source cryptography. Because it benefits the entire community and the whole ecosystem. What I call the commonwealth of what we are as an open‑source.

Nathaniel:  Just to add to what Mike was saying, this notion that with enough eyes, all bugs are shallow, this is predicated on the ratio of eyes to the amount of code.

When you are dealing with secure code, one of the things that you want to ensure is that because there are a limited number of eyes, we also need to try to limit the amount of code as much as possible.

This is why a project like an Enarx is all about reducing the trusted computing base. We want there to be a lot less code that you have to trust, which means that we need less eyes to review it to make sure it's secure.

Gordon:  We've certainly seen recently low levels of hardware in the light, that you get into these very complex pieces of engineering and it gets harder and harder to predict or to figure out every possible security exploit.

Let’s go maybe a little bit far afield as we wind this down.

Coming back to trust, what are some of the other areas in this low level, the software stack, in terms of trusted execution environments, in terms of firmware, in terms, perhaps, of CPUs themselves, where work is being done, or you think that there are possibilities to increase the security?

Mike:  Let me start with one, which is TPMs. TPMs have been around for quite a long time and people have not been using them. Partly because within the open‑source world, there was a great concern, 10, 15 years ago now, I guess in the early 2000s, that they were going to be used for DRM.

DRM has long been anathema to much of the open‑source community. They never got taken up usually within Linux and the open‑source community. There's a new version of TPM 2.0, which is much improved, and people are beginning to realize there's great benefit in using them.

The thing about a TPM is it's a hardware root of trust. It's really good for that if you need to be building up levels of trust because you can't do everything in Enarx, yet. There are times you need to build up trust, and it's a very good building block for those sorts of things. That's one example. Nathaniel, have you got some others?

Nathaniel:  There's been a variety of technologies, even besides the TPM. Unfortunately, none of them have really gone very well. Most of them have been hard to use, they've been hard to enable on the system, and they've been driven by a lot of concerns like DRM. Concerns that don't put the user first. This is why one of the key principles of the Enarx project is to make sure that we always put the user first.

Mike:  For years now, we've understood about encrypting data at rest when it's stored, encrypting data in transport when it's going over the network. We're now moving into a world where we need to encrypt data and algorithms in process.

That's what TEEs are for and that's what Enarx aims to make it easy for you to do as a developer.

Gordon:  Thank you for listening to this episode of Innovate @Open. For future episodes, subscribe to Innovate @Open on your favorite podcast app.

Keeping my herbs alive: An indoor watering system

As I was again reminded in a recent Twitter thread, I like having fresh herbs. But you often don’t need a lot of them, which in turn means that it’s nice to have some pots of them growing at home so you can snip off just a little bit rather than buying 20 times what you need at the store. 

The problem is that 1.) I travel and 2.) I forget to water plants. One common low-tech method for automatically watering is to fill a wine bottle (or soda bottle, etc.) upside down in the soil. This works reasonably for a few days but my issue was the 2-week plant-killing excursions. So I looked around for commercially-available solutions.

There were a few systems online but at least one of them had to sit up above the plants, which wasn’t really feasible for my setup. And none of them seemed to have very good reviews. So I decided to see what I could come up with myself. I already had a small hydroponics system in the same room which turned my thoughts to supplying the water from an aquarium pump in a bucket of water on the ground hooked up to a timer. This is indeed what I ended up doing but getting to a system that actually worked took a fair bit of fiddling and experimentation.

My first thought was to “borrow” the aquarium pump I already had to keep my hydroponics system topped off in the summer and use some T-connectors to fan out the tubing so that I could water multiple plants. To make a long story short, the pump I had wasn’t powerful enough to lift water and force it through a network of T-connectors. Elevating the bucket on a stool helped. Sort of. But if I lifted it too high, the water started siphoning when the pump turned off. Furthermore, as I discovered when I experimented with different tubing sizes, in a network of tubing like this, it’s hard to get relatively even flows out of the different lines.

What ended up working the best had three basic elements:

A more powerful aquarium pump (head of about 8 feet)

A manifold intended to split the output of an aquarium air pump

A digital timer with 1 second resolution

The manifold was really the key thing here because it splits the flow pretty evenly. Furthermore, each of the outputs has a little flow control valve that lets you tweak the flows so you can account for different line lengths or different amounts of water for different pots. You just have to experiment to see how long you want to run the pump for. For me, it’s about 15 seconds which is probably a little on the short side; I sometimes end up watering a little every week or two. 

Now, in practice, actually building this was more complicated because of getting all the tubing sizes right. The big culprit was the pump. In order to get an aquarium/pond pump with sufficient head (pressure) you need to get one capable of vastly more flow than needed for this application. And because the pump is designed for that higher flow, its output size is fairly large so you need to adapt the tubing down to the significantly smaller diameter that’s appropriate for this application (and the manifold). I got it almost right; I had to use some epoxy paste in one place because I couldn’t find an adapter that was quite what I needed.

(There’s also a lot of inconsistency in how sizes are advertised, e.g. both the check valve and the manifold are supposed to be 3/8” but the tubing fits easily on one and was a very tight squeeze on the other. You may need to fiddle around.)

Finally, I added a check valve though it probably isn’t needed.

In addition to the parts listed above, here’s what you need:

• Short piece of 3/4” tubing (I had some black pond corrugated pond tubing from earlier experiments)
• You may need some epoxy paste if a connection is a bit loose connecting this tubing to the adapter
• Multi-hose adapter
• 3/8” tubing (probably ID)
• Aquarium airline tubing (I believe it’s 3/16” diameter)

Finally I was trying to figure out what I could stick the pots in so that I didn’t need to worry if there was some overflow. I was coming up more or less blank. The options either weren’t really the right shape or they were a lot deeper than I needed or wanted (or both). Then I found the perfect thing: a 26” water heater drain pan. Just cover up the hole for the drain with duct tape. To make it even more perfect, I happened to  have a round 26” table up in my attic.

Now my herbs are happy and I’m happy. 

More 2008 redux: Open APIs

Given the discussion going on around API openness these days, I thought I’d resurrect yet some more text from an "Open Source vs. the Cloud" research note that I wrote in 2008. (See also "Open Source vs. the Cloud Redux” and this twitter thread.)

At the same time, to focus on source code is to focus on a specific type of openness and freedom that was important historically—but may not be as important going forward. Indeed, in the case of Web services running on massive server farms and cooperating over a network with all manner of other code, services, and data, the value of code is questionable. After all, you can hardly just load it up on a server and do anything useful with it anyway. One needs all those servers and interlocking pieces. Also, the ability to view, modify, and redistribute source code is only one of many rights or protections to consider in a Cloud Computing world. For example, consider these other things that might matter more:

...

Open APIs. Open Source as we know it today evolved largely in the context of Unix-like operating systems and the programs that ran directly on top of them using “libc” and other system libraries. While we may run monolithic programs over the network, much of the action in Web 2.0 has been in services such as Facebook, Flickr, Google Maps, and Salesforce.com that expose application programming interfaces (API) at a higher level. This allows developers considerable freedom to extend these platforms. Thus, whether a platform or application is Open Source or not, given public APIs, it can be extended and consumed in ways that are very analogous to Open Source. At the same time, the predictability and transparency of the terms of service for APIs—especially in the case of consumer-oriented services—raise their own issues.

The cloud vs. open source redux

If you’re reading this, you’re probably aware that there is a fracas going on around open source licensing. Quite a bit has been written on the topic and I won’t rehash the specific details here; they have been well covered by:

• Stephen O’Grady
• Bryan Cantrill
• John Mark Walker
• Kevin McLaughlin

However, to net l’affaire out, long-simmering issues associated with building businesses on the back of open source software are boiling over. In particular, cloud providers like Amazon Web Services (AWS) not only rely on vast amounts of open source software to run their infrastructure but they’re increasingly offering cloud services that directly compete with the companies that created much of that open source software in the first place. Furthermore, there’s a widespread (largely justified) perception, that some of these providers in particular are taking from the open source commons far more than they’re giving back.

Some thoughts.

This is not a new concern

As an industry analyst, I wrote a research paper titled “The Cloud vs. Open Source” in 2008. That’s just two years after AWS debuted. Much of the paper cautions against getting too fixated on source code when thinking about user freedoms and openness generally. This remains true today and is the subject of an entire chapter of my book How Open Source Ate Software that I published last year.

However, I also argued that throwing up roadblocks to making use of open source software was ultimately unproductive.

Today, Open Source is widely embraced by all manner of technology companies because they’ve found that, for many purposes, Open Source is a great way to engage with developer and user communities—and even with competitors. Therefore, the concern that, left to their own devices, companies will wholesale strip-mine Open Source projects and “take it all private” seems anachronistic. That’s not to say that everyone will always contribute as much code without copyleft as with it, but the suggestion that copyleft is all that’s holding the whole Open Source process together just doesn’t square with the facts.

Was I just wrong?

Now, at this point, you might turn around and say: “But wholesale strip-ming is exactly what’s happening. We need even stronger protections if the commons is not to be ruthlessly exploited!"

One problem is that all the evidence suggests this doesn’t work. Permissive licenses like Apache, MIT, and BSD have gained in popularity over time. There’s a reason for this. Much of modern open source’s success isn’t about the ability to view source code. It’s about its collaborative development model. And the Eclipse Foundation's Ian Skerrett argues that "projects use a permissive license to get as many users and adopters, to encourage potential contributions. They aren't worried about trying to force anyone. You can't force anyone to contribute to your project; you can only limit your community through a restrictive license."

Another data point is the AGPL. At the time the new version of the copyleft GPL came out (GPLv3 in 2007), the Affero General Public License was introduced as a new GPL variant. Copyleft basically says that if you distribute software, you have to make the source code available. This includes any changes you made. The rub is that, under the GPL’s terms, “distributing” basically means shipping software on a disc or offering it for download. This creates what some saw as a loophole because offering the software as a cloud service isn’t distribution as traditionally defined.

(Is this starting to sound familiar? I told you none of this was new.)

Enter the AGPL, which was just like the GPL except the definition of distribution was broadened to include offering software as a service.

However, the AGPL hasn’t been much used. Ironically, one of the users was MongoDB, which is one of the current companies that have relicensed their software to prevent its use by cloud providers. In general, lots of companies are nervous that the AGPL could potentially interact with internal code that they don’t want to make publicly available. So its often on the license no fly list for application development.

Which is all to say that the overall direction in open source has been away from restrictive licenses. Leaving aside whether an even more restrictive license could still be reasonably considered “open source,” there just seems very little appetite for such a creature.

Words matter

In my view, a lot of the heat around licenses like the Commons Clause comes about because the companies involved seem to be, on the one hand, trying to gain the perceived value of a proprietary license while also getting credit for still being open source. “Open core” arguably plays the same parlor trick.

It still might have been news if one or more of these companies simply relicensed some or all of their software to a license that was unabashedly proprietary even if it retained some aspects of open source. But I suspect it would have been much less of a tempest.

Whether or not doing so would have been a good idea is a separate question. But it’s their software. Their business challenges are real. Own it. If you’re not going to have an open source development model, I’m not sure why you particularly even care if it’s technically open source or not.

Can we make cloud providers do better?

While the software vendors are taking heat from one side. Cloud providers are taking it from another. There is indeed a widespread view that most cloud providers are takers rather than givers. AWS, as the #1 cloud provider, takes particular heat. It’s mostly deserved. Although Adrian Cockcroft’s team has arguably moved the needle in making AWS play better with open source communities, much more could be done.

However, publicly shaming Amazon will not be a very effective strategy to drive change. If you can't sell the business value of participating in open source, you've pretty much lost the battle. Shaming might net you some contributions for the PR value but certainly no real commitment. Pinning your hopes on Jeff Bezos’ altruism is not a winning move.

Instead, as Linux Foundation Executive Director Jim Zemlin  told me during an interview at the Open Source Leadership Summit last year: 

The epiphany that many companies have had over the last three to four years, in particular, has been, "Wow. If I have processes where I can bring code in, modify it for my purposes, and then, most importantly, share those changes back, those changes will be maintained over time.

"When I build my next project or a product, I should say, that project will be in line with, in a much more effective way, the products that I'm building.

"To get the value, it's not just consumed, it is to share back and that there's not some moral obligation, although I would argue that that's also important. There's an actual incredibly large business benefit to that as well." The industry has gotten that, and that's a big change.

In closing

None of this is to dismiss the underlying challenges that these changes came in response too. There will always be challenges at the level of the individual company trying to build a business no matter what the product. But there are more macro dynamics here as well. 

This shift of computing towards public clouds recreates a new type of vertically integrated stack. One-time chief technology officer of Sun Microsystems, Greg Papadopoulos, one suspects hyperbolically and with an eye towards something IBM founder Thomas J. Watson probably never said, suggested that “the world only needs five computers,” which is to say there would be “more or less, five hyperscale, pan-global broadband computing services giants” each on the order of a Google.  

Some cloud giants have indeed made significant contributions to open source projects. For example, Google originally created Kubernetes, the leading open source project for managing software containers, based on the infrastructure it had built for its internal use. Facebook has open sourced both software and hardware projects.

But, for the most part, these dominant companies use open source to create what are largely proprietary platforms far more than they reinvest to perpetuate ongoing development in the commons. And they’re sufficiently large and well-resourced that they mostly don’t depend on cooperative invention at this point.

It’s easy to dismiss free-riding as a problem given that organizations are missing out in some ways if they do so. However, to the degree that large tech companies, both cloud providers and others such as Apple, take far more from the open source commons than they contribute back, this at least raises concerns about open source sustainability.

The last section is based in part on content from How Open Source Ate Software (Apress 2018)

2019 New Year updates

A new year. A few updates.

In 2018, for the second year in a row, I published a book. How Open Source Ate Software with Apress. Buy early, buy often as the saying goes. If I do anything along these lines in 2019, it will probably be a mini-book about the lessons we can take away from Intel’s Itanium processor that I followed closely while an analyst. I’ve written an outline and we’ll have to see if I get the energy to do something more.

In part because of my open source book, this blog has been a bit inactive of late. I’ve also been publishing in a number of other places, including opensource.com, The Enterprisers Project, and TechTarget. I plan to continue doing so but I’m going to try to get back into the swing of jotting down quick thoughts here—related to both professional interests and otherwise. (I had planned to kick off a separate travel and food blog last year but I pretty much got no further than registering a domain. I’m just going to post here as the mood strikes.)

One decision I made over the holidays was that I’m going to pull the plug on my Cloudy Chat podcast. It’s gotten pretty irregular and unfocused and I don’t really see that changing. Furthermore, while my work focus remains fairly broad, it’s shifted towards emerging technology topics, which “cloud” really isn’t any longer even if some of its newer aspects are. This isn’t to say that I won’t do the occasional published interview. But I think a podcast implies a schedule and topic focus that I don’t really see being in the cards for the immediate future.

What may take its place is some more work with video. I haven’t fleshed out what that means exactly but it’s an idea I’ve noodled on over time. I’m hoping that taking “I really ought to do a new podcast” off my omnipresent mental to-dos will free up some cycles to create short educational videos related to emerging tech areas.

Contact and social media information is unchanged. I’m on twitter as @ghaff. My photos are on Flickr. (I’m hopeful for Flickr’s revitalization post-Yahoo.) I’m on LinkedIn but, if I don’t know you, you send a form invite, and our connection isn’t obvious to me, I’ll probably ignore it. I’m still on Facebook but I’m pretty selective about friend requests; if I ignore you don’t take it personally if you’re a casual and/or professional acquaintance.

I occasionally do product and book reviews. Feel free to inquire if you have something you think might interest me. But I’m don’t do a lot and I emphatically don’t write about things that I haven’t gotten hands-on with. I also sometimes do interviews at events. (See above re: podcasts however.) I mostly limit these to discussions about open source projects and other non-commercial topics. Otherwise, there’s just too much opportunity for conflicts of interest. 

I’m always open to speaking at/attending events on professional topics of interest. Some presentations are up on Slideshare. I do have a busy travel schedule though so I have to prioritize where I spend my time. (Somehow, the first few months of the year ended up especially crazy.)

To be done: Updating my overall website. It’s still a good source for links but I never really liked the design I used and I’ve held off on updating it recently as a result. 

What was new at Serverlessconf?

Cue the obligatory “There still servers in serverless” and “There is no cloud; it’s just someone else’s computer.” Like many, I’m not a fan of the term but it’s seemingly here to stay. I’ll get over it, just like I did with private cloud. In any case, the more interesting question is what’s going on with serverless—which is what took me to Serverlessconf in cool, gray San Francisco last week.

This is the point where it makes sense to introduce serverless for anyone who may have heard of the term but hasn’t studied it closely. And that turns out to also be a good segue for discussing the conference as a whole. As recently as last December, I was on a panel and a member of the audience asked us about the difference between serverless and Function-as-a-Service. We were able to offer the beginnings of an answer at the time—not that everyone was quite singing from the same hymnal—but my sense is that most everyone is in alignment now, with a caveat that I’ll get to presently.

So what are serverless and FaaS? For that I’ll turn to a recent blog post by my Red Hat colleague William Markito Oliveira who wrote in a recent blog post:

Functions-as-a-Service (FaaS) is an event-driven computing execution model that runs in stateless containers and those functions manage server-side logic and state through the use of services. Serverless is the architectural pattern that describe applications that combine FaaS and those hosted (managed) services. MartinFowler.com has a great article that provides more details and the origin of the terms.

In other words, FaaS is one component of serverless. But an application written using serverless patterns will also generally use a variety of standard building blocks to provide common services across many applications. Databases, authentication, and proxies are examples of services that many different applications require. These will be managed by an operations team; from a developer’s perspective, it doesn’t matter if that team works for the same company and runs them on-premise or if the team is employed elsewhere, such as a public cloud provider.

This brings us to my caveat. While most everyone agrees on what makes for a serverless architectural pattern, there’s far less unanimity on the degree to which this pattern mostly applies to public clouds only, fits with various hybrid application development and deployment models, and over what timeframe the assumed shift to increased serverless usage takes place.

Serverlessconf itself rotates fairly far on the public cloud angle. But that really shouldn’t be surprising. The organizers of the show are a training organization focused on public clouds. Of course, they’re not the only ones to see serverless as part and parcel of the rich set of public cloud services that complement FaaS on public clouds. The finer pricing granularity of many of these cloud services (including FaaS) relative to per-hour or even per-minute on Amazon Web Services EC2, for example, is also seen as a feature that doesn’t really translate to an on-premise environment. (That said, there was far more emphasis at this conference on developer productivity advantages than on pricing models; I want to say this represents at least a shift of degree from this conference in New York last year.)

At the same time, a number of talks expressed a pragmatic recognition that serverless and FaaS isn’t for everyone, at least today. Amiram Shachar disused “Shipping Containers as Functions.” Yochay Kiriaty pointed out “Mistakes and Anti-Patterns in Serverless (or when NOT to use Serverless).” Kiriaty noted that serverless best fits with a specific async and event-driven programming model. Other characteristics that are mostly needed to benefit from serverless include: stateless logic, idempotence of functions, one task per function, and functions that finish quickly and avoid recursion.

More generally, Erica Windisch argued that 12 factor provided guidelines. But serverless enforces them. 12 factor is most associated with early hosted platform-as-a-service, so this may give you some sense of the type of applications that are primary serverless targets today.

A tweet from Andrew Clay Shafer that I assume was inspired by this or another talk stated: “Serverless is a particularly opinionated PaaS.” This is worth pondering, if only because I think there are a lot of hard and fast lines being drawn around what are essentially architectural patterns: serverless, containers, PaaS, VMs, IaaS, CaaS (containers-as-a-service) whereas it’s more of a continuum with adjacencies that blend into each other and combine features. A topic for another day.

Google Next: Enterprise, ML/AI, open source, and hybrid

Things I learned at Google Next (or at least I think I did). Think of these as preliminary observations based on sessions I watched, people I spoke with, or tweets I read/replied to.

One thing about the show reminded me of an AWS re:Invent from four or so years ago. Earlier re:Invents mostly trotted the usual suspects up on stage: Netflix, SmugMug, other startups that probably are no longer with us. Suddenly, we had NASDAQ and other well-known large enterprise customers who demanded mission-critical out of their infrastructure. This year at Google Next saw a similar transformation. There were young companies doing cool stuff; Indonesia's Go-Jek made a particular splash. But there were also plenty of speakers from companies like Nielsen and Target (which is leaving AWS in favor of Google). [ADDED: At least that was the image projected from the main tent stage. As a colleague correctly noted to me, the show floor told a much more startup-centric story relative to where AWS and Microsoft Azure are today.]

I'm not sure I heard any direct, or even oblique, references to competitors but I think it was pretty clear where Google thinks its differentiation lies. The most prominent area was ML and AI which was omnipresent. There were some explicit announcements of ML services, mostly aimed at making ML more accessible to the millions of developers who are not data scientists. But elements of AI/ML were pervasive whether as part of Google Maps or GSuite. The second area was open source. Open source as a central strategy was strongly reflected in the small Community day I was invited to (and gave a lightning talk at) on Monday but it was also front-and-center on the opening day keynotes as well.

Google announced that Cloud Functions was GA and took the covers off Knative. (Knative essentially helps create a common building block for serverless on top of Kubernetes across hybrid clouds. My colleague William Markito Oliveira has a nice piece up that discusses FaaS, serverless, and Knative in more detail.) Google sort of soft-pedaled serverless (which I'll use as the general term even if I don't like it) though. They announced Knative on Day 1 in press release. But serverless only got a short segment in the Day 2 keynote. I'm not sure what to make of it. One theory I heard, which I sort of like, is that given AWS' FaaS time-to-market and mindshare lead with Lambda, Google is taking advantage of Kubernetes' container mindshare to enter the market through that door rather than take on Lambda directly.

To expand on the previous point slightly, here's a quote from the Day 2 keynote: "Containers are the universal platform for cloud." I'm pretty sure neither Microsoft nor AWS would make that statement. And I think I'm not over-analyzing to say that Google views serverless as part-and-parcel of a broader container infrastructure and cloud-native app dev environment as opposed to a discrete technology. For what it's worth, I agree with this view. I think there's too much drawing of hard lines between these different approaches to writing and running services going on--but that's a topic for another day.

Finally, the hybrid thing. Google announced an on-prem version of its Google Kubernetes Engine (GKE). First of all, I think I should be getting royalties for some of their messaging; it sounds a lot like various things I've written over the years. But I digress. It's a good story and one with which I obviously agree. There's clearly an appetite for being able to run workloads portably across different environments. But I'd just observe that this is very new territory for Google. Enterprise customers bring a lot of quirks, integration needs, and customization requests to their in-house infrastructure. Heck, if they are happy with a fully standardized offering, they probably should be looking at just using a public cloud. So, strategically this makes sense. But it's not really in Google's wheelhouse and they may find this sort of offering less amenable to the sort of technical solution they're accustomed to creating.

More to come but these are some observations after the first couple of days.

Links for 07-11-2018

• Blockchain beyond the hype: What is the strategic business value? | McKinsey & Company - "Despite the hype, blockchain is still an immature technology, with a market that is still nascent and a clear recipe for success that has not yet emerged. Unstructured experimentation of blockchain solutions without strategic evaluation of the value at stake or the feasibility of capturing it means that many companies will not see a return on their investments. With this in mind, how can companies determine if there is strategic value in blockchain that justifies major investments?"
• Miniservices as a Realistic Alternative to Microservices
• Google Is Building a City of the Future in Toronto. Would Anyone Want to Live There? - POLITICO Magazine
• 7 Features that Make Kubernetes Ideal for CI/CD - The New Stack

Podcast: Patrick Maddox of Twistlock on container security

Twistlock does lifecycle vulnerability and compliance management for containers. In this podcast, we talk about balancing the needs of developing fast and operating securely, including how to use tooling as a common framework for driving discussions between those writing apps and those responsible for operating them.

Links:

• Twistlock

 

Listen to MP3 [13:09]

Listen to OGG [13:09]

Podcast: Arpit Joshipura on open source networking

Arpit Joshipura, general manager of networking at The Linux Foundation, takes us through the evolution of networking from its proprietary beginning. He describes the networking layers that make up the full stack and explains how technical capabilities like disaggregation, the software-defined phenomenon more broadly, and virtual functions have led to the big changes in the telco world and increasingly networking more broadly that we see today.

Listen to podcast MP3 [00:12:25]

Listen to podcast OGG [12:25]

Huawei Chief Strategy Officer Bryan Che talks the China market and open source

I worked for Bryan Che for my first seven years at Red Hat. We caught up with each other at the Open Source Leadership Summit in Sonoma. Bryan thought it would be interesting to share his observations and experiences on open source adoption in China, the China cloud market and cloud portability, and what it's like living in China. One of the interesting dynamics we discuss is that China is starting afresh in areas such as public clouds where, in the US, patterns were well-established before the current generation of open source software became available. 

Listen to podcast MP3 [00:22:26]

Listen to podcast OGG [00:22:26]

[Transcript:]

Gordon Haff:  Today I am joined at the Open Source Leadership Summit in Sonoma by Bryan Che. Bryan Che is actually the person who hired me on at Red Hat, about eight years ago.

He is now working at Huawei in ChinaHe's going to give us some insights about the China market, about open source in China, his personal experiences in China, and things in transformation. Welcome, Bryan.

Bryan Che:  Thanks, Gordon. It's great to see you again here, and a lot of old friends from Red Hat, as well as many new friends from the area.

Gordon:  Maybe you can start by introducing yourself, and tell us about your journey.

Bryan:  I'm currently the Chief Strategy Officer at Huawei. I'm working out in their headquarters in Shenzhen, China, which borders with Hong Kong and southern China. I was at Red Hat for 15 years. Before that, straight out of MIT, where I did my bachelor's and master's in computer science. Spent the last 20 odd years in Boston, and never figured I'd move overseas to work.

This was a really interesting opportunity to be able to work in open source software, but also in everything from consumer electronics, to telecommunications and hardware, and also see a lot of the things that were happening on the other side of the world. I'd read about it. I'd visited a few times ‑‑ on business trips and vacations ‑‑ but it's another thing to actually be in the midst of it, day by day.

Gordon:  We were talking the other night about the China market, and how it's so very different from ‑‑ it's probably fair to say ‑‑ every place else in the world. I thought it'd be interesting, now that you've had some time, some perspective...With your work, you've obviously spent a lot of time looking at it in depth.

What is the few‑minute‑or‑so summary of how you see the Chinese market around telecoms, around computing, around consumer apps...all of that kind of stuff?

Bryan:  I think one of the fascinating things for me that I've observed, is that technology space is really booming. A lot of the really high‑level themes are similar to what we've seen in the US, in Europe, and elsewhere. There's a big focus on digital transformation, on machine learning and AI, and IoT.

The way these technologies are evolving and being deployed is very, very different from what I've seen elsewhere. Just as one example in the public cloud space, we've had AWS and a lot of these other public clouds like Google and Microsoft and IBM and so on starting 10, 11 years ago in the US. It's a more recent market in China.

One of the things that's been really interesting for me as I've been working with our cloud business in terms of their public cloud strategy and what we do around open source is that if you had the opportunity to do public cloud again, now with 10 years hindsight and now with all these new open source technologies and architectures and microservices and Kubernetes and OpenStack, what would you do differently?

Being able to have the chance to do that in China where adoption is growing very, very quickly in the public cloud standpoint, but is still relatively new compared to the adoption in the US and other parts, it provides a really fun thought experiment as well as a way to be able to try new things.

On the other hand, we're also seeing because China doesn't have as much incumbent technology, in many cases, it's leapfrogging a lot of the things that I've experienced here in the US.

Just as one example, at Huawei, we've been working with some of the bike sharing companies like Ofo. If you come around Shenzhen or Shanghai or Beijing or any of the major cities around China, you'll see all these bikes everywhere.

They're just parked in the streets and not at a dock. One of the things that has become really popular is you can just pick up a bike, unlock it with your smartphone, ride it to somewhere else and then immediately drop it off there.

If you take a look at what they've had to enable from a digital standpoint to do those kinds of things, first off, they have to be able to track where all the bicycles are and so they built in all these new sensors and chipsets into them. They have to work indoors and congested areas. They've got to be able to work with their phones in an easy way to do mobile payments around that.

Then, you've got to be able to have the logistics necessary to manage OK, if they're not just going from a fixed point A to B, how do you manage all that?

These kinds of things where it's been sort of leading in China as opposed to first coming from the US over to the China market, it's interesting to see how digital technology is really changing a lot of people's lives in ways that are very different compared to other parts of the world.

Gordon:  It seems particularly when you talk about payment methods and things like that, there is a huge amount of inertia and custom and we've always done it this way. I always find it interesting even between countries in Europe, there's very different attitudes towards cash and credit cards and things like that.

Of course, the residents of any given country, whether it's the US or the UK or Sweden, just can't understand how messed up the rest of the world is compared to what's obviously the right thing. China's had an amazing transformation. Just really, I think, totally different from any place else in that regard.

Bryan:  Yeah. Most of the people that I interact with have gone completely cashless in Shenzhen and in Greater China. For example, just to give one story about how mobile payments has totally integrated, when I go to a restaurant, sitting on the restaurant table will be QR codes.

You scan the QR code and up shows the menu on your phone. There'll be pictures of all the dishes. Sometimes, there'll be reviews from other people. You just order the dishes straight from your phone.

Because there's a QR code for every single seat, the restaurant knows where you're sitting so the food just arrives at your table. You pay for the food directly on your phone, so you never have to get a bill and it just goes away.

The entire process of how do you get your food, how do you order it, how do you pay for it is totally streamlined and integrated into your phone. Afterwards, if you want to split the bill with somebody else, you can just send money from your phone to somebody else's.

There's ways to be able to have groups set up so you just automatically send money to everyone at the same time. It's really, really easy to be able to do all these things.

When you think about the US process, typically you pay by credit card, you have to add your tip, you've got to wait for somebody to bring your menu and order for you. It's taken a lot of the friction out of even just how do you sit down and order a meal.

Gordon:  I imagine it must be very frustrating for a lot of Silicon Valley startups because they can do the backend infrastructure for this sort of thing, but people just don't do it because habits are established. Let's switch to you personally.

What's it been like? This was a big change for you from Cambridge Massachusetts and Red Hat right out of school to moving to Hong Kong.

Bryan:  Yeah, it's been really different. It's been really fun. Hong Kong is a world‑class city with tons of food. Everyone who knows me knows that I like to go out and eat and sample things. Then, Shenzhen is a really fascinating city just with all the hyper growth and technology that's going on there.

Working within a Chinese company has also been different. Working in the US, of course, for example, everyone speaks English. All my communications were in English and emails.

Working at a Chinese company, the vast majority of my emails come across in written Chinese. My Mandarin, it's spoken Mandarin. It's not very good. I've actually become super dependent on machine learning and AI.

Huawei has built out its own internal machine learning trained translation tools. Every day, I'm sitting on those tools translating everything back and forth between Mandarin and English. That's been one change.

Other things, obviously, when I go to the company cafeteria, I'm not eating salad bars anymore every day but trying all sorts of other things like that. Some of the other ways that people work over there, the company cultural values I find a little bit different from the US.

Just one thing that really surprised me, in the US, every tech company talks about being customer‑centric. In the US, when we talk about being customer‑centric, that means we build the best product for our customers, we listen to them, we pay attention to them, we provide good customer support.

At Huawei, their number one priority is also being customer‑centric and being dedicated to customers. But when I started learning about what it means to be customer‑centric at Huawei, we have a very different view on that compared to any place I've seen in the US.

For example, all the stories that Huawei has started telling us when I went to orientation about what it means to be customer‑centric is things like during the Ebola crisis in Africa a few years ago, all the Huawei engineers voluntarily stayed behind and didn't evacuate because they wanted to keep the mobile communication infrastructures up and running in Africa.

Or during the tsunami and subsequent nuclear crisis in Japan a few years ago, Huawei engineers were the very first ones to go back to Japan and Tokyo and other parts where the radiation was really high in order to reset up their infrastructure.

Huawei is able to be successful because they stick by their customers even at personal costs. It's about know how dedicated are you to your customer and how loyal are you to them when they talk about being customer‑centric.

A very, very different way of looking at the world, which I just found really surprising and fascinating.

Gordon:  We're at the Open Source Leadership Summit. Let's talk a little more about open source. What's the open source story in China?

Bryan:  Open source has been booming in China. Last year, for example, a lot of the key conferences by Linux Foundation...They started hosting parallel events over in Beijing, in Shanghai, and other places. Really, really, high attendance.

If you look at a lot of the companies, Huawei was one of the very, very early companies in to open source. Now it's amongst the top contributors to all the major projects, like Kubernetes, and OpenStack, and so on.

It's not just Huawei anymore. We're seeing a lot of other companies that are building their businesses using entirely open source technologies. Many of the Chinese companies are also starting to get involved in the open source community.

I know that when I was at Red Hat and working with you, we often met with a lot of different customers who were asking, "How do we get involved in open source? How do we contribute? How do we strategically adopt open source?"

I'm seeing those exact same conversations happening at many of the companies across China. Many of them are now starting to see open source is a very strategic way to build platforms because they're seeing all the innovations that can be possible and what happens as you start to collaborate together.

Gordon:  Do you see differences in the approach to open source? You mentioned earlier about cloud providers may be doing things a bit differently because they didn't have the 10 years of, "This is how we did things in 2006."

There's a lot of legacy carried forward because of that. When it comes to adopting open source, are you seeing different patterns in China compared to the US, given that it's fair to say, a lot of conservative US companies resisted open source for a long time. Some still are.

Bryan:  I actually think you'll see a higher percentage deployment of open source at many of these companies. Just to use public cloud as an example, the public cloud market, as I mentioned, is relatively nascent. There's many, many different companies also competing in the space, all building public clouds.

Unlike the US market, the vast majority of these public clouds are all built on open source, at the core, using OpenStack, for example. Most of the top clouds in China, whether from Huawei or from Tencent, or from China Mobile, and so on, they're all building on top of OpenStack which is not the case in the US where most of the public clouds preexisted, these open source technologies. They all built their own stuff.

That's just one example where you see, because these platforms came along a little bit later. Then when they saw that, "Hey, somebody's already invented this stuff in open source. Let's just take advantage of that, and then go and build the other things that matter to our customers."

You're starting to see default to open source in many of these places where it wasn't possible, markets where you had a longer term of no incumbency.

Gordon:  Of course, the fact that if you're buying proprietary software, you are probably going to have to buy a lot of it from US companies. I imagine that plays a role in open source as well.

Bryan:  Definitely. One of the great benefits of open source is that it puts you in control of your own destiny. Just like all the typical startups today in the US and elsewhere, open source is very natural for them.

It's very much a startup mentality almost, especially in Shenzhen where there's so many new businesses being formed all the time. There's pretty much a maker's mentality in terms of how do you hack hardware, and how do you hack software?

Open source does play along very, very nicely with a lot of that dynamic in terms of what people are trying to do.

Gordon:  You mentioned language, in your case, earlier on. How are you seeing that affecting...There are a variety of reasons why it's probably harder for someone in China or a company in China to fully interact with a number of the open source communities.

How do you see that playing out?

Bryan:  That can be one of the big challenges. I see both language as well as communication medium as one of the challenges. Obviously, English has become the de facto language everyone uses in open source.

If you come across it in China, it's a little bit uneven in terms of how fluent people are, being able to communicate in English, let alone be able to persuade, or evangelize, or say "This is why this commit should be good" or to take some of the leadership positions. The good thing is that the open source communities have, in general, been welcoming.

The other dynamic is a lot of the communication channels that people typically use in China, WeChat is dominant in terms of the major communication channels.

Then some of the other popular forms, like Twitter or Google, are not even accessible in China. It makes "How do you even connect with each other" a little more difficult at times. There's a couple things that have been happening.

One is that you're starting to see many of the open source foundations, like Linux Foundation here, set up. It's supporting infrastructure in China, a greater Asia‑Pacific, and so on to try to foster a lot of the communities.

One of the effects of that has been now you're starting to see pockets of projects that are initiated by collections of Chinese companies. Then coming into other parts of the world, instead of everything just happening from US, or Europe, or Latin America, or some other part, and then coming back into China.

I think that's good because now it means that contributions, and innovations, and leadership is coming from everywhere. I still think that there's a lot of things to be figured out in terms of how do you best incorporate people from all over the world into a community when the language, and communication mediums, and other things like that are just barriers.

Gordon:  What do you see ahead? You're doing strategy, without giving away any secrets. What are some of the things related to China broadly that you think people should be thinking about in looking out over the next few years? I'm not sure it makes much sense. In this industry, you don't talk about more than a few years.

Bryan:  Huawei, obviously. We're based in China, so we think a lot about China. Huawei is a very interesting company within there in that the vast majority of its business is actually overseas and not in mainland China, plus when I take a look strategically I think across a few different dimensions.

One is from our overall technology portfolio. How do we make it useful for everyone in the world? Obviously, we take advantage of the fact that a different market dynamics exists, whether in China, the UK, or Brazil, or something like that.

The good thing is that a lot of these macro trends around digital transformation, around technology, is like IoT, or machine learning and cloud computing. Those are the same things that are happening everywhere. It's more about, "What is the actual solution deployment that you get into these other areas?"

Based on that, there's a few things that we look at in terms of, "OK, how do we create a good baseline of technologies, or good platforms, that can support all these different areas?" Some of the principles that we think are very useful to build these generic platforms being open is hugely important.

This is why Huawei has invested so much in open source because if we want to go to these different markets and be able to have it adapt, the only way that's possible is to build on an open foundation, and to make it so that people can do what they want with it in their own individual markets.

We also take a look at how do we take the benefits of one market and bring it to the other. A good example is in public cloud. Huawei operates its own public cloud in China, as I mentioned, for the software built on OpenStack on top of the CMCF Stack, and so on.

Then Huawei also sells an OEM, this exact same technology platform to many of our partners around the world. For example, Deutsche Telecom or France Telecom. Telefonica, China Mobile, and many of these large Telcos around the world. All building the same public clouds and operating them in their local markets.

One of the reasons for this is that if you take a look at a lot of the existing public cloud today. The dominant one is in the US, for example, if you just look at AWS, and Microsoft, and Google. First off, Google's not even present in China.

If you look at Amazon and Microsoft, they're much weaker in the China market. They can't even operate their own data centers there.

What we think is the best approach is to say, "Well, let's let the specialists in their own geographies operate their own leading class, public clouds. Let's figure out a way so that everything works together, so that if you want to buy all your cloud capacity through Deutsche Telecom but you happen to be a multi‑national like Volkswagen.

Also, when you need to deploy into data centers in China, you can take advantage of a first‑class operator like Huawei, a first‑class operator in Germany, like DT, versus a cloud provider that happens to be strong in one market but relatively weak or non‑existent in another.

This is our approach to say, "How do we get the best local experience, but on that same technology platform, whether we operate it ourselves, or we resell it, and OEM it with others." It's using that shared platform, but then offering specialized experience to be able to deliver that in a best in class, local customer experience.

Gordon:  As you well know, that's something we see a lot at Red Hat. A lot of people go, "Oh. The public cloud market, that's AWS, Google, and Microsoft. If they're thinking more internationally, “there's a couple of people in China, too, whose names I forget.”

Obviously, we do a lot of business with regional Telcos, regional cloud providers running portable platforms, like OpenShift and, obviously, Red Hat Enterprise Linux

You do have this portable and transferrable experience among clouds in different regions, different countries, what have you.

Bryan:  Absolutely. This is one of the reasons why Red Hat and Huawei have both been partnering as well. Red Hat's open hybrid cloud strategy and portfolio around it, being able to enable customers to use open source technology so they can run their applications in all these different environments.

That's very consistent at Huawei as well as we're looking to enable these platforms to run workloads like Red Hat and others around the world. We very much believe as well that being open and being able to give customers that flexibility and the best in class experience. However they want to use it, that's the most critical thing.

.