William Henry on open source innovation, the role of standards, and consuming software

William Henry is a senior distinguished engineer at Red Hat who has been involved with open source since Slackware. In this podcast, William talks about some of tensions in the open source world, including those between innovation and standardization, which have taken many forms over the years.

Show notes:

• From Pots and Vats to Programs and Apps: How Software Learned to Package Itself (free ebook)

Podcast:

• William Henry [MP3 - 31:49]

Transcript:

-->

Gordon Haff:   I'm very happy today to have with me a co‑author of mine, a frequent collaborator in many things, open source. That is, William Henry, senior distinguished engineer at Red Hat. Why don't we start with some background in your part? How did you get involved with open source?

William Henry:  It was probably two different areas going on at the time. It would have been late '90s and then early 2000 when I was working in the whole distributed computing space and CORBA and JEE.

I was on one side obviously experimenting with Linux. I had downloaded Slackware probably in '97 and was playing with this at home on an old IBM Aptiva.

At the same time at work, I was seeing a lot of open source initiatives in the CORBA space for example and also in JEE or J2EE at the time or JBoss. Then of course there was a lot of other projects like Fuse and Camel that were coming out on the SOA side for things like messaging and Web services.

I was getting involved somewhat with those but not maybe as much hands‑on but more as a user and a person who was advising folks the new upstream community from companies I worked with.

Then of course, I joined Red Hat in 2008 and that was a completely different level of experience with open source, complete explosion. It was like drinking from the firehose. Because I'd been working so far up the stack, on the SOA side you almost assumed that a lot of thing were already done in operating system.

We had lots of cool things like Solaris, IBM AIX, and HP‑UX, those sort of things on the Unix side surely, all the real innovation was kind of done.

Perhaps Linux was trying to catch up, in some ways, to some of the innovations on those platforms, but there wasn't any real innovation going on.

Of course, I discovered very quickly there was massive innovation in terms of real time, then containers come along, and we still see that we're innovating quite a lot of Linux platforms while there's an explosion of technologies on top of that, as well.

Gordon:  That seems to be one of the watershed changes that's happened with open source over the last, maybe 10, 15 years. But, even more so recently is that open source software, Linux, various message buses, and things like that. Really, what their "innovation" was, initially, much lower cost. That was really a disruptive factor of open source.

The big change that we've been seeing over the last 10 years, and I'd argue it's accelerated in the last 5, is open source has become where much of the innovation in the tech industry is happening.

William:  Yeah, I think it's still a combination though. In other words, yes, there's an explosion of innovation, particularly if you look at gravitating around the Linux platform. In many ways, some of the innovation has already been invented before, and we've just come up with different flavors of it. Less expensive flavors of it.

The whole Linux‑based platform, be that everything up front, the Linux kernel to things like containers or virtualization of the cloud, it's almost like a reinvention of technology we were doing in the '60s and '70s, just at a much greater scale. We've reinvented the mainframe.

At the same time, you're right. Some of the innovation is around the scale. It is a massive scale that we can do now because of the cheapness, and because of the availability of infrastructure as a service, where the access to the time sharing of a platform is much broader. It's very simple for me to go to the cloud today, just for a simple demo. I don't have to go out and acquire anything or do anything.

I can get that resource in Sydney or in Ireland on the cloud there. It doesn't really matter. The scale and the amount of technology out there provides a huge, layered platform for innovation on top of that. At the same time, we're still doing things we did 20 or 30 years ago, just cooler in color with video and streaming and everything else.

Gordon:  In a way, I think the meme that everything has been done before is a little bit tired. Of course, there's many echoes and many instantiations of concepts which may go back decades and decades. For instance, to say, "Oh, public Clouds are just like timesharing." Well, they're really not.

William:  Right. We've got the type of multi‑tenancy that you're seeing on clouds today. Almost certainly to the user, the opaqueness of the geographical distribution of those assets is incredible. The types of tools and innovation and availability of software and services on top of that are a lot different.

Gordon:  We just talked about, I'm not sure if it's a tension, but certainly there's these two faces of open source about easy to acquire, low cost, very accessible on the one hand, and this engine for innovation on the other hand.

I'd like to take us into some of the other dual aspects that we see around open source today. We just touched on one of them, which is also relevant to your work with container tools like Podman, Buildah, and so forth.

That is, when do you standardize and when you innovate, and how does standardization and innovation play against each other?

William:  This is a tough one because you can see certain innovations out there thriving because of standards. Yet, you can see other innovations out there dying despite standards. Not so much dying but perhaps not delivering where people thought they would deliver.

Then the de facto standards, for example, you could look at things like container area for a start, where that was driven by an open source, but very non‑standard technology called Docker. Docker then moved with the rest of the community that were developing it into an open source standard OCI.

That has obviously become very successful. You can see other things like Kafka which comes out of the Apache Software Foundation. They came out with essentially another messaging type system which was living in Apache alongside things like Qpid.

Qpid became perhaps less interesting despite the fact that it was based on the AMQP standard. Less interesting from a commercial perspective than something like Kafka which, as we know, exploded.

Just because you build a standard, it needs a huge community behind it. In some ways, a large community can drive the standards. You have projects like Kubernetes which obviously came out of Google and open source.It wasn't like a standard, but companies like Red Hat and others saw how powerful it was going to be. We jumped in on that project. Now Kubernetes has become the de facto standard. Of course, the whole Cloud Native Computing Foundation which is part of the Linux Foundation has essentially grown up out of that.

It's a complex area where you have communities. You have commercial. You have standards. It's about trying to find a perfect storm of bringing those three things together that really drive the popularity or the success of open source project.

Gordon:  It's not universal, and there certainly are areas that are standards first. I think you are highlighting an important way in which standardization has evolved in many cases. You talked about some of your work in the middleware, in the eventing space.

If you look at the first iteration of that SOA 1.0 if you would, that was very much standardized in principle, but it was very heavy weight, very big vendor‑driven type of standardization. Whereas what we're seeing with the OCI container specs, obviously what we're seeing with Kubernetes.

What we're seeing with many many of the projects in cloud‑native space, is some company, some individual is going out and writing software that scratches some edge.

Other people are using that. Other people participating in the community. People are going, "Hey, this works. Let's make it a standard now." There's code‑first approach to standardization.

William:  It's really fascinating because when you look at it the whole...When you talk about SOA 1.0 and you talk about things like CORBA for example, there was a massive consortium, a huge standards effort with CORBA with massive commercial backing, with banks and telecommunication companies etc.

Perhaps it slowed down because of that. It didn't have a clear vision and scale‑level approach to it. Also, you can look at the W3C around standards as well. That was almost dead on arrival. Lots of lesson learned.

I was involved with the WS policy work. You can see again massive collaboration, huge organizations, massive money behind it. Folks like IBM and Microsoft and others, but they never really took off.

What had happened instead was the free market and innovation in the industry to solve problems first. For example, REST‑based approaches won. Standards doesn't always solve the problem. What I tell you what it will do though is what standard will help figure out is whether something will last.

When you look at the lessons learned from things like CORBA or things like W3C, it's very easy to get a cool "hello world" demonstration up and going in the space. When it comes to massive scale and transactions and security and all those other things, you pull in from all these standards bodies and industry experts and all that.

That's where the drag comes in, the lag comes. In many ways, the innovation today benefits from those early standards not because they were successful, [laughs] but in many ways because they were successful for a time and there were these lessons learned from them.

When you look at some of the hugely scalable architectures we have today, they are ‑‑ as you say ‑‑ built on the backs of the knowledge we gained from those quite frankly successful standards from the past.

It's not that they were all closed‑sourced sided because you had things in the CORBA space and other areas there. Obviously, we still got things like JBoss around today in the JEE.

Essentially lots of lesson learned there, but not exactly a lot of open source products that you will regard as massively successful in today's enterprise computing deployment.

Gordon:  OSI [Open Systems Interconnection, in this context] was another one I was involved with back in the day somewhat. The network model, people use it. It's a pretty good model, but the products that came out of it… There was an awful lot of money wasted on that.

William:  We still learned the lesson though. We still talk about L3 or whatever level layer 3 on today when we talk or whatever layer.

We still use that as a standard way of talking about how we're going to communicate in a distributed way and whether a piece of open‑sourced software you used in today's speaking at one of those layers etc. They are not OSI projects as you say.

Gordon:  Let's talk about some of the other trade‑offs or challenges or conflicts that we see in the space. You've mentioned community a number of times, what are some of the challenges you see with communities around factors like standardization, innovation, stability and trading off all of those things?

William:  Again, it's a tough one because I tell you that one of the areas that a lot of community people don't understand fully is the marketability of the technology they are using and how they are bringing it to market.

An example of that that I would use would be on the Qpid side. Here you had open source projects with a huge backing from the many of the banks out there because of AMQP, the standard.

You had an AMQP standard, you had an open‑sourced project called Qpid. When you take that to market, it was a very long sale cycle. There were people who would understand it. The low latency nature, all that coolness, it's open source, super‑fast, scalable, had a fault tolerance and all that stuff built in.

Unless you know how to sell that and sell it at scale, it comes hard to take it to market. You have a salesforce that's selling lots of different products and some of them are easier to sell than others and maybe are bigger price point.

Maybe your Qpid‑based product is less interesting. Something on the other hand like Kafka comes out and not so much geared towards a massive commercial product sale but more as a service, when it comes to thing like cloud or how to build it into a platform.

Suddenly it becomes a hugely more popular way of doing things because the way it's brought to market was different.

Gordon:  Maybe this would serve a good segue to talk about business models and some of the trade‑offs there a bit. We'll hear an argument with some of my colleagues about whether open source is a business model or not.

It's certainly fair to say that whether or not software's is open source or more broadly how software is licensed enables and forecloses certain pass in a business model.

That said, I still find it's useful to separate the open sourceness and the business modelish because while they interact with each other, they are not the same thing. Open source is not a business model by itself.

William:  No, it is not. The other thing I would say is just because your open source project isn't able to be directly marketed doesn't mean that there's not a market for it within something else. For example, Qpid, it's still got a market there as something that's deployed within other larger platforms.

For example, the Proton Project there can be used extensively for a non‑brokered, more distributed messaging pattern. It's very good. What you have to understand is that when you're taking things to market in the open‑source world you're competing with a lot of other different stories. Particularly, you as a community, in terms of how successful this will be commercially, you're very dependent on the people who are taking it to market for you.

A lot of communities will build really wonderful technology. You're sending a guy off to the bazaar. He's taking his rugs, his baskets, or whatever it is to it. How is he presenting those in the bazaar at his stall? How is he showing them?

Is your product sitting down on a back shelf because he looks at it and goes, "I don't know how to sell it. If somebody asks me for it, I'll sell it to them, but really I don't want it up front on my countertop. What I want up on front of my countertop are the things that sell maybe easily or maybe bring me a lot of markup." Whatever it might be.

How you're taking your open source project from the community, and how you expect that to be delivered in the market is super important. Sometimes it may be that you're not selling it direct. You're selling it as part of platform or something else that you're doing.

Messaging becomes an example of that where people really don't want to handle the complexity of setting up and managing complex messaging systems, but they may certainly consume a messaging service because it's easy, and they don't have to worry about it.

Gordon:  We've been seeing things play out in the Hadoop space, for example, about this recently. That seems to be a difficult stand‑alone sale. In a way, this doesn't even have that much to do directly with open source because I can look up other areas of software, like developer tools for example, that have historically been very difficult to sell for the most part.

Another trade‑off that we frequently see is between the speed of innovation ‑‑ rapid change, come out with a new incompatible build every day ‑‑ and stability, particularly for enterprise customers who just want something that works.

It doesn't necessarily need to be the latest and greatest. Fairly or not, open source communities' projects have sometimes gotten a little bit of a bad rap for being, maybe, a little too focused on the change ‑‑ early change, often, rule of development. What are some of your experiences with those trade‑offs?

William:  Obviously, there's a thirst, particularly the bleeding edge curve of that innovation, where people want newer features faster. They want to be able to turn around and consume. We want to do fault tolerance. It needs to be multi‑tenant or it needs to be more secure.

They're hungry for these new features but they also are struggling with how they're going to consume it now. There's two aspects of that too. There's the cloud‑native world or whatever, and your traditional apps, where you want more stability.

You still want more stability but at the same time you want to innovate. You want to be able to consume these newer technologies faster. One of the things I would say that's changed is that, in the past, communities would want people to catch up. The consumer would say, "No. We wanna slow down."

That's why companies like Red Hat were so successful because we were able to provide stability for 10 years. For example, on Linux, when Red Hat opened Red Hat Enterprise Linux. We're able to provide that stability for them.

At the same time you had innovation on the cloud space which has accelerated including DevOps and, as you say, the break it early, fix it and deploy often ‑‑ all that good stuff or whatever. Anyway, what I'm seeing now is a very different trend.

We've gone through three stages. One is the stability side with a, "We want the innovation but we can't really handle this in our infrastructure," to the world of DevOps and CICD where it was more of a, "Yes, we want to consume it and we can consume it. Give it to us faster."

To now, almost a situation where it has become, "Yes, we want to consume it faster, but we don't want to own it anymore. We just want to consume it as a service."

We're almost at this third phase of...It's like this comedian that I heard who was talking about a joke...his name is Gary Coleman, I believe. He's talking about how this generation will say "I want all of my music on my phone now," and they're going "What do you mean all of the music on your phone?"

He says, "Sorry, what I meant was all the music on my phone, right now, this moment." "How much are you willing to pay for it?" "Pay for it? Nothing."

That's the joke. My final offer is nothing. If people want everything now to be consumable as a service for free. It's a struggle on the...It's great in some ways for the upstream communities, but it also means they have to be innovating faster. It's interesting for customers like Red Hat as they have to try to pivot to perhaps more of a services‑based model.

Software as a service, providing messaging as a service, for example, or container build as a service, or whatever that might be. It obviously fits nicely into some of the cloud vendors, but also puts a challenge on the consumers as to "How do we want to standardize all this?" If we want to innovate fast and consume these things as a service what are they tradeoffs?

Do we expect a standardized consumption model across all of this or do we expect that we're going to have different pipelines into these different deployment platforms.

Gordon:  That's why the critical tradeoffs you have here is we went into near the end of our book that I'll put a free download link in the show notes. You have this ease of consumption in public clouds and software as a service, but in order to get that ease of consumption you are in many ways locking yourself into that single provider.

One of the great both opportunities and challenges for open source broadly is how do you get those attractive experiences to customers in a way that has a sustainable business model while allowing the end users the ability to move their work loads, move their data, move their intellectual property to wherever they want to run it.

William:  I think that's going to be a challenge for a while. That's where the industry is in a waiting mode a little bit. Obviously they're not waiting for it in terms of business. They're expecting someone to solve that for them.

Whether that's their open source vendor or someone like Red Hat or whether it's the cloud provider themselves, but as they rushed to the cloud and as they're embracing open‑hybrid cloud and multi‑cloud approaches. They are hitting that whole, just how portable is this container stuff, for example? How easy is it for me to move my instances across these various clouds and to bring them back on‑prem?

We've done a very good job, people like you and me, of talking about the benefits of open hyper‑cloud and multi‑cloud but the industry itself is still trying to catch up with that model. Things like OpenShift can provide a single consolidated platform across public clouds and private clouds, etc. The question is whether in this approach how will these developers and operators and essentially CIO offices respond to that?

They're trying to look at a consumption model. Other things like OpenShift Dedicated and other things like Azure OpenShift or Red Hat OpenShift also help with that model. I still think it's a, "Hurry up with my business." "Keep going everybody." "Go fast." "Do things quickly." But really, are you all making the right decision here?

Because there seems to be, "I don't want to own all this stuff anymore, I want somebody else to own it." Are we doing it right? I think that a lot of the people I've been talking to are struggling with that decision.

Gordon:  Well, there's really been this fundamental shift with Cloud, with software as a service. To provide a different way of consuming software and for that matter, really consuming services to your music example earlier. That changed environment. I'm not sure...In fact, I am sure it really hasn't been internalized by everyone.

We run software differently than we used to. Even when we do it on premise and a lot of those implications are still being thrashed out there.

William, anything that you'd like to add before we close? There's a couple of topics I would love to dive down to but I think those are entire podcast by themselves. So let's hit those another day.

William:  I would just say that what's exciting is that open source is running strong. Never before have we seen such validation of open sources we do today. In terms of the innovation, there's enterprise‑type projects for almost everything and they continue to improve. You're seeing things like Federation getting added to things like Kubernetes.

We're seeing some good collaboration around these open source projects with companies. With a view to moving more and more of these community‑led projects into a standards approach. All that has been wonderful.

On the other hand, we have this consumption model that there's some hesitation in. Because despite the appetite for all of this innovation there is still struggles, as there always has been with open source world of how people are going to consume this. How they want to consume it and who do they want to own the problem.

Before it was, "Can you own this from an upstream management perspective?" Now it's, "Yes, can I have it on‑prem but can you also own it off‑prem in the cloud so I don't have to do anything up there to play it because I really don't want to be in this mess of complicated IT business."

On the other hand, you’ve got small practitioners and consumers out there that are just quite happy to roll up their sleeves, deploy a bunch of servers and build all the cool stuff on top of it. It's really an exciting place to be. There's all sorts of different people in the marketplace consuming this. There's some super smart people in the communities upstream.

Of course, there's some excellent standards been driven out of these projects that everyone's going to benefit from and continue to make money in the market on.

Hyperledger's Brian Behlendorf on starting Apache, foundations, and blockchain

Brian Behlendorf has a long history in open source going back to his co-founding of Apache. Today, he's the executive director of the Hyperledger Foundation. In this podcast, Brian takes us through some of his motivations in the early days of Apache, the tension between pragmatism and idealism in free and open source software, and why he's excited about distributed ledgers.

[Photo: Used with permission of the Linux Foundation.]

Show notes:

• Hyperledger Foundation
• Wikipedia article for Brian

Podcast:

• Brian Behlendorf [MP3 - 28:42]

Transcript:

Gordon Haff:   I'm particularly excited to have with me today Brian Behlendorf, who is the executive director of the Hyperledger Foundation, but also has a rather illustrious history in open source.

Many people in this call probably know who you are, Brian, but could you give us the Reader's Digest version, assuming talking Reader's Digest isn’t a complete anachronism at this point.

Brian Behlendorf:  [laughs] I'm old enough to get the reference, but I'm sure many people won't,  so Brian Behlendorf, as you mentioned, Executive Director of Hyperledger which is actually part of the Linux Foundation.

The Linux Foundation has its own illustrious history of...I've only joined it about three years ago to do this project, but obviously it's been around since 2000 trying to serve the needs of the Linux ecosystem, both the developers and the companies around it.

Then starting about six, seven years ago expanding into a whole bunch of related technology domains, cloud computing, software‑defined networking and with Hyperledger blockchain technology.

My background, I have done a bunch of different things, I'm perhaps most known for being one of the cofounders of the Apache Project, which became the Apache Software Foundation, then serving as its president for the first couple years.

That was always a night gig. It was never a paid gig. As it's true for almost everybody involved with Apache. It's an entirely volunteer gig.

My day job has varied from starting one of the first website design companies called Organic. Starting a company called CollabNet, which you could think of as maybe GitHub, but perhaps two or three generations too early. [laughs]

We did kick out the Subversion project and get that started up and brought that over into Apache. I've also served as CTO for the World Economic Forum for a couple years. That was based in Geneva. That was a very fun and very different kind of organization.

I worked in the White House for about a year, and then the Department of Health and Human Services for a year. Mostly been either starting companies or working for open source foundations in one form or another. Usually volunteer.

The Linux Foundation is the first time I've actually worked for a paycheck for an open source organization, and it's really fun, so lots of different things.

I guess I put the first ad banner online and I've been apologizing ever since. Please don't hold it against me. I've had a lot of fun in my career.

Gordon:  Brian, I tend to let the topics in these podcasts wander where they want to go. I do try and stay somewhat centered in this new podcast as it intersects the innovation and open source. To what degree did inventing new and better things influence you early on with Apache versus user freedoms, having an alternative to big vendors, and that kind of thing?

Brian:  The Internet culture in the 1990s was definitely one of...First off, most people didn't know about it, didn't know what it was going to be. I think there's, among people who are online, a real conviction that this is bigger than AOL, this is bigger than CompuServe, this is bigger than telephones even. This is something that will be the substrate for a society in the long term.

There may have been different opinions on how quickly that would arrive, but probably not that much difference of opinion on if it would arrive. There's this real deep conviction that we're on to something and it was going to be big, but a lot of concern at the same time because the desktop computing world at that time was maybe 96 percent Microsoft Windows.

We might have had a more beneficent view of technology companies at that moment. We still thought of them as leading the fight for individual empowerment, the whole Apple "Think Different" campaign. We kind of saw tech companies as the good guys, so to speak.

I think there was still a concern that, as the web grew, it would lose its character and its soul as this kind of funky domain, very flat space, supportive of freedoms of speech, freedoms of thought, freedoms of association that were completely novel to us at the time, but now we take for granted or even we have found weaponized against us. [laughs]

What I think drove the founders of Apache early on was two things. One, a very pragmatic base, I was working at Wired magazine building this web company, Organic, and we simply needed a better web server, and iteratively improving upon the NCSA web server was just easier and certainly a lot cheaper than buying Netscape's commercial web server or thinking about IIS or any of the other commercial options at the time.

There is this pragmatic thing which was, this is just simpler. This feels easier. It's nice to have other people out there who can review my code and work together with, but none of us really wanted to be full‑time web server developers. That seemed boring in a way. We were having much more fun building cool websites.

The second was this idealistic notion that tapped into that zeitgeist in the '90s but it was a bit of, this is a printing press. We can help people publish their own blogs, help people publish their own websites and get as much content liberated as possible, and digitized as possible, by getting this out there.

That was kind of the web movement, but, in particular, we felt it would be important to make sure that the printing presses remained in the hands of the people. That everybody could use this stuff for free, and that it didn't become a single‑vendor web, the way that it felt like the desktop had kind of collapsed to a single vendor.

That dual mentality, the pragmatic and the idealistic, I think was a driver for Apache. I don't know that there was a same kind of driver for the Free Software Foundation. That was more Stallman's moral, the indignity of having his code proprietarized in the mid '80s and going, "Never again."

I'm certainly sympathetic to the moral worldview, that when you have software inside of every car, every doorknob, every thermostat, that pretty soon access to source code, and the ability to modify it, starts verge on a human right.

I totally get that worldview. I think there were more pragmatic and more operational kind of benefits that were key to Apache growing, and I'd say arguably the rest of the open source world. I carry that forward to today.

With Hyperledger, one thing that pulled me in and got me excited, was this notion that there are some really important problems we can solve with distributed systems, with distributed Ledgers, and smart contract techniques.

It wasn't programmable money, it wasn't regulatory arbitrage. It wasn't many of the things people associate with crypto currencies that was the driver here. It was the sense that the digitalization of society had led to a future that looked a lot more like big central systems.

Everyone loves to make fun of them but Uber and PayPal and that sort of thing ‑‑ not that they're the boogeyman, necessarily ‑‑ but like that architecture of all of us going through central services to connect with each other. It was a very un‑Internet kind of worldview, but it seemed to be the trend line we were on.

Blockchain technology seemed urgent to get involved in that lined up with these idealistic and pragmatic impulses that I've had, and I think other people in open source have had. That's kind of why I've dedicated the last three years of my life to it.

Gordon:  I will talk more about Blockchain in just a second, but right before that I'd like to talk a little bit about foundations. Foundations are a big thing these days in the open‑source world. Some will argue maybe they're too big a thing in the open source world.

What was your thinking with the Apache Software Foundation initially, in terms of what your priorities were, and how you've seen the foundation landscape evolve over the years?

Brian:  In 1998, Apache had been around for three years. In those three years, it had grown by ‑‑ Netcraft, the company that does this survey of the web once a month ‑‑ Netcraft's measure to be something like 70 percent market share.

Which seems weird to use air quotes around market share, because no one was paying any money for Apache, but in terms of installed base, that's 70 percent of the web is running on top of Apache HTTPD, which was pretty awesome.

It was still being built by a group of people whose only connection to each other was that they were all on an email mailing list. All had commit to a CVS repository. All had shell on a Unix box that I maintained off of Wired's Internet connection, [laughs] and otherwise had no formalism between us.

In a way that was liberating, in a way, we were like, "Yeah, you know, we don't need overhead, we don't need stuffy bureaucrats." We do want process, and we had developed kind of a sense of how to work together, because we want the efficiency that comes from not having to rehash the same arguments over and over.

Having lack of clarity about who does what. It was fundamentally about transparency and an open door to anybody to get involved in. I think there was a healthy degree of skepticism that a foundation, or any sort of corporate structure, would have the same advantage.

Certainly, we didn't want to incorporate as a for‑profit company, because then you get into thorny questions of, "How much do you pay people? How much equity share do they have?" all that kind of stuff when we all had our own agendas and startup businesses anyways. It was when we started seeing more and more people using it, asking harder and harder questions.

We'd always defer to other people to provide support. There were these thorny questions around, "What happens if somebody who owned a patent decided to file a patent lawsuit against the developers of Apache and wanted something as simple and modest as a dollar per copy?"

If they won, and given patent laws, they certainly could win, they'd seek those tens, or hundreds of millions of dollars from the Apache developers. For that crime of giving away free software, we could lose our homes. There was no corporate shield to protect the activities of the developers involved in Apache.

If somebody said, "I've got a patent claim against something, and you have to remove it." The laws are the laws. We might fight it if we felt it was a weak patent. If it wasn't a weak patent then what else can you do?

Without any sort of corporate shield around our activities, we all stood the risk of losing our homes or losing financial cushions or other things that just would have sucked.

That was one motivation creating it. The second was, the project had grown beyond being just about the web server. Pretty early on, there is a module to do Perl, a module to do Java, Tomcat which was a Jakarta kind of...We're coming in these where whole new Java code bases.

Coming from Sun and other participants asking to get involved in a way that causes us to ask, "What are we really doing that's scalable?" or, "Have we tapped into something here?" or, "Beyond the individual heroics of the people involved, is there something repeatable that's worth trying to take to more and more software projects?"

The answer was kind of yes. I didn't think we were too full of ourselves to say we had actually found something that was a happy medium between the idealism of the free software movement, and the pragmatism of getting code built and then embedded inside of large companies' projects.

In fact, we didn't even mind the idea of somebody like a Microsoft or an IBM or a Sun ingesting our code and putting it into their commercial products, as long as they didn't abuse our brand by calling it Apache plus plus or Apache prime or anything like that, as long as they didn't try to shuffle their support request queue just down upon us.

As long as they followed the license we were actually enthusiastic about the idea of those vendors coming in on the presumption that it would mean additional development resources, as well. That it would help the idealistic side.

If you could really get not just the rebels and the indie operators, but the actual establishment to use open‑source code then maybe you get faster to a world where everybody's got printing presses, everybody has that freedoms that we all consider essential.

When thinking about this, we realized that having some sort of corporate structure around us, that was nonprofit in nature… That was benign, beneficent, universally recognized as a way to be protective, rather than exploitative would be the right thing.

That's where started forming as a 501(c)(3) charity made sense, and forming it specifically as something that was very much a membership‑based organization. That's not the only way to do it. There are other approaches.

The Linux Foundation for one is more of an industrial consortium than a membership‑based charity. Mozilla also is a charity, the Mozilla Foundation, but it also does most of its operations through a for‑profit wholly‑owned subsidiary called the Mozilla Corporation.

There's all these different models out there. It's been great to see those grow. In general, if you're doing anything meaningful in open‑source software your activities should be parked somewhere where there is a protective structure around it that helps answer the questions and the needs of the broader‑user community.

I'm pretty happy with that approach, also happy at the same time to see quite a few foundations out there and new ones showing up overtime.

Gordon:  I do still want to get to blockchain and open source, but as a way of getting there as we fast‑forward today. The Hyperledger Foundation, you've mentioned it, I'm obviously very familiar with it just having been in Tokyo with you. Can you tell our listeners what it is and what its goals are? Because I think there is sometimes some confusion there.

Brian:  Three years ago, I jumped on this project. It had been announced actually about six months earlier, like December 2015. The first code drop was February 2016. I joined it in May of 2016.

Hyperledger was announced at a time when the Ethereum community was just getting launched as well, when Bitcoin was just before its big run‑up in price, when there was a lot of excitement in the blockchain and cryptocurrency space.

The emergence of a set of use cases beyond programmable money that can jump across borders easily. That really started to speak to some things that were much harder to otherwise do. I think the one that pulled me in was land titles and emerging markets.

Where a distributed database that was not just “Here is a master MySQL node and slaves that hang off of it,” was not just a multi multi-write kind of system, but one that actually supported consensus, one that actually had the network enforcing rules about valid transactions versus invalid transactions. One that was programmable, with smart contracts on top.

This started to make sense to me, and was something that was appealing to me in a way that financial instruments and proof-of-work was not. Hyperledger was announced by a set of large companies, along with the Linux Foundation to try to research this space further, and try to figure out the enterprise applications of these technologies.

What's possible and start, let's start coming up with code that would meet those needs. Let's think about, what are the different architectures required. It was bootstrapped with a couple of different pieces of code that came, one piece internally that had been developed by IBM and other that have been internally developed at Intel.

When I came in, I said, "We should try to decide do we want to be about a single architecture?" kind of in the way that the Linux kernel is about a single architecture. "Do we want to be about a basically a portfolio approach of different architectures, different approaches, to threading this needle, to solving these use cases, and let the market decide?"

Over time, if we have multiple winning solutions, we just kind of weaved them together in some The community came back and said, "We want the latter." From that point forward, the mission of Hyperledger has been, be a home for a portfolio of technologies of software that implements distributed ledger and smart contract functionality.

Kind of like Apache we have an open door to new projects coming in. Have a more of a thematic focus on this domain. Put more intentional effort into weaving these solutions together over the course of time. That's what Hyperledger is.

Sometimes it can seem confusing, because we have right now 14 different technology initiatives, some of them very mature, like Hyperledger Fabric. Some of them still emerging and starting to use in production environments like Sawtooth and Iroha and Indy and Burrow.

Some of them just very supportive as tooling, like Explorer and Composer, and some of them still very young. It's the portfolio overall that is the Hyperledger community that's the Hyperledger code‑base.

Actually innovative open‑source community has this spectrum of different technologies under their wing. It's, I think, ultimately, the right approach.

Our goal, long‑term, is if anyone is building distributed ledger systems, they're probably using one or more, or perhaps all [laughs] of the technologies coming up from Hyperledger.

Gordon:  I was at the event run by "The Economist" magazine a little while back. At that event's keynote, there was this distinction drawn by invention in the sense of new technology, na ew open‑source project, for example.

Innovation is something broader that can involve things like collaboration, new practices, new types of ecosystems, and so forth. My observation would be that blockchain, of course, does require technology but also requires a lot of that latter type of innovation.

Brian:  Open‑source software has shown that you can't really separate the code from the people behind it and you can't really separate the code from the zeitgeist of its movement. Today, whether we trust using Linux, Apache or any of these pieces, sometimes it just comes from pure market share. If everyone else is using it, then it can't suck too badly.

For that first wave of users and the early adopters to later adopters, knowing that there is individuals and organizations committed to a body of code is pretty important to deciding whether to use it or not. A software product isn't just some fixed point of flag‑in‑the‑ground that says, "Here's where we are. Get used to it." Software is more like a stream.

You might decide to use a version of software based not just on what it does today, but on its likelihood of meeting your needs in the future. You'd probably do.

When people look at using code, they want to see that there is this active community around it that is making regular software releases, that is pushing forward on something ambitious, but also answering the very pragmatic and prosaic needs of its end‑users, that has this this vector to it, that has this this momentum.

There are lots of companies out there that are today actually building their own products and services on top of Hyperledger Fabric and other pieces of Hyperledger. Showing people that there is this living, breathing core to these projects, this fountain of functionality and bug fixes, and how they can plug into that, is essential to getting that broader adoption.

We think it's equally important to be innovative in terms of new features as it is to actually be showing here's the community processes, and being an open‑source project actually lends to itself to better quality code and to real people's needs being met and the entire market place actually being effectively deserved.

Gordon:  Of course, if we talk about blockchains specifically, it is also interesting that these permissioned distributed ledger systems need these new ecosystems of companies cooperating and working together and giving up some level of ownership really, and there's lot of good lessons from open source generally in why you have to do that and how you can benefit from doing that.

Brian:  That's right. Certainly one of these blockchains that works, that people are setting up, for them to actually have any point to using blockchain technology wants them to be decentralized at some point. They want them to not just be based at a single vendor's cloud to maybe want to actually have nodes that are on different clouds.

Probably also nodes that are being run by different technology partners or even in some cases nodes being run by end‑user organizations themselves, because otherwise, why not just use a central database run by a single vendor or a single technology partner.

We've been trying to do a lot to work with the cloud community. Right now, there are pretty much every major cloud provider offers Fabric as a managed service which was a nice little milestone to hit this past year.

In doing that, it helped us establish Fabric as standard in the space, but it's also important that when you bootstrap a Fabric network that on day one, you're able to add nodes from other cloud providers or other places.

We are running a certification process for cloud providers this year that will help to guarantee to end users that degree of decentralization, that degree of flexibility around the deployment, because we think that's important to get out there.

Really, this is just the network services equivalent of the same degree of transposability and flexibility and anti‑vendor lock‑in that people expect when they use open source solutions, as well. It requires the same degree of thinking amongst the technology vendors of how do I reassure my customers that I'm not going to trap them in something vendor specific.

Yet, I will still be able to sell them some additional value that keeps them as a customer. Something that Red Hat has been extremely good at in the last 20 years. That Red Hat has been in existence at 25 years at this point and I think they are certainly a big part of what Red Hat will be bringing to IBM. It's something that every IT vendor, every enterprise software vendor, has to get really good at in this era.

Gordon:  One last topic I'd like to touch on. We look at blockchain generally, essentially everything is open source. We're seeing this in other places, too. Cloud Native Computing Foundation, also under the Linux Foundation, pretty much has all of the innovation happening in Cloud Native Computing, at least all the innovation outside of the big cloud providers, but then they're using the software internally.

What do you think the characteristics of these new ecosystems are, these new combinations of technologies, and projects, and companies, that are so defined by open source. There's not just an open‑source alternative to proprietary software, but really everything happening in those spaces being open source. Why has this happened?

Brian:  I first want us to be cognizant that it might feel like open source has won. [laughs] It might feel like Linux has won. That Apache has won. Linux is still not the dominant operative system out there.

It may be by device. If you count lots of IoT devices running embedded Linux, but the mobile phone market and the laptop computer market, and others that are still very rife with proprietary software, which is fine.

You might take the moral point of view like Stallman does, that it is a problem. Or you can take the pragmatic point of view, which says as long as we have critical mass, then an open‑source solution is likely to emerge and is likely to provide benefits.

It's been very reassuring to see that argument playing out, not just in operating systems and web servers and databases, but now playing out at the level of...or cloud containerizations, which is where Kubernetes and the Cloud Native Computer Foundation has really set its mark.

Increasingly in machine learning, in AI tools, increasingly in fields as conservative as automotive, or Telco, open‑source options have started to become at the very least industry competitive in the same way Linux is industry competitive.

In some cases just as dominant as say Kubernetes is now with cloud containers. That doesn't mean that Tesla is going to wake up one day and open source all of their automated driving software, or their end‑user interfaces. Nor do I think that's necessarily a goal for anyone, except Tesla car hackers, and the Tesla user community I think would love that.

If Tesla were to use more the Automotive Grade Linux as a project at the Linux Foundation for their software for underlying communications bus with third‑party part suppliers, more of the nav system and voice control stuff that AGL is emerging with, that would certainly complement what I'm sure is a ton of other open‑source software that Tesla is already using inside their vehicles.

Maybe make it cheaper for Tesla, maybe as a result, make it so Tesla cars are less expensive, and other electric vehicles are less expensive. At the end of the day I think this is about companies deciding there are things that are simply table stakes.

Things that we all need to do to be able to be in this industry together. It's much more efficient for us to work on those common bits of plumbing, so that we can spend more money, or balance more of our investment at the levels above that, and creating stuff that is truly feature full for end users and differentiated.

I think it's an entirely rationale, non‑idealistic business argument for why we're seeing more and more companies, even the ones we traditionally associated with very proprietary business models, be it Microsoft, be it Uber, be it Facebook, actually recognizing open‑source is strategically interesting to them.

That feels like this continuation of the same thinking on our minds 20 years ago at the Apache Software Foundation, that hey if we just involve some of these parties in our projects and kept to our core principles of how to build software, of how our licenses work, how our dev processes work publicly.

If we made them play by our rules, we may still end up in a much better place and move further faster. I think that's been the story the last 20 years.