Podcast: Red Hat's Matt Hicks talks multi-tenancy in PaaS

Efficient and secure multi-tenancy is one of the big operational challenges in Platform-as-a-Service (PaaS) environments. Principal Architect Matt Hicks describes some of the key tools used by Red Hat to operate its OpenShift PaaS including SELinux. Matt covers:

• What multi-tenancy is
• Why virtual machines by themselves aren't sufficient
• The important benefits that SELinux can deliver
• Best practices for PaaS operations

Listen to MP3 (0:12:03)
Listen to OGG (0:12:03)

[TRANSCRIPT]

Gordon Haff:  You're listening to the Cloudy Chat Podcast with Gordon Haff. Hello, everyone. This is Gordon Haff, cloud evangelist with Red Hat. I'm sitting here with Principal Architect Matt Hicks on our OpenShift platform. Welcome, Matt.

Matt Hicks:  Thanks, Gordon.

Gordon:  So, Matt, we're going to talk about multi‑tenancy today. I suppose, if we're going to talk about multi‑tenancy, it would be good to start talking about, what is multi‑tenancy? What do we mean by it?

Matt:  Multi‑tenancy, it's a tough term because it's fairly abstract. I think, for this conversation, when we talk about multi‑tenancy, it's good to frame it. My definition would be being able to run multiple workloads on the same instance of an operating system. That operating system might be a virtual instance, it might be a bare‑metal instance, but multi‑tenancy being that you can run these workloads, they're segmented from each other, they're secure, they can't access each other's data, they can't access the other processes, and they each have somewhat of a feeling of they own the entire machine.

Gordon:  Let's drill down on that security aspect a little bit, because that's been getting quite a bit of attention recently. For that matter, Larry Ellison, Oracle, very recently essentially made the statement that the only real way to isolate workloads was by using virtual machines.

Matt:  Yeah, I saw that. I think traditional logic is we know operating‑system segmentation really well. Virtual machines are an important layer. They provide a great means of essentially separate operating systems, since sysadmins know how to segment them. In one aspect, it's somewhat of a true statement, because VMs are great at segmentation. 

The challenge with VMs is, especially in the PaaS space, our density requirements, the amount of stuff that you have to run, is extremely high, and the cost pressure to get your costs low is very intense as well. A virtual machine carries a lot of operational costs for doing that segmentation. You have sysadmins that are putting up firewall rules and putting them in separate networks, and they have to be patched and updated. If you run a workload per VM, it's very secure, it's very well segmented, but it'll probably be very expensive in a PaaS model.

When we look at multi‑tenancy, one of the things that worries me about multi‑tenancy is the people that just run traditional, Unix‑style segmentation. They take a VM, they run a bunch of processes on it, and then they basically pray that permissions and everything are set right and there is security between them. That's what we tend to call discretionary access control; you'll see the acronym, DAC. 

Discretionary access control requires that you're essentially perfect. You have all the permissions right. You have all the users properly segmented. The machine is always patched. There are no backdoors for somebody to get from one app to the other.

I think that's very risky. We see a lot of that in the market. That's what people are doing for multi‑tenancy. I think that's a security problem just waiting to happen.

Luckily there's a very industry‑standard way of solving this. That's moving from discretionary access control to mandatory access control with SELinux. The power of doing that, it's like moving from a blacklist model, where you have to say all the things that aren't allowed. SELinux moves stuff to more of a whitelist model, where you list the things that are allowed on those machines, and it brings with it a tremendous amount of security in a multi‑tenant space.

In PaaS, we know what applications are doing. It's a very effective thing for us to be able to list the actions that they should take and then block everything else. I think, with SELinux, there's a ton of security and segmentation ability with normal multi‑tenancy. You can get the best of both worlds there.

Gordon:  Organizations like the National Security Agency have been involved in the development of SELinux, so some pretty high‑security people have had a big hand in this.

Matt:  Yeah, absolutely. It's becoming best practice across the board. Even if you're using virtualization, you want your hypervisors and those things controlled by SELinux because it is that good at helping to avoid exploits. Combining that with the power of being able to segment Unix processes, it's a great combination. You get the density benefits of avoiding VM sprawl. You have a smaller list of VMs that you have to carry that operational cost of updating and maintaining them on, and you can carry a wide variety of workloads within those VMs and get a tremendous amount of segmentation between them just with SELinux. It's not new, like new stuff that's having to be built. It's really using the capabilities that's in the Linux operating system.

Gordon:  Matt, I want to ask a question very specifically related to platform‑as‑a‑service, since that's what you're involved in the operations of. Nobody should take this to be a statement about all of enterprise IT or all of cloud, but what are the best practices for isolation that you're seeing in the platform‑as‑a‑service space?

Matt:  In the platform‑as‑a‑service space, we're really seeing multi‑tenancy as sort of an evolving standard in that space. The way it's achieved is very different, but the major players, from Google to Heroku to VMware's CloudFoundry, are all using process segmentation, to one degree or another, to achieve the density that's required in PaaS. I think what we'll see going forward is, when you're in the PaaS space, the demands of being able to segment based on multi‑tenancy are going to be the standard. I think that the techniques right now are different across the board. Some people fork the frameworks themselves to take out the insecure things. Some people are just using technologies like LXC with nothing else. Our view is we use basically every tool in the toolkit plus SELinux to be able to have the most secure option. I think that will still evolve a little bit, but I think it's pretty safe to say that multi‑tenancy in this space is probably here to stay.

Gordon:  Because in the PaaS space, you really aren't thinking about the operating system as a user or as a developer. Unlike the infrastructure‑as‑a‑service space, for example, the virtual machine isn't an obvious construct that you care about.

Matt:  Right. In the PaaS space users interact with components of the operating system, but it's pretty well accepted that you don't have control of the full machine. You might need to get access to ports, but you don't have every port on the system. You might need to get access to HTTP routing, but you don't own the actual top‑level Apache instance. I think that's been pretty well established in the market. That benefit of limiting the use case lets us make multi‑tenancy much more powerful. If we didn't have any limits, we'd have to give each user their own virtual machine because they would expect to be able to control everything on it.

Gordon:  At that point, we're talking infrastructure‑as‑a‑service.

Matt:  Absolutely.

Gordon:  You mentioned LXC. What is that?

Matt:  LXC is actually a project that is focused on segmentation, to some extent, and workload management. It's a combination of a lot of different technologies that are in Linux. LXC is the name of the project, and it uses technologies like Linux control groups to help segment the processes themselves, put them in different groups, so users don't see each other's processes. It uses things like kernel namespaces and technology bind mounting, to be able to make parts of the file system appear like you own them‑‑for example, giving each user, they see their own temp directory instead of seeing a big shared one. LXC has been this conglomerate of segmentation technologies. The one challenge with LXC that I don't think a lot of people realize today is that it does not include the SELinux layer. In my presentations a lot of times, I compare it to Japanese walls security, where it's very nice. It's a nice privacy segmentation [but not security isolation].

Gordon:  One of the things I find interesting about this discussion is that, historically on Unix systems, there was a wide range of isolation techniques and, essentially, this idea of this trade‑off between physical separation on the one hand and, basically, Unix process control on the other hand, with a whole range of continuum in between. By the way, virtual machines were somewhere in the middle of that continuum. They were not historically the be‑all to end‑all for maximum isolation. Then it seemed that people were going down this road of "everything's going to be VMs." Now, with platform‑as‑a‑service, with PaaS, and with other new types of operational models, we seem to be coming back to this realization that, different horses for different courses. There is no one‑size‑fits‑all.

Matt:  I agree completely. I think virtual machine has a great role in being able to provide segmentation. But just like you said, all of the traditional hosting techniques that were used 20 years ago to segment stuff are still being used by us today, plus this newer generation of tooling, like Linux control groups and SELinux and kernel namespaces which help bring that continuum a little bit closer to, "We do traditional process segmentation," but we're able to give users a lot more control than they would typically get. They don't have full control of the machine, like they would in a VM, but it helps us strike the balance a little bit better and let users have a lot of ability even though they're in this sandbox on the machine.

Gordon:  Ultimately, any of this is about striking a balance. If you want perfect isolation, you run your workload on a single physical server locked in a vault that's disconnected from the network. The problem is that's not going to be very useful under most circumstances.

Matt:  Absolutely.

Gordon:  Anything else you'd like to add, Matt?

Matt:  I think it's an exciting space to watch. One of the things I love about PaaS is the demands of things like density are really driving this resurgence in tools that have been around, in some cases, for a couple decades. I think it's an exciting space to see the combination of those tools with newer technologies being brought together. It makes that spectrum a lot more powerful, whether physical hardware is what you need for your use case, or whether you can do it with purely virtual machines, or whether you have the need to start packing density in controlled use cases and go more down the LXC and SELinux‑type model. It's great. I have more fun with Linux these days than in a long time.

Gordon:  You're talking about something we see a lot of in this industry. The basic concepts have been around forever, but they get re‑imagined, and with new technologies they can be put to new uses.

Matt:  Absolutely.

Gordon:  Great, Matt. Thank you.

Matt:  Thanks a lot, Gordon.

Gordon:  Bye‑bye.