DOP 46: Making Containers Great Again - A Conversation With Phil Estes

Transcript

Darin Pope 0:00
This is episode number 46 of DevOps Paradox with Darin Pope and Viktor Farcic. I am Darin

Viktor Farcic 0:06
and I am Viktor.

Darin Pope 0:07
And today we have a guest. Viktor, who is our guest?

Viktor Farcic 0:12
Our guest is Phil. That's a mysterious person that I happen to meet in random places of the world in random moments in time. That's the definition. So we know each other from being Docker Captains early on I think, and as I said before, then we meet each other at random places. And that's about it. He can say more about himself, I guess.

Darin Pope 0:39
like his last name, too.

Viktor Farcic 0:40
Ehh, yes, Estes. Estes.

Phil Estes 0:45
Yeah. So yeah, Viktor and I have bumped into each other all around the world. We met each other originally in the Docker Captain's program, I guess, officially. And yeah, I've worked in the open source container world for five or six years now, on behalf of IBM, and sit, I guess officially in the CTO Office of our cloud platform. And so the connection is the work I do in containers upstream, is used in our, you know, Kubernetes and manage container offerings in our cloud. So that's kind of my connection to opensource and to containers and cloud native for IBM.

Viktor Farcic 1:28
and you'll be nominating Chair of OCI board right this year?

Phil Estes 1:34
Yeah. So, the open container initiative, OCI, has a technical oversight board that's actually an elected position. I've been a member for a couple years and then this year, just a few weeks ago, I took over the chair position from Michael Crosby, who, who we all know and is taking a little break. Just had his first child. So yeah, OCI, you know, created a runtime spec and an image spec, you know, couple years ago and I think a lot of people have kind of maybe forgotten it existed. But there's, there's some new activity there that's, I think exciting in the sense of, you know, people are still wanting to standardize, you know, what it means to use containers, no matter if they're using Docker or something else to create and run containers. So yeah, that's, that's kind of the newest thing going on in my world.

Darin Pope 2:36
So basically, now you're a politician. And you're, and you're telling us that you're going to be able to make containers great again.

Phil Estes 2:44
Yeah, something like that.

Viktor Farcic 2:48
So what's, what's been going on with containers? I mean, is that are we talking about standardizing new features or we're talking about standardizing something else?

Phil Estes 3:02
Yeah, yeah, good question. So, you know, as I mentioned, the runtime spec, you know, kind of everyone has agreed that runC is kind of the de facto standard for how containers get run no matter what, you know, runtime you're using at a higher level, whether it's Docker, some Kubernetes abstraction that you don't even know what's, you know, runtime is running or CRI-O, from Red Hat and the OpenShift side of things or containerd, which, you know, is the project I'm most active in these days and a maintainer of. So, yeah, that that's kind of all decided, and I would say there's, there's really no, the container wars are over from a runtime spec standpoint. So the work that's happening now, one is there was a lot of kind of foot dragging in the early days of the OCI whether we really wanted to standardize the registry interaction like push, pull, you know, all the things you do with a container registry. You know, with Docker being the de facto standard, there are some other ideas around distributing images, you know, takeover. So that discussion ended last year when the OCI accepted the distribution spec, which again starts with the Docker API, the Docker registry API, and is being finalized. So we hope in 2020, there will be a distribution spec release. And there's some great work that's been going on to actually produce reports from all your popular registries. Whether we're talking about the cloud providers or open source projects, like Project Harbor, you can actually run a conformance test and see exactly which API's work or don't work or don't return the right return code. And so people are actually in the past few weeks fixing opening PRs, issues against various registry projects to make sure that everyone's conformant to the distribution spec. So that's goodness, just from a UX. You know, it shouldn't matter what registry you use, it shouldn't matter what tools you use. If we can all agree that that's how we talk to registries then migrating from, you know, GitHub's new registry to DockerHub to Azure, or Amazon's registry, or IBM's registry should be, you know, no bumps in the road for users. So that's kind of a big deal. And then the second thing that's newer, is, you know, I don't know how much you've played around with image signing, but the whole supply chain security topic is is really popular these days. And Docker had created this, you know, notary and the update framework and donated that to the CNCF. And then a bunch of people said, Hey, this stuff is too hard, you know? Azure and IBM both implemented it in our registries. But then actually using it and helping end users use it has been challenging to say the least. And Red Hat said, forget, you know, Docker and notary, we're going to do our own PGP based signing. Anyway, there was a meeting in December where we had representation from every registry, every cloud, and basically sat down and said, What would notary v2 look like if we can rethink it today, and all agree on what the use cases are? And how to come up with a simple, simpler, you know, UX for developers and image signing tools. What would that look like? So anyway, that's kicked off something called Notary v2 and that's under the umbrella, that OCI as well as the CNCF projects related to it. So yeah, there's a lot of positivity coming out it. They're doing a weekly meeting, having good attendance, again, good representation from the different registries. So I hope again, in 2020, by the end of the year, there will be a spec and some projects actually underway that say, here's how we can all agree to sign images. And you know that that would bring hopefully some commonality of tooling and process for what I think will be a very important part of the supply chain security story in the future.

Viktor Farcic 7:36
So whenever you standardize something, you make things below it commodity in a way, kind of. It doesn't matter anymore which container runtime you use. Soon, it might not matter which tool for signing your images you use. Kind of they're all gonna be almost the same, right?

Phil Estes 7:54
Yeah, yeah, sure. I think you're right. The lower layers really become commodity and the interesting pieces like which tools are going to make it really easy to integrate this into your DevOps pipeline, or who's going to make the most beautiful, you know, CI/CD flow that gives you everything you need for image build, image signing, you know, runtime protection and security capabilities. You know, so those higher layers are where, you know, hopefully the interesting work happens once we kind of all agree on these standard kind of basic layers at the bottom.

Darin Pope 8:34
So you're assuming that people are going to agree?

Phil Estes 8:38
yeah

Darin Pope 8:38
let's get real. Is it really going to happen?

Phil Estes 8:43
So yeah, I'm a, I will admit to being a horrible crystal ball person. I never write those end of year blog posts with the next year's predictions. And you know, here's what's going to happen in the industry. But that said, We also pretty sure Viktor was around back then, you know, in 2015, it looked like the future of container runtimes was going to be pretty ugly, you know, there's going to be Docker there was going to be Rocket, there was potentially going to be some other ideas about runtimes. And it was painful. But we, by 2016, we kind of had all those people on board with the OCI runtime spec. And today, like I said, a few minutes ago, it'd be hard, you're hard pressed to find some runtime out there that doesn't claim OCI compliance or even just use the runC binary underneath, you know, some abstraction such that we're all running the same things using the OCI specs. And developers don't even have to think or know about those things anymore, and then just kind of expect it to work. So we've gotten agreement before from a scenario where it looked like things were going to go south pretty pretty quickly. So I, I'm fairly optimistic, especially given when we're talking about the image signing discussion. Everyone who's involved has been like, Yeah, let's do this. Let's let's make it happen. So I'm hoping for for good results there.

Darin Pope 10:27
Why do you think image signing is now the I'm going to call it the hot topic? Is it just the next evolution or the next step down the path?

Phil Estes 10:35
Yeah, I think there's a couple things. One, you know, look at KubeCon agenda or any popular cloud native conference. How many talks on security are there or security infrastructure or securing your workloads, or, you know, shift left ideas about security. So there's that marriage of like, all of a sudden, enterprises are adopting this cloud native world and they're saying, wait, you know, we want some of the same protections we had in our kind of hand built, you know, prior to cloud native or our little world, we had this set of security guarantees about mapping source to an image to making sure that the image that's running in my production scenario can be tracked back to a very specific version of source code. So I think image signing is one piece of that puzzle to provide those same guarantees in the current world, that effectively are missing today. Or Or like I said, you know, Docker has Docker content trust, kind of built into their variant of, you know, their enterprise stack, but no one else really adopted it and so having a standardized general purpose, you know, image signing specification, I think is attractive because it's it's you're hard pressed to find an enterprise who doesn't want this kind of capability that says, you know, my CI pipeline builds, tests, and then signs images. And then I can validate all the way back to my source repository all the way into my production workloads that I'm running the same code I tested and built and verified. So I think, yeah, I think it's part of that whole security focus that we see in the industry right now.

Darin Pope 12:37
Let me go one step further. Because I know somebody at some point is going to believe this is a good idea. When do you think we're going to start seeing malware and virus scanners being required within a container?

Phil Estes 12:54
Oh, so I hope we never get to that point where, you know, we're actually inserting antivirus daemons into our, into our containers like like we did with VMs or, or bare metal systems.

Darin Pope 13:11
But you know its going to happen

Phil Estes 13:13
Yeah, but but i think i think what we'll see is that that kind of thing gets I mean, you could, you know, Viktor may have seen this out in conference land but you know, there's people showing off demos with these new eBPF tools to say, hey, look, I can report on this binary just got started in your container and I bet you don't even know what that is. And so I think we're going to see integrated at the host side with all these fancy tools that use eBPF and host level awareness of what a container is to go digging around and do you know whether you want to call it anomaly detection or runtime scanning. We're definitely going to see those features come into tools from Aqua and Twistlock and all the you know, Sysdig and all the major vendors I think will have what are effectively malware identifiers or scanners that will work at that level.

Darin Pope 14:14
I'm crying already.

Phil Estes 14:18
I mean, you know, we, we've seen reports of was a Tesla that, you know, this wasn't even like a container level exploit it was just somebody left the keys to a Kubernetes dashboard somewhere and someone was deploying workloads on their infrastructure running Bitcoin mining, you know, containers and Tesla's infrastructure so, so definitely, you know, I guess what, one report I heard was, you know, what we're not seeing is, is all the scary container breakouts we thought we'd see. What we are is the same old, same old, you know, somebody has a weak password or credentials were left somewhere. And instead of stealing data, people are saying, hey, compute is really expensive. I'm just going to use your compute to run my, you know, Bitcoin miners or whatever that generates money for me.

Darin Pope 15:19
Well, it's people being lazy,

Phil Estes 15:20
yeah,

Darin Pope 15:21
they just go and grab an image from DockerHub, thinking it's okay.

Phil Estes 15:24
True. Yeah. There was the interesting twist as well. So, again, that's where some of the signing and validation and verifying that what I'm running in production is something that came from my organization and not some random guy in the internet.

Viktor Farcic 15:41
Signing is at least that's that's probably the most basic thing you can expect in any enterprise. I want to I want you to confirm that that that what you're that what I'm running is what I'm supposed to run. And then whether there is an exploit in your code and things like that. That's already I mean, that's important to know, but that's already an added bonus. Beyond the most basic, which is, is this really what you wanted to run?

Phil Estes 16:09
Right. Right, exactly.

Viktor Farcic 16:11
And then, you know, many of the other things are so human dependent is that ultimately it really depends on how, how much you trust your people. Because if you don't, it's almost impossible to prevent your own people from exploiting your system. That's, that's close to impossible.

Phil Estes 16:34
Yeah, exactly. You know, as has been talked about a lot in the whole DevOps world is you know, culture and people and processes is as important or maybe more important than, than how fancy or tools can can become, you know how much we can promise, you know that we have the best tools and most secure infrastructure. Like you said, it comes down to at some point trusting your people and, you know, having a culture where hopefully, they're following the security processes and things that you put in place.

Darin Pope 17:19
We would like to think that people want to follow security. But sometimes that security process is so convoluted that nobody can figure it out.

Phil Estes 17:32
Yeah, I mean that. Yeah, definitely. That's one of the diff, you know, long before cloud native, you know, the discussion around you know, the lack of usable security is no security. If people can't use it, they'll just turn it off.

Viktor Farcic 17:50
I would say that the only reasonable way for something to be secure for us to expect that people are adopt security is to make it fully transparent. You cannot expect the developer to spend 25% of his time on security related tasks. It just needs to happen without him doing anything or her, right? If you don't do it like that, it just a waste of time. Because that person is going to spend instead of 25% of those task, he's going to spend 10% figuring out the workaround.

Phil Estes 18:25
Yeah. Yeah. And, you know, this is where I definitely give a lot of credit to Docker's early ideas about security, you know, the fact that they said, Hey, we're going to create an AppArmor profile that just works and just is applied every time you run a container. We're going to create a seccomp profile that turns off a bunch of sys calls that you should never touch and use anyway and we're just going to apply that. You're not gonna have to know about it, you know, just that idea that if if someone doesn't do the hard work, the developers isn't going to do that. They're not going to go figure out AppArmor and seccomp. And so I think you know, there is a lot of wisdom in trying to make Docker's engine secure by default. And even Notary and the whole signing, you know, the signing thing they came up with, you know, obviously ended up with a lot more complexity behind the scenes if you try and operate it. But the idea again, was this trust on first use, you know, you just docker build, docker push, and if you turned on Docker content trust, keys were created, images were signed, and you didn't have to like go read 10 blog posts to figure out how to do that.

Viktor Farcic 19:43
Exactly.

Darin Pope 19:44
That were all outdated by the time you read them.

Phil Estes 19:46
Right.

Darin Pope 19:48
Let's stay on that signing just a little bit longer because I think it's important. Going back to where the December meeting happened. They want to keep it simple. We had an episode recently about making everything an API. If they can actually come to an agreement about the contract of what that API is, and they actually implement it, what are the chances that somebody doesn't do just one thing a little bit different.

Phil Estes 20:12
Yeah, I mean, that's where, and this is. I mentioned one positive aspect of the OCI was this new compliance tool that's extremely simple to say, I have a registry. I want to check if it actually follows the OCI spec for distribution. You run the tool, it generates an HTML report, it opens in your browser, you see exactly which API's, you didn't do the right return code or you didn't handle the right parameter. I think signing will have to have that same level of simplicity with compliance tooling that says, You can't be a Notary v2 implementer unless you pass this compliance suite and make it as simple as possible to, to validate that, because you're right as soon as. And we have that today, the Docker client registry client accepted a lot of looseness around return codes and how auth data was passed. And for example, in the containerd project, we tried to go, we removed all those kind of hacks and immediately start getting issues like, hey, Docker works, but containerd doesn't work against my special registry. Oh, well, you know, you're returning a 403 when you should return a 404 for this, you know, API. So it's pretty painful after you know, when you allow kind of a community to build up a set of looseness around an API. And then you try and go back to conform to a very specific API. So hopefully, hopefully we can do that with signing and not not make it a, you know, anybody can do anything. And we'll just make the client kind of deal with a bunch of weird conditions.

Darin Pope 22:14
That's bad for everybody.

Phil Estes 22:15
Yeah, yep.

Darin Pope 22:17
Now let me ask this. I've got six providers of v2. They all pass. And then a couple of them decide to add on their own extras. As long as they're still compliant with as long as they're all still green, is that okay?

Phil Estes 22:34
Yeah, I mean, it. I guess this is where we have to define what what it would look like to have extra capabilities or features, and whether those impact the core capability of saying I know how to sign images. And that's where I think Kubernetes has gotten where there's a very stable API, it's very clear to go look at what the API is. But now there's a now there's operators. Operators are way to actually change, you know, these custom resource definitions, you know, my installation of Kubernetes understands Istio's special custom types. And I install a vanilla Kubernetes over here. It's same API, same version, but I didn't install Istio's operators and custom resource definitions. So extensibility is both powerful and useful. But I think maybe that is a good analogy for what does it mean when we say add ons or extra features? Does it mean that I can't actually run your tool because your tool expected these plugins or extra capabilities that I don't have over here.

Viktor Farcic 23:47
I mean unlike container runtime, which is basically noncommercial in most cases. security is all about enterprise software at the end of the day. If there is no, if anybody tries to I might be wrong, so correct me. But if anybody tries to standardize things, without leaving the possibility that different products will distinguish themselves with special features, then we are going to see a problem.

Phil Estes 24:17
Yeah, no, I, that's a great point. I mean, the the creation of specifications and standards should not disallow there to be differentiation at the higher layer abstractions. And so I think, you know, if we try and think about a practical application of that in image signing, maybe some provider is going to say, hey, by the way, I allow multiple signatures. So you can actually say, in my Kubernetes cluster, it has to be signed by the developer ,has to be signed by the CI, you know, some tag that the CI puts on and signs it with a different key. And it has to be signed by some production train, you know, authorizer that says, Yeah, this this, this application is ready for production. So that's fine, you know, know that enterprise suite would have capabilities to support multiple signatures. They would have a Kubernetes webhook that when you try to deploy an image would check some policy engine that says, you know, are all three signatures there? You know, I think those kinds of, you know, that may be a good example of something that the core spec should just say, here's how you sign. But maybe the the enterprise feature is, hey, we check for multiple signatures and you can set up different policies for different you know, namespaces in your cluster, you know, allowing that differentiation and, and productization of a standardized feature.

Darin Pope 26:01
Okay, how many companies going to be implementing that after they listen to this episode?

Phil Estes 26:06
Well, I, that I'm cheating because I think even Notary, the current version had some, some nice ways that you can do, you know, staged signatures. So the general example of something that could be more specific in the future.

Viktor Farcic 26:28
Anything else? Viktor, did you have anything? Other questions?

Viktor Farcic 26:31
No, I'm, I'm still recuperating from last week, so I'm good.

Darin Pope 26:37
So by the time you're listening to this last week was actually last week, but not really last week. Time travel. We were in Vegas as of last week, which isn't last week by the time you're listening to this. So if we're still dealing with last week, when you hear this, it's not a good thing. Let's just say Viktor was politely requested to leave a flight and we don't have to say anything else about that. But it wasn't because of him being rude. Viktor's never rude.

Viktor Farcic 27:11
I'm always rude but this is the first time I'm not asked out because I'm rude.

Darin Pope 27:18
You're never rude on a flight. You're rude everywhere else, because you want to get from A to B.

Viktor Farcic 27:23
That's true.

Darin Pope 27:24
If you're listening via Apple podcast, please subscribe and leave a rating and review. All of our contact information including Twitter and LinkedIn can be found at https://www.devopsparadox.com/contact. Also in the show notes for today, we'll have Phil's contact information if you want to talk to him about all the OCI stuff. If you'd like to be notified by email when a new episode is released, you can sign up at https://www.devopsparadox.com/ The signup form is at the top of every page. There are links to the Slack workspace, the Voxer account, and how to leave a review in the description of this episode, as well as Phil's contact information. A couple of things that I sort of pulled out from today's episode. I heard Project Harbor mentioned. There was the the four letter E word or E acronym. What was that?

Phil Estes 28:09
Oh eBPF. Yeah. So extended Berkeley packet filters. It's a Linux kernel feature that a lot of security tools are being built around.

Viktor Farcic 28:19
That's one of those things that if you don't know what it is, you probably don't want it in the first place.

Darin Pope 28:24
Okay, so I may or may not put a link to that one, but I'll put a link to a Project Harbor. Is there an OCI site to where people can follow along?

Phil Estes 28:33
Absolutely. So I can provide some links that we can share with people about the OCI and even some of the new activity that I mentioned.

Darin Pope 28:42
Cool. Yeah, we'll put all that down in the in the show notes and the description of the episode. There's also a transcript if you want to go back. If I'm not totally lazy, I'll embed the links in the transcript as well. Just as you're reading in context. So Viktor's getting well so I'm gonna leave Viktor to go ahead and go to sleep. I also noticed as we're recording today that Viktor is not drinking his beverage of choice today. Its the red can and not the blue can. He's drinking Coca Cola instead of Pepsi. So, Phil, you have any last words?

Phil Estes 29:15
No, that was great to chat with Viktor, without having to travel anywhere usually I have to travel a long distance to bump into him. So nice to chat from my home office. And yeah, it was great to share a bit about what's going on in the OCI.

Darin Pope 29:30
You probably have seen Viktor face to face more than I have, even though we talk every week. I'm trying to stay as far away from Viktor right now. Yeah. After last after what our last week was, nobody needs to see it. Phil, thanks again for hanging out today. Viktor get well, and thanks to you for listening to episode number 46 of DevOps Paradox.

DOP 46: Making Containers Great Again - A Conversation With Phil Estes

Show Notes

Links from the episode

Guests

Phil Estes

Hosts

Darin Pope

Viktor Farcic

Links

Rate, Review, & Subscribe on Apple Podcasts

Signup to receive an email when new content is released

Transcript