DOP 103: Knative in Action

Posted on Wednesday, Apr 14, 2021

Show Notes

#103: What is there was a way that you could harness the power of Kubernetes without having to learn all the ins and outs of Kubernetes? Enter Knative. Today with speak with Jacques Chester, the author of Knative in Action, about that at much more.

Order your copy of Knative in Action at:

https://www.manning.com/books/knative-in-action

and be sure to use the code “podparadox20” to save 40% off of Knative in Action and any other purchases at Manning Publications.

Guests

Jacques Chester

Hosts

Darin Pope

Darin Pope

Darin Pope is a developer advocate for CloudBees.

Viktor Farcic

Viktor Farcic

Viktor Farcic is a member of the Google Developer Experts and Docker Captains groups, and published author.

His big passions are DevOps, Containers, Kubernetes, Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD).

He often speaks at community gatherings and conferences (latest can be found here).

He has published The DevOps Toolkit Series, DevOps Paradox and Test-Driven Java Development.

His random thoughts and tutorials can be found in his blog TechnologyConversations.com.

Rate, Review, & Subscribe on Apple Podcasts

If you like our podcast, please consider rating and reviewing our show! Click here, scroll to the bottom, tap to rate with five stars, and select “Write a Review.” Then be sure to let us know what you liked most about the episode!

Also, if you haven’t done so already, subscribe to the podcast. We're adding a bunch of bonus episodes to the feed and, if you’re not subscribed, there’s a good chance you’ll miss out. Subscribe now!

Signup to receive an email when new content is released

Transcript

Jacques: [00:00:00]
Kubernetes itself is a toolkit. It's a collection of things that can be used to construct something. It is not a construction. Knative is much more like a machine that you feed what you want in one end and out the other end comes a functioning system with most of the modcons installed. .

Darin:
This is DevOps Paradox episode number 103. Knative in Action

Darin:
Welcome to DevOps Paradox. This is a podcast about random stuff in which we, Darin and Viktor, pretend we know what we're talking about. Most of the time, we mask our ignorance by putting the word DevOps everywhere we can, and mix it with random buzzwords like Kubernetes, serverless, CI/CD, team productivity, islands of happiness, and other fancy expressions that make it sound like we know what we're doing. Occasionally, we invite guests who do know something, but we do not do that often, since they might make us look incompetent. The truth is out there, and there is no way we are going to find it. PS: it's Darin reading this text and feeling embarrassed that Viktor made me do it. Here are your hosts, Darin Pope and Viktor Farcic.

Darin: [00:01:17]
Viktor, everyone knows your love for all things Google Cloud Run.

Viktor: [00:01:22]
Oh yeah. It's just the simplest way to deploy containers in cloud today, at least from my perspective. Not necessarily the best. It doesn't necessarily fit all the scenarios and all those things, but nothing beats simplicity of Google Cloud Run.

Darin: [00:01:37]
Why is that?

Viktor: [00:01:38]
Because they hide many of the complexities that normally people would need to deal with. There are many solutions out there that say, Hey, here is a mechanism for you to specify an image and it will run. That's easy. Everybody does that. But making sure that it is highly available, that it scales, that it doesn't break your budget and quite a few other things that are required are working pretty well in Cloud Run. There are, I cannot say completely hidden from users, but part of it is really hidden. If you compare that with, let's say, Azure Container Instances which conceptually are the same. Here's my image. Run it. Container instances are not doing a fraction of what Cloud Run or Knative behind it is doing. It's literally, oh, okay. I'm going to give you a VM with Docker where that container image will become a container, which is a completely different level than what let's say Cloud Run is doing.

Darin: [00:02:40]
You just called out that Cloud Run is hiding Knative.

Viktor: [00:02:44]
Yes. It is a, let's say, provider specific implementation of Knative.

Darin: [00:02:52]
On today's episode, we have somebody that's really smart in Knative. We have Jacques. Jacques. I have asked Jacques Okay

Jacques: [00:03:02]
I blame my parents

Darin: [00:03:05]
Jacques Chester. He is the author of the Manning Publications book Knative in Action. By the way, I'll go ahead and tell you now. If you haven't bought the book yet, go to Manning, use the code podparadox20. It'll save you 40%. I'll remind you later about that too. Knative in Action. Jacques, why the heck did you want to write a book about Knative?

Jacques: [00:03:30]
That's a good question. When I was talking to the acquisitions editor, Mike Stevens, I said I'd be crazy to say no and he said you'd also be crazy to say yes. You're stuffed coming and going as far as these things go. Essentially, it was an opportunity that was too good to pass up one and two I don't know. There was a space for it and I felt that I could do a good job and I think I've done an okay job as far as such things go.

Viktor: [00:03:54]
Is this the first book you're publishing?

Jacques: [00:03:56]
Yeah. This is the first book I've written. Before that, most of my creative non-fiction, as it were, has been arguing with people on Hacker News which is one of life's greatest and noblest time sinks. There's a lot of time and words that I'll never get back. I think the real sport on Hacker News is like which one of us is going to be the best at highfalutin correction of the other one. It's a bit like snark at 20 paces.

Viktor: [00:04:19]
I'm curious, and this is completely off topic. Do see that the work involved in writing a book the same at the beginning as now?

Jacques: [00:04:29]
Oh no. No. I had no idea how much work it would be. None. It's sort of interesting. It's very much like going to school. There are parts that are easy because I was fascinated in the topic and there were parts that were hard because the topic was necessary for a complete coverage but I didn't have a native passion for it. I won't say which section is which. I'll leave it as an exercise to the reader to guess at that. I'd be very interested in people's answers. It meant that there were days where I could bang out 3000 words and there were days where I'd be lucky to get through three sentences, but I always tried to make sure that I did a little bit every day so that I didn't end the day thinking, well, that was a waste of time. The closest experience I've had before this was doing an honors dissertation, kind of like a mini-thesis I guess really and that seemed like all the work in the world and that too was difficult but just on a different scale.

Darin: [00:05:16]
So how did you end up with Knative? How did that one end up in your lap? Was it that you pitched that to Manning? They pitched it to you?

Jacques: [00:05:24]
They pitched it to me. I've been afraid to ask them why they picked me, in case it's a mistaken identity or something, so I've just let them slide on with the impression that I know what I'm talking about. The connection I had with Knative was like many things in an industry, purely coincidental. I had been working on a functions as a service group at Pivotal, as it then was, called Project riff. We were approached by Google who said Hey we've got this thing we're doing and we want you to be part of it and so leadership said well who've we got around that does stuff like this? So they grabbed us in this group and said go take a look. Kick the tires and check the oil and poke at it and tell us what you think. We had the feeling that yeah this is pretty promising. We should be on board. Essentially, it was a case of right place, right time. I want to say that I sort of had this brilliant vision and that everyone shared it and there was a grail hovering in the distance. That sort of business but no it was basically just dumb luck initially. Just a coincidence.

Darin: [00:06:23]
So, what is your real relationship to Knative?

Jacques: [00:06:26]
At the moment mostly as an author. In the early days, I was heavily involved as someone helping to bootstrap the community, so I did a fair amount of work around helping with the development of governance. It's not as complex as the governance of something like Kubernetes is or Istio for that matter, but there was still a feeling out the stones with our toes as we crossed the river had to be done and I was involved with that and also with auto scaling. I'm sort of spoiling my earlier exercise and a topic I'm fairly interested in. I spend a lot of time talking to folks about auto scaling and talking to the then leader of the auto scaling working group, Joe Burnett, about auto scaling and what I thought we could do or how we might do it. I also built an autoscaler simulator harness at that time, which for reasons related to deep sigh go.mod was difficult to keep up to date. Yes. You're sad and knowing laughter is appropriate because go.mod, there are swear words that are yet undiscovered by humankind that will barely scrape the outside of go.mod and how I feel about it. The story being that I'd worked on an autoscaler simulator harness and I'd used that with a few versions of the Knative pod autoscaler, the inbuilt autoscaler it comes with. It'd been an interesting exercise but when it became possible to do so, I turned my focus to the book and that was my principle focus for about a year, year and a half.

Viktor: [00:07:51]
If we ignore the ability of Knative to scale to zero, which doesn't really exist in a standard Kubernetes cluster, does Knative autoscaler bring something to the table that you don't have with horizontal pod autoscaler?

Jacques: [00:08:07]
At the time, yes. You can do custom metrics with the HPA now. I can't remember if custom metrics didn't exist or were very preliminary at the time, but for the Knative pod autoscaler, that was one consideration. The default is that it's based on concurrency, so the number of requests that are currently somewhere between arrival and having been served is the main thing that it's structured around. You can also now structure it around requests per second, but as I explain in the book, those are actually more or less interchangeable depending how you go at it, so you might as well go with concurrency. That's what the default is. That's what the documentation talks about. The other reason is that the Knative pod autoscaler has an intimate knowledge of how Knative is structured, so it is able to get the statistics it needs quickly, so it knows where to look. With the HPA, it's slightly more at arms length. That's not life or death. There's a notion that at some date in the future it could be retired in favor of the HPA but that day has taken much longer than anyone expected, so work on the Knative pod autoscaler continues.

Viktor: [00:09:10]
For listeners who are not familiar with Knative, how would you describe it? Why would anybody bother?

Jacques: [00:09:15]
I started the book with what I call the Onsi Haiku Test after Onsi Fakhouri who first said it. Here's my source code. Run it on the cloud for me. I do not care how. It's a beautiful and elegant and simple statement of how the world ought to be and how the world in some places it used to be. Knative brings Kubernetes much closer to that vision. Kubernetes itself is a toolkit. It's a collection of things that can be used to construct something. It is not a construction. Knative is much more like a machine that you feed what you want in one end and out the other end comes a functioning system with most of the modcons installed. That to me is very attractive. My background for context is working on Cloud Foundry before I worked on Knative. In fact there are quite a few ex Cloud Foundry people from multiple companies who have a hand in Knative or who are working on Knative, so we've brought that Onsi Haiku Test with us as a North star. We want it to be easy. We feel that if you're a developer, you ought to be able to just develop. It's a controversial notion, but Knative makes that easier, makes it better.

Viktor: [00:10:24]
I like what you just said, because usually when I speak with people about Knative, the first association is hey, it is some form of serverless, right? It scales to zero when you don't use it and to something else when you do. In my head, the thing I like the most about Knative is not that. I mean, that's definitely there and it's important, but it's more that compact way of describing what you need. If I compare Knative YAML with equivalent YAML that would involve deployment and HPA and custom metrics, services, virtual services, gateway. All the stuff I get from 20 to probably 100, 200 lines more of YAML. Nobody knows how many right?

Jacques: [00:11:13]
It's even less if you use the kn CLI. In the book, I go out of my way to avoid YAML and not always successfully. I do see the virtue and the necessity for a declarative document style, but I also wanted to demonstrate that you can get a lot of day to day development done at the command line and that the experience can be very simple and very straightforward. The interesting thing to me was could I write a book that didn't need you to understand Kubernetes first. It's difficult for me to know if I succeeded. It's difficult to know first of all whether Knative has succeeded in being able to completely abstract Kubernetes. I think there are various Kubernetes-isms that sprout up through the ground of necessity but could you get most of the work done without needing first of all to have a mastery of Kubernetes which is a lot of material. In the very early days of Kubernetes, there wasn't that much to get your head around. You pretty much had to understand pods and then maybe later on you had to understand ReplicaController and then it became ReplicaSet and then you had to know Deployments and then you had to know Services and then you had to know ingress and then and then and then and then and then. These features are not added just for the hell of it to Kubernetes. They solve particular problems that are encountered by people in real production scenarios but they still impose cognitive load on everybody because they have to be like well I have to know enough about this to know whether I need to know about this. The nice thing about Knative is that there's very little you have to know and once you've know it you're like, all right, well, now I can take it or leave it as the case may be. There's something like 400 kinds I think. I haven't counted them but there are hundreds and hundreds of kinds or types. Kind is the Kubernetes term inside the API of Kubernetes out of the box and then you add custom resource definitions, custom kinds, which can be installed. Knative is an example. Of course everybody and their dog and their sister's dog and their sister has been creating these and so on top of the hundreds that are built in, you've got to know about a bunch of others and also got to understand how they interact with the ones that are installed. So in the book I wanted to avoid that. I wanted to basically say how close can we get to the Onsi Haiku Test as an experience? How far can I get you without requiring to teach you the whole of Kubernetes? There are asides or places where the joke I made somewhere that in order to teach you Knative chemistry, I sometimes have to dip into Kubernetes physics, but most of the time hopefully I was able to avoid that.

Viktor: [00:13:41]
In an organization, in a company, somebody still needs to know those things because things go wrong. But I guess what you're alluding to is that for the majority of people in a team in a company whatever you can just say deploy right?

Jacques: [00:13:54]
Yeah exactly. We go through these experiences of you need to know the details and there's an abstraction you need to know the details and the abstraction and then you need to know the abstraction and then people say you still need to know the details and then eventually the details become so reliable and so standardized that people forget they exist except for the specialists who continue to shake their fists and grind their teeth that people have forgotten in other ways. An example of this is NetBoot. Your computer starts over network. Now I have not got the faintest idea how this miracle is achieved and I am wonderfully glad that I do not have to. That is only possible through the advancement of abstraction. There will come a day where you don't need to know Kubernetes. That seems impossible now but it seemed impossible that a day would come where you didn't need to know everything there is to know. You needed to have TCP/IP Illustrated on your bookshelf with many bookmarks in it to get anything done and that age has passed. Now I just pull up the fetch API in JavaScript or I use a HTTP client in Java and off I go and I literally do not care. That is the world that is always coming to us and that we should always be striving for.

Viktor: [00:15:08]
That's something similar like with containers. People had to know a lot about containers and underlying technology, namespaces, until recently. Now Docker just works. It's not working in a distributed way. It's not cluster but if you look at only I need to run a single container, nobody knows anymore how containers work because it's just working right?

Jacques: [00:15:30]
Yeah exactly. I think there were two things or maybe three things that were critical about Docker. One was that developer experience where it just worked. It just happened without you needing to first read a dissertation on isolation mechanisms in the Linux kernel and how they had tied them together into the solution. Then there's Dockerfiles which I have objections to but they're very easy to use. The container image format turned out to be critical. That was really the big novelty compared to previous generations of technology that used these kernel primitives, of which there were quite a few. There was LXR and LXC and is it Linux VPS maybe? I've forgotten all the names. Google had one called LMGTFY. Let Me Containerize That For You was the name of it. There's a long history. It was funny because I was working in Cloud Foundry when Docker popped up. We were using containers. That was the technology of the runtime. I was perplexed as to what the fuss was about because I was like this is an implementation detail. Why does anyone care so much. I didn't fully understand the value of the image format and its utility yet, but I was definitely flummoxed by it. I think a lot of people like me had that blind side. We underestimated the impact and the importance but I agree with you that the thing that's nice now about Docker and other runc based runtimes and things that use all sorts of mechanisms now is that at some level doesn't matter. You go run the thing and the thing runs.

Viktor: [00:16:57]
Is that where we are going now with Kubernetes then in a way? At least how I see it is that Docker simplified certain things a lot but then in that simplification it failed to satisfy people who do need to know about those things, almost as if Docker was too simple to satisfy that small number of people who do really care about systems.

Jacques: [00:17:22]
There are such folks always and they always have a legitimate complaint. There are always cases where you need to know the details and you need to be able to change the details. There's really no sort of answer that satisfies everybody, so I very much sort of fall on what you might think of as the populist side of developer centric technologies. I'm trying to uplift the masses as it were rather than the elites who deeply understand everything. That's just a simple economic reality. It is not tractable for everybody to be an expert on everything. It just doesn't work. That's not how society and civilization progresses. You need specialization and specialization requires abstraction between specialties. The journey for Kubernetes is not done. Knative is the next evolutionary link in that journey I believe.

Darin: [00:18:14]
When you have conversations with people that think they need to know what's going on under the hood but just from your experience you know that they really don't. How do you talk them off of that ledge?

Jacques: [00:18:27]
I don't. I say, look I believe it's done for you. There's the joke about tell a man that there's 400 billion stars in the galaxy and he'll believe you. Tell him that the paint is wet and he will have to touch it. On some level telling people don't pay attention to this draws their attention to it. I think it's fine. I think there are people who will take it on faith that the hard work has been done and is worth taking at face value. There are people who will not take it at face value, go and investigate and discover through that journey that things made sense. Both are fine. The ones that are most frustrating though to deal with emotionally as a person involved is when somebody tells you that what you're doing is wrong but also very simple. I could do that in a weekend or why don't you just do X which takes us elegantly back to riffing negatively about where everything can be achieved in a weekend by the unlimited singleton geniuses. I faced a lot of that in my Cloud Foundry days. A lot of people would say well how complicated could it be and I still see it a lot. I saw one the other day where it's sort of like oh I've written a replacement for Heroku and it's just like, you haven't. You have done no such thing. Ah man. We competed with Heroku. They're good. Really good. The same with Cloud Foundry. It's like oh I could replace that. No you can't. Let me pull up the dozens of backlogs with collectively tens of thousands of stories, bugs, and chores and just pick them at random and tell me that you can do that in a weekend because I promise you, you have no idea how deep the rabbit hole can go.

Viktor: [00:19:57]
That usually comes from people who either do not understand the complexities of certain things or are maybe arrogant because it sounds silly to say I can do single-handedly in a week what I don't know hundreds of people were doing for a year. Now I understand that when you do something only for you, the scope is smaller but still no matter how much you divide the effort in let's say Knative or anything else, it's still tremendous, right?

Jacques: [00:20:28]
Yeah I mean, there's a level at which they're not lying because they tend to mentally simulate the things that they will do. What they're mostly unaware of is all the things they don't know that they don't know. The vast endless, endless stew of details of things that you have to think about as the implementer of a platform. I remember once one of my experiences was working on buildpacks and there's the cat being annoying in the background. Sorry about that. Just adds some character I believe. I had an experience working on buildpacks, the previous generation. I also got to work a little bit on the new generation and there was some bug deep in the guts of Python or node or something like that that we were working on. One person had reported it and it took six steps to recreate it and we were struggling to work out how to solve it and we'd been on it for a week and I thought why are we doing this. This is ridiculous. My pair turned to me and said so that nobody else has to ever again. That's the sort of details that you deal with as the implementer. If you've done your job well, people think they can replace it in a weekend because they don't see all the stuff behind the curtain. So in some ways, it's a journey of self-defeating but also victory over detail.

Viktor: [00:21:44]
Speaking about the amount of work and things missing, will Knative ever get volumes?

Jacques: [00:21:51]
That's a good question. That's an open question I believe. I have not kept up with the volumes discussion. One of the trickiest discussions I think in Knative is how much of Kubernetes to expose. There is always a constituency who want a particular Kubernetes capability to be exposed through Knative's primitives and always a case that that will essentially make it more complicated for everybody else. So that's the thing. You gotta sort of think about where does the cost fall for any given change and if you add something from Kubernetes that reduces the cost for the person who got it and increases the cost of everyone else because now they've got to say do I need to know this which means I need to know it. Again in my experience we went through an evolution on Cloud Foundry where the original doctrine that we had was no. No permanent volumes of any kind. It's a 12 factor application. You wrote a 12 factor application right and if you didn't, well that was your problem. You need to get hip. Get with the times. That was at some level true as a matter of dogma but also most of our customers were like that's great but I've got code that uses the Java transactions API and the Java transactions API uses the File class and the File class is final and cannot be sub classed into something else that understands ephemeral volumes and so I need an actual volume. Thank you very much. I don't know is the answer to your original question will there be volume support. Maybe. I think one of the things that I have mixed feelings about in the Knative API is that it partly works in serving by exposing some but not all of what you can set on a pod in the shape of the pod. So It directly embeds the kind and exposes some of the fields but that means the shape is the same as a Kubernetes pod and there were a lot of things in Kubernetes pods that are positioned where they are and named what they are and work the way that they do because of implementation convenience rather than consistency or user experience. I feel mixed feelings about bringing that sort of history that path dependency into Knative. So maybe. Maybe not.

Darin: [00:24:04]
I want to go back to one point you said earlier about when you were writing the book you tried to stay on the command line versus going YAML and that you're not anti declarative but I heard a subtext to that It felt like you were actually anti YAML.

Jacques: [00:24:19]
I like it when it's small. I have been at the receiving end of multi thousand line YAML documents in the days long before Kubernetes actually. Both the people listening who know what BOSH is, the original BOSH 1 deployment manifests were humongous. Humongous. I recognize the necessity of a format that can be snapshotted and diffed. I think that's useful. As an actual interface for working with a system, YAML ceases to be human friendly very quickly. It's an interesting case study in ergonomics. It's pro-ergonomic in the small and anti-ergonomic in the large. That being said, the CLI is not going to be effective at very large scale. If you have a portfolio of a hundred, a thousand, 10,000 different Knative services, you need automation and that automation is going to work in YAML. Actually in the last chapter of the book chapter nine, I switch and I sort of say look well I focused on the CLI because it's interactive. It lets you poke things and see what happens very quickly and very easy with very little ceremony, but as you move into a production environment, you're going to be working with YAML and that YAML is going to be templated and that template is going to be managed by some CI/CD system and you're going to be writing through a Git repository so that you have snapshots of your intentions at different points in time. Not necessarily the actuals but definitely at least snapshots of the intentions. That works pretty well. That works very well. I mean I again was in a world where we were doing that before it was common in Kubernetes. We didn't have a cool name for it unfortunately or I might've had a startup. We just called it the thing we do. I think It just depends. I'm not fully bought into the idea that YAML can be replaced. I think we're keeping it. As I say in the book, people I trust have sung to me the virtues of alternatives. So I've had people tell me that Dhall is fantastic and having glanced at it I believe them. I've had people tell me that qlang is the best thing since sliced bread and can prove it and I believe them. I've looked at it and other people are telling me that Pulumi or Pulumi I'm not sure how it's pronounced I haven't heard it said out loud is fantastic and I believe them too. I've used none of these things in anger so I couldn't give you an opinion which was backed up by anything than well, I like the way it looks.

Viktor: [00:26:36]
I tend to agree with you that YAMLs quickly become a nightmare. But where we might disagree is that I think that's a reflection of some other things. Because if you have a hundred lines YAML which is a nightmare to manage. A hundred lines is nothing compared to big systems. Still doing the same thing through CLI would be hundred arguments or 95 arguments. I think that rather the problem is not that much whether it's CLI or YAML or this or that but that the way how we define systems no matter what is the technology are convoluted by unnecessary things. I understand the need for labels and how all those things connect in Kubernetes, but whenever I look at labels, I lose a bit of my hair. They are everywhere. Repeated all over over and over again. I would need to create them with CLI also in a way, unless I have something let's say like Knative that simplifies. What I'm really trying to say is not that it's that much whether it's Pulumi or YAML or this or that but that we need constructs that remove unnecessary things.

Jacques: [00:27:45]
Yes. I broadly agree. The difficulty is always agreeing on which parts are necessary and which parts are not. So for example are volumes necessary, tying it back to a previous conversation, is an open question. I think YAML is here to stay. I like YAML. There are a number of tools I use where one of for me the bonuses is that it uses YAML or at least it uses a simple declarative approach, but in the book my intention was like this is a starting book. I want to teach you the concepts and not the syntax of the documents. So I do have examples in some of the chapters of what the YAML looks like but really if you need to know what the YAML looks like, the documentation site is really good and that's it's focus. It focuses on the YAML rather than the CLI, whereas the CLI focuses on an interactive experience. I mean if you're using the CLI in production, then shame on you. That's not what it's for. You should be using YAML in production, but for getting your feet wet for interrogating something quickly without having to remember wait where's that field, I think the CLI is great.

Darin: [00:28:46]
Just as a reminder, Jacques is the author I don't know why I am talking like this. Jacques is the author for Knative in Action from Manning Publications. If you haven't bought it yet, go buy it. Go buy two. And Viktor, what do we always say and then expense it. Expense

Viktor: [00:29:06]
Expense Expense it to your manager.

Darin: [00:29:10]
If you are expensing it and you still want to save your manager a little bit of money, use the code podparadox20. The link directly to the book on the Manning site and the code are down in the show notes. If you haven't looked at the show notes you can go do that That code is only good by the way at the Manning publication site. You can't use it at Amazon but you can buy it at Amazon. You can buy it anywhere you buy your books but if you want to save 40% the only place you can do that is on Manning and be sure to use the code podparadox20. Jacques, any final words for today and how many times did I mess up your name?

Jacques: [00:29:52]
No you did pretty well. I've heard some pretty creative interpretations of my name

Darin: [00:29:57]
And by the way your name is Jacques which is a French name and you're not French. You're from uh you're from the North Yeah Well Northern

Jacques: [00:30:15]
the Southern North Yeah I mean it was North of me like all right. You know as a young man I would talk about southerners the way that southerners here talk about northerners. So there's always another place that you resent. That's a law of nature. Yes so my mother was from the Seychelle Islands which is a French speaking country out in the middle of the middle is not accurate out way out in the Indian Ocean and she moved to Australia with the rest of her family before I was even a thought.

Darin: [00:30:47]
Very interesting. So if you wouldn't have hung out until the end you wouldn't have learned this about Jacques today. I don't know what the the award. Would be but

Jacques: [00:31:05]
Well I mean if you haven't got Vegemite of your own, that's sad

Darin: [00:31:10]
Viktor, do you even know what Vegemite is?

Viktor: [00:31:12]
All I know about Vegemite is that I'm not supposed to try it. That's what I was told. Kind let not your curiosity, get the better that's the

Jacques: [00:31:24]
Well I disagree. On behalf of all Australians I invite you to try Vegemite done properly on some medium crusty toast with a bit of butter and don't spread it too thickly. Just start easy if you're new to it. Delicious. Really great stuff.

Darin: [00:31:40]
And this is where I I butt back in and say being from the South we eat all sorts of weird things Vegemite was simple compared to a lot of other things It's I I like it, especially in the little packs. You can just get the a like a jelly spread sized thing and you just just a little bit of Vegemite and like you just said, start small. Don't don't glob it on like you do with jelly. That's not what you want to first time through. Uh also all of Jacques' contact information, Twitter, LinkedIn, all of that is also down in the show notes. Jacques, thanks for hanging out with us.

Darin:
We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and the link to the Slack workspace are at https://www.devopsparadox.com/contact. If you subscribe through Apple Podcasts, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at https://www.devopsparadox.com/ to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox.