DOP 101: What to Do When Technology Fails

Transcript

Nicolas: [00:00:00]
Previous summer, I was doing a code and architecture review. You cannot imagine the real world. The logging was completely non-existent. I have no clue how they could actually debug when a failure happened in production. Of course no containers. No nothing. You would have died, really from the shock.

Darin:
This is DevOps Paradox episode number 101. What to Do When Technology Fails

Darin:
Welcome to DevOps Paradox. This is a podcast about random stuff in which we, Darin and Viktor, pretend we know what we're talking about. Most of the time, we mask our ignorance by putting the word DevOps everywhere we can, and mix it with random buzzwords like Kubernetes, serverless, CI/CD, team productivity, islands of happiness, and other fancy expressions that make it sound like we know what we're doing. Occasionally, we invite guests who do know something, but we do not do that often, since they might make us look incompetent. The truth is out there, and there is no way we are going to find it. PS: it's Darin reading this text and feeling embarrassed that Viktor made me do it. Here are your hosts, Darin Pope and Viktor Farcic.

Darin: [00:01:25]
Viktor, I saw this blog post the other week about saying goodbye to minikube. You use minikube all the time.

Viktor: [00:01:33]
I use Docker Desktop and minikube all the time and Kind as well, but not as much as the other two.

Darin: [00:01:38]
Wait, which don't you use as much as the other two? Kind?

Viktor: [00:01:41]
Kind, yes.

Darin: [00:01:43]
Why don't you use Kind as much as the other two?

Viktor: [00:01:45]
Multiple reasons. To begin with, I'm used to minikube. It exists much longer and it is in a way easier than Kind. I get ingress just by executing addon instead of trying to install ingress in a special way. I get IP that I can reference which is to me easier than referencing localhost. Other than that, they're very similar. Similar for basic usage. Now when we get into multi cluster stuff, then it's a bit different story. That's when I'm jumping to Kind.

Darin: [00:02:17]
We decided to ask the author of that blog post to come on with us today. Nicolas, how are you doing today?

Nicolas: [00:02:25]
Fine. Thanks for having me.

Darin: [00:02:27]
What was the background for you writing this blog post called goodbye minikube?

Nicolas: [00:02:35]
It got a lot of traction. I will tell you what I did and what was my intent. I was using minikube for demos for a long time. At the beginning of the week where it happened, I had a talk and minikube was acting quite slow but it was okay. Two days later, I had this other talk and I also use minikube and it was like really, really, really slow. So slow when I tried to stop it, it took about two minutes. I'm a developer advocate, so it's very important to me that my demos, they work because developer advocates, they might have this reputation as saying a lot of stuff but when you really try to implement what they tell, it's not that great. So for me it's about my reputation and when my demo fails, which happens of course from time to time, I feel really, really bad inside. After this happened, I felt really, really, really bad. I apologized a lot to the audience and I had to do something. I just tried to stop minikube, remove the cluster. I was using the VirtualBox driver. I tried to create a new cluster and it was again, very, very slow. It was enough for me. I wrote the post out of anger. It was a catharsis because it was a way to remove my emotion. I was very, very angry at myself and at minikube. I had to do something. This was my way to cope with that. Then I wrote the blog posts and I write blog posts every week. On Sunday, I publish my blog posts and this one for I don't know which reasons, it got super popular. So I do write some blog posts that take me days to write. This one I wrote it in 20 minutes and it got traction. The others not. So that's the background.

Darin: [00:04:35]
The background for that was you were fed up with minikube. You switched to Kind. Why did you choose Kind over fill in the blank?

Nicolas: [00:04:48]
The background is that I'm running on Mac and I tried to see how there was this K3s also that was on my radar for a couple of weeks or months. I tried to install it and it didn't work just right out of the box. I installed Kind. It was a Homebrew recipe. It just worked. I decided, okay, let's redo the same demo that I did and wow. It was really as I would have expected. In previous versions with minikube when I started the MySQL container, I could really wait for each logging line at startup time, but with Kind, it was as it would happen on my machine. It was really, really fast. I settled on Kind because it just worked. Of course, it's not exempt of cons. For example, I dislike the fact that I need to first load every image from my local Docker daemon into Kind but otherwise it gets the job done, so I'm quite happy. Also, one of the issues I had with minikube before is that I don't remember the exact circumstances but the IP of the cluster changes. So the IP is incremented every now and then and you have the ways to reset it but I couldn't configure my /etc/hosts to have Kubernetes host or something. So I need every time to remember to check which IP is it. minikube IP? Okay. Give me the IP. That's fine, but it's one more hurdle along the way. Whereas with Kind, I just access it with localhost and it works for me. I mean it's good enough. That's the point.

Viktor: [00:06:29]
Just to play devil's advocate, you're using Kind with Docker Desktop, right?

Nicolas: [00:06:34]
Yes, correct.

Viktor: [00:06:35]
Yeah. So why not Docker Desktop as your Kubernetes cluster?

Nicolas: [00:06:39]
That's a good question. When I had time to think it out, I asked myself why I've been using that. Honestly I don't know. When minikube was working out for me, I was already wondering a bit about Hey are there any other alternatives? I knew about Kind. I knew about K3s. The first thing that came in my mind when I had this big failure was to try something out and it was not the Docker Desktop Kubernetes cluster. That's a good question. There is no real reason.

Viktor: [00:07:11]
From my perspective, if you talk about Linux, Kind is almost certainly the best choice because basically you run Kind directly on your OS. Actually, it's much better than minikube because for minikube you are spinning up a VM and Linux you're just running containers which end up being your Kubernetes cluster. Now for Docker Desktop, the doubt I'm having is if I have to use Docker Desktop and Docker Desktop runs in a VM and Docker Desktop gives me Kubernetes, is Kind just additional layer that doesn't bring any tangible value on top of Docker Desktop, excluding now the use case of multi cluster setup. That's absolutely awesome.

Nicolas: [00:07:50]
I'm not choosing multi cluster honestly. As I mentioned, the thing is I didn't have any rational thought thinking at the time. I just wanted something that worked. I used Kind. It worked and now I settle for it. As I mentioned, I wrote the blog post in anger. I might give it a shot like some people told me, Hey, why aren't you using minikube with the Docker driver? I said that's a good question. Probably because I got burned and now minikube is associated in my mind with failure. I try to be as rational as possible. As I mentioned when I wrote the blog post, I was not rational at all. I was very emotional.

Viktor: [00:08:30]
Those are the best moments to write a blog post. Those are the best blog posts. At least in my case the popularity of a video or a blog post is directly opposite of the amount of time spent on it. The more time I spend, the less popular it is.

Nicolas: [00:08:45]
I'm afraid that yes I might have the same experience which is not great.

Darin: [00:08:52]
So why do you think that is? We were joking before we pressed record of well that's just the way the internet is. We like producing long form content but people could care less about long form content or about things that we've thought about. They want to hear the rants. Why is that?

Viktor: [00:09:13]
Rants are not something we didn't think about. Quite the opposite. Rants are culmination of something that you've been working on. Right? So you're working on something for days, months, weeks, at least I'm explaining myself now. Not everybody else. But you work on something then you are passionate about it and you're deep into it. Then something terrible happens and then you rant about it. But it's not really that you're ranting only about it is ranting based on some experience. So it's not really that it's not thought through. Maybe the specific content is not thought through, but ranting is usually based on experience and usually longer experience. You don't rant about something that you have absolutely no experience about.

Nicolas: [00:09:57]
That's a good comment but I wrote sometime in the past, like more than a couple of years in the past, a rant about some thing I had no experience about which was Gradle in the Java ecosystem. I am a Java developer by trade. There was this Maven build tool. It improved a lot of stuff compared to Ant. I was a consultant for nearly two decades. You came to a customer. You came upon the Ant build tool and you needed half a day or one day to understand how it was built. All Java projects are structured in the same way. You can read the pom and you need at most one hour if it's a really, really complicated one. Then Gradle came and said Oh but you can code your builds and it was back to Ant again. I wrote a rant about that. It also got pretty popular but it was of course very controversial especially all the Gradle lovers on the face of the earth told me that I had no clue what I was writing about. So sometimes it's also about not a direct experience but like viewpoints that you want to rant against.

Viktor: [00:11:04]
I would challenge even that. If you think about it, you had a lot of experience with Maven and you wanted to jump into Gradle, which is a natural progression for everybody. Almost nobody starts with Gradle from scratch without having previous experience with Maven or Ant. You're jumping into Gradle as a potential substitute for Maven. Your rant is actually based on that experience. Hey, I have years of experience with Maven. For me jumping to Gradle was painful for this and that and I can expect other people also not to spend three months before they make a conclusion whether they're going to switch. You're going to spend a couple of days max on a new tool before you choose to switch, right?

Nicolas: [00:11:48]
Yes but I want to correct. I think that Gradle is a step backwards actually. Let's not go into another rant that of course. We want this podcast to be popular So it could be a pretty popular way to advertise for this podcast. And by the way I listen to it when I run. I really really like it, so keep up the good work. But yeah, Gradle is a step backwards because you again get back to a time where you need to spend one day, two days, to understand the build. Everybody seems to have an opinion regarding how an application should be built. Whereas, Maven told hey it's our model. It's the golden standard. Of course, you can disagree with that but if you want to use Maven, you need to adapt to Maven. You can write plugins but if you have ever wrote a Maven plugin, it's going to be super painful. So before you invest time into writing a Maven plugin, you're probably have a good engineering decision saying okay plugin, I will spend that much time adapting to Maven. I won't like it but it will be less painful. Whereas in Gradle, anybody, and when I say anybody any really anybody, can write some Groovy code, or now Kotlin code into their Gradle build and decide Hey that's how I want my special snowflake project to be and it's never the case. Never the case. Now Gradle is in the Android ecosystem. You cannot move it out of there because Google decided it's the way we do that. But most projects that adopt Gradle for some reasons it's probably the wrong reason. But again I don't want to delve deeper into that, otherwise it will be another flame war of a Gradle versus Maven. We already have one of minikube versus Kind so let's not open another front.

Darin: [00:13:46]
Come on. We need to have more ranting. By the way I picked Gradle over Maven anyway.

Nicolas: [00:13:54]
Oh that's interesting. Why?

Darin: [00:13:56]
Why? Mine was very simple. Going back to Ant. I was at the point to where I had Ant builds and we were getting fed up and Gradle was just coming out. So I completely bypassed Maven and went straight from Ant to Gradle. Now did I love it? No, but it was fine.

Nicolas: [00:14:21]
Okay That's that's understandable. So you didn't have this phase of standardization that everybody has the same way to structure their application, which then makes sense. If Maven didn't exist then of course Gradle would be an improvement over Ant. I agree.

Viktor: [00:14:38]
I must also admit that and this is from 10 years ago I haven't worked with Java for a long time, but back in the days when I worked, I started by liking Gradle more to be honest because to me writing Gradle build script is easier and more readable than Ant than Maven but over time, I started disliking it because I could not find information on internet how to do stuff precisely because everybody does it completely different and that's completely silly. So, I like Gradle syntax more than Maven syntax but I also like standards. There is no reason for me to build my jar file anyway differently than everybody else.

Nicolas: [00:15:24]
You have a good point. Actually you nailed it. In Gradle, you code your builds. You code your scripts. Whereas in Maven, you declare it. I think that's a very important difference. I don't want a Java application to be scripted for the build to be scripted because nearly 99.99% of them are all the same. I just want to configure. They're all the same. Again, if you really really want to have something specific or not want but to need to have something specific then you can always write a Maven plugin and again it's going to be painful, but at least you know it and before you make that step you must really really think about it deeply. The fun part is at some point I had to touch Gradle anyway because somebody in the company I was working for decided it would be the best thing in the world. Also it was an Android project, so in that case it made sense. A lot of the counter arguments of the Gradle folks they say Oh but you know nothing prevents you from writing a plugin on your own and that's completely true. But first as you mentioned documentation at least at the time was a bit lacking and when I say a bit it's an English understatement. Also the thing is it's so easy to write code directly in the Gradle builds that nobody ever does. So in this company I was working for they had created an Android project and they had written code to interface with Sonar which made a lot of sense. Of course then came a new Android project and you would think that at this time somebody would have made a plugin out of these custom Gradle code so that the plugin would be used by the two projects. Of course not. They just copy pasted the Gradle build because it's so easy. Then Hey you've got a third application. Who was the idiot to try to factor that into a plugin? Well the guy who just told you that he hated Gradle. I tried to do it. Not fun.

Viktor: [00:17:33]
I think that the situation is kind of a bit different today in terms, I'm curious whether you agree with this what I'm about to say, that many of the things that we were doing in language specific build tools like Gradle, Maven, Make for Go, they're these days outside of the domains of those languages. We're moving towards direction that language specific build tools are more about building the binaries and most of the other tasks are somehow outside now of that domain. More generalized. You don't build container images using Maven, Gradle. You build it using Dockerfile. You communicate with Sonar through some pipelines and things like that. So I have impression that the amount of things we do in language specific build mechanisms is smaller than it was before.

Nicolas: [00:18:27]
It's definitely smaller. I think we are in this blurry area where yes there are some things that might go away. For example, for Java which I happen to know or Kotlin or whatever. I happen to know more than other languages. If you create a Dockerfile to containerize your Java application, guess what? They're all going to appear exactly the same. You will always have this compile phase, this package phase, perhaps you will run exploded. I think it's probably a good idea to run exploded than to package a jar and to run this jar. In the Java ecosystem, you have this Maven Jib plugin and it can understand the structure of your pom and it can create the layers image. So on the top and depending on how you look at it and at the deepest level you will have the libraries because the libraries they don't change that much. Then you will have the resources .Then you will have your code. So you have this layer. With Spring Boot for example now you can also directly create a Docker image from Maven and even better you can create a GraalVM, a binary without any JVM code inside. Let's not complicate stuff. An executable binary inside the container from the Maven command line. Of course it probably will be used by some external build tool, GitHub Actions, Jenkins, whatever. It's meet in the middle. The build tool, Gradle, Maven, they are going toward a more integrated way of building and the generic build tools, they are trying to get deeper into different languages, in my opinion.

Viktor: [00:20:18]
Yeah because we have potentially on one hand if you're a let's say Java developer, it's much easier to let's say build container image just as example by executing Maven something, Gradle something, but on the other hand almost every company today is polyglot company. As a minimum we have JavaScript as well as a minimum. It's almost unavoidable to have some JavaScript for front-end. Then we have those decisions to make Hey should I have Maven something and npm something to build container images and both of them are easier for people using those languages or should we use something else to build images that will work for both, if you understand what I mean.

Nicolas: [00:21:05]
Yes but I think we are biased. We all three are biased because we are completely at I wouldn't say at the tip of the iceberg but close to the top. Unfortunately I don't know but really in the real world most companies, they're stuck with Java 8. Perhaps half of them might already use containers. The other half don't. That's not completely true. We are looking at the conferences folks. We are getting used to this but if you get back to do consulting for the company close to you, you are going to notice that we are light years ahead of their time.

Viktor: [00:21:50]
I'm so so happy that you mentioned that because that's one of the arguments I often have with people. People tell me Hey majority is using Kubernetes. Absolutely not. Maybe 5% is using Kubernetes. Majority of companies have something in Kubernetes, which is not the same thing as majority is using it. I would even say that probably more than half of workloads are not even containers. Not even getting into no half, I mean third, quarter, right?

Nicolas: [00:22:19]
I try to do some consulting still a couple of days here and there just to keep in touch with reality. Previous summer, I was doing a code and architecture review. You cannot imagine the real world. The logging was completely non-existent. I have no clue how they could actually debug when a failure happened in production. Of course no containers. No nothing. You would have died, really from the shock. We must realize that yeah by going to conferences, checking how the latest advancement in technologies makes us a bit out of touch with the real world. The real world as you mentioned third of containers fourth I'm not even sure. We might be interested in the results. If you go to Spain to a big city with big customers and you check how they are doing, well probably you will find perhaps half. In Switzerland, for example, I'm probably sure it's less.

Viktor: [00:23:26]
Yeah I wouldn't even say half. The bigger the company is, the more legacy they have. That's the golden rule.

Nicolas: [00:23:33]
Yeah, correct. Also, the more resources they have to put into new stuff. There might be a bigger chance that they have somebody who wants to make things go forward and to shake the tree. I think that's perhaps the bigger the company, the higher the chance that at least in some team, in some places, there is somebody doing that kind of stuff.

Viktor: [00:23:55]
Exactly, Exactly. Cause for them, having a team of 10 people that is experimenting with service mesh is nobody even notices that expense but then everybody forgets that there is 9,990 people who are not there yet in that same company.

Darin: [00:24:14]
As we start wrapping up today, you've gone from minikube to Kind. You've hated on Gradle because Maven is the truth which I disagree with and we wrapped back around that Kubernetes is not running everywhere. So if Kubernetes is not running everywhere, why are you even using Kubernetes in your demos?

Nicolas: [00:24:37]
Because I'm part of the conference circuit. I need to use the tools that the cool people are using if I want to make it into the conference. Now imagine today I want to make a token Hey you are using Java 8 and you want to move from Ant to Maven. Nobody is ever going to listen to such a talk. No conference committee is even thinking about accepting such talk. That's not what people want to listen to. People who go to conferences, they want to hear the latest news. That's very interesting. We are in an industry that changes very very often. Changes like Kubernetes five years ago was nothing, but most people they are not really interested in those changes. They just want to work from nine to five. Do their job. Get their pay and that's perfectly fine. The conference circuits, the podcast, the blog posts, they don't cater to that crowd. The interest the people who actually want to listen want to change. My audience today is not the people who run on Java 8 and are happy to run on Ant. Whether it's from a professional point of view or my own personal blog post. They are the people who are interested in how things are moving. So that's the reason why like two years ago I started to teach myself minikube and Kubernetes.

Viktor: [00:26:08]
That's normal because people running Java 8, they either already know everything there is to know about Java 8 when there is nothing for you to tell them or they don't but that's by choice, so they're not going to change their choice. If you're running Java 8 for 10 years you either choose not to know more than you should or you already do know everything. That's almost life in every area. When you look at a show about cars, you're not going to look at them present a car that you already have. You're completely uninterested $15,000 car that is 10 years old because you have it. No. You're interested in Tesla, even though you're most likely not going to buy it.

Nicolas: [00:26:53]
That's true. That's correct.

Darin: [00:26:54]
So now we have completely devolved technical conferences down to a car show.

Viktor: [00:27:01]
Yeah, you want to see what is the latest and greatest and most of us cannot afford latest and greatest but Hey you want to see where the industry is going or where people are going and it makes you feel better because you feel Hey one day I'm going to buy Tesla.

Darin: [00:27:18]
After it's been on a used car lot three times, you'll buy it in 15 years. That does sound like our industry.

Nicolas: [00:27:25]
Well actually, I'm very risk adverse and I prefer at least when I was a consultant I prefer when other companies, teams, organizations, try a technology and first got burned and got experience and could tell about it than to do that myself. I'm never at the tip of the iceberg. I'm just like well well before that and just wait for other people to first get burned before me.

Viktor: [00:27:58]
There is also that thing about investment. If we go back to our previous examples and I would like to see whether you agree. Let's say that you have 10% big company 10% or 5% of that company is into experimenting with something new and vast majority is in something old like mainframe whatever, but that 5% or 10% is actually getting 90% of the budget because companies actually buying licenses for Kubernetes, not buying new licenses for mainframe. So financially actually that small number of people is usually getting much bigger investment than the large number of people in the company.

Nicolas: [00:28:40]
Well that's probably because you never paid the maintenance costs of a mainframe.

Viktor: [00:28:45]
Yeah I meant new investment, just to clarify. I meant where are we going to allocate those extra X of the budget. Of course, maintenance is extremely expensive, but when you think about let's say which new tool we are going to invest in? We have 10 million extra. We want to invest in some new tools. You're most likely not going to buy new tools for mainframe.

Nicolas: [00:29:08]
That's true but if you've got 10 millions extra like that and you just want new ideas on where to invest them, I want to know the company you are working for because I'm very interested. I have a lot of ideas to invest 10

Viktor: [00:29:23]
Split it

Nicolas: [00:29:25]
Okay Okay. Six for you. Four for me. It would be good enough since you designed the idea in the first place

Darin: [00:29:33]
So Nicolas if people have that extra 10 million and they're ready to talk to you, how can they get in touch with you?

Nicolas: [00:29:40]
They can reach me on Twitter and Nicholas. So yeah it's I'm French so it's written with an S but it's not pronounced So it's N I C O L A S at Frankel F R A N K E L dot C H or yeah that's the easiest way to reach me on Twitter. Otherwise I will receive too many LinkedIn invites. I already receive too many so please don't send me any LinkedIn invite unless you write down why you want to connect. Yeah Twitter is good. Twitter is fine. Twitter is perfect.

Darin: [00:30:19]
And we will link to all of Nicholas' So this is weird I am saying to S when I'm making it possessive but it doesn't sound right after you've been correcting me already today trying to make it make sense. The links to all of Nicholas' information is going to be down in the show notes, Nicholas, thanks for hanging out with us today.

Nicolas: [00:30:44]
Thanks for your kind invite and now I'm really, really eager to know which title, how, how the, you will name this podcast episode, because I think we were going in many different directions, so I will be waiting impatiently to know about it. Thanks Viktor. Thanks Darin. Have a good day.

Darin:
We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and the link to the Slack workspace are at https://www.devopsparadox.com/contact. If you subscribe through Apple Podcasts, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at https://www.devopsparadox.com/ to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox.

DOP 101: What to Do When Technology Fails

Show Notes

Guests

Nicolas Frankel

Hosts

Darin Pope

Viktor Farcic

Links

Rate, Review, & Subscribe on Apple Podcasts

Signup to receive an email when new content is released

Transcript