DOP 44: Is It Possible To Make On Premise Great Again?

Transcript

Darin Pope 0:00
This is episode number 44 of DevOps Paradox with Darin Pope and Viktor Farcic. I'm Darin.

Viktor Farcic 0:06
And I'm Viktor

Darin Pope 0:08
Viktor's been on the road. We're time traveling again. He's been on the road for two weeks, even though you just heard the podcast last week. So we actually haven't talked for two weeks. And one of the recurring themes that he heard, and I hear a lot is, we can't do cloud, fill in the blank. Everything we do has to be on premise. And in today's world of Kubernetes, I'll say it, that sounds pretty insane.

Viktor Farcic 0:41
So assuming that there is even a remote option to go to cloud, I would never run my Kubernetes on prem, simply because it's hard. Why would they do that? So let's start with why not pay somebody to make sure Kubernetes is running and somebody who knows very well how to do that, like Google or Amazon or Azure, right? Running Kubernetes is not anybody's business or almost anybody's business. Let somebody else take care of that. And on top of that, you don't get all the real benefits of Kubernetes. You cannot scale your cluster when it's on prem. Assuming that you don't have a magic wand and say materialize more servers, right? So it's not scalable. It's complicated and just trouble for no obvious reason. The only reason I know of and we can talk about others later, but the only reason somebody might be able to convince me that cloud is not an option would be Europe because all cloud providers, I mean all big three, are US based and even if their data centers are in Europe, data protection law might not...they might not be able to apply European protection laws because US government theoretically could request that data, even though it's not physically in US. Now, I'm not rambling now between different countries, stuff like that. But that's the only valid reason I heard.

Darin Pope 2:19
I think that's true. And even with whatever happened in California, which is very similar to GDPR, I believe we're going to start seeing more of that in the US. But let's go back one step. We're talking about magically materialize more servers.

Viktor Farcic 2:41
Yeah

Darin Pope 2:41
You sort of said that.

Viktor Farcic 2:43
I did.

Darin Pope 2:44
Now one thing that I've started seeing recently, which it always things tend to happen in clusters. You drive along, you don't see any red cars, and then all of a sudden you see three red cars. One of the clusters I've seen recently, not a bad cluster an interesting cluster, is I'm starting to see OpenStack pop its head back up again. And OpenStack gives you that capability as long as you actually have the underlying capacity.

Viktor Farcic 3:13
So when I say you cannot really scale on premise it's not true because both on prem and any cloud provider have a limited capacity. Nobody can do magic. But I've yet to see somebody setting up clusters in plural, on prem, when they're elastic, and then one cluster gets reduced capacity so that it frees the space for another cluster that needs it more. So you could do it. I just never saw on prem cluster that is truly elastic. Sorry, clusters in plural.

Darin Pope 3:50
Right. And I think the reason why is unless you're somebody that is that doesn't, let's let's go back to the European or the California rules. You can't use cloud because of paranoia. I'll call it that, or legal, which sometimes is the same thing. You would have to have, you would have to employ a certain skill set that is probably more valuable right now than somebody that knows how to program COBOL on the mainframe, because to have somebody that has the skill set, that not only does it in a single data center, but if you're large company, you're going to have multiple data centers. And probably not from a straight DR perspective, but a active active perspective. It's going to be an interesting problem to solve, but the skill sets to solve those problems goes well beyond "can you write hello world?"

Viktor Farcic 4:49
Oh, way beyond. Way beyond. There's a small percentage of people can do that really well. And here comes the problematic part. They're not going to work for you. They're likely not going to work for you. I'm not trying...very often I'm offensive. I recognize that, but this is really now I'm being honest. You know, most likely you're not getting the top talent, because top talent is working in somewhere else, in Amazon and Google and Netflix. So unless you're one of those, you're not gonna get that guy or that people. This is now the part why I like cloud. Actually, you can get those guys if you take it as a service.

Darin Pope 5:31
Absolutely. Because with that, you've lost that risk. Usually the answer back to that is "well, this is proprietary to us. We're special. We need to have all the skills in house in case one of the big in case the cloud provider we pick goes away."

Viktor Farcic 5:51
Look, if Amazon goes away tomorrow, internet is gone. I mean, if Amazon, Google...so if Amazon goes away there are no more servers, so there is no more internet. If Google goes away, there is no search. So there is no more internet, and so on and so forth. So trust me, if one of those guys go away, you have much bigger problems than whether you know how to set up a cluster.

Darin Pope 6:15
Yeah, I mean, there used to be a...now this, this may be completely incorrect. So don't quote me on this. But I remember seeing at one point that Amazon...it was either Amazon or Azure, probably both...installed more servers in a day than most companies will install in years,

Viktor Farcic 6:40
I don't have the data but I would be very surprised if that's not true.

Darin Pope 6:44
Because they're not buying off the shelf servers. They've got custom built servers. They've got custom built racks. I know Azure has their you know, they have all these different cooling and different data center structures, right? That just make things more better. But yet, you know, we'll bring up the CTO episode we had a while back, you've got a CTO that's always worked in big company, and they've never been on the front side of the curve and never really had to try to innovate. I had a client recently say, you know, a lot of the stuff that that we do, we think we're innovating, but really we're just imitating.

Viktor Farcic 7:29
Exactly. And even if you are lucky few who can innovate, and you're probably not, really what is it you want to innovate on? Is your business really infrastructure, so that's where you're going to innovate? You're going to create a better Kubernetes? You're not. Your business is whatever your business is and most likely it is about writing some killer application, not infrastructure. I mean, for some companies, that is a business. That's true. But for most, no. Why would you waste effort and money and everything on something that brings no value at all. Infrastructure is a cost. It's a necessary evil.

Darin Pope 8:13
And the problem is most of the time you can't tell people that that is truth. They're their answers back will be, "oh, it's regulation, we have to have everything on site or it's in the US. It's a HIPAA law, that's a health care issue. Nothing can be in the cloud."

Viktor Farcic 8:27
So some of those things might be true, but more often than not, they're not. It just is, there is a regulation somewhere but nobody knows what it is. It's just that somebody interpreted that regulation long time ago as you cannot be on the cloud or you cannot do this or you must do that. That's rarely the real thing. Or at least that the real regulation. I mean, how many regulations says you need to be... is there a regulation that says that your data center needs to be up to 200 meters from your office? Is that the regulation?

Darin Pope 9:08
I would hope not.

Viktor Farcic 9:10
Least I never heard something like that. So your stuff is running on a server somewhere. And that server can be below your desk, 500 meters from it, or two kilometers or 200 kilometres from you. But it's still it's not below your desk. So it's somewhere. And then I hear those things like...my favorite is "cloud is not secure. You know, they have no idea what they're doing. Those guys have no idea. We know how to make things secure." That's my favorite. That's just so silly. But I cannot even start explaining.

Darin Pope 9:48
I can explain it. Do you want me to try?

Viktor Farcic 9:50
Yeah, go ahead.

Darin Pope 9:52
The way we secure our servers is we don't connect them to a network.

Viktor Farcic 9:57
Oh, yeah, you're right. Or we make it so secure that nobody can do anything, including our developers. Yeah, that's true. That is truly secure. But I doubt that anybody really wants wants it to be so secure that you cannot do anything.

Darin Pope 10:13
The one scenario I have seen, and it was at a client. It was very interesting. was nobody was allowed root access. Once the machine was installed, was stripped away from everyone, including the system administrators. No sudo, no nothing else. Which that's an image or a template, an image, whatever you want to call it.

Viktor Farcic 10:38
Actually, I don't think that's such a bad idea.

Darin Pope 10:41
I think that's actually a great idea because then you can't break the freaking thing and snowflake it.

Viktor Farcic 10:46
The only thing is that what I'm observing in companies is that yeah, nobody has root access, but we are not providing you a service how you can do your stuff without root access. So kind of like I don't know. I go on site and they say, "can you help me solve this? Yes, I can. Give me root access. I will not. Is there a service? Is there any button I can click? No." And then yes, then it's super, super secure. Nobody has root access and nothing is running.

Darin Pope 11:16
But then, gee, doesn't that sound like what the cloud providers give you out of the box?

Viktor Farcic 11:23
That is what cloud providers give you. You don't have root access to their infrastructure. But there is a service to do everything you need to do. And everything happens within seconds.

Darin Pope 11:36
We've talked about this before.

Viktor Farcic 11:37
Yeah.

Darin Pope 11:38
People are trying to recreate the cloud in their four walls without realizing it a lot. They're being imitators again.

Viktor Farcic 11:46
Because that's their job. What happens with them if you move to cloud? What happens with hundreds of people managing current infrastructure in a company?

Darin Pope 11:57
Welcome to McDonald's. Would you like fries with your order?

Viktor Farcic 12:00
There you go.

Darin Pope 12:03
The answer is yes, by the way, always say yes. Okay, so that's we went down a really dark, dreary path there. Let's go to the other side. Let's say there's a legitimate reason you've got to run on premise.

Viktor Farcic 12:17
Okay.

Darin Pope 12:18
Right, we'll say it's GDPR for right now. As much as we're poking fun at everybody else, privacy is a big deal.

Viktor Farcic 12:26
True

Darin Pope 12:27
I'm a very reasonable conspiracy theorist, if you will. I'm not tin foil, but I have some aluminum foil. The way I also look at it is I have a driver's license. I have a passport. The government already knows everything about me anyway. That's another conversation probably,

Viktor Farcic 12:45
If I would have ambition to become president, I would be much more concerned than I am right now.

Darin Pope 12:52
Everybody go start googling Viktor, and you'll find out all the things about Viktor. Okay, so back to the story. Reasonable reason to go on prem.

Viktor Farcic 13:05
It happens, yes

Darin Pope 13:06
and let's keep it scoped to Kubernetes for the moment. Today's world, there are a handful of options, named options. We can say Rancher. We can say OpenShift. Those are probably the two bigs. I can't think of anything else off the top of...well, Docker Enterprise.

Viktor Farcic 13:27
I mean, I think that Docker Enterprise is going away. But I mean, all the traditional players are coming back. After years of ignoring the situation, they're realizing that Kubernetes is here to stay. So I'm seeing now PKS and Oracle and IBM and basically, all those who already have established business with companies are now coming back to those companies saying, "Oh, I'm having a Kubernetes as well. My Kubernetes is better simply because you're already locked into my ecosystem. So why not continue being locked into my ecosystem? So use this.", whatever that is. So all those are going...I mean, VMware is also offering something, whatever it's called. Everybody does. And I think that their influence and their usage is going to increase, but I doubt that that will increase drastically. The only player that really matters in on prem today, that's OpenShift, which happens to be my least favorite option.

Darin Pope 14:38
be nice.

Viktor Farcic 14:39
I'm nice. Look, I just said least favorite. I didn't start saying why.

Darin Pope 14:45
Alright, so I'll ask this question point blank. If you had to do pure bare metal today, not Kubernetes...if you had an application and it was either in go on bare metal/virtual or you're going to put it on OpenShift, which would you pick?

Viktor Farcic 15:02
OpenShift

Darin Pope 15:02
Okay. It's not that bad, it's just your least favorite. It's your least favorite.

Viktor Farcic 15:08
Yeah and it's not because it's better or worse. I mean, we can discuss that as well. But more is because it is the one furthest away from Kubernetes API. That's my problem. It's not about quality. It's about how far or close you are to Kubernetes API.

Darin Pope 15:28
Right. There are some abstractions that seem a little strange. My words not Viktor's.

Viktor Farcic 15:35
It's not even about being strange. It's about being different and the main reason why I like Kubernetes is not because it runs containers, but because I acquired some Kubernetes skills and they're almost equally useful everywhere, except in OpenShift. So I need to invest a lot of time to be able to do the things that I already know how to do, or the other way around. It's more about me being lazy not wanting to learn the same thing twice than about quality.

Darin Pope 16:10
Yeah, I think with the move towards, we'll call it Kubernetes right now, because I don't want to just say containerization because we talked about on the Kube API episode that containers is just one part of what Kubernetes can do for you.

Viktor Farcic 16:28
Yep.

Darin Pope 16:28
But as people move towards Kubernetes these people are having to re-skill themselves or not as the case may be in a lot of places. CTO said we have to do Kubernetes Okay, we're going into Kubernetes I think one of the big things in today's world and I see this every once in a while, not very often. It's like oh, yeah, so we've got to run on prem, so we are just going to use the baseline Kubernetes from the Kubernetes project. We can't afford, we don't want pay for any kind of support. Is that a bad idea or a good idea?

Viktor Farcic 17:05
Paying for support I think is a good idea. I was very much against it in the past, but I've worked with so many big companies that kind of that's peanuts money, compared to how much trouble you can be in without it. So I'm not against support. I'm not even against enterprise something on top of open source. That's all good. Yeah, so why not? I mean, if you're talking about thousands of people working in a company, why not pay support and worry a bit less. In a way, going to cloud is also getting somebody support which is either answering your questions or giving you some service or something like that. So I'm perfectly fine with that, assuming that you are big enough that that makes sense. You know, if you are having a ten people company, you're not going to buy support for everything.

Darin Pope 18:03
Well, you touched on it right there. If you go cloud, a lot of the arguments that we're talking about just sort of magically, either go away or change in a different direction. Now, instead of having to pay somebody to manage my servers, it's just part of the API call.

Viktor Farcic 18:22
But that's kind of I would actually say a big part of the reasons why you want to go cloud is the support. And the reason why it works so well, and you rarely use it, is because they know exactly what you have because it's their service.

Darin Pope 18:40
I'll take it one step further. It's their business.

Viktor Farcic 18:43
Exactly. I mean, you know, its business of when you have a vendor on premise is their business as well. The problem is that when you're on prem, then you're having random things that nobody really knows how to solve from outside your company. So support might be a bit less effective or a lot less effective when you're on prem because whomever is giving you that support doesn't know what you have.

Darin Pope 19:08
Well, I think with the on prem just because you have support, you're still the one operating it

Viktor Farcic 19:15
exactly

Darin Pope 19:15
versus cloud. They're supporting it and they're operating it

Viktor Farcic 19:19
exactly

Darin Pope 19:20
As long as it's real service. If you've taken and built from, we'll use AWS, if you if you've taken and built a bunch of EC2 instances, and you use kubeadm to install Kubernetes, well, number one you did it wrong. Number two, you've put yourself and your company into a corner.

Viktor Farcic 19:44
Yeah, and AWS is not going to help you with that at all. Just to make it clear, because they're going to help you with EC2 instances because that's what you're using. They're not going to help you with your Kubernetes problems because that's out of their SLAs. At the end of the day, why would you even use kubeadm to run Kubernetes cluster in AWS without additional expense, you get the service that they are managing or very small increment. AWS is not even a good example because but let's say Azure or Google. Not only that you're getting it for free actually, running a Kubernetes cluster as a service is cheaper than running your own because you're not paying for control plane.

Darin Pope 20:29
But even in AWS where you're paying for the control plane. It's $150USD a month, roughly.

Viktor Farcic 20:37
Yeah, it's not much

Darin Pope 20:38
it's, it's not much. I mean that won't even buy sodas for your crew during the week.

Viktor Farcic 20:45
The only thing problematic with that is that in AWS, it limits you to which options you have on the table. So for example, in Google, I can say maybe I want to run two huge clusters or maybe it makes more sense for me to have hundreds more clusters. Let's say give each team a cluster. Now in AWS, I would not likely make that decision because of that additional cost of control plane. So I'm incentivized to have shared clusters instead of smaller, individual clusters.

Darin Pope 21:22
Right. Okay, let's come back from cloud and go back to on prem. I put us there and now I'm pulling us back. So we're on prem, we're running a vendor supported version of it. I have my people that I've trained to operate it. They've built up their skills to a point to where they leave.

Viktor Farcic 21:48
They leave the company?

Darin Pope 21:49
They leave the company, because now they can go somewhere else and get more money. Now me as a business owner, I'm stuck. I'm having to rehire and trying to get somebody to help out. Now, at least I've got support. But I'm going to not be kept up to date probably for a long time as I'm onboarding somebody to take care, I mean, that there's these risks, that people not playing the long game just don't see.

Viktor Farcic 22:20
First of all, you would have much bigger problem if people leave that were in charge of something that is very specific to your company. People leaving that were in charge of your Kubernetes stuff, that's relatively easy to replace as long as people want to work in your company and you're willing to pay for that people. And that's one of the big advantages. We all know, or will all know, Kubernetes right? So it's a it's a skill that is acquirable in the job market. I wouldn't be worried about that. And on top of that, I'm much more worried about your people who don't want to leave. That would be the people that worry me. Do you really want to have an employee that decided that he will never leave? Because probably the decision that he will never leave or she is based on "I have no opportunities outside of this company. That's how bad I am. Therefore, I'm never gonna leave." You want people who want to leave.

Darin Pope 23:27
You're harshing my buzz man. No, you want people that...back when I was doing consulting just for myself, the very first thing I did day one, I would sit down with the whoever had bought the service like, hey, look, I am working to get myself out of here as quickly as possible because you need to be self sustaining and move on. I don't think most employees think that way.

Viktor Farcic 23:59
They don't, which is a pity and its a bad strategy and all those things. We've already spoke about it. But yet, you want to increase your knowledge and your skills so that you, you're better at what you do. And when you're better at what you do you get more offers from outside the company you work and you can potentially leave. That's a good thing. Companies should be concerned. How do you create an environment in which people who would normally leave don't?

Darin Pope 24:30
That sounds like great content for another episode. Anything else about on prem? We're not saying on prem is bad. We're saying make sure you can't do cloud before you go on prem.

Viktor Farcic 24:44
A good thing about being on prem and moving to Kubernetes. So what I'm going to say next applies only if you're moving to Kubernetes on prem is that you are moving to operating things through a standard API and going somewhere else shouldn't be a big deal. And sooner or later you will go cloud. Switching to Kubernetes is a good step, on prem or not.

Darin Pope 25:12
because it's a stepping stone, potentially, other than OpenShift. But even OpenShift, you can run OpenShift in cloud.

Viktor Farcic 25:21
Yeah, so you can you can go OpenShift on cloud. That's absolutely not a problem. We can have a separate episode where I would explain how that makes no sense to me. But yes, it can be OpenShift on cloud and you should be fine.

Darin Pope 25:34
So on prem, not completely evil. But

Viktor Farcic 25:38
So let me rephrase that. I don't think that on prem is bad. I think it's brilliant, if you manage to make it great. It's just I've yet to see on prem that is great. That's the problem.

Darin Pope 25:50
And by great, we're defining that as everything's already a service that people just can use as if it's a big 3 right now,

Viktor Farcic 26:01
The definition of anything being great is always yes because when I compare it with something truly great, this is great as well. So what is our reference for greatness in infrastructure? That's Amazon, Google and Azure. That's a reference. So when I say you're doing great, that means that you're doing all the things, or most of the things, that those providers are doing just as good or better, limited to the features that you need, because you will never need everything that Amazon offers, or Azure, right? So that's great. So you as infrastructure department, you're providing just as good service as they are and just as cheap.

Darin Pope 26:41
That's another episode man, too, because people think "Well, how do I do this? So let's go read an ITIL book. Let's go find a book from the 80s." No, just open up aws.amazon.com or go open up cloud.google.com. That's what you need to do.

Viktor Farcic 26:58
You know that you're doing great as infrastructure department in your company if nobody calls you on the phone, nobody opens a Jira ticket, nobody asks you to create something. That it's just happening. You're kind of just creating new services.

Darin Pope 27:16
Let's stop because I'll keep going on there. All right. If you're listening right now via Apple podcast or Spotify or wherever, and you can subscribe, go ahead and subscribe and leave a rating and review. All of our contact information including Twitter and LinkedIn can be found at https://www.devopsparadox.com/contact. And if you'd like to be notified by email when a new episode is released, you can sign up at https://www.devopsparadox.com The sign up form is at the top of every page. There are also links to the Slack workspace, the Voxer account if you want to leave us a voice message for a question or comment and also how to leave a review in the description of this episode. Anything else about on prem?

Viktor Farcic 27:57
You do it if you can. Run away if you can't. Make it as good as it should be.

Darin Pope 28:04
Make it as good as it should be. Make on prem great again?

Viktor Farcic 28:10
Oh, yeah. There you go.

Darin Pope 28:12
I don't know. There's a certain tipping point where on prem pulling things out of cloud and bringing them back on prem makes financial sense.

Viktor Farcic 28:24
Oh, yeah. But you know what the difference is when you pulling from cloud to on prem, then you're going to on prem with a different experience and different expectations. I would encourage everybody to try to go on prem after years being on cloud. But that's a completely different thing.

Darin Pope 28:44
And with that, we'll go ahead and wrap it up. Thanks for listening to episode number 44 of DevOps Paradox

DOP 44: Is It Possible to Make on Premise Great Again?

Show Notes

Hosts

Darin Pope

Viktor Farcic

Links

Rate, Review, & Subscribe on Apple Podcasts

Signup to receive an email when new content is released

Transcript