DOP 48: Regulations Aren't An Excuse For Not Doing The Right Thing

Transcript

Darin Pope 0:00
This is episode number 48 of DevOps Paradox with Darin Pope and Viktor Farcic. I am Darin.

Viktor Farcic 0:06
And I am Viktor.

Darin Pope 0:07
Viktor, we actually got a request from one of the people that we work with like, Man, you guys never really have any guests. And I was like, well, we do have guests. She was like, Oh, I probably just never listened to them.

Viktor Farcic 0:18
Because nobody is worthy enough to join this podcast.

Darin Pope 0:23
But yes, but I can't believe I just said yes to that. That was a slip. But today we do have guests. Viktor, do you want to introduce our guests, please?

Viktor Farcic 0:34
Yes, those are two guys. Actually, one of the two guests I met because he promised he's gonna write a chapter in my book. And I'm still waiting. That's like three years later.

Darin Pope 0:46
You mean the book you've already taken off for sale?

Viktor Farcic 0:50
Yeah, exactly. The one that is already deprecated and not in use anymore. Exactly. that one, that one. So I'm hoping that bringing him to the podcast might actually help speed up the process.

Tigran Mnatsakanyan 1:03
That's me.

Darin Pope 1:05
And who is me?

Tigran Mnatsakanyan 1:07
That's me Tigran.

Viktor Farcic 1:10
introduce yourself man.

Darin Pope 1:12
Yeah, introduce. Yeah, you must have a last name unless you're a Beyonce. I don't know. Do you only have one name?

Tigran Mnatsakanyan 1:16
Last name is pretty long. It's almost like an exercise for linguistics exercise. Mnatsakanyan. I am DevOps.

Darin Pope 1:31
You are DevOps

Tigran Mnatsakanyan 1:32
I am DevOps. I'm an engineer and that they call me DevOps depending on the contract date. They call me different things, including DevOps. And I met Viktor, I think, five or five years ago. And at the end, mostly it was online afterwards, but we did met a few times offline as well. So yeah.

Darin Pope 1:57
We have one we have one other person sorry.

Tigran Mnatsakanyan 2:00
Yeah, Viktor, go ahead with another person.

Viktor Farcic 2:04
Other person can say his name. That's okay.

Roger Day 2:07
Hey guys, it's great to be on the DevOps Paradox. I'm Roger Day. Most of my career I've been a programmer. I identify as a Scala developer. But the last couple of years I've been doing YAML and Groovy and Jenkinsfiles. YAML for Kubernetes and that's how I met Tigran and Viktor. Tigran "DevOps" Mnatsakanyan.

Darin Pope 2:29
I'm so sorry that you're doing YAML. That's just like, it's not today's COBOL.

Viktor Farcic 2:37
But that's kind of increasing right? More and more, I meet more and more YAML developers. Like I know how to write YAML.

Darin Pope 2:45
It's not a language.

Viktor Farcic 2:47
I know but there's a lot of lines to write. Somebody needs to do it. It's new Word document, I guess.

Tigran Mnatsakanyan 2:55
I guess it's because there is no autocomplete. There's lots of things to remember. That makes it a different career path to remember all the syntax for the YAML and make it correct.

Roger Day 3:09
Well, it's its lowest common denominator. It doesn't matter if you're a Ruby guy or C# guy, you can deal with YAML. So, you know, if you're trying to if you're trying to get people to adopt your Kubernetes platform, then YAML is a safe bet, right? At least it's not JSON.

Darin Pope 3:25
I could disagree. I think JSON would probably be a better solution, because people can't get tabs or spaces, right?

Tigran Mnatsakanyan 3:32
Yeah. Yeah.

Darin Pope 3:34
At least with JSON, I can screw that up.

Tigran Mnatsakanyan 3:38
Yeah.

Roger Day 3:41
I struggle. I struggle with the with the quotes and then having to escape the quotes and escape the escapes and so forth.

Darin Pope 3:47
I didn't say was perfect. It's just better than tabs and spaces. But let's not talk about YAML and JSON because that's well, we'll probably end up talking about them anyway. But we're going to talk about today, in general, both of you guys are, I'm assuming hired guns. We haven't talked before we started recording today. Is that a fair assessment?

Roger Day 4:08
Yep.

Darin Pope 4:09
Okay.

Tigran Mnatsakanyan 4:10
Yeah.

Roger Day 4:11
mercenaries

Darin Pope 4:12
Mercenaries. Okay. There's there's a name for that actually over in the states in the Washington DC area. It was called the Beltway bandits back in the 70s. And 80s. It was just crazy money. I'm sure you're not making crazy money, but you're probably making good money because you guys primarily work in financial services, right?

Roger Day 4:32
That's correct.

Viktor Farcic 4:33
Oh, we are going from YAML to something even more depressing now.

Darin Pope 4:37
Yes. So yeah, that's actually a good segue of how bad is DevOps and development and everything else that's associated with your contract and stuff in financial services?

Roger Day 4:55
So I think it's a mixed bag. When you have companies those size, and I've worked for a large American bank in the past, and they had, I think, a quarter of a million employees if you include the retail stuff. So you could ask any question. You could ask, do they use Kubernetes? Do they use Docker? Do they do agile? Whatever the question is, the answer is yes in an organization that size there's, there are pockets of excellence. Though I the thing I'm interested in actually finding out from, from you guys who might have seen a wider range of companies is what is the difference between the financial sector and other large companies, which are not in the financial sector? I mean, I assume there's a bit more regulation. And the banks used to have, before the financial crash in 2008, the banks used to have money and they used to do innovation, and they used to compete with the tech companies for the top talent. I'm not sure that that's still the case.

Viktor Farcic 5:48
I know the answer. I know the answer.

Darin Pope 5:50
Let's see if your answer and my answer sound the same.

Viktor Farcic 5:55
Can I do it? Yeah. Okay.

Darin Pope 5:56
Go ahead.

Viktor Farcic 5:57
Yeah. The difference between financial institution and non-financial institution is that the financial institutions have a good excuse not to do stuff.

Tigran Mnatsakanyan 6:03
correct.

Viktor Farcic 6:05
They can hide behind the policies and other things that officially prevent them from doing things. So so you have an excuse kind of like Yes, I would like to do what the rest of the world is doing. But you know, there is this policy

Roger Day 6:21
But we're a bank. We're regulated. That's what they say, we're not Amazon, Roger, we're not Amazon.

Viktor Farcic 6:28
And I didn't know what the regulation is. I don't know what it is, but I know that it prevents me from doing the right thing. Is that a good explanation?

Tigran Mnatsakanyan 6:36
Yeah. There was, I think that excuse is is is hard to, to use nowadays, because there are unicorn banks now which are working in the same regulation or almost in the same regulations, especially in retail. There are very good examples. And so those excuses I think are more or less deprecated. But there it was a yes. You're absolutely right. There was a biggest excuse saying we are a bank. We can't be. We can't be like Google. I was I was working in a FinTech five or six years ago. It was a good idea. It was it was medium or small company. But the way the technology was working, it was artificially bubbled up, like something big. So instead of instead of RESTful services, it was artificial SOAP services because it's complicated. It's more banky. It's more it's it is more XML. And the excuse was, yes, we need to act like a big institution. We need because we are working with banks. Everything should be like a bank, the management layers, endless layers of management should be like a bank. Technology should be like a bank. Everything was bank oriented. Even the source code was bank oriented, which was wrong. And then and there back then the excuse was working very well. Well, we are we are regulated industry, everything should be slower and there should be less innovation there should be less automation. I think now it's slowly disappearing because the small unicorns are taking markets from from large, large, large banks. And me and Roger can see that internally, from the clients we're working with now.

Viktor Farcic 8:36
To me, financial sector in London is especially interesting, because London probably has more FinTech companies than any other place in the world. And the large large banks are probably under a bigger pressure in London than anywhere else. But at least that's my impression. So I would guess, and I'm not there as much as you guys that all those fintechs must be creating a more positive effect in London among big banks than than anywhere else.

Roger Day 9:15
I think so. I think it's a mixed bag. There's the banks are under some of them are under cost pressure. And they are looking they've been they've been heavily engaged in outsourcing and offshoring and so forth for years and years. And then as you say, you have you have and it's not it's not just financial services, there are lots of, you know, that sort of second phase startup where where they are starting to compete with, with bank salaries, and even at a small discount as a developer you think, well, I could go and work for an actual tech company where you know, where I don't have to fill out forms to run my container. So that I think that aspect of it is interesting, but I don't think I don't think that's that's unique to London. I I've tried to hire developers to work in banks in New York. I think I think in the US, there's been pressure from Google and Facebook and Amazon and lots of other companies for a long time. I think I think London is starting to experience that. I think the banks have had it their own way there. But the other thing going going back to the you know, we use the word excuse, and I was saying that the people who say these things to me who say, Roger, you know, what are you talking about? We're not Amazon. We can't do this. The thing is, it's not speculation the jury is no longer right. The this has been studied. And I'm a huge fan of the Accelerate the science of DevOps, a book by Nicole Forsgren and Gene Kim, Jez Humble, who are obviously, you know, well known names in this space. And that, you know, she's a statistician, and this stuff is rigorous. It's peer reviewed. And one of the things it said is that there is no evidence that that regulation means you can't be good at this stuff. And they call out these 24 capabilities. And it's a mix of culture of leadership of bureaucracy and the normal technical things you would expect. It's kind of funny, I think because someone's gone written down and said, Look, here are 24 things that companies do and the ones who do them make more money and have happier employees. So I, I sort of wonder why people don't just do those things. Are people talking about that? And what you know, when you guys go on your travels, are you are you hearing them talking about Accelerate, these these 24 capabilities, Western organizational culture, and so forth.

Viktor Farcic 11:25
One of the problems I believe, is that the bigger the company, the more closed it is. The less it looks for external resources for information and practices. So kind of normally, those big companies, they tend to have everything in their wiki. So if it's not in their own wiki, then it doesn't exist. So Accelerate does not really penetrate. And I'm saying Accelerate as example. It can apply to anything else. External practices are not really coming in because it's a very closed environment. Kind of we have all the knowledge I need for my work is internal. Why would I go outside?

Darin Pope 12:07
That's very common too. It's rare that I hear anybody. In fact, 10 years in, I'm still amazed at the number of people have never heard of Phoenix project.

Roger Day 12:19
All right.

Darin Pope 12:20
Yeah, that's just it's, it's people like, well, I never heard of it. It's like 10 years old. Okay. What's it about? Then I said, it's your company. I usually try to say it a little nicer than that but not a whole lot. I want to go back just for a second because there was a phrase I don't remember who said it. Roger I think it was you. It was interesting. You said pockets of excellence.

Roger Day 12:46
Mm hmm.

Darin Pope 12:47
A lot of companies, you know, pride themselves on being having Centers of Excellence or some other phrasing of that. But the concept of a pocket of excellence. Let's dive into that just a little bit more.

Roger Day 13:00
Yeah, okay, so I think I think you said a center of excellence is a public thing you say you call yourself a center of excellence. It's been identified as I understand it in the research as an anti pattern. So you say, We're the Cloud Center of Excellence, and we will teach you how to do cloud. What What I mean by a pocket of excellence is someone senior enough to hire a few teams, and to build relationships across some of the silos and hire maybe 20 people 100 people and do things well. Hire good people, give them enough air cover that they can they can be programmers that they can focus on delivering useful software frequently to the people who are trying to run the business.

Darin Pope 13:45
So basically, a good pocket of excellence is run some run by somebody that's doing nothing but running interference all the time. And it's like, just go get it done. But in a good way, in a good way of saying go get it done.

Roger Day 13:59
Yeah. in these in these big org and these big organizations if you've got to have that you've got to have someone senior to protect you from the from the machine. Like for example, this guy this guy was able to was senior enough to that I worked for was senior enough to go against company policy that that he could ignore the fact we weren't allowed to run virtual machines. So we were running CentOS on our our huge surfboard shaped laptops. And it was a hiring advantage. I was doing the hiring at the time, and we would walk people in and I once I figured out what was happening. I did deliberately, I would I would walk people to the interview room, and I'm taking a winding path through all of the desks. So they would see they would see our build monitors. So they would see the Linux desktop. So they would see the monitoring system we'd setup. And then by the time they got to the to the interview room, they were saying you got it you guys use Linux? You're allowed to do that? Yeah, yeah. It's a VM. I'm sorry, it's not perfect, but you know, that really helped me hiring. You get it you get inertia then like good people, produce good people, and so on. It's good.

Viktor Farcic 15:00
From what I've seen in practice, not necessarily in theory, Center of Excellence is usually yet another department that is created by the Corporation, and follows the same rules and as anybody else, except that they call themselves Center of Excellence and they have some mission, but they have to follow the same rules, they have the same restrictions and so on and so forth in practice. While, what you call Roger, pockets of excellence, that is usually somebody going rogue, somebody going wild and saying, I'm not gonna follow those rules and I have sufficient authority and maybe budget and what so not to actually bring couple of people over to my side, but I'm out out of this company. I'm not following your rules type of stuff.

Roger Day 15:47
Or you don't you don't you don't you don't you don't talk about it. You just say okay, within this group with these, this is how we're going to do things you're not you're not making it public. You're not saying we're going to change the rest of the organization and we're going to have milestones and you know, That's not your job. A Center of Excellence I think I think your job is to change the org.

Darin Pope 16:05
Yeah, whereas a pocket a pocket of excellence is sort of like Fight Club. You don't talk about Fight Club.

Viktor Farcic 16:11
Exactly

Darin Pope 16:12
You just fight. it basically means you're just getting the stuff done. While everybody else is having a bunch of ceremony around you, all of a sudden, things are happening because of this pocket,

Roger Day 16:24
I think I think the pockets have a short TTL as well. I've learned that in two places.

Darin Pope 16:29
Well, is it is it short because they're in and out or is it short because it gets stomped squashed out.

Tigran Mnatsakanyan 16:36
I think budgets are changing quite often and and when the change comes then then it is affecting all the pockets. So if it can be that the priority is not delivering software anymore anymore. priority is saving money and that can be pushed everywhere. And when this kind of priority is is a thing everyone is suffering. It's genuinely just changing everything just to save money It doesn't matter if as a result of saving money or losing money next year that's next year that's just backing off the problem but if within a year you are saving money you are showing that you saved more money on engineering on travel on on good engineers now you have less good engineers so some work would be moved to offshore then you have done your part and everyone is happy and then the next year, yet another digital transformation will start. It will never end. The cycle is I think what what I saw is cycle is two years to start some good initiative and then it's it's being killed anyway. I never a good result good delivery in. on that scale of, I don't know, some center of excellence. I never saw that in my life.

Darin Pope 18:11
So if you've not seen it and you have the magic wand, what should it look like?

Tigran Mnatsakanyan 18:19
Yeah, that's a good question. I, I never under okay. There are technological problems that are there is there is bureaucracy and politics of course, but I never can understand the problems with When, when, when it comes to money, restricting money on technology. I think that's, that's the fundamental problem when when someone is trying to save money and they are going to take to save that money on technology and on on people. I think that will be the number one thing to change to acknowledge that software is not number one place to go to save money. In fact, software can earn you more money if you do it correct. Usually the cut where things are stopped and Okay, everything is closed projects that go is when when the freeze on the money comes. I think that's what that's what I will reconsider. There should be some algorithm or some coaching to people who are taking those decisions that because they are so detached from technology in the real problem problems Well, maybe I am detached maybe I can. I don't see that way as they see it. But usually they go after reducing headcount, reducing headcount, offshore, offshore, and then that that's obviously yeah, exploding in many ways.

Viktor Farcic 19:56
Fundamental problem there I think is that software is still perceived in quite a few companies as a cost, and when you want to save money and everybody wants to save money and everybody has to save money every once in a while. That's normal, but you always go for departments that are cost not not instead of departments that are making money. And that's the problem I have with financial institutions. T hey're no doesn't matter what they say officially and publicly but in practice, they did not realize yet that software is their business. If they would realize that then they wouldn't go for whomever is cheaper per hour type of way of saving money because I'm pretty sure that for the for the management of the bank itself, they don't replace their CEO with cheaper CEO to save money. They don't do that. Because CEO can earn you money. But for software, yes, that's that's kind of ok, it's a cost. We don't earn any money from it. So why not save?

Tigran Mnatsakanyan 21:02
Yeah, but it does the story has two sides, right? Because their experience from their experience, it's not all. It's not all software is a good software for them. Because I saw it myself as well. There was digital transform, you know, every everything now is called digital transformation. So, so, one transformation when they hired I think three or 400 people without having a good plan what these 400 people are going to do it was just like, Okay, this is new department, new laptops, new management. Let's call everything agile. Let's hire these agile coaches that will be coaching these 400 people in the business and they will be bridging gap between clients. All these buzzwords, but there was no plan what to do. So those people 400 people were inventing a work for themselves. And there was as Roger is calling pocket pockets of excellence. There was people like alright, 10 people are doing dashboard for Jenkins for two years and other 10 people invented okay that would then we can connect that dashboards to some some stream of data. There was it was almost like a dotcom bubble. Tt was useless work within 400 people and everyone hiding that it is useless because it's a job they do. They had been hired and then the product people are saying Yeah, yeah, we are we are doing the good thing. Engineers are saying, Oh, yeah, our code is pretty good. It is unit tested. But then at the end, they found out the whole digital transformation was just a bubble. Everyone was just doing something. And all the layers were just doing this illusion of illusion of good illusion of DevOps, automation and agile, but with zero value. So they just obviously at the end, someone just killed all the project and

Roger Day 22:50
a little bit cargo cultish like, we

Tigran Mnatsakanyan 22:54
They saw that from both sides so that they add this question mark to anything about the technology. Because they saw that people are faking a lot. And and I think lie is the fundamental problem in those kind of big orgs. And because they lied so much to each other, and it's very hard for them to find out is it actually worth pushing forward or is it just another fake bubble in our org. I don't know what Roger. Roger would you agree? Have you seen those kind of things?

Roger Day 23:27
Absolutely. Yeah. Yeah. Yeah. I've been thinking about the difference between you know, because some of the tech companies not as huge they have large numbers of staff they have more money than the banks and the you know, the difference is perhaps that they are tech companies. So they they don't view IT as a cost. The view it as an asset. They've built their success on being better than other people with you know, with software, with IT. Like there's a there's a story everybody talks about Blockbuster and Netflix, but there's there's a story I heard about them and I don't know if it's true but but it's really interesting. And it goes that there was a point in time after Netflix had made their pitch to Blockbuster, saying, you know, we'll work with you and we'll distribute the DVDs and envelopes and you won't have any, you won't have any late fees and Blockbuster went, now hold on, we make a couple of million a year in late fees. We quite like them, and then laughed them out the door. There was a point after that when when Blockbuster actually had really good IT. They had they had lots of money. They were doing really well. And they hired a great IT department. And there was a brief point in time when they had a better streaming service than Netflix. But management didn't understand it. They weren't techies. I mean this was this was back when smartphones I guess were relatively new. They didn't they didn't understand that all the young people wanted to do everything on a smartphone or everything on a laptop. They just didn't prioritize it and it just went away. And I think i think that i think that experience if that you know the pockets of excellence, I think I think in a way, that's what's happened to them. The management don't have don't understand them don't understand the potential and don't care.

Viktor Farcic 25:07
It is definitely not understanding the potential, but also sometimes many attempts of advanced teams fail because they're too early. I don't know for Blockbuster, but for all I know, maybe what Blockbuster was doing was too early at that time. At least I've seen in other areas that Yeah, we have a, we mentioned that, like Heroku Heroku is a good example of something that in my opinion is amazing. Fantastic. It was never seen before. Too early to make business at that time. Basically, Heroku gave us what Kubernetes is trying to give us right now. Not in that scope, not that scale, but that simplicity that Docker Kubernetes gives gave us we got with Heroku. And where is Heroku right now? So it's also about finding the right moment to do something.

Darin Pope 26:02
Let's let that be the stopping point, I think, because people are getting sad when we talk about Blockbuster and Heroku. Let's, let's end on up thing. So one of the good things out of today is pockets of excellence. So if you find yourself working in a pocket of excellence, be happy. If you find yourself working in a center of excellence, run for your life. That's my, I'm not speaking for everybody else, but that's my thinking anyway.

Roger Day 26:34
I think that's good advice.

Darin Pope 26:35
I think one of the things that I think was Roger, you said this, it's like Center of Excellence is just yet another department. It doesn't really mean anything. It's just like another training internal training department or something. It's like, hey, let's do things this way. Alright, so again, I'm bringing us down. Let's be happy again. Unicorns and rainbows, unicorns and rainbows.

Roger Day 26:57
Pockets of excellence

Darin Pope 27:00
What?

Roger Day 27:01
Pockets of excellence

Darin Pope 27:02
Pockets of excellence. Pockets of excellence.

Roger Day 27:05
Unicorns have pockets

Darin Pope 27:07
Unicorns have pockets. What if I could draw draw something? Go ahead.

Viktor Farcic 27:13
The expression I'd like the most that was always in my head kind of silly and never working was when I was hearing it was very popular in the past to say islands of happiness in a company

Tigran Mnatsakanyan 27:27
islands

Viktor Farcic 27:28
that's how those teams were called right?

Roger Day 27:30
Yes, yes,

Viktor Farcic 27:31
This is a horrible company, but there are there are those few islands over there where people are happy.

Darin Pope 27:37
The continents are horrible, but the islands are wonderful. Yeah. Okay. All right. So right now if you're listening via Apple podcast, please subscribe and leave a rating and review. All of our contact information including Twitter and LinkedIn can be found at https://www.devopsparadox.com/contact. If you'd like to be notified by email when a new episode is released, you can sign up at https://www.devopsparadox.com/. The signup forms at the top of every page. And finally, there are links to the Slack workspace, the Voxer account, to leave us a question, please leave us a question. And how to leave a review are all in the in the...1,2,3...it's easy for me to say, isn't it...in the description of this episode. Now finally, we've had good pockets of excellence today. We have islands of happiness. We have unicorns. How is it that you're working in financial services again?

Roger Day 28:30
We live in London.

Darin Pope 28:31
Oh, you live in London?

Tigran Mnatsakanyan 28:34
Yeah, yeah. It's trendy here.

Darin Pope 28:37
Its trendy. Yeah, I understand again, in the town I live in it's a banking town. So that's if you don't work for the banks, you you pretty much don't work for anybody.

Viktor Farcic 28:47
No, if you don't work for a bank, you work remote.

Darin Pope 28:50
You work remote.

Roger Day 28:52
But it's changing. It's changing. The banks brought developers as a critical mass now there are startups, there's FinTech and now there are other kinds of startups. London is becoming a city of tech. Well, we'll see if that holds over the next few years. If we don't all get COVID-19.

Darin Pope 29:09
So it's no we're not talking about that. So it's March 2020. So we'll see in two years if we're still if London still a financial services town. Okay. Okay, guys, thanks for hanging out. We'll also have the links down to your Twitter's tweets, your Twitter, your Twitter's, Twitter's, your LinkedIn, whatever you want, just send me some information. I'll put it in the show notes. So if you want to follow Tigran or Roger, go do that. Anything else final Viktor, you don't get to talk. Tigran. Anything else to say?

Tigran Mnatsakanyan 29:43
Thanks for inviting. That was a good chat.

Darin Pope 29:45
Yep. Roger, anything?

Roger Day 29:47
Yeah. Thank you very much privileged to be on the DevOps Paradox.

Darin Pope 29:52
Privilege? I think that's the first time anybody's ever said privilege.

Viktor Farcic 29:56
That's new. That's new. I like that guy. We need to invite him again.

Darin Pope 30:03
And to you listening today, thanks again for listening to episode number 48 of DevOps Paradox

DOP 48: Regulations Aren't an Excuse for Not Doing the Right Thing

Show Notes

Guests

Tigran Mnatsakanyan

Roger Day

Hosts

Darin Pope

Viktor Farcic

Links

Rate, Review, & Subscribe on Apple Podcasts

Signup to receive an email when new content is released

Transcript