The Confusion of DevOps

After spending a long enough time on /r/Sysadmin, Twitter or in professional Sysadmin communities like LISA & LOPSA, invariably, someone comes across the term DevOps. Here, we will try to demystify this term, cover what it means to the relevant parties & what takeaways there are to be gained so that you can have a better understanding of this concept when people mention it in the workplace.

My first experience with this term was on Reddit & more recently at LOPSA-East a few years ago where, a few System Administrators were talking about their environments & sharing how much of a struggle it is to deal with developers — who, by their nature, either request root on their machines, or spin up their own servers, much to the chagrin of System Administrators.

At the time, I was somewhat confused, as, I didn’t really see the point: Sysadmins use code & they wear many hats. Is that DevOps? Am I a DevOp? Is it a concept? I started to doubt myself & my future in this field. Titles & jobs would pop up commanding ridiculous (e.g. overly inflated) salaries for a ”DevOp” person when, the requirements were standard Sysadmin duties (things we can do with our eyes closed) with some programmer mumbo-jumbo (being comfortable with Ruby, PHP, Docker, Unit Tests, Agile, Configuration Management)

Further, the idea that developers should be in control of things like configuration management to push out changes to infrastructure was asinine to me (it still is). So, I decided to do some research. Breaking down the term, it is generally accepted combination of terms “Software Developer” and “IT Operations” (or Ops)

The confusion should be readily apparent. For someone with a Sysadmin background, this poses a conundrum: why do we need or care for Dev? A Sysadmin (as I’m now learning) should be fairly comfortable at scripting & know how to code at a very basic level – that is to say, be able to read a script & understand what it may be doing.

In a smaller shop where the company in question may not deal with the IT field or industry, they may not have a development team. Non-profits & companies that produce physical goods in some manner are good examples of where DevOps may not be entirely relevant, unless there is a developer team for their website. But in places where you are designing a product or service, either in house to be used by outside users; or developing a platform for scaling applications? DevOps is essential. Let’s look at these two different perspectives, offering myself as an example.

In my first ”real” Sysadmin job, I was a Sysadmin of a small office. There was an IT Director & one other Sysadmin at our HQ. We had no developers, because we didn’t produce any software internally for ourselves or for end users. We are a shop that had no need for a developer team. The Sysadmins took care of everything, as they should.

In a different job as System Administrator, I work with a ticketing system that is coded by a team of developers internally.  The developers control the hardware & the code that runs our infrastructure internally & externally facing. The Sysadmins don’t do any Sysadmin work. We’re just button pushers. There is a literal glass wall between the teams.

In, what I would consider an ideal version of this scenario, the developers would make changes to their code, run unit tests & push it out to production. If they needed hardware, root access to equipment, or additional servers, etc. they would engage the Sysadmin team. This isn’t the case & it makes no sense. Some key developers have root on all systems & make changes that have large effects to our internal ecosystem that change functionality.

For example – a developer may make a code change that causes a function to break. Customers complain. This broken function causes a cascading failure, albeit momentary preventing customers from properly deploying new machines. Assume no change management controls (if there are, we don’t know about it)

A traditional Sysadmin would cringe at the thought of this. However, if you take into consideration a key developer was a Sysadmin first and/or a founder to the organization, would it make a difference?*

Do we need DevOps? Do we benefit from it if there are struggles to an organization internally? I don’t know. Is the person who takes the risk to traverse the glass wall, a DevOp now?

A solid example of applying these Dev & Ops principles is over here — I’ve reached out to the author to see if there are other, good examples of what each team can do to understand the other side. I’ve yet to see such a concise article written about the subject.

Another well written piece on this that I found was here. An individual in the comments seems to have the right idea which seems to coincide with my example/experiences at work. (Reddit thread here) However I would posit that Sysadmins being pushed out of the Operations process entirely & not being trusted (no architecture access, no Puppet access, no router/switch access, etc) is not a way to run an organization.

Unfortunately, the example I explained is a real thing & it happens more & more in shops that develop software products especially. Developers cut out the middle man: they don’t need Operations anymore. They can just grab Docker, run their code & ask Operations to ship it as is, a container with old, crusty, unreliable code. Another fear of DevOps is well amplified in this comment.

If you’re still confused — or don’t necessarily see what I’m getting at, I highly recommend watching the talk “The Future of Sysadmin” by Adam Moskowitz (relevant Reddit thread)– the general concept he describes is identical to the current work environment that I deal with. It shows me, that I need to adapt in some capacity before I die (before I am replaced by a shell script).

The issue, again, somewhat unique to my environment — is that, I am unable, as an individual to effect change where I am unless I make myself a developer/completely different team. I’m sitting here (grumpy Sysadmin), pressing buttons & functions that someone else designed. What if I want to learn how it works? I have no access to that. What if I see something wrong or want to work on implementing something for quality of life for us, internally? Can’t do that unless I am a developer.

If you’re a Sysadmin doing this sort of transition professionally or you understand what I’m getting at, you might find threads like these fascinating, along with the sub-reddit /r/DevOps.

Obviously, the things I mentioned are not relevant to all organizations, applicable to all types of infrastructure or are relevant to each situation. I’m sure I am probably wrong. If I am, or if this post wasn’t clear, let me know. Have you successfully made a transition or are a SRE who plays on both sides? Let me know.

Ideally, I’d love to take some of these concepts & work on creating an environment where people can spin up virtual machines (with some automated tasks) to do malware analysis and/or building a vulnerable lab to emulate SANS/OSCP learning labs. If anyone wants to work on some of this stuff with me, I’d love the help &/or to work along with you on it. I’m already helping one or two people with something similar. 🙂

Thoughts? This is also a useful talk on the subject matter.