Getting started in SRE: My SRE Journey

AzureGoogleSREDevOps

As I write this I’m not an SRE.

I’ve been a software developer for years, luck struck me, I was jobless, living in a grotty single room in a cold damp bedsit.

I was jobless.

When I was around 14-16, I played an online game called Ultima Online, my family is poor, so the £10 a month subscription fee, just wasn’t something that we could afford. I ended up finding out that you can update your game to work with free servers. I found Neverlands, it was fantastic, loads of people online, lots of custom development.

My life wasn’t great, this isn’t a woe is me, I used Ultima Online as an escape, I spent 20 hours a day playing this game. So when the server went down, which is did from time to time, I was lost. Friends that I had made in this game introduced me to someone. He had his own server.

Being young and impressionable, joining this server was amazing, even more so as I got given power I was a game master, this meant I could do things players couldn’t.

The server was fresh, almost vanilla, the owner wanted help and I was willing. The server in use was POL which is scripted in a language called eScript. I had never written a line of code before in my life.

But the owner, took me under his wing and mentored me, I worked hard, I learned eScript, I learned the bespoke UI configuration. This is where the inception of my development started.

After Ultima Online, I got hold of books, I read about C, C++, ASM. Eventually I went to college.

Problem with college was, I was still poor, it meant getting the bus everyday which cost around £4 a return. £20 a week, if I can’t afford £10 a month, I certainly can’t afford £80.

The bus was also a problem, I was unskilled, So limited to jobs where you don’t need skills to start, stacking shelves etc. Why was the bus a problem? It stopped at 6pm. These low-skilled jobs are rota based, that was never going to fly.

I didn’t have a bike, I could have walked the 8 miles a day, many have, i’ve read many stories about people who do, so, compared to some, I’m quite lazy.

So I went to college when I could. It in itself was pointless, nothing taught led to where I am, this was a course which featured a module on programming, but it was more drag and drop with classic VB.

College ends, I end up moving in with a girlfriend, getting a temp job. life had many downs during that time.

I’m now 21. Cold damp bedsit, struggling to feed myself on what was then “job seekers allowance” and also pay the top up of the rent, because although I was in a single room, which had no heat, it cost more than the housing benefit I was entitled to.

Every two weeks I went to the job centre. I would be given a grey booklet, fill this out with all the jobs you’ve applied for, I would fill that book every time. with zero responses.

Eventually I cycled onto a special programme, where to get money for food and rent, I had to go and spend all day long in a room hunting for jobs under supervision. This was awful, It doesn’t take 8 hours a day, 5 days a week to apply for a limited pool of a jobs.

So mostly, you sat bored, but one good thing came out of this. A gentleman there seemed to take a shinning to me, perhaps because I genuinely was applying for jobs, I was clean, polite. He forced me to apply for a job I had zero chance for getting.

It was for a senior/lead web designer.

I got an interview…

I went, was told immediately, this isn’t that job, we need a web designer. I didn’t have the experience, I don’t know what they saw, but at the end of the day… they offered me a fixed 6 months contract. It minimum wage. I didn’t qualify for housing help any more, or job seekers.

I was worse off taking this job, but I had to take it. If I refused then no job seekers allowance.

So I got on with it, I went to work every day, I was polite and worked hard, The person who got the lead role was forced to have me in their team.

He tried to get me to design banners for the homepage showing sales, I didn’t know photoshop, I don’t have that designers eye. I failed.

Instead, I spent my days remove the backgrounds from around items of clothing, models, resizing the uploading to the website.

It was mind numbing.

I was get the odd break, the content manager would need help making changes to the html, I knew some html from messing around in my own time. My work was appreciated.

I hated, do the images, it was slow and boring work.

The cataloge team for 99% of the images had already traced paths around the images, I just had to open then, remove the background and resize, save.

This had to be automatable right? Turns out, with photoshop it has a tool which you can use to script with javascript.

Over my lunchtimes, I worked hard on this tool, I didn’t know javascript, so it was challenging. But on the 4th lunch break, I had it working.

Now, when the other designers had finished their designs they would help me do these images, there were hundreds per week, it took most of the week to get it down with 4 of us.

The tool, because it was so fast… 20 minutes. I’d automated myself out of a job. Or had i?

I demonstrated it to my team lead and my team.

My team lead, is a very decent bloke, he told the marketing director what I had done, didn’t try to claim it as his own, I got thanked by the marketing director. It was clear to them I was needed, but I was useless in the design team.

I got a full perm job, as a junior content editor.

Luck got me the job, and luck and hardwork got me a full time job.

I worked hard, but I was bored. Copy/paste of text into html, the odd bit of formatting, eventually a development team was created, they existed 60 miles away. They changed how things worked, the html became xslt to allow more control within the content team on how things should display.

Work had it’s interesting challenges, but I wanted to be a developer, I knew it, I found out they used C#, so every day, would lug a huge C# book with me to work, every lunchtime I would read it, every morning and evening on the train I would read. At home I would, I would do the code samples.

The site would occassionally not work, bugs here and there, there weren’t unit tests, the deployments weren’t CI/CD. Having this basic C# knowledge, allowed me to look at the code, to work out what was wrong, and suggest fixes. I sent these in emails and probably was hated by the dev team.

I was valued, because of how hard I worked, how I would get stuck in and help no matter what. So when I got through an interview for another company, offered a 50% pay raise. The company panicked and wrote me somewhat of a blank cheque.

I said, I want the pay to match AND I want to be a developer, how do we make this happen? I probably could have negociated better pay, I was drastically underpaid at that point in my life. My manager at the time would invite me to the local pub during lunch breaks, I would sit and listen to the conversations and banter with everyone. I didn’t drink.

I couldn’t afford to, with food, rent and train travel, I didn’t have much spare. I certainly couldn’t waste it on beer. He knew, so would buy me a pint, another fantastic person.

I was offered the same pay and a secondment to the development team, I spent 3 days 60 miles away and 2 days on my normal job. I had to proof myself, because after 6 months if I hadn’t I was back to square one.

So I worked hard, continueing to study on the train, except now the train took 90 minutes with a switch in the middle, usually wet and freezing cold, I left at 6am, got to the office at 8:30-ish, one of the developers would pick me up from the train station and take me to the office, he’s still a friend today. I would get home around 8pm.

I was a fish out of water, the chaps were great, knew their onions, but incredibly busy. They didn’t know what to do with me, so just through small issues my way, threw me in the deepend, here’s a bug, work it out and fix it.

I had ups and downs, i made mistakes, serious mistakes, I was tired, I didn’t know if I was doing well. I just kept working hard and learning everything I could.

4 months on, I got an email, It was the dev manager, I was at my normal job, it was a full contract, no more secondment, I was going to be based in the other office full time. It was a bump in pay, I was over the moon!

I found a place to rent, friends helped me move close to the development office, I didn’t drive then, couldn’t afford to.

I had done it, I was a dev, a junior dev. But I had made it.

Luck, lots of it, and hard, hard work.

I didn’t stop working hard, I constantly learned new things, I grew and I outgrew that company. I moved on but the financial crisis happened and the company I went to got rid of the dev team. I ended up being asked to come back to the company I cut my teeth with.

The last words from my manager at the new place, “You are being underpaid here, you are worth so much more, you have no idea how good you are. Do not let them pay you less than you are on now”.

Those words didn’t really sink in, and I didn’t have the gusto to stand my ground, So I went back on the pay I was on at the new place, which was more than I was on before at this place. I found out that If I had pushed, I could have gotten more, much more, I was very valued because of how hard I worked, at that moment, the ball was in my court and I fumbled.

I spent a year there, I was happy, a senior dev got promoted all the way to dev manager, I worked hard, he was supportive. Kept telling me I was a senior level dev, he was going to get me sorted out out, a new title and pay aligned to a senior at that company.

It didn’t happen, somewhere along the road, he changed his tune, I needed to act like “some contractor”, but that wasn’t me. I felt low, every milestone I was given I hit only to be told to jog on. I knew I had to leave, I was still being paid under the average pay for a developer at that company let alone others, I felt I couldn’t grow there, I had no mentors.

My original mentor had left to go contracting a few years back, it was a huge blow, I felt that. The loss of him and the other lead, left a black hole of knowledge. The rest of the team weren’t anywhere near their level.

So I left, when I left my manager said “I always knew you would leave, you never seemed comitted”, that hit hard, I was working 60-70 hour weeks, I was on call and while paid for it, even if I was up all night I was at work for 8am. I couldn’t give more. But also, surely if he felt I was going to leave, why not find a way to keep me? why not get involved early. shrugs

I went onto a few other companies, none really fitted, that company was special, or rather the team were, when I left that time, the developer who used to pick me up from the train station always left, this chap was the heart and soul of the team.

Now, how does this fit to SRE? Well, that’s my start, that’s how I became a developer. While I was a junior, I didn’t know CI/CD, dependency injection, unit testing, I learnt it off my own back, I moved the team from subversion to Mercurial, git was an option, but it wasn’t huge as it is now and it was easier to get yourself into trouble with git than it was with mercurial.

So, mercurial made sense for that team at that time.

I introduced hudson/jenkins, now the team did bring in some contractors years back to do this, they tried with cruise control, and it never worked, it was poorly introduced, poorly implemented.

So, I headed up the CI/CD movement, I pushed for automated UI testing.

The start of my DevOps and SRE journey, I just didn’t know those terms back then, this is 2009-2013 that all of this happened.

Contracting

So, I’ve introduced monitoring, testing, automated ui tests, CI/CD and more to the team, I’ve been an advocate for process improvement, but I couldn’t move up, I was aware I was underpaid. So I left.

I joined another company and I hated it, it wasn’t for me. There was always a weird air, like I didn’t belong, there was no support. It was funny, the dev manager who hired me, really liked me, but within my team, I was the weak link in the chain.

One evening I got a call from my old mentor. He said “You once said to me that you wanted to earn £100k, you always impressed me and I need help, I’m about to start a contract at (some place) and while I could do it alone, I think you could help, do you want to be a contractor”.

So I handed in my notice and started my own Ltd co. The company I provided services for had no dev team, this was very much come in, produce tooling and systems for them to take their business forward. We were successful.

I came to the clients office one day to find out that the contract for me was to end very soon, they had done some checks and there wasn’t enough work to really cover the two of us, so my mentor stayed on, and I went on to contract for the best part of a decade.

During each engagement, I was always involved in improving coding standards, mentoring developers, introducing monitoring, improving/introducing CI/CD, load and capacity testing.

My last contract

During what would turn out to be my final contract covid hit. I was lucky again, I had a client all the way through. Why I’ve been so lucky, I don’t know.

This contract was going well, but at times, work would dry up, I was close with another team who did all of the DevOps bits, So I would pitch in and do the off bit of DevOps, modify an azure devops pipeline, improve monitoring in azure.

I had previously worked with this client and done some load testing on another project, one day, I was asked to help with load testing on this project.

We had some contractual goals to meet, which meant we had to communicate with multiple third parties within a time window and a target message count.

When I started, I was told it wasn’t possible, no one believed we’d be able to hit that mark.

I wrote some JMeter tests and started within a few requests per second a service died, it couldn’t cope with the load, it was a third party, that we hosted.

We didn’t have the luxury of time, so I wrote a tool to generate the json required to populate the cosmos db with all the data needed for the test run. When push came to shove the main test was this communication one, there were others for other parts of the system, but this was the main issue.

So, I loaded the data, and setup the test. It didn’t work, we couldn’t hit the numbers for multiple reasons. I suggested changes, then created proof of concepts of those changes, I improved the monitoring so we knew from our log analytics how many messages went out, how many went to each third party, etc.

Within a few weeks, we had proved the numbers, the changes were implemented into the main develop branch.

I really enjoyed this work, the dev work was boring, so I tried to move over the DevOps, I love the DevOps side it brings me enjoyment.

Being a contractor this move wasn’t going to be easy, at the end of my contract I would reengage with the client, but under a different offer of services.

Before then though, I picked up any DevOps work needed on the projects I was working on.

I found out that the monitoring, approach I was taking to errors (think error budgets), and load/capacity testing were common place under SRE. I quickly dove in deep, read the books, found influencers, youtube videos, and I am still to study SRE as hard as I can.

Now, Covid was still in full force, I have a wife and child, my child hadn’t seen the grand parents, aunts, uncles for over year. My wife wanted to move “home”. This meant moving to a very remote location, a location so remote, contracting would be next to impossible. While I am never at a clients site full time, Only a day or two a week, I needed to be remote 100% of the time. Not something I felt was possible with contracting post covid. I was also tired of developing software, essentially throwing it over the fence to the core development team and leaving, I wanted to see it in production, to manage it, improve it.

So I hunted out 100% remote permnant positions. I found one. During the interview I showed my passion for SRE, talked through what I had done, where I could bring skills. Thanks to contracting, I have a very broad range of skills, which my employer is enjoying.

When I joined, I found that they have no CI/CD which didn’t show in the interview, while a set of team city builds exist, they exist to build, the environments aren’t managed or to deployed to via CI/CD, it’s a right click -> publish from visual studio.

The cloud architecture was a message, very much an R&D project pushed live.

There are many many things that need to be done here and I’m excited to do them. I’ve introduce an epic to contain all of the tech debt that i’ve noticed, so that it can be brought into sprints. I’ve looked at documenting monitoring approaches.

I’ve introduce Azure Devops and started to create build pipelines, I wonder and communicated with the team leads about the best approach, ARM templates? Bicep creating those ARM Templates? This is a multi-cloud team, so I chose Terraform, with Terratest, as even your infrastructure as code should be tested.

I was granted a contractor to help create IaC, I instructed the contractor on Terraform use, modules and testing.

That contractor, fantastic chap, worked with him before, I knew his goals, they weren’t DevOps. So when the perfect job showed up for him he took it.

I found out the day, I sent an email to my manage saying, I want to be your Lead SRE. Here’s why I am already doing this role, here’s what we need to do.

A day later, I have a call with my manager. He loves the idea, this is exactly what the team and the company need.

We are developing a product for customers and reliability is the number one feature, also the number one feature people forget about.

So, am I an SRE now? No… has my journey begun? Maybe…

The pre-negociation

In the meeting, we spoke about how SRE is an extension of DevOps. How DevOps has somehow become Operations who write arm templates, and this is exactly what my manager had noticed when trying to fill the DevOps position.

This meeting was going so well, I was so excited, until the close, I had hoped that my manager would bring up pay. Being poor most of my life, money is important to me, I never want to be poor again.

I invest heavily into my pension, I use tax wrappers available to me so ISA and LISA’s, I am building wealth so that I never have to be homeless, barely able to feed myself again.

A lot of people. Will say that you should be ruled by passion not money. I have a mortgage, a wife and child. Passion isn’t going to keep them housed, passion isn’t going to keep them fed and warm.

You can be passionate, but also want to be paid fairly. There’s nothing wrong with that. You should know you worth and fight for it.

So with that in mind, I brought it up. I said what the average pay an SRE got, based on research on the job market that morning. I said how much an SRE Lead gets based on the few roles I could find.

We spoke about on call, as this was important to him. I mentioned, I’d been on call, that it wasn’t free, that I got paid to be on call and then paid per hour I was actively investigating.

Alarm bells rang when “It’s apart of the job” was mentioned, yeah on call is expected of DevOps/SRE’s it’s not free though, either there is a financial incentive, or the pay is increased. Hell even google pays overtime to their SRE’s if they need to.

More alarm bells when “Are you doing this for the money?” as I started to defend myself he interjected with “Or do you want fair compensation”

It’s been 3 weeks since that conversation, he was interested but had to talk to his manager.

So has my journey begun? Time will tell, I’ll keep learning, I’ll make sure to document the resources I’ve used on this site, and whether or not I become an SRE.

This is the first time, throughout my career where I’ve known my worth, I know I am a good developer, I know DevOps and SRE. This knowledge grows daily. I’m aware of what is required from me to take up SRE within this company, I’m aware of the need to have a junior under me and the work required to mentor them. I watched a video about a how a junior was nurtured and grown to a fully fledged member within a year. Something I would look to do, as well.

I know how much work there is to do, So my ask was for fair compensation. I walked away feeling like I was being greedy, that by offering a whole range of skills for the company to use but asking for fair compensation for them, was wrong.

As I said, this is the first time I’ve had the gusto and the foresight to know my worth and to state that fact. We’ll see if it pays off!

Training

To become an SRE I need to be educated. I’ve been doing C# for over a decade now, DevOps has come through practice and formal education.

I’ll list training providers and courses I’ve used, it won’t cover everything, but should give you a good shot.

https://www.youtube.com/playlist?list=PLLNq9CBV7AFwyRzICyCRKdcsAPAlG5bPu - SLO Conf https://www.pluralsight.com/paths/microsoft-azure-deployment - Microsoft Azure https://www.pluralsight.com/paths/failover-conf-2020 - Failover Conf

https://www.pluralsight.com/courses/site-reliability-engineering-measuring-managing-reliability - Site Reliability Engineering: Measuring and Managing Reliability https://www.pluralsight.com/courses/allthetalks-session-41 - SRE - Using Error Budgets to Prioritize Work https://www.pluralsight.com/courses/developing-google-sre-culture - Developing a Google SRE Culture https://www.pluralsight.com/courses/logging-monitoring-observability-google-cloud - Logging, Monitoring and Observability in Google Cloud https://www.coursera.org/learn/site-reliability-engineering-slos - Site Reliability Engineering: Measuring and Managing Reliability https://learn.cantrill.io/ - AWS courses https://training.antonit.com/ - GCP Courses https://cloudskills.io/ - Azure, Terraform, Kubernetes, Cloud native, and much more https://courses.skylinesacademy.com/ Azure, Kubernetes, GCP, Terraform

Buy Me A Coffee