The DevOps spaghetti
It was a lonely monday, not enough to commit and release just went live and all were lazing around by the pool. Head of engineering splurged a bit more cash, real cash(not the virtual ones we are usually promised during the hire times, call it the Dumb Money unless company goes public). I was trying to be in the zone, just to match the Gen Z folks who were wired in with Hifimans with a beer but somehow it looked like, i will never be like them, even not here when we are 1000 kms away from corporate buzz , off at a private island, rented to relax the overworked mental muscles for the past 3 months. Why so, my pager buzzed again with a p1 alert and well, its that time of the day :). Looked like my peace of mind just got a period. Well, its nothing critical, db seems to be unreachable and prod was responding too slow. Well, what the hell happened now !!.
Looked like the last untested commit was approved and the feature mistakenly opened up a random port and alas, the bad guys scraping network turned up the heat on us, it was a DDOS on that bad untested, son of a b** port that crash landed on our prod that release night. :) .
Well, how does this becomes a spaghetti, well the branch from which it was committed was never a part of the regression, well, is it can of worms now :). Or the Jenkins approvals were messed up and the upgraded plugin messed it all up. I donno, that was a different guy looking at that one !!. Pages flooded and squeezed the blood out of my mind. What was happening !!!. Was it the commit, the branch, the jenkins, approvers, the merge tree or something else. I called up Dan, the party man alas head of engineering and we were on a swat call in a matter of 10 min.
Fired up the lap but somehow it was not charged and charger was somewhere !!. Anyways,, it still has 30% left. So lets get on it. Prod was flimsy, the dusk was intimidating and I was chained , the waves were calling me hard but I was somehow in the zone (SRE zone) and not the real zone for which we were there. It was a real spaghetti !!. a Real one. whats next...