E04 - Dr Strangecron or: How I Learned to Stop Worrying and Love the lastcomm

stack.io CEO and show host Hany Fahim puts on his Sherlock Holmes hat in this episode of Tales from the Ops Side.

He's investigating a missing file of sensitive data, while his client and their partner point fingers at each other over fault.

This client had an agreement to provide sensitive data on a nightly basis to a partner. This was automated, the process called a cronjob. Hany's lunch was interrupted in late 2017 when an urgent request came in from this client.

The problem: One night the data was never received by the partner.

Hany shifted into troubleshooting mode, working with his client and examining the guts of their system. It appeared the job did run as scheduled, even though the partner did not receive the file.

What happened?

Digging deeper into alert files, database logs and network graphs didn’t shed any more light on the problem. After examining all the evidence and chasing down logical leads, Hany was no further ahead.

That night at home, he was distracted by the problem. After a late-night of research, he was no further ahead. Over a cup of strong coffee the next morning, he spotted an obscure forum post he had disregarded the night before. The post gave his investigation a new avenue to explore - Linux Process Accounting. 

With more sleuthing, hiking through the historical bowels of the internet, and combing through over 1000 lines of code, Hany was rewarded with an answer and knew what the client's problem was and who was at fault.

For the whole story, all the technical details, and Hany's insider view, listen to the episode.

Connect with Hany at his company stack.io and LinkedIn.
 
If you enjoyed this episode, please share it with anyone you think will enjoy it. And if you can give us a review on Apple Podcasts, we’ll be grateful!