In the Wayfair Network Operations Center (NOC) we wear a lot of hats: Code Deployers, Debuggers, Production Triage, and Incident Coordination, to name a few. Just this month we’ve responded to an average of 256 alerts a day and are staffed 24×7 with a team spanning two continents, servicing several locations. At the end of a long day of deploying code and ensuring production stability, nothing beats an ice-cold beer/cider/seltzer/soda from our very own NOC Fridge.
Our intrepid Director of the NOC, Tameem Hourani, posed the following idea on our team’s Slack channel: “Let’s put a magnetic lock on the NOC Fridge”. He wanted a way for us to have a physical outcome be triggered by our production environment. This way, we can see our hard work when automating some aspects of our workload, and also be rewarded for “working smart, not hard”.
I know what you’re thinking: “Why lock up a fridge when we have free snacks and drinks in the office?“.
His idea came with a twist: if we leverage our stack to auto-resolve common production issues as they arise, then we will automatically unlock the fridge for a given time span, 30 seconds or so, which is plenty of time to grab a beverage.
So how did we go about building it? To give more technical context, here is what we’re using to achieve the dream of not only working smarter, but also having a smart fridge that rewards a team for their hard work:
- ServiceNow – Our Incident Management system that receives email alerts from a number of monitoring systems including Kibana, Graphite, vCenter, and more.
- Rundeck – A tool we are leveraging to provide our Engineering teams a frontend to run pre-approved scripts for performing routine commands, like clearing out a file directory or restarting a service.
- Flask – A Python framework to create RESTful APIs. Flask makes it devilishly easy to spin up a proof of concept API in minutes.
- NGINX and Gunicorn – Battle-hardened open source web servers to serve the Flask API application.
With the software in mind, we assembled our NOC Fridge Lock using the following hardware:
- Raspberry Pi 3 – Small ARM-based computer. Definitely overkill for this project but having the flexibility of a full Linux environment made development, debugging, and deployment a snap.
- Electromagnetic Lock – 12V electromagnet that engages when supplied with 12 volts (~0.15 amps)
- 5V Relay – Used to provide a means to shut off power to the electromagnet lock. We found a pre-made powerstrip with a built-in relay was preferable to splicing AC cables and playing with high voltage and amperage.
- Beer Fridge – Conveniently, Wayfair sells exactly what we needed.
- Fun Visual Indicator – We picked a police light as a visual indicator that the fridge has been unlocked.
In order to mesh the software and hardware together, here is the workflow:
- An alert email comes into ServiceNow from a monitoring service.
- ServiceNow determines what kind of issue it is and sends an HTTP request to Rundeck to run the appropriate predefined command.
- If the command is successful and the monitoring system confirms the issue is no longer present, the monitoring service sends an OK email to ServiceNow.
- ServiceNow runs a business rule to send an HTTP request to a ServiceNow MID Server on the Raspberry Pi, which performs an HTTP request to the unlock endpoint on the local Raspberry Pi.
- The unlock endpoint on the Flask application uses the GPIO to send a low bit to the relay, which cuts power to the electromagnet lock, unlocking the fridge.
- The fridge remains unlocked for a predefined time, then the Flask application returns power to the relay, locking the fridge, awaiting the next auto-closed alert.
The software part of this project was simple. A Flask application was spun up, providing an endpoint for ServiceNow to hit. The bulk of the work was wiring up the simple circuit to trip the lock – waiting on Amazon deliveries for the component parts was the primary blocker.
The hardware portion was a matter of using jumper cables to connect from the GPIO on the Raspberry Pi to the relay. Here is a simple wiring diagram for how the Raspberry Pi, relay, lock, and light are set up:
So, why build this? Having a mini fridge is a great way to have a visual indication that the work we are doing effects Wayfair as an organization. Automatically fixing the easy, repetitive stuff frees up time to work on the more interesting, challenging problems, like automating a fridge. This project is representative of the culture at Wayfair; if you have an idea, run with it and use the experience to create and innovate.
What else do we get up to in the NOC? Watching over the operations of our entire Engineering department can be thirsty work. If you’re interested in solving technical challenges in creative and original ways, check out our open roles in Infrastructure today.