Wayfair’s software engineering history demonstrates a common theme throughout its time: It moves fast and is a core trait of our business. To create and support the right foundations, we need our infrastructure to move faster than our ideas, and this is especially true for our hybrid environment objectives in 2019 and beyond. Hybrid for Wayfair means using Google Cloud Platform (GCP) to deliver extra capacity to our customer-facing website and related systems, while still supporting and maintaining our on-premise data centers. This includes splitting traffic between GCP and existing data centers to provide adequate support during peak retail periods.
In the engineering world, everyone needs compute. Whether it’s a cloud instance, a virtual machine for testing, or a physical server for production, each Wayfair product or service depends on a software engineer to run code. With this in mind, our Infrastructure organization, as part of our larger hybrid roadmap, has been working to continuously deliver infrastructure in a rapid, automated fashion, while being agnostic to whether its origin was cloud, virtual on-premises software, or physical on-prem.
The result of all this work? Project Argo: Our on-premise automation solution that gives our engineers the ability to spin up servers and tear them down on demand, quickly, with any operating system, on any environment. Argo maps out the entire work pipeline, from business production request to “go-live” in a data center. Unlike previous projects that centered on individual workstations in the pipeline, Argo unifies every team’s effort in infrastructure to concentrate on accelerating the overall pipeline. Automated testing verifies outputs passing through the pipeline, versus manual verification, and all of this is achieved while using the exact same tech stack we’re relying on for our GCP compute.
HashiCorp and Wayfair – A Collaboration Story
Infrastructure as code (IaC) is popularly promoted for cloud computing, allowing developers to be more involved in the configuration of their servers and the visibility of state. Knowing that we wanted a piece of the IaC pie, we collaborated with HashiCorp via our own open source contributions to Terraform.
We first contributed to HashiCorp’s Google Cloud provider in February of 2018, adding features we had needed but weren’t available at the time. In parallel, we wrote our own on-premise driven providers for EMC, Cisco, Foreman, and Device42, allowing Terraform, which is cloud software natively, to interact directly with on-premise applications and bare metal. We recently demonstrated this to HashiCorp’s CTO in September 2018 during a local meetup hosted by Wayfair, making quite the impression. They were floored, quoting, “You are far ahead of others for on-prem usage of our products”.
What We’ve Been Working On
Project Argo is meant to simplify the Wayfair infrastructure pipeline so that all employees can be serviced faster, and with better results. Speed is key, here – we don’t want infrastructure to ever be the bottleneck of a new innovative project or service.
To reach our ultimate goal of a unified tech stack for production servers and applications within a hybrid environment, we wanted to think outside of the Wayfair ecosystem. By open-sourcing our Project Argo framework and the Foreman Terraform Provider, we hope to allow for easy interaction with on-prem resources via their REST API. The framework itself allows anything with a REST or API frontend to be incorporated into our Argo pipeline. We built it this way purposely, to be able to hot swap out technologies as improve upon them over the years.
What’s Coming Next
This release is the first of many more to come – more utility packages and providers are on the horizon, such as Cisco ACI and EMC Storage. On-prem automation is a long term, strategic step for Wayfair, both in terms of satisfying engineering needs and making the right move to a scalable, reliable, sustainable solution.
Using the same stack as GCP is no coincidence: We know there’s a desire for this type of framework, having worked closely with HashiCorp up to today. And we’ll be collaborating further with HashiCorp in the future: We’re in the process of a joint effort to build an HP/Dell provider for additional on-prem bare metal usage.
We’ve released Project Argo under the Mozilla Public License (MPL), much like Terraform itself. We’re excited to have this out in the world and would openly welcome contributions and feedback – head on over to Wayfair’s GitHub to get started.
After a successful Way Day in 2018, we’re on track to delivering a hybrid solution for further peak periods of traffic. We’re declaring 2019 as the Year of Hybrid at Wayfair, with the support of open source software like Project Argo. Here’s to an on-prem automated future!