You would think data replication is a piece of cake these days given all the advances in database technology, and that’s true for the most part when you’re dealing with databases of the same type, but when you have to replicate parts of your product catalog with other companies, things get a bit tricky. At Wayfair Engineering we’ve figured out how to make it happen by creating a great software solution that keeps our retail partner operations working like a well-oiled machine.

The biggest challenge we face is the number of different ways of communication and data formats we have to support to accomplish the integration with our partners. For example, with Amazon we build XML messages and send them using web services, for Walmart we create XML transaction files to upload via FTP, for Sears we submit XML transactions via REST API services, for eBay we use SOAP messages over HTTPS, and for we generate tab-delimited text files and send them via FTP.

We’ve established a baseline of data structures and processing steps that we can use as a starting point for each new partner integration. But we didn’t get there overnight or even close to the beginning of this endeavor. For our first few partnerships we tried to accommodate their specifications independently, and that’s when we realized we wouldn’t get too far if we kept doing it that way forever. So we started building a common set of rules and data structures to use as a foundation for future integrations.

We also started building consolidated services designed to provide the necessary data for all our partners without duplicating efforts. For example, when we had about six partnerships already integrated, we had six different systems for preparing inventory information, and in many cases we were retrieving the same exact data six times every night. This was obviously overwhelming our servers and limiting our ability to scale and grow our partnerships. So we created a single system to function as an inventory data service for all partners. Before the improvements the individual inventory processes took between 8 and 16 hours to run. After the consolidation, we got the whole process done in under 4 hours. Eventually we created similar services for price change tracking, product variations, kits, etc. We now have a collection of services available for any new retail partner integration. Bring them on!