Next week, Wayfair Data Science will be at the Computer Vision and Pattern Recognition conference in Salt Lake City. We’re bringing data scientists and engineers from several different teams, plus a VR rig to try out the shopping experience of the future (come find us at booth 224!). We pride ourselves on being a disruptive technology company that is changing the way people shop for their homes, and computer vision plays a major role in that. Here’s a round-up of six interesting projects at Wayfair that incorporate computer vision into their work.

1. Visual Search

Launched in 2017, Wayfair’s Visual Search tool lets customers take photos of items they like and find visually similar products on Wayfair’s website or app. The first version of Visual Search recommended that users take close cropped photos of individual items, but this year we rolled out Object Detection, which automatically selects crop-areas for search and can identify multiple items in each picture. This is the application of computer vision that our customers are most likely to use themselves.

2. Algorithmic Style Detection

Our Style Graph team is working on a project that will teach computers to identify the style of products. We want customers to come to Wayfair and, once they find a product they love, easily identify items in other classes that are in the same style. It’s a challenging question—what are the computer-readable features that make a room, look bohemian, nautical, modern, traditional? What counts as a “minimalist” coffee table that will match a customer’s minimalist sofa?

To accomplish this, we generate style clusters using product data, aggregate customer shopping behavior, and visual embeddings generated by the computer vision team. We also leverage this information to create a dynamic online model to determine customer style preference. The techniques applied include convolution neural networks (CNN) and unsupervised and semi-supervised learning

3. Automatic Selection of Lead Images

Each item on Wayfair has multiple images on its Product Detail Page. Which is the best one to show to customers first? Data scientists on our demand recommendations team are working on figuring that out, and the first step is to generate better tags for the photos that tell us more about the photo: is it an environmental shot (featuring a whole room or space)? What angle is the product at? Is the product in the foreground or the background?

This desk has three images: a “silo” image (just the desk on a white background), a fairly close shot of the desk in a room, and a photo of the desk with a chair in front of it.

To answer that last question, we use two different computer vision models, both of which are also used for other purposes at Wayfair. First, the Room Detection model generates estimated probabilities that a given photo is a certain type of room—a living room, a bedroom, a kitchen, etc. If a photo has a high probability of being a living room, then it’s probably not a detail shot of just a single product. Second, the Object Detection model gives the coordinates of an object within an image. It the object takes up most of the image, then it’s presumably featured, and not in the background.

4. Visual Recommendations

The Visual Recommendations algorithm uses the Visual Search team’s work to serve high quality recommendations. The algorithm suggests products that are visually similar to ones that a customer has already viewed. This lets us incorporate a customer’s style preferences into our recommendations.

Multiple products make up a Browse Context, and each Browse Context is used to generate its own set of product recommendations. The recommendations are then concatenated, sorted ascending by distance, and then filtered to a final set of product recommendations.

Data Science Manager Vinny DeGenova covered our computer-vision-powered recommender system in much more detail in his blog post back in December. Of note: the algorithm performs quite well across classes (different categories of products) as well as within single classes.

5. Deduplication in the Wayfair Catalog

Our deduplication algorithm was covered in more detail on the blog last month, but recently we released a update to the deduplication algorithm which both expands the algorithm’s coverage of the Wayfair catalog—to every product on the U.S. site—and speeds it up significantly, so it still cycles through every pair in the catalog about every three weeks.

screenshot of the duplicate review tool UI

Since its launch, the deduplication algorithm has found over 40,000 duplicate products. The new algorithm, based on Spark ML rather than scikit-learn in Python like V1, is expected to identify tens of thousands of new duplicate pairs in the next few weeks.

6. Wayfair Next: 3D Computer Vision

The members of the Wayfair Next team are the masterminds behind our augmented reality shopping app, which lets customers view products at accurate scale in their own spaces. Behind this is a large library of 3D models of our products, usually produced using photogrammetry. But we are working towards generating 3D models based on images, which would hopefully be faster than physically scanning each item.

3D-influenced computer vision can also help improve our site merchandising and the information available about each product on our website. Wayfair Next is investigating ways to automatically crop and warp photos of products like pillows, wall art, and rugs to produce full-on shots that will be more helpful to our customers. We are also looking at determining product dimensions from their photos, estimating camera perspective based on images, and detecting characters on product images for use in monograms.

Want to learn more, and meet the teams behind these projects?

Come find us at CVPR–we’ll be in Salt Lake City all week! Data scientists, researchers, and recruiters will be at our booth at the expo, #224. You might even get a chance to try our our virtual reality rig.