As has been discussed at great length on this blog recently, performance is a key part of the work we do here at Wayfair. Recently, we’ve been putting a lot of extra effort into our technology developments to make our pages load faster and more reliably. One of the most recent releases to this end was an update to how we store a lot of our data in our various caching systems. I am going to focus this post on the introduction of a new technology to our caching layers: CDB (Constant Database). CDB is a key/value store built for quick lookup speeds, utilizing a perfect hash function. CDB is a supported PHP extension that has been tested in a number of environments, generally outperforming other key/value store options (e.g. http://qdbm.sourceforge.net/benchmark.pdf). Our interest in adapting another caching mechanism stemmed from some issues we were having with APC (Alternative PHP Cache). APC is a powerful local opcode cache, but in general the APC User Cache does not perform very well under heavy load. Our performance metrics were affirming this fact, so we needed to replace it with some other system. CDB boasts great performance metrics under heavy load, so we went ahead and mapped out a plan to incorporate CDB into our set of caching technologies.
At the onset of this project our cache system was essentially a two-tiered system with local APC caching on all servers, backed up by remote memcached servers. One benefit of APC is that during code execution and page loads we can very easily set and get values in cache, which makes its use very dynamic. CDB works very well on heavy load for just read operations, but fails if multiple processes attempt to update or write new key/value pairs on the fly. Therefore we envisioned a system in which both APC and CDB were utilized as local cache options, with memcache as a backup for both. CDB would handle our constants, which in general are static and do not change often. We update them once an hour with an asynchronous job, which keeps them fresh enough for our needs. If CDB could handle the constant load well, we planned to move our language resources (how we localize our site for other countries) over from APC to CDB as well and observe how performance was affected.
While the initial move of constants to CDB did not show a huge performance improvement s, we did notice some small gains once we added language resources to CDB. On top of a marginal decrease in load time, CDB performed much better under load, showing significantly less periodic behavior than what we had seen previously with APC.
Below are graphs of homepage performance over an 8 hour window, before and after our change. The huge drop seen in the first graph is due to clearing APC on our servers. Clearing APC on a somewhat regular basis became necessary to keep our site performance stable under load. With CDB taking over a huge portion of the storage that lived in APC previously, we have achieved a level of stability that was not possible with the APC user cache.
Ignoring the blue line from the initial graph, what we see here is twofold. First off, over the course of a couple months, our team has made huge strides to improve the performance of our servers and our code, decreasing the overall load time for the Wayfair.com homepage. CDB contributed to this vast decrease in load times, but many other factors/projects contributed as well. CDB’s main contribution comes with improved stability and reliability for our sites. Moving forward, we have more ideas in the works to make our pages faster and more stable, and we plan to continue our efforts in this area well into the future. As one of our mottos states very clearly, we are never done.