When a Drupal site begins to be overwhelmed by high volumes of traffic, one of the worst bottlenecks is Drupal core's menu system. A menu rebuild during high traffic can easily causes a stampede on your database, which is exacerbated by the architecture of core's Lock API.
The order of operations that leads to the stampede behavior is as follows:
- Menu rebuild is triggered by a user action (e.g., change to menu item).
- Lock API creates a lock in semaphore table in MySQL, temporarily preventing other menu rebuild requests.
- Before lock is released, a page request is made that calls
menu_get_item()
. This attempts to rebuild menu, but must wait until current lock is released. - Subsequent requests snowball by spinning up additional processes that 1) wait for lock to be released, and 2) rebuild menu again after release.
- Lock stampede ensues.
In some cases, a single page load can trigger up to 10 write requests to the semaphore table. Imagine the number of writes that would be generated by a high traffic event combined with this stampeding behavior. Your site can jump from 20 idle MySQL processes to 150+ in the blink of an eye--bringing your site to its knees.
How do you know if this is happening to you?
- Your site sporadically becomes extremely slow
- You see slow INSERT queries performed on the semaphore table
- You see unexplained spikes in the number of writes to your MySQL database
- You see a large number of sleeping MySQL processes
What can you do about it? One solution is to swap out the backend for the Lock API. MySQL just isn't a good backend for a locking system that requires this volume of writes, while also serving a large number of selects. If you're using Memcache with Drupal, there's an out of the box solution for you:
$conf['cache_backends'][] = './sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['lock_inc'] = './sites/all/modules/contrib/memcache/memcache-lock.inc';
$conf['memcache_stampede_protection'] = TRUE;
?>
Note the two lines at the end of that snippet--they change the Lock API to use Memcache as a backend. Usage of Memcache as a backend for the Lock API changes the behavior in the following way:
- Menu rebuild is triggered by a user action (e.g., change to menu item).
- Lock API creates a lock in semaphore Memcache, temporarily preventing other menu rebuild requests.
- Before lock is released, a page request is made that calls
menu_get_item()
. Since the resource is locked, the menu is served from cache. No additional rebuilds are triggered.
The combination of providing a faster backend with the new anti-stampede logic can make a significant difference in your site's stability.
Add new comment