When a Drupal site begins to be overwhelmed by high volumes of traffic, one of the worst bottlenecks is Drupal core's menu system. A menu rebuild during high traffic can easily causes a stampede on your database, which is exacerbated by the architecture of core's Lock API.

The order of operations that leads to the stampede behavior is as follows:

• Menu rebuild is triggered by a user action (e.g., change to menu item).
• Lock API creates a lock in semaphore table in MySQL, temporarily preventing other menu rebuild requests.
• Before lock is released, a page request is made that calls menu_get_item(). This attempts to rebuild menu, but must wait until current lock is released.
• Subsequent requests snowball by spinning up additional processes that 1) wait for lock to be released, and 2) rebuild menu again after release.
• Lock stampede ensues.

In some cases, a single page load can trigger up to 10 write requests to the semaphore table. Imagine the number of writes that would be generated by a high traffic event combined with this stampeding behavior. Your site can jump from 20 idle MySQL processes to 150+ in the blink of an eye--bringing your site to its knees.

How do you know if this is happening to you?

• Your site sporadically becomes extremely slow
• You see slow INSERT queries performed on the semaphore table
• You see unexplained spikes in the number of writes to your MySQL database
• You see a large number of sleeping MySQL processes

What can you do about it? One solution is to swap out the backend for the Lock API. MySQL just isn't a good backend for a locking system that requires this volume of writes, while also serving a large number of selects. If you're using Memcache with Drupal, there's an out of the box solution for you:

$conf['cache_backends'][] = './sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['lock_inc'] = './sites/all/modules/contrib/memcache/memcache-lock.inc';
$conf['memcache_stampede_protection'] = TRUE;
?>

Note the two lines at the end of that snippet--they change the Lock API to use Memcache as a backend. Usage of Memcache as a backend for the Lock API changes the behavior in the following way:

• Menu rebuild is triggered by a user action (e.g., change to menu item).
• Lock API creates a lock in semaphore Memcache, temporarily preventing other menu rebuild requests.
• Before lock is released, a page request is made that calls menu_get_item(). Since the resource is locked, the menu is served from cache. No additional rebuilds are triggered.

The combination of providing a faster backend with the new anti-stampede logic can make a significant difference in your site's stability.