dm-cache

dm-cache uses solid-state drives (SSDs) as an additional level of indirection while accessing hard disk drives (HDDs), improving the overall performance by using fast flash-based SSDs as caches for the slower mechanical HDDs based on rotational magnetic media. As a result, the costly speed of SSDs becomes combined with the storage capacity offered by slower but less expensive HDDs.[1] Moreover, in the case of storage area networks (SANs) used in cloud environments as shared storage systems for virtual machines, dm-cache can also improve overall performance and reduce the load of SANs by providing data caching using client-side local storage.[2][3][4]

dm-cache is implemented as a component of the Linux kernel's device mapper, which is a volume management framework that allows various mappings to be created between physical and virtual block devices. The way a mapping between devices is created determines how the virtual blocks are translated into underlying physical blocks, with the specific translation types referred to as targets.[5] Acting as a mapping target, dm-cache makes it possible for SSD-based caching to be part of the created virtual block device, while the configurable operating modes and cache policies determine how dm-cache works internally. The operating mode selects the way in which the data is kept in sync between an HDD and an SSD, while the cache policy, selectable from separate modules that implement each of the policies, provides the algorithm for determining which blocks are promoted (moved from an HDD to an SSD), demoted (moved from an SSD to an HDD), cleaned, etc.[6]

When configured to use the multiqueue (mq) or stochastic multiqueue (smq) cache policy, with the latter being the default, dm-cache uses SSDs to store the data associated with performed random reads and writes, capitalizing on near-zero seek times of SSDs and avoiding such I/O operations as typical HDD performance bottlenecks. The data associated with sequential reads and writes is not cached on SSDs, avoiding undesirable cache invalidation during such operations; performance-wise, this is beneficial because the sequential I/O operations are suitable for HDDs due to their mechanical nature. Not caching the sequential I/O also helps in extending the lifetime of SSDs used as caches.[7]

Another dm-cache project with similar goals was announced by Eric Van Hensbergen and Ming Zhao in 2006, as the result of an internship work at IBM.[8]

Later, Joe Thornber, Heinz Mauelshagen and Mike Snitzer provided their own implementation of the concept, which resulted in the inclusion of dm-cache into the Linux kernel. dm-cache was merged into the Linux kernel mainline in kernel version 3.9, which was released on April 28, 2013.[6][9]

In dm-cache, creating a mapped virtual block device that acts as a hybrid volume requires three physical storage devices:[6]

• Origin device – provides slow primary storage (usually an HDD)
• Cache device – provides a fast cache (usually an SSD)
• Metadata device – records the placement of blocks and their dirty flags, as well as other internal data required by a cache policy, including per-block hit counts; a metadata device cannot be shared between multiple cache devices, and it is recommended to be mirrored

Internally, dm-cache references to each of the origin devices through a number of fixed-size blocks; the size of these blocks, equaling to the size of a caching extent, is configurable only during the creation of a hybrid volume. The size of a caching extent must range between 32 KB and 1 GB, and it must be a multiple of 32 KB; typically, the size of a caching extent is between 256 and 1024 KB. The choice of the caching extents bigger than disk sectors acts a compromise between the size of metadata and the possibility for wasting cache space. Having too small caching extents increases the size of metadata, both on the metadata device and in kernel memory, while having too large caching extents increases the amount of wasted cache space due to caching whole extents even in the case of high hit rates only for some of their parts.[6][10]

Operating modes supported by dm-cache are write-back, which is the default, write-through, and pass-through. In the write-back operating mode, writes to cached blocks go only to the cache device, while the blocks on origin device are only marked as dirty in the metadata. For the write-through operating mode, write requests are not returned as completed until the data reaches both the origin and cache devices, with no clean blocks becoming marked as dirty. In the pass-through operating mode, all reads are performed directly from the origin device, avoiding the cache, while all writes go directly to the origin device; any cache write hits also cause invalidation of the cached blocks. The pass-through mode allows a hybrid volume to be activated when the state of a cache device is not known to be consistent with the origin device.[6][11]

The rate of data migration that dm-cache performs in both directions (i.e., data promotions and demotions) can be throttled down to a configured speed so regular I/O to the origin and cache devices can be preserved. Decommissioning a hybrid volume or shrinking a cache device requires use of the cleaner policy, which effectively flushes all blocks marked in metadata as dirty from the cache device to the origin device.[6][7]

As of August 2015 and version 4.2 of the Linux kernel,[12] the following three cache policies are distributed with the Linux kernel mainline, out of which dm-cache by default uses the stochastic multiqueue policy:[6][7]

multiqueue (mq)

The multiqueue (mq) policy has three sets of 16 queues, using the first set for entries waiting for the cache and the remaining two sets for entries already in the cache, with the latter separated so the clean and dirty entries belong to each of the two sets. The age of cache entries in the queues is based on their associated logical time. The selection of entries going into the cache (i.e., becoming promoted) is based on variable thresholds, and queue selection is based on the hit count of an entry. This policy aims to take different cache miss costs into account, and to make automatic adjustments to different load patterns.

This policy internally tracks sequential I/O operations so they can be routed around the cache, with different configurable thresholds for the differentiation between random I/O and sequential I/O operations. As a result, large contiguous I/O operations are left to be performed by the origin device because such data access patterns are suitable for HDDs, and because they avoid undesirable cache invalidation.

stochastic multiqueue (smq)

The stochastic multiqueue (smq) policy performs in a similar way as the multiqueue policy, but requires fewer resources to operate; in particular, it uses substantially smaller amounts of main memory to track cached blocks. It also replaces the hit counting from the multiqueue policy with a "hotspot" queue, and decides on data promotion and demotion on a least-recently used (LRU) basis. As a result, this policy provides better performance compared to the multiqueue policy, adjusts better automatically to different load patterns, and eliminates the configuration of various thresholds.

cleaner

The cleaner policy writes back to the origin device all blocks that are marked as dirty in the metadata. After the completion of this operation, a hybrid volume can be decommissioned or the size of a cache device can be reduced.

Logical Volume Manager includes lvmcache, which provides a wrapper for dm-cache integrated with LVM.[13]

• bcache – a Linux kernel's block layer cache, developed by Kent Overstreet
• Flashcache – a disk cache component for the Linux kernel, initially developed by Facebook
• Hybrid drive – a storage device that combines flash-based and spinning magnetic media storage technologies
• ReadyBoost – a disk caching software component of Windows Vista and later Microsoft operating systems
• Smart Response Technology (SRT) – a proprietary disk storage caching mechanism, developed by Intel for its chipsets
• ZFS – a cross-OS storage management system that has a similar integrated caching device support (L2ARC)

• Linux Block Caching Choices in Stable Upstream Kernel (PDF), Dell, December 2013
• Performance Comparison among EnhanceIO, bcache and dm-cache, LKML, June 11, 2013
• EnhanceIO, Bcache & DM-Cache Benchmarked, Phoronix, June 11, 2013, by Michael Larabel
• SSD Caching Using dm-cache Tutorial, July 2014, by Kyle Manna
• Re: [dm-devel] [PATCH 8/8] [dm-cache] cache target, December 14, 2012 (guidelines for metadata device sizing)