[linux-2.6-block.git] / Documentation / device-mapper / cache-policies.txt

Guidance for writing policies
=============================

Try to keep transactionality out of it.  The core is careful to
avoid asking about anything that is migrating.  This is a pain, but
makes it easier to write the policies.

Mappings are loaded into the policy at construction time.

Every bio that is mapped by the target is referred to the policy.
The policy can return a simple HIT or MISS or issue a migration.

Currently there's no way for the policy to issue background work,
e.g. to start writing back dirty blocks that are going to be evicted
soon.

Because we map bios, rather than requests it's easy for the policy
to get fooled by many small bios.  For this reason the core target
issues periodic ticks to the policy.  It's suggested that the policy
doesn't update states (eg, hit counts) for a block more than once
for each tick.  The core ticks by watching bios complete, and so
trying to see when the io scheduler has let the ios run.


Overview of supplied cache replacement policies
===============================================

multiqueue (mq)
---------------

This policy is now an alias for smq (see below).

The following tunables are accepted, but have no effect:

	'sequential_threshold <#nr_sequential_ios>'
	'random_threshold <#nr_random_ios>'
	'read_promote_adjustment <value>'
	'write_promote_adjustment <value>'
	'discard_promote_adjustment <value>'

Stochastic multiqueue (smq)
---------------------------

This policy is the default.

The stochastic multi-queue (smq) policy addresses some of the problems
with the multiqueue (mq) policy.

The smq policy (vs mq) offers the promise of less memory utilization,
improved performance and increased adaptability in the face of changing
workloads.  smq also does not have any cumbersome tuning knobs.

Users may switch from "mq" to "smq" simply by appropriately reloading a
DM table that is using the cache target.  Doing so will cause all of the
mq policy's hints to be dropped.  Also, performance of the cache may
degrade slightly until smq recalculates the origin device's hotspots
that should be cached.

Memory usage:
The mq policy used a lot of memory; 88 bytes per cache block on a 64
bit machine.

smq uses 28bit indexes to implement it's data structures rather than
pointers.  It avoids storing an explicit hit count for each block.  It
has a 'hotspot' queue, rather than a pre-cache, which uses a quarter of
the entries (each hotspot block covers a larger area than a single
cache block).

All this means smq uses ~25bytes per cache block.  Still a lot of
memory, but a substantial improvement nontheless.

Level balancing:
mq placed entries in different levels of the multiqueue structures
based on their hit count (~ln(hit count)).  This meant the bottom
levels generally had the most entries, and the top ones had very
few.  Having unbalanced levels like this reduced the efficacy of the
multiqueue.

smq does not maintain a hit count, instead it swaps hit entries with
the least recently used entry from the level above.  The overall
ordering being a side effect of this stochastic process.  With this
scheme we can decide how many entries occupy each multiqueue level,
resulting in better promotion/demotion decisions.

Adaptability:
The mq policy maintained a hit count for each cache block.  For a
different block to get promoted to the cache it's hit count has to
exceed the lowest currently in the cache.  This meant it could take a
long time for the cache to adapt between varying IO patterns.

smq doesn't maintain hit counts, so a lot of this problem just goes
away.  In addition it tracks performance of the hotspot queue, which
is used to decide which blocks to promote.  If the hotspot queue is
performing badly then it starts moving entries more quickly between
levels.  This lets it adapt to new IO patterns very quickly.

Performance:
Testing smq shows substantially better performance than mq.

cleaner
-------

The cleaner writes back all dirty blocks in a cache to decommission it.

Examples
========

The syntax for a table is:
	cache <metadata dev> <cache dev> <origin dev> <block size>
	<#feature_args> [<feature arg>]*
	<policy> <#policy_args> [<policy arg>]*

The syntax to send a message using the dmsetup command is:
	dmsetup message <mapped device> 0 sequential_threshold 1024
	dmsetup message <mapped device> 0 random_threshold 8

Using dmsetup:
	dmsetup create blah --table "0 268435456 cache /dev/sdb /dev/sdc \
	    /dev/sdd 512 0 mq 4 sequential_threshold 1024 random_threshold 8"
	creates a 128GB large mapped device named 'blah' with the
	sequential threshold set to 1024 and the random_threshold set to 8.
Commit	Line	Data
f2836352 JT	1	Guidance for writing policies
	2	=============================
	3
	4	Try to keep transactionality out of it. The core is careful to
	5	avoid asking about anything that is migrating. This is a pain, but
	6	makes it easier to write the policies.
	7
	8	Mappings are loaded into the policy at construction time.
	9
	10	Every bio that is mapped by the target is referred to the policy.
	11	The policy can return a simple HIT or MISS or issue a migration.
	12
	13	Currently there's no way for the policy to issue background work,
492d48db	14	e.g. to start writing back dirty blocks that are going to be evicted
f2836352 JT	15	soon.
	16
	17	Because we map bios, rather than requests it's easy for the policy
	18	to get fooled by many small bios. For this reason the core target
	19	issues periodic ticks to the policy. It's suggested that the policy
	20	doesn't update states (eg, hit counts) for a block more than once
	21	for each tick. The core ticks by watching bios complete, and so
	22	trying to see when the io scheduler has let the ios run.
	23
	24
	25	Overview of supplied cache replacement policies
	26	===============================================
	27
bccab6a0 MS	28	multiqueue (mq)
bccab6a0 MS	29	---------------
f2836352	30
9ed84698	31	This policy is now an alias for smq (see below).
f2836352	32
9ed84698	33	The following tunables are accepted, but have no effect:
01911c19	34
78e03d69 JT	35	'sequential_threshold <#nr_sequential_ios>'
	36	'random_threshold <#nr_random_ios>'
	37	'read_promote_adjustment <value>'
	38	'write_promote_adjustment <value>'
	39	'discard_promote_adjustment <value>'
f2836352	40
bccab6a0 MS	41	Stochastic multiqueue (smq)
	42	---------------------------
	43
	44	This policy is the default.
	45
	46	The stochastic multi-queue (smq) policy addresses some of the problems
	47	with the multiqueue (mq) policy.
	48
	49	The smq policy (vs mq) offers the promise of less memory utilization,
	50	improved performance and increased adaptability in the face of changing
492d48db	51	workloads. smq also does not have any cumbersome tuning knobs.
bccab6a0 MS	52
	53	Users may switch from "mq" to "smq" simply by appropriately reloading a
	54	DM table that is using the cache target. Doing so will cause all of the
	55	mq policy's hints to be dropped. Also, performance of the cache may
	56	degrade slightly until smq recalculates the origin device's hotspots
	57	that should be cached.
	58
	59	Memory usage:
492d48db	60	The mq policy used a lot of memory; 88 bytes per cache block on a 64
bccab6a0 MS	61	bit machine.
bccab6a0 MS	62
492d48db	63	smq uses 28bit indexes to implement it's data structures rather than
bccab6a0	64	pointers. It avoids storing an explicit hit count for each block. It
492d48db	65	has a 'hotspot' queue, rather than a pre-cache, which uses a quarter of
bccab6a0 MS	66	the entries (each hotspot block covers a larger area than a single
	67	cache block).
	68
492d48db	69	All this means smq uses ~25bytes per cache block. Still a lot of
bccab6a0 MS	70	memory, but a substantial improvement nontheless.
	71
	72	Level balancing:
492d48db MS	73	mq placed entries in different levels of the multiqueue structures
	74	based on their hit count (~ln(hit count)). This meant the bottom
	75	levels generally had the most entries, and the top ones had very
	76	few. Having unbalanced levels like this reduced the efficacy of the
bccab6a0 MS	77	multiqueue.
bccab6a0 MS	78
492d48db MS	79	smq does not maintain a hit count, instead it swaps hit entries with
492d48db MS	80	the least recently used entry from the level above. The overall
bccab6a0 MS	81	ordering being a side effect of this stochastic process. With this
	82	scheme we can decide how many entries occupy each multiqueue level,
	83	resulting in better promotion/demotion decisions.
	84
	85	Adaptability:
492d48db	86	The mq policy maintained a hit count for each cache block. For a
bccab6a0	87	different block to get promoted to the cache it's hit count has to
492d48db	88	exceed the lowest currently in the cache. This meant it could take a
bccab6a0	89	long time for the cache to adapt between varying IO patterns.
bccab6a0	90
492d48db	91	smq doesn't maintain hit counts, so a lot of this problem just goes
bccab6a0 MS	92	away. In addition it tracks performance of the hotspot queue, which
	93	is used to decide which blocks to promote. If the hotspot queue is
	94	performing badly then it starts moving entries more quickly between
	95	levels. This lets it adapt to new IO patterns very quickly.
	96
	97	Performance:
492d48db	98	Testing smq shows substantially better performance than mq.
bccab6a0	99
8735a813 HM	100	cleaner
	101	-------
	102
	103	The cleaner writes back all dirty blocks in a cache to decommission it.
	104
f2836352 JT	105	Examples
	106	========
	107
	108	The syntax for a table is:
	109	cache <metadata dev> <cache dev> <origin dev> <block size>
	110	<#feature_args> [<feature arg>]*
	111	<policy> <#policy_args> [<policy arg>]*
	112
	113	The syntax to send a message using the dmsetup command is:
	114	dmsetup message <mapped device> 0 sequential_threshold 1024
	115	dmsetup message <mapped device> 0 random_threshold 8
	116
	117	Using dmsetup:
	118	dmsetup create blah --table "0 268435456 cache /dev/sdb /dev/sdc \
	119	/dev/sdd 512 0 mq 4 sequential_threshold 1024 random_threshold 8"
	120	creates a 128GB large mapped device named 'blah' with the
	121	sequential threshold set to 1024 and the random_threshold set to 8.