bio: skip atomic inc/dec of ->bi_remaining for non-chains
authorJens Axboe <axboe@fb.com>
Fri, 17 Apr 2015 22:15:18 +0000 (16:15 -0600)
committerJens Axboe <axboe@fb.com>
Sat, 18 Apr 2015 01:41:24 +0000 (19:41 -0600)
commit411875b29fc4ef290c6d683af7e0a2ec07a6cb2c
treeed2a381f48f53782e83d0a70d8b11a6680231fe6
parent04b7fe6a4a231871ef681bc95e08fe66992f7b1f
bio: skip atomic inc/dec of ->bi_remaining for non-chains

Struct bio has an atomic ref count for chained bio's, and we use this
to know when to end IO on the bio. However, most bio's are not chained,
so we don't need to always introduce this atomic operation as part of
ending IO.

Add a helper to elevate the bi_remaining count, and flag the bio as
now actually needing the decrement at end_io time. Rename the field
to __bi_remaining to catch any current users of this doing the
incrementing manually.

For high IOPS workloads, this reduces the overhead of bio_endio()
substantially.

Tested-by: Robert Elliott <elliott@hp.com>
Acked-by: Kent Overstreet <kent.overstreet@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
block/bio.c
drivers/md/dm-cache-target.c
drivers/md/dm-raid1.c
drivers/md/dm-snap.c
drivers/md/dm-thin.c
include/linux/bio.h
include/linux/blk_types.h