bpf: Use this_cpu_{inc|dec|inc_return} for bpf_task_storage_busy
authorHou Tao <houtao1@huawei.com>
Thu, 1 Sep 2022 06:19:35 +0000 (14:19 +0800)
committerMartin KaFai Lau <martin.lau@kernel.org>
Thu, 1 Sep 2022 19:16:12 +0000 (12:16 -0700)
commit197827a05e13808c60f52632e9887eede63f1c16
tree3557bd01bd7498389f9e04d93a03fcb9d44b6ba8
parentc9ae8c966f05c85c5928c8f1790b13b71cc5ccd5
bpf: Use this_cpu_{inc|dec|inc_return} for bpf_task_storage_busy

Now migrate_disable() does not disable preemption and under some
architectures (e.g. arm64) __this_cpu_{inc|dec|inc_return} are neither
preemption-safe nor IRQ-safe, so for fully preemptible kernel concurrent
lookups or updates on the same task local storage and on the same CPU
may make bpf_task_storage_busy be imbalanced, and
bpf_task_storage_trylock() on the specific cpu will always fail.

Fixing it by using this_cpu_{inc|dec|inc_return} when manipulating
bpf_task_storage_busy.

Fixes: bc235cdb423a ("bpf: Prevent deadlock from recursive bpf_task_storage_[get|delete]")
Signed-off-by: Hou Tao <houtao1@huawei.com>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/r/20220901061938.3789460-2-houtao@huaweicloud.com
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
kernel/bpf/bpf_local_storage.c
kernel/bpf/bpf_task_storage.c