Set affinity before doing buffer and io_u allocs