lib/vsprintf: refactor duplicate code to special_hex_number()