fio.git
20 hours agoMerge branch 'fix_devdax' of https://github.com/harish-24/fio master
Jens Axboe [Tue, 14 Jul 2020 16:10:23 +0000 (10:10 -0600)]
Merge branch 'fix_devdax' of https://github.com/harish-24/fio

* 'fix_devdax' of https://github.com/harish-24/fio:
  Fix: dev-dax engine building with make

20 hours agoFix: dev-dax engine building with make
Harish [Tue, 14 Jul 2020 16:00:24 +0000 (21:30 +0530)]
Fix: dev-dax engine building with make

With recent changes to Makefile dev-dax skips building. This patch
fixes that.

Signed-off-by: Harish <harish@linux.ibm.com>
46 hours agoMerge branch 'io_uring-opt' of https://github.com/antonblanchard/fio
Jens Axboe [Mon, 13 Jul 2020 14:01:52 +0000 (08:01 -0600)]
Merge branch 'io_uring-opt' of https://github.com/antonblanchard/fio

* 'io_uring-opt' of https://github.com/antonblanchard/fio:
  io_uring: Avoid needless update of completion queue head pointer

2 days agoio_uring: Avoid needless update of completion queue head pointer
Anton Blanchard [Mon, 13 Jul 2020 01:34:48 +0000 (11:34 +1000)]
io_uring: Avoid needless update of completion queue head pointer

I'm seeing a slowdown in io_uring performance on a POWER9 box when
the userspace and kernel polling threads are on two cores that
share an L2 cache.

fio_ioring_cqring_reap() always stores to the completion queue head
pointer, even if nothing was reaped and the value hasn't changed.

Changing this to only update the head pointer when it changes results
in a 95% improvement in performance on this particular test.

Signed-off-by: Anton Blanchard <anton@ozlabs.org>
6 days agot/io_uring: make bs and polled IO configurable at runtime
Jens Axboe [Wed, 8 Jul 2020 21:48:11 +0000 (15:48 -0600)]
t/io_uring: make bs and polled IO configurable at runtime

Signed-off-by: Jens Axboe <axboe@kernel.dk>
10 days agoMerge branch 'windows' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 4 Jul 2020 23:29:32 +0000 (17:29 -0600)]
Merge branch 'windows' of https://github.com/bvanassche/fio

* 'windows' of https://github.com/bvanassche/fio:
  os/windows/posix.c: Use INVALID_SOCKET instead of < 0
  os/windows/posix.c: Strip trailing whitespace

10 days agoos/windows/posix.c: Use INVALID_SOCKET instead of < 0
Bart Van Assche [Sat, 4 Jul 2020 03:20:00 +0000 (20:20 -0700)]
os/windows/posix.c: Use INVALID_SOCKET instead of < 0

From
https://docs.microsoft.com/en-us/windows/win32/winsock/socket-data-type-2:
"Because the SOCKET type is unsigned, compiling existing source code from,
for example, a UNIX environment may lead to compiler warnings about
signed/unsigned data type mismatches.

This means, for example, that checking for errors when the socket and
accept functions return should not be done by comparing the return value
with -1, or seeing if the value is negative (both common and legal
approaches in UNIX). Instead, an application should use the manifest
constant INVALID_SOCKET as defined in the Winsock2.h header file."

This patch fixes the following build error for the latest version of
the MinGW64 cross-compiler on Cygwin:

os/windows/posix.c:886:17: error: comparison of unsigned expression < 0
  is always false

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
10 days agoos/windows/posix.c: Strip trailing whitespace
Bart Van Assche [Sat, 4 Jul 2020 03:16:07 +0000 (20:16 -0700)]
os/windows/posix.c: Strip trailing whitespace

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
10 days agoMerge branch 'workqueue' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 4 Jul 2020 19:46:46 +0000 (13:46 -0600)]
Merge branch 'workqueue' of https://github.com/bvanassche/fio

* 'workqueue' of https://github.com/bvanassche/fio:
  workqueue: Fix race conditions in the workqueue mechanism
  workqueue: Rework while loop locking

10 days agoworkqueue: Fix race conditions in the workqueue mechanism
Bart Van Assche [Sat, 4 Jul 2020 16:44:56 +0000 (09:44 -0700)]
workqueue: Fix race conditions in the workqueue mechanism

From https://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_cond_wait.html:
"The application shall ensure that these functions are called with mutex
locked by the calling thread; otherwise, an error (for
PTHREAD_MUTEX_ERRORCHECK and robust mutexes) or undefined behavior (for
other mutexes) results.

From https://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_cond_signal.html:
"however, if predictable scheduling behavior is required, then that mutex
shall be locked by the thread calling pthread_cond_broadcast() or
pthread_cond_signal()."

Hence always hold the associated mutex around pthread_cond_wait() and
pthread_signal() calls.

This patch fixes the hangs reported by Travis and Appveyor for test case
t0013.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
10 days agoworkqueue: Rework while loop locking
Bart Van Assche [Sat, 4 Jul 2020 16:39:54 +0000 (09:39 -0700)]
workqueue: Rework while loop locking

Reduce the number of pthread_mutex_lock() and unlock calls by moving
these outside while loops. Other than protecting the all_sw_idle() call with
the wq->flush_lock, this patch does not change any functionality.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
10 days agoMerge branch 'overlap' of https://github.com/bvanassche/fio
Jens Axboe [Sat, 4 Jul 2020 14:28:43 +0000 (08:28 -0600)]
Merge branch 'overlap' of https://github.com/bvanassche/fio

* 'overlap' of https://github.com/bvanassche/fio:
  Add a test for serialize_overlap=1
  Enable error checking for the mutex that serializes overlapping I/O
  Fix a potentially infinite loop in check_overlap()
  t/run-fio-tests.py: Increase IOPS tolerance further
  .appveyor.yml: Select the latest software image

11 days agoAdd a test for serialize_overlap=1
Bart Van Assche [Sun, 14 Jun 2020 19:09:45 +0000 (12:09 -0700)]
Add a test for serialize_overlap=1

This test causes fio to call check_overlap().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
11 days agoEnable error checking for the mutex that serializes overlapping I/O
Bart Van Assche [Sat, 13 Jun 2020 18:33:37 +0000 (11:33 -0700)]
Enable error checking for the mutex that serializes overlapping I/O

Since it is nontrivial to verify that pthread_mutex_lock() and unlock
calls are paired for the overlap_check mutex, enable runtime verification.
The PTHREAD_ERRORCHECK_MUTEX_INITIALIZER_NP initializer causes
pthread_mutex_lock() to fail with EDEADLK if the mutex was already locked
and causes pthread_mutex_unlock() to fail with EPERM if the mutex is owned
by another thread.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
11 days agoFix a potentially infinite loop in check_overlap()
Bart Van Assche [Sat, 13 Jun 2020 18:23:39 +0000 (11:23 -0700)]
Fix a potentially infinite loop in check_overlap()

If the following happens:
* check_overlap() finds an overlap.
* All other threads finish after the overlap has been found and before
  the next iteration of the do/while loop starts.

Then the do/while loop in check_overlap() will iterate forever. Fix this
by rewriting check_overlap() such that this cannot happen.

This patch fixes the following Coverity complaint:

CID 184174 (#2 of 2): Double lock (LOCK)

Fixes: c06379a65d46 ("fio: enable overlap checking with offload submission")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
11 days agot/run-fio-tests.py: Increase IOPS tolerance further
Bart Van Assche [Sat, 4 Jul 2020 04:19:23 +0000 (21:19 -0700)]
t/run-fio-tests.py: Increase IOPS tolerance further

From an Appveyor build:

DEBUG:root:Test 12: iops1: 984.676882

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
11 days ago.appveyor.yml: Select the latest software image
Bart Van Assche [Sat, 4 Jul 2020 04:13:31 +0000 (21:13 -0700)]
.appveyor.yml: Select the latest software image

This is necessary to make the overlap test (t0013) pass.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
11 days agoconfigure: fix bad indentation
Jens Axboe [Fri, 3 Jul 2020 14:34:56 +0000 (08:34 -0600)]
configure: fix bad indentation

Signed-off-by: Jens Axboe <axboe@kernel.dk>
11 days agofio: better info when engine not found
Yigal Korman [Fri, 3 Jul 2020 12:38:43 +0000 (15:38 +0300)]
fio: better info when engine not found

Jens suggested we provide the user with the likely reasons why we failed
to load the given engine.
This should help better resolve these issues.

Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
11 days agoconfigure: new --dynamic-libengines build option
Yigal Korman [Fri, 3 Jul 2020 12:38:42 +0000 (15:38 +0300)]
configure: new --dynamic-libengines build option

When enabled, some of the more dependency-heavy internal engines are
converted to "plugin" engines, i.e. they are built into separate object
files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not
force a long list of package dependencies from the base fio package.

Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
11 days agoconfigure/Makefile: engine LIBS consistency
Yigal Korman [Fri, 3 Jul 2020 12:38:41 +0000 (15:38 +0300)]
configure/Makefile: engine LIBS consistency

Some engines were adding their LIBS in the Makefile and some were
declaring them in the configure script.
This is inconsistent and confusing and prevents easy bulk changes to the
engines.

Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
11 days agofio: don't retry engine search on failure
Yigal Korman [Fri, 3 Jul 2020 12:38:40 +0000 (15:38 +0300)]
fio: don't retry engine search on failure

ioengine_load will try to load a given engine and if it doesn't match
the current one (default is always psync), it will remove the current
one and try again.
This makes sense when the engine is loaded successfully, but if it's
not, we try to load it twice for no reason.

Fix by failing if we were unable to load requested engine and not
retrying.

Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
12 days agoMerge branch 'num2str' of https://github.com/bvanassche/fio
Jens Axboe [Fri, 3 Jul 2020 00:08:14 +0000 (18:08 -0600)]
Merge branch 'num2str' of https://github.com/bvanassche/fio

* 'num2str' of https://github.com/bvanassche/fio:
  num2str(): Add the E (exa) prefix
  Add a num2str() unit test
  num2str(): Fix overflow handling
  num2str(): Remove the fmt[] array
  num2str(): Use asprintf() instead of malloc()

12 days agonum2str(): Add the E (exa) prefix
Bart Van Assche [Thu, 2 Jul 2020 23:31:35 +0000 (16:31 -0700)]
num2str(): Add the E (exa) prefix

This change guarantees that all 64-bit integers that should be assigned a
multiplier are assigned a multiplier.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
12 days agoAdd a num2str() unit test
Bart Van Assche [Thu, 2 Jul 2020 22:18:37 +0000 (15:18 -0700)]
Add a num2str() unit test

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
12 days agonum2str(): Fix overflow handling
Bart Van Assche [Thu, 2 Jul 2020 21:53:09 +0000 (14:53 -0700)]
num2str(): Fix overflow handling

This patch changes the output for the arguments (UINT64_MAX, 4, 1, 0,
N2S_NONE, "18.4") from "18.4(null)" into "18.4".

This patch fixes the following Coverity complaint:

CID 169307 (#1 of 1): Out-of-bounds read (OVERRUN)
27. overrun-local: Overrunning array of 6 8-byte elements at element
    index 6 (byte offset 55) by dereferencing pointer unitprefix + post_index.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
12 days agonum2str(): Remove the fmt[] array
Bart Van Assche [Thu, 2 Jul 2020 21:56:16 +0000 (14:56 -0700)]
num2str(): Remove the fmt[] array

This patch does not change any functionality.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
12 days agonum2str(): Use asprintf() instead of malloc()
Bart Van Assche [Thu, 2 Jul 2020 21:47:01 +0000 (14:47 -0700)]
num2str(): Use asprintf() instead of malloc()

This patch reduces the size of the buffer allocated by num2str().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 weeks agoMerge branch 'pmemblk' of https://github.com/bvanassche/fio
Jens Axboe [Sun, 28 Jun 2020 13:08:36 +0000 (07:08 -0600)]
Merge branch 'pmemblk' of https://github.com/bvanassche/fio

* 'pmemblk' of https://github.com/bvanassche/fio:
  Unbreak the pmemblk engine

2 weeks agoUnbreak the pmemblk engine
Bart Van Assche [Sat, 27 Jun 2020 14:26:24 +0000 (07:26 -0700)]
Unbreak the pmemblk engine

Reported-by: Yi Zhang <yi.zhang@redhat.com>
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Fixes: e9c7be0e32e6 ("pmemblk: Fix a memory leak")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 weeks agoMerge branch 'compiler' of https://github.com/bvanassche/fio
Jens Axboe [Wed, 24 Jun 2020 15:02:06 +0000 (09:02 -0600)]
Merge branch 'compiler' of https://github.com/bvanassche/fio

* 'compiler' of https://github.com/bvanassche/fio:
  Merge compiler/gcc4.h into compiler/compiler.h

3 weeks agoMerge compiler/gcc4.h into compiler/compiler.h
Bart Van Assche [Mon, 22 Jun 2020 15:21:06 +0000 (08:21 -0700)]
Merge compiler/gcc4.h into compiler/compiler.h

Follow the example of the Linux kernel and merge the file compiler/gcc4.h
into compiler/gcc.h. Additionally, enable __must_check,
__compiletime_warning() and __compiletime_error() for clang.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoMerge branch 'master' of https://github.com/safl/fio
Jens Axboe [Tue, 23 Jun 2020 18:41:35 +0000 (12:41 -0600)]
Merge branch 'master' of https://github.com/safl/fio

* 'master' of https://github.com/safl/fio:
  Changed signedness of seqlock.sequence fixing comparison-warning

3 weeks agoChanged signedness of seqlock.sequence fixing comparison-warning
Simon A. F. Lund [Tue, 23 Jun 2020 17:50:22 +0000 (19:50 +0200)]
Changed signedness of seqlock.sequence fixing comparison-warning

Signed-off-by: Simon A. F. Lund <simon.lund@samsung.com>
3 weeks agocompiler/compiler.h: include clang 6.0 and above
Jens Axboe [Tue, 23 Jun 2020 17:49:14 +0000 (11:49 -0600)]
compiler/compiler.h: include clang 6.0 and above

Fixes: baa16486051f ("compiler/compiler.h: minimum GCC version is 4.9")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agooslib/linux-blkzoned: fix bogus poiter alignment warning
Jens Axboe [Tue, 23 Jun 2020 17:41:56 +0000 (11:41 -0600)]
oslib/linux-blkzoned: fix bogus poiter alignment warning

GCC 10.x reports:

oslib/linux-blkzoned.c: In function ‘blkzoned_report_zones’:
oslib/linux-blkzoned.c:146:18: warning: taking address of packed member of ‘struct blk_zone_report’ may result in an unaligned pointer value [-Waddress-of-packed-member]
  146 |  blkz = (void *) &hdr->zones[0];

which is totally fine, but we can easily avoid this warning by just
setting the 'blkz' pointer to the end of 'hdr'.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agocompiler/compiler.h: minimum GCC version is 4.9
Jens Axboe [Tue, 23 Jun 2020 17:29:36 +0000 (11:29 -0600)]
compiler/compiler.h: minimum GCC version is 4.9

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agoMerge branch 'atomics' of https://github.com/bvanassche/fio
Jens Axboe [Mon, 22 Jun 2020 02:48:05 +0000 (20:48 -0600)]
Merge branch 'atomics' of https://github.com/bvanassche/fio

* 'atomics' of https://github.com/bvanassche/fio:
  Optimize fio_gettime_offload()
  Optimize the seqlock implementation
  Add a test that sets gtod_cpu=1
  t/run-fio-tests.py: Increase IOPS tolerance further
  fio: Use atomic_load_acquire() and atomic_store_release() where appropriate
  engines/io_uring: Use atomic_{load_acquire,store_release}()
  engines/libaio: Use atomic_store_release() instead of read_barrier()
  arch/arch.h: Introduce atomic_{load_acquire,store_release}()
  fio_sem: Remove a read_barrier() call
  Make __rand_0_1() compatible with clang
  configure: Use -Wimplicit-fallthrough=2 instead of -Wimplicit-fallthrough=3

3 weeks agoOptimize fio_gettime_offload()
Bart Van Assche [Sun, 21 Jun 2020 21:31:08 +0000 (14:31 -0700)]
Optimize fio_gettime_offload()

This patch not only an optimization but also a bug fix because it guarantees
that fio_gettime_offload() never returns an inconsistent timestamp, something
that could happen before. In this context inconsistent means returning the
.tv_sec value from one gettimeofday() call and the .tv_usec value from another
gettimeofday() call.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoOptimize the seqlock implementation
Bart Van Assche [Sun, 21 Jun 2020 21:24:36 +0000 (14:24 -0700)]
Optimize the seqlock implementation

Use atomic_load_acquire() and atomic_store_release() instead of barriers
where appropriate.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoAdd a test that sets gtod_cpu=1
Bart Van Assche [Mon, 22 Jun 2020 01:28:08 +0000 (18:28 -0700)]
Add a test that sets gtod_cpu=1

None of the existing tests sets gtod_cpu=1. Hence add a test that sets
gtod_cpu=1.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agot/run-fio-tests.py: Increase IOPS tolerance further
Bart Van Assche [Mon, 22 Jun 2020 02:45:58 +0000 (19:45 -0700)]
t/run-fio-tests.py: Increase IOPS tolerance further

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agofio: Use atomic_load_acquire() and atomic_store_release() where appropriate
Bart Van Assche [Sun, 21 Jun 2020 20:55:04 +0000 (13:55 -0700)]
fio: Use atomic_load_acquire() and atomic_store_release() where appropriate

The write_barrier() in io_completed() and also the read barriers in
verify.c are misplaced: the write barrier should occur before the flags
update instead of after and the read barriers should occur after the flags
read instead of before. Fix this by using atomic_{load_acquire,
store_release}() instead of read and write barriers.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoengines/io_uring: Use atomic_{load_acquire,store_release}()
Bart Van Assche [Sun, 21 Jun 2020 21:17:58 +0000 (14:17 -0700)]
engines/io_uring: Use atomic_{load_acquire,store_release}()

This patch improves performance by using acquire and release semantics
instead of barriers and also brings the io_uring engine code closer to that
of liburing.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoengines/libaio: Use atomic_store_release() instead of read_barrier()
Bart Van Assche [Sun, 21 Jun 2020 21:16:59 +0000 (14:16 -0700)]
engines/libaio: Use atomic_store_release() instead of read_barrier()

This is a micro-optimization that does not affect the functionality of the
code.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoarch/arch.h: Introduce atomic_{load_acquire,store_release}()
Bart Van Assche [Sun, 21 Jun 2020 21:38:41 +0000 (14:38 -0700)]
arch/arch.h: Introduce atomic_{load_acquire,store_release}()

Implement atomic_load_acquire() and atomic_store_release() with C11
atomic operations. These two primitives will be used in later patches.
This patch increases the minimum requirement for the compiler fio is
built with from C90 to C11.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agofio_sem: Remove a read_barrier() call
Bart Van Assche [Sun, 21 Jun 2020 20:54:15 +0000 (13:54 -0700)]
fio_sem: Remove a read_barrier() call

Since pthread_mutex_lock() already provides load-acquire semantics, calling
read_barrier() just after pthread_mutex_lock() is not necessary. See also
commit 4d4e80f2b426 ("Revamp file locking").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoMake __rand_0_1() compatible with clang
Bart Van Assche [Mon, 22 Jun 2020 02:01:00 +0000 (19:01 -0700)]
Make __rand_0_1() compatible with clang

This patch fixes the following clang compiler error:

crc/../arch/../lib/rand.h:109:25: error: implicit conversion from
      'unsigned long long' to 'double' changes value from 18446744073709551615
      to 18446744073709551616 [-Werror,-Wimplicit-int-float-conversion]
                return (val + 1.0) / (FRAND64_MAX + 1.0);
                                      ^~~~~~~~~~~ ~
crc/../arch/../lib/rand.h:9:22: note: expanded from macro 'FRAND64_MAX'
 #define FRAND64_MAX     (-1ULL)
                         ^~~~~

Fixes: e7b240474543 ("Fixups for poisson rate")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
3 weeks agoconfigure: Use -Wimplicit-fallthrough=2 instead of -Wimplicit-fallthrough=3
Bart Van Assche [Mon, 22 Jun 2020 02:09:48 +0000 (19:09 -0700)]
configure: Use -Wimplicit-fallthrough=2 instead of -Wimplicit-fallthrough=3

This makes clang accept /* fallthrough */ as a fallthrough annotation.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
4 weeks agoPriority bit log file format documentation update
Phillip Chen [Fri, 12 Jun 2020 18:30:24 +0000 (12:30 -0600)]
Priority bit log file format documentation update

Updates the description of log file formats to include and
describe the priority bit entry.

Signed-off-by: Phillip Chen <phillip.a.chen@seagate.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
4 weeks agoMerge branch 'zbd' of https://github.com/bvanassche/fio
Jens Axboe [Fri, 12 Jun 2020 14:48:44 +0000 (08:48 -0600)]
Merge branch 'zbd' of https://github.com/bvanassche/fio

* 'zbd' of https://github.com/bvanassche/fio:
  pshared: Improve mutex_init_pshared_with_type()
  zbd: Fix zoned_block_device_info.zone_size documentation
  zbd: Fix spelling of the "zonemode" job option

4 weeks agoMerge branch 'master' of https://github.com/raphael-nutanix/fio
Jens Axboe [Fri, 12 Jun 2020 14:48:20 +0000 (08:48 -0600)]
Merge branch 'master' of https://github.com/raphael-nutanix/fio

* 'master' of https://github.com/raphael-nutanix/fio:
  Fix typo in libiscsi error message

4 weeks agoFix typo in libiscsi error message
Raphael Norwitz [Thu, 11 Jun 2020 00:59:03 +0000 (20:59 -0400)]
Fix typo in libiscsi error message

On authentication failure when connecting to an iscsi server, fio will
print "sicsi: failed to connect to LUN : Failed to log in to target.".

This change fixes a typo, changing sicsi to iscsi.

Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
5 weeks agopshared: Improve mutex_init_pshared_with_type()
Bart Van Assche [Sat, 30 May 2020 23:03:48 +0000 (16:03 -0700)]
pshared: Improve mutex_init_pshared_with_type()

Skipping the pthread_mutexattr_settype() call if type == 0 is questionable
because it makes it impossible to initialize a mutex with type == 0 (type 0
corresponds to PTHREAD_MUTEX_TIMED_NP on Linux). Use symbolic names for
pthread mutex types and leave out the type == 0 check.

Fixes: 3ed6894b00c4 ("pshared: Add mutex_init_pshared_with_type()")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 weeks agozbd: Fix zoned_block_device_info.zone_size documentation
Bart Van Assche [Sat, 30 May 2020 22:54:09 +0000 (15:54 -0700)]
zbd: Fix zoned_block_device_info.zone_size documentation

Commit ee3696bdfd84 changed the unit of 'zone_size' from 512 bytes into one
byte. Bring the documentation of that member variable in sync with the code.

Fixes: ee3696bdfd84 ("zbd: Use bytes unit")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 weeks agozbd: Fix spelling of the "zonemode" job option
Bart Van Assche [Sat, 30 May 2020 22:24:07 +0000 (15:24 -0700)]
zbd: Fix spelling of the "zonemode" job option

The comment above zbd_adjust_ddir() refers to the fio "zonemode" option.
Make sure that the name of that option has been spelled correctly.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 weeks agoMerge branch 'latency_window' of https://github.com/liu-song-6/fio
Jens Axboe [Wed, 10 Jun 2020 01:43:52 +0000 (19:43 -0600)]
Merge branch 'latency_window' of https://github.com/liu-song-6/fio

* 'latency_window' of https://github.com/liu-song-6/fio:
  init: fix unit of latency_window

5 weeks agoMerge branch 'nowait' of https://github.com/koct9i/fio
Jens Axboe [Wed, 10 Jun 2020 01:43:37 +0000 (19:43 -0600)]
Merge branch 'nowait' of https://github.com/koct9i/fio

* 'nowait' of https://github.com/koct9i/fio:
  engines: pvsync2 libaio io_uring: add support for RWF_NOWAIT

5 weeks agoMerge branch 'master' of https://github.com/bvanassche/fio
Jens Axboe [Wed, 10 Jun 2020 01:43:16 +0000 (19:43 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio

* 'master' of https://github.com/bvanassche/fio:
  client: Make skipping option appending in handle_job_opt() more selective
  client: Fix two memory leaks in handle_job_opt()
  Make json_object_add_value_string() duplicate its 'value' argument

5 weeks agoMerge branch 'client-leak-fix' of https://github.com/bvanassche/fio
Jens Axboe [Wed, 10 Jun 2020 01:42:35 +0000 (19:42 -0600)]
Merge branch 'client-leak-fix' of https://github.com/bvanassche/fio

* 'client-leak-fix' of https://github.com/bvanassche/fio:
  client: Fix another memory leak in an error path
  client: Fix a memory leak in an error path

5 weeks agoclient: Fix another memory leak in an error path
Bart Van Assche [Sat, 30 May 2020 22:09:00 +0000 (15:09 -0700)]
client: Fix another memory leak in an error path

Duplicate the hostname after if (...) goto err instead of before that check.
This was found by inspecting get_new_client() callers. It is not clear to me
why Coverity did not complain about this function.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 weeks agoclient: Fix a memory leak in an error path
Bart Van Assche [Sat, 30 May 2020 21:58:34 +0000 (14:58 -0700)]
client: Fix a memory leak in an error path

This patch fixes the following Coverity complaint:

CID 300985 (#1 of 1): Resource leak (RESOURCE_LEAK)
leaked_storage: Variable client going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
5 weeks agot/run-fio-tests.py: Accept a wider range of IOPS values
Bart Van Assche [Sun, 7 Jun 2020 19:11:04 +0000 (12:11 -0700)]
t/run-fio-tests.py: Accept a wider range of IOPS values

This patch prevents that test 11 sporadically fails as follows:

DEBUG:root:Test 11: return code: 0
DEBUG:root:Test 11: skipped 0 lines decoding JSON data
DEBUG:root:Test 11: iops1: 997.670549
DEBUG:root:Test 11: ratio: 8.163684

Test 11 FAILED:  iops value mismatch,

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoSwitch to Python3
Bart Van Assche [Sun, 7 Jun 2020 19:11:03 +0000 (12:11 -0700)]
Switch to Python3

Support for Python2.7 in Linux distributions is being phased out. As an
example, some Linux distributions do not provide Python2.7 versions of
some of the Python packages fio Python scripts relies on. Hence switch
to Python3. Most of the Python code changes in this patch have been
generated by running the Python scripts through 2to3, the Python2 to
Python3 converter from python.org.

Notes:
- Python is only used by fio test scripts and not by fio itself.
- Older versions of RHEL / CentOS 6 do not support Python3.
  Python3 binaries for RHEL 6 / CentOS 6 are available e.g. in the EPEL
  repository (https://fedoraproject.org/wiki/EPEL).

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoci/travis-install.sh, MacOS: Install the Python 'six' package
Vincent Fu [Sun, 7 Jun 2020 19:11:02 +0000 (12:11 -0700)]
ci/travis-install.sh, MacOS: Install the Python 'six' package

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
[bvanassche: edited commit message / only retained "pip install six" command /
 changed 'pip' into 'pip3']
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoci/travis-install.sh: Install python3-six package
Bart Van Assche [Sun, 7 Jun 2020 19:11:01 +0000 (12:11 -0700)]
ci/travis-install.sh: Install python3-six package

Additionally, report the Python3 path and version.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks ago.travis.yml: Move shell code out of .travis.yml
Bart Van Assche [Sun, 7 Jun 2020 19:11:00 +0000 (12:11 -0700)]
.travis.yml: Move shell code out of .travis.yml

Shell code in a .travis.yml file is hard to test. Make it easy to run the
shell code locally and to verify that code with shellcheck by moving the
shell code into separate files.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agoMakefile: include linux-blkzoned.c for Android, if set
Jens Axboe [Fri, 5 Jun 2020 13:06:46 +0000 (07:06 -0600)]
Makefile: include linux-blkzoned.c for Android, if set

Fixes: https://github.com/axboe/fio/issues/1008
Signed-off-by: Jens Axboe <axboe@kernel.dk>
5 weeks agozbd: Fix max_open_zones checks
Shin'ichiro Kawasaki [Wed, 27 May 2020 01:20:13 +0000 (10:20 +0900)]
zbd: Fix max_open_zones checks

Commit 219c662d3b12 ("zbd: introduce per job maximum open zones limit")
introduced job_max_open_zones option which limits the number of open
zones per job. It has similar role as max_open_zones option which limits
the number of open zones for all jobs. It was intended that these two
options both work, but the commit replaced some checks for max_open_zones
simply with checks for job_max_open_zones. Because of this, when
max_open_zones is set and job_max_open_zones is not set, fio fails to
limit the number of open zones. This resulted in test case #29 failure
of t/zbd/test-zbd-support script for regular null_blk devices.

To fix the failure, modify the checks to target both job_max_open_zones
and max_open_zones.

Fixes: 219c662d3b12 ("zbd: introduce per job maximum open zones limit")
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agogfio: don't have multiple versions of main_ui
Jens Axboe [Tue, 2 Jun 2020 14:20:03 +0000 (08:20 -0600)]
gfio: don't have multiple versions of main_ui

Also comment a fall-through case appropriately, so newer GCC
don't warn.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoengines: pvsync2 libaio io_uring: add support for RWF_NOWAIT
Konstantin Khlebnikov [Mon, 25 May 2020 17:29:22 +0000 (20:29 +0300)]
engines: pvsync2 libaio io_uring: add support for RWF_NOWAIT

Add bool option "nowait" into engines which could support this feature.

Add test for libaio >= 0.3.111 into configure script to be compatible with
prior versions where field iocb.aio_rw_flags was declared as "__pad2".

By default if a request cannot be executed immediately (e.g. resource starvation,
waiting on locks) it is queued and the initiating process will be blocked until
the required resource becomes free.

This option sets the RWF_NOWAIT flag (supported from the 4.14 Linux kernel) and
the call will return instantly with EAGAIN or a partial result rather than waiting.

It is useful to also use ignore_error=EAGAIN when using this option.

Note: glibc 2.27, 2.28 have a bug in syscall wrappers preadv2, pwritev2.
They return EOPNOTSUP instead of EAGAIN.

For cached I/O, using this option usually means a request operates only with
cached data. Currently the RWF_NOWAIT flag does not supported for cached write.

For direct I/O, requests will only succeed if cache invalidation isn't required,
file blocks are fully allocated and the disk request could be issued immediately.

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Link: https://lwn.net/Articles/724631/
6 weeks agoMerge branch 'pshared1' of https://github.com/kusumi/fio
Jens Axboe [Fri, 29 May 2020 14:16:10 +0000 (08:16 -0600)]
Merge branch 'pshared1' of https://github.com/kusumi/fio

* 'pshared1' of https://github.com/kusumi/fio:
  pshared: fix comment on supported platforms

6 weeks agopshared: fix comment on supported platforms
Tomohiro Kusumi [Fri, 29 May 2020 14:08:34 +0000 (23:08 +0900)]
pshared: fix comment on supported platforms

FreeBSD supports it, but NetBSD/OpenBSD don't.

 # uname
 FreeBSD
 # ./configure | grep "POSIX pshared"
 POSIX pshared support         yes

 # uname
 NetBSD
 # ./configure | grep "POSIX pshared"
 POSIX pshared support         no

Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
6 weeks agoMerge branch 'python3-testing' of https://github.com/vincentkfu/fio
Jens Axboe [Thu, 28 May 2020 19:27:15 +0000 (13:27 -0600)]
Merge branch 'python3-testing' of https://github.com/vincentkfu/fio

* 'python3-testing' of https://github.com/vincentkfu/fio:
  travis: install python3 scipy for Linux and macOS tests
  testing: change two test scripts to refer to python3

6 weeks agotravis: install python3 scipy for Linux and macOS tests
Vincent Fu [Thu, 28 May 2020 14:12:52 +0000 (10:12 -0400)]
travis: install python3 scipy for Linux and macOS tests

Since the test scripts triggered by TravisCI now all rely on python3,
make sure we always install scipy for python3.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
6 weeks agotesting: change two test scripts to refer to python3
Vincent Fu [Thu, 28 May 2020 13:05:06 +0000 (09:05 -0400)]
testing: change two test scripts to refer to python3

Since python2 is no longer supported we should now use python3 in our
test scripts. Change the shebang lines for two test scripts to refer to
python3.

Note that t/sgunmap-test.py and t/sgunmap-perf.py still refer to
python2.  I no longer have the means to test those two scripts and am
leaving those unchanged.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
6 weeks agoMerge branch 'testing' of https://github.com/vincentkfu/fio
Jens Axboe [Thu, 28 May 2020 17:07:31 +0000 (11:07 -0600)]
Merge branch 'testing' of https://github.com/vincentkfu/fio

* 'testing' of https://github.com/vincentkfu/fio:
  .travis: enable arm64 architecture builds
  t/run-fio-tests: pass-through arguments to test scripts
  appveyor: use on_finish section to upload artifacts

6 weeks ago.travis: enable arm64 architecture builds
Vincent Fu [Tue, 26 May 2020 20:55:58 +0000 (16:55 -0400)]
.travis: enable arm64 architecture builds

The travis-ci containers do not support the cmdprio_percentage option.
So skip latency_percentile.py tests using that option.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
6 weeks agot/run-fio-tests: pass-through arguments to test scripts
Vincent Fu [Tue, 26 May 2020 20:54:44 +0000 (16:54 -0400)]
t/run-fio-tests: pass-through arguments to test scripts

Add an option to pass-through arguments to specified test scripts. This
can be used to alter the behavior of tests on different platforms.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
6 weeks agoappveyor: use on_finish section to upload artifacts
Vincent Fu [Tue, 26 May 2020 17:22:38 +0000 (13:22 -0400)]
appveyor: use on_finish section to upload artifacts

We cannot rely on the artifacts section to upload test artifacts because
when a test failure occurs, the entire build process stops and the
artifacts are not uploaded. Use the on_finish section instead to upload
test artifacts.

Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
6 weeks agozbd: Fix compilation error on BSD
Shin'ichiro Kawasaki [Thu, 28 May 2020 12:56:42 +0000 (21:56 +0900)]
zbd: Fix compilation error on BSD

Commit b76949618d55 ("fio: Generalize zonemode=zbd") enabled zbd.c
compilation on other operating systems than Linux. This caused a
compilation error on NetBSD as follows:

ld: zbd.o: in function `parse_zone_info':
fio/zbd.c:422: undefined reference to `pthread_mutexattr_setpshared'
ld: zbd.o: in function `init_zone_info':
fio/zbd.c:378: undefined reference to `pthread_mutexattr_setpshared'
gmake: *** [Makefile:483: fio] Error 1

Same error is expected on other BSD OSes.

Fix this by initializing mutex using helper functions pshared.c provides.
To initialize mutex with POSIX_MUTEX_RECURSIVE attribute type, utilize
mutex_init_pshared_with_type().

Reported-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Fixes: b76949618d55 ("fio: Generalize zonemode=zbd")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agopshared: Add mutex_init_pshared_with_type()
Shin'ichiro Kawasaki [Thu, 28 May 2020 12:56:41 +0000 (21:56 +0900)]
pshared: Add mutex_init_pshared_with_type()

To initialize mutex to be shared across processes, the helper function
mutex_init_pshared() is available. However, it does not allow to set
mutex attribute types such as POSIX_MUTEX_RECURSIVE.

To allow setting mutex attribute types, introduce another helper function
mutex_init_pshared_with_type(). It initialize mutex for sharing across
processes and set attribute types specified as its argument.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agot/zbd: make the test script easier to terminate
Dmitry Fomichev [Mon, 25 May 2020 21:32:56 +0000 (06:32 +0900)]
t/zbd: make the test script easier to terminate

Very often, it takes more than one ^C to terminate test-zbd-support
script. Just a single ^C does end the test that is currently being
executed, but then the script proceeds to the next test. This commit
adds a simple signal handler to exit the test loop after receiving
a Ctrl-C.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agot/zbd: beautify test script output
Dmitry Fomichev [Mon, 25 May 2020 21:32:55 +0000 (06:32 +0900)]
t/zbd: beautify test script output

The test printout columns are better aligned now. Also, the test
result, PASS/FAIL, is now color-coded and that makes it easier
to spot failures.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agolibzbc: fix whitespace errors
Dmitry Fomichev [Mon, 25 May 2020 21:32:54 +0000 (06:32 +0900)]
libzbc: fix whitespace errors

Make checkpatch happy... no functional change.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agolibzbc: cleanup init code
Dmitry Fomichev [Mon, 25 May 2020 21:32:53 +0000 (06:32 +0900)]
libzbc: cleanup init code

Make sure every allocated data structure gets freed in case of
unsuccessful libzbc ioengine initialization.

Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agoclient: Make skipping option appending in handle_job_opt() more selective
Bart Van Assche [Fri, 7 Feb 2020 14:54:29 +0000 (06:54 -0800)]
client: Make skipping option appending in handle_job_opt() more selective

Instead of not appending an option to the option list if JSON output is
disabled, only skip appending an option to the JSON option list. See also
commit b127b679769c ("client: fix segfault for !json output").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
7 weeks agoclient: Fix two memory leaks in handle_job_opt()
Bart Van Assche [Sun, 24 May 2020 20:35:54 +0000 (13:35 -0700)]
client: Fix two memory leaks in handle_job_opt()

Do not leak p if pdu->global != 0.

This is an improvement for a previous attempt to fix handle_job_opt(). See
also commit ebae36a28aee ("client: Fix memory leaks in handle_job_opt()").

Do not leak strdup(pdu->name) when calling json_object_add_value_string().
That function namely (indirectly) duplicates its 'name' argument.

This patch fixes the following Coverity complaint:

CID 169311 (#1 of 1): Resource leak (RESOURCE_LEAK)
9. leaked_storage: Variable p going out of scope leaks the storage it points to.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
7 weeks agoMake json_object_add_value_string() duplicate its 'value' argument
Bart Van Assche [Sun, 24 May 2020 20:25:37 +0000 (13:25 -0700)]
Make json_object_add_value_string() duplicate its 'value' argument

Having some json_object_add_value_string() callers duplicate the 'value'
argument others not is dubious because it can lead to dangling pointers,
e.g. when passing strings to json_object_add_value_string() that live less
long than the created JSON object. See e.g. the 'time_buf' stack array that
is passed by fio_client_json_init() to json_object_add_value_string().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
7 weeks agoMerge branch 'parse-and-fill-pattern' of https://github.com/bvanassche/fio
Jens Axboe [Sun, 24 May 2020 18:03:56 +0000 (12:03 -0600)]
Merge branch 'parse-and-fill-pattern' of https://github.com/bvanassche/fio

* 'parse-and-fill-pattern' of https://github.com/bvanassche/fio:
  Do not read past the end of fmt_desc[]
  Declare a static variable 'const'
  Fix spelling in a source code comment

7 weeks agoDo not read past the end of fmt_desc[]
Bart Van Assche [Sun, 24 May 2020 03:39:47 +0000 (20:39 -0700)]
Do not read past the end of fmt_desc[]

Callers of parse_format() pass a size in bytes while the parse_format()
function itself expects a number of elements. Fix this by making the
fmt_desc[] array NULL-terminated. This patch fixes the following Coverity
complaint:

CID 300986 (#1 of 1): Out-of-bounds access (OVERRUN)
overrun-buffer-arg: Overrunning array fmt_desc of 1 24-byte elements by
passing it to a function which accesses it at element index 23 (byte
offset 575) using argument 24U.

Cc: Roman Pen <r.peniaev@gmail.com>
Fixes: 634bd210c17a ("lib/pattern: add set of functions to parse combined pattern input")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
7 weeks agoDeclare a static variable 'const'
Bart Van Assche [Sun, 24 May 2020 03:30:17 +0000 (20:30 -0700)]
Declare a static variable 'const'

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
7 weeks agoFix spelling in a source code comment
Bart Van Assche [Sun, 24 May 2020 03:28:21 +0000 (20:28 -0700)]
Fix spelling in a source code comment

Change two occurrences of 'descritor' into 'descriptor'

Cc: Roman Pen <r.peniaev@gmail.com>
Fixes: 634bd210c17a ("lib/pattern: add set of functions to parse combined pattern input")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
7 weeks agoFio 3.20 fio-3.20
Jens Axboe [Sat, 23 May 2020 17:14:14 +0000 (11:14 -0600)]
Fio 3.20

Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agoMerge branch 'master' of https://github.com/ffontaine/fio
Jens Axboe [Sat, 23 May 2020 17:13:25 +0000 (11:13 -0600)]
Merge branch 'master' of https://github.com/ffontaine/fio

* 'master' of https://github.com/ffontaine/fio:
  Makefile: fix build of io_uring on sh4

7 weeks agoMakefile: fix build of io_uring on sh4
Fabrice Fontaine [Sat, 23 May 2020 17:07:40 +0000 (19:07 +0200)]
Makefile: fix build of io_uring on sh4

SuperH compile currently fails with:

/usr/lfs/hdd_v1/rc-buildroot-test/scripts/instance-0/output-1/host/opt/ext-toolchain/bin/../lib/gcc/sh4-buildroot-linux-uclibc/8.3.0/../../../../sh4-buildroot-linux-uclibc/bin/ld: t/io_uring.o: in function `submitter_fn':
/usr/lfs/hdd_v1/rc-buildroot-test/scripts/instance-0/output-1/build/fio-3.19/t/io_uring.c:131: undefined reference to `arch_flags'
/usr/lfs/hdd_v1/rc-buildroot-test/scripts/instance-0/output-1/host/opt/ext-toolchain/bin/../lib/gcc/sh4-buildroot-linux-uclibc/8.3.0/../../../../sh4-buildroot-linux-uclibc/bin/ld: /usr/lfs/hdd_v1/rc-buildroot-test/scripts/instance-0/output-1/build/fio-3.19/t/io_uring.c:367: undefined reference to `arch_flags'
collect2: error: ld returned 1 exit status

Fix that by ensuring we have a stub arch.o with the necessary arch flags

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
7 weeks agozbd: make zbd_info->mutex non-recursive
Alexey Dobriyan [Thu, 21 May 2020 23:17:16 +0000 (02:17 +0300)]
zbd: make zbd_info->mutex non-recursive

There is no reason for it to be recursive. Resursiveness leaked
from struct fio_zone_info::mutex initialisation.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agozbd: introduce per job maximum open zones limit
Alexey Dobriyan [Thu, 21 May 2020 23:17:15 +0000 (02:17 +0300)]
zbd: introduce per job maximum open zones limit

It is not possible to maintain sustained per-thread iodepth in ZBD mode.
The way code is written, "max_open_zones" acts as a global limit, and
once one or few threads open all "max_open_zones" zones, other threads
can't open anything and _exit_ prematurely.

This config is guaranteed to make equal number of zone resets/IO now:
each thread generates identical pattern and doesn't intersect with other
threads:

zonemode=zbd
zonesize=...
rw=write

numjobs=N
offset_increment=M*zonesize

[j]
size=M*zonesize

Patch introduces "job_max_open_zones" which is per-thread/process limit.
"max_open_zones" remains per file/device limit. Both limits are checked
for each open zone so one thread can't kick out others.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agozbd: don't lock zones outside working area
Alexey Dobriyan [Thu, 21 May 2020 23:17:14 +0000 (02:17 +0300)]
zbd: don't lock zones outside working area

Currently threads lock each other zones even if their working areas as
defined by [f->file_offset, f->file_offset + f->io_size) don't intersect.
This leads to unnecessary quiescing.

Patch clamps every zone to [->min_zone, ->max_zone) when doing search
for opened zone and more importantly adds an assert so that any
unnecessary zone locking becomes very visible.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
7 weeks agozbd: bump ZBD_MAX_OPEN_ZONES
Alexey Dobriyan [Thu, 21 May 2020 23:17:13 +0000 (02:17 +0300)]
zbd: bump ZBD_MAX_OPEN_ZONES

128 opened zones is not enough for us!

4096 opened zones is OK for 64×iodepth=64 stress testing.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>