fio.git
65 min agoMerge branch 'update-fio-ioops-version' of https://github.com/diameter/fio master
Jens Axboe [Wed, 25 Nov 2020 18:01:00 +0000 (11:01 -0700)]
Merge branch 'update-fio-ioops-version' of https://github.com/diameter/fio

* 'update-fio-ioops-version' of https://github.com/diameter/fio:
  ioengines: increment FIO_IOOPS_VERSION

11 hours agoioengines: increment FIO_IOOPS_VERSION
Ivan Andreyev [Sun, 22 Nov 2020 15:14:26 +0000 (18:14 +0300)]
ioengines: increment FIO_IOOPS_VERSION

Signed-off-by: Ivan Andreyev <ewantlf@gmail.com>
3 days agoerror out if ENOSPC during file layout
Kushal Kumaran [Fri, 20 Nov 2020 05:00:05 +0000 (21:00 -0800)]
error out if ENOSPC during file layout

Hi,

When I run fio with --create_only=1 and it runs out of space, it still
exits with status 0.  I could not figure out if this was intentional
(except for the fill_device case, where this is obviously the expected
behavior).

$ cat ~/fio
[global]
ioengine=posixaio
rw=readwrite
size=2g
directory=${HOME}/mounts/testdisk
thread=1

[trivial-readwrite-1g]
$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop6      976M  2.6M  907M   1% /home/kushal/mounts/testdisk
$ fio ~/fio --create_only=1
trivial-readwrite-1g: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pos
ixaio, iodepth=1
fio-3.12
Starting 1 thread
trivial-readwrite-1g: Laying out IO file (1 file / 2048MiB)
fio: ENOSPC on laying out file, stopping

Run status group 0 (all jobs):

Disk stats (read/write):
  loop6: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
$ echo $?
0

A trivial patch for this gives me the behavior I expect.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
12 days agoMerge branch 'segmented-threads'
Jens Axboe [Fri, 13 Nov 2020 17:06:26 +0000 (10:06 -0700)]
Merge branch 'segmented-threads'

Pull in the segmented segments code. This avoids having to setup a
thread area segment for the max (4096) number of jobs. Instead it's
done in chunks of 8. This results in a net reduction of more than 1G
of memory used for the normal use case.

* segmented-threads:
  Kill off 'max_jobs'
  Add thread_segments as needed
  Wrap thread_data in thread_segment

12 days agoKill off 'max_jobs' segmented-threads
Jens Axboe [Fri, 13 Nov 2020 16:17:11 +0000 (09:17 -0700)]
Kill off 'max_jobs'

We don't really need this anymore - we cap the number of segments
separately, and since the chunk size is small enough, this removes the
need to have per-OS limits on the thread area size.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
12 days agoAdd thread_segments as needed
Jens Axboe [Fri, 13 Nov 2020 16:02:35 +0000 (09:02 -0700)]
Add thread_segments as needed

Setup segments in units of 8, and add extra ones as needed. This avoids
having to setup one huge segment upfront for the maximum number of jobs.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
12 days agoWrap thread_data in thread_segment
Jens Axboe [Fri, 13 Nov 2020 15:33:50 +0000 (08:33 -0700)]
Wrap thread_data in thread_segment

No functional changes in this patch, just in preparation for having
multiple shm segments for the thread_data structures. They are getting
very large, and since it's hard to know upfront how many we need, let's
instead refactor the code a bit to allow us to add chunks of them as
needed when parsing the job file (or command line).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 days agot/latency_percentiles.py: tweak terse output parse
Jens Axboe [Thu, 12 Nov 2020 18:26:58 +0000 (11:26 -0700)]
t/latency_percentiles.py: tweak terse output parse

Seems like we still have some wonkiness on whether or not the fio version
string is printed. For now, let's just cut this down to '3;' for
detecting terse (v3) output, don't have time to fully investigate this
one right now.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 days agoFio 3.24 fio-3.24
Jens Axboe [Thu, 12 Nov 2020 16:40:58 +0000 (09:40 -0700)]
Fio 3.24

Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 days agoMakefile: fix fio version gen
Jens Axboe [Thu, 12 Nov 2020 16:36:45 +0000 (09:36 -0700)]
Makefile: fix fio version gen

A previous commit moved the cflags override, but neglected to move the
version generation that is being used by it.

Fixes: f4bd2c3d80bc ("fix dynamic engine build")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agot/latency_percentiles.py: correct terse parse
Jens Axboe [Mon, 9 Nov 2020 19:13:25 +0000 (12:13 -0700)]
t/latency_percentiles.py: correct terse parse

For some reason the script expects '3;fio-' when it should be latency.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoMake sure we do libaio engine compatability names
Jens Axboe [Mon, 9 Nov 2020 18:47:50 +0000 (11:47 -0700)]
Make sure we do libaio engine compatability names

This goes for both the dynamic engines, and the builtin ones. If
'aio' or 'linuxaio' is used, we want 'libaio'.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoconfigure: remove libaio-uring remnant
Jens Axboe [Mon, 9 Nov 2020 17:42:03 +0000 (10:42 -0700)]
configure: remove libaio-uring remnant

This was missed in the previous patch.

Fixes: d4946e79e833 ("Remove the "libaio over io_uring" mode")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoMakefile: ensure that external libs are linked properly with dynamic engine
Jens Axboe [Mon, 9 Nov 2020 17:05:11 +0000 (10:05 -0700)]
Makefile: ensure that external libs are linked properly with dynamic engine

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoRemove the "libaio over io_uring" mode
Jens Axboe [Mon, 9 Nov 2020 16:59:07 +0000 (09:59 -0700)]
Remove the "libaio over io_uring" mode

Not going to maintain this mode in liburing, so just get rid of it. It's
also caused some confusion where users think they need this enabled to
run io_uring, which isn't the case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoFIO_EXT_ENG_DIR should be default path
Jens Axboe [Mon, 9 Nov 2020 16:54:10 +0000 (09:54 -0700)]
FIO_EXT_ENG_DIR should be default path

Distros will override this anyway, make sure the default is what
will work on a normal build out of git.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agolist all available dynamic ioengines with --enghelp
Eric Sandeen [Mon, 9 Nov 2020 16:46:30 +0000 (10:46 -0600)]
list all available dynamic ioengines with --enghelp

When dynamic engines are enabled, "fio --enghelp" does not list all
available engines as the man page says it will:

If no ioengine is given, list all available ioengines.

Fix this by opening FIO_EXT_ENG_DIR and attempting dlopen_ioengine
everything that's found there.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agofix dynamic engine loading for libaio engine etc
Eric Sandeen [Mon, 9 Nov 2020 16:46:29 +0000 (10:46 -0600)]
fix dynamic engine loading for libaio engine etc

The dynamic engine loading for libaio (and some others) currently
fails because the dlopen routine is looking for ("lib%s", enginename)
which translates into "liblibiscsi.so":

openat(AT_FDCWD, "/usr/lib64/fio/liblibiscsi.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Engine libiscsi not found; Either name is invalid, was not built, or fio-engine-libiscsi package is missing.
fio: engine libiscsi not loadable
IO engine libiscsi not found

The Makefile decide to name this engine "iscsi" instead of "libiscsi",
which leads to "libiscsi.so" not "liblibiscsi.so" hence the mismatch.

OTOH, "liblibiscsi.so" seems a bit bonkers.

Try to resolve all this by:

1) make all of the engine names match the documented engine names
   in the Makefile, i.e. "iscsi" -> "libiscsi"
2) change the created library filenames to "fio-$(ENGINENAME)"
   from "lib$(ENGINENAME)" to avoid the "liblib" prefix.

So now we consistently have the libraries named "fio-$(ENGINENAME).so"

fio-http.so
fio-libaio.so

etc.

Fixes: 5a8a6a03 ("configure: new --dynamic-libengines build option")
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agofix dynamic engine build
Eric Sandeen [Mon, 9 Nov 2020 16:46:28 +0000 (10:46 -0600)]
fix dynamic engine build

Builds with --dynamic-libengines are currently broken, because the rule
to build the libraries was moved out of the template that creates them:

make: *** No rule to make target 'engines/libpmem.so', needed by 'all'.  Stop.
make: *** Waiting for unfinished jobs....

Fix this by moving the rule back into the template, calling the template
to create rules only after FIO_CFLAGS has been incorporated into CFLAGS,
and using CFLAGS + -fPIC in that rule rather than FIO_CFLAGS.

Fixes: 8a2cf08d29ac ("Makefile: introduce FIO_CFLAGS")
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoRevert "Windows: update dobuild.cmd to run the configure/make"
Jens Axboe [Thu, 5 Nov 2020 22:33:00 +0000 (15:33 -0700)]
Revert "Windows: update dobuild.cmd to run the configure/make"

This reverts commit 38c2f9384db8dbd93f59d965d70ab0d3a53343fa.

It's causing issues for CI, revert it for now.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 weeks agoMerge branch 'master' of https://github.com/albert-chang0/fio
Jens Axboe [Thu, 5 Nov 2020 22:17:11 +0000 (15:17 -0700)]
Merge branch 'master' of https://github.com/albert-chang0/fio

* 'master' of https://github.com/albert-chang0/fio:
  engines/hdfs: swap fio_hdfsio_init and fio_hdfsio_io_u_init
  Makefile: fix usage of JAVA_HOME environmental variable

2 weeks agoMerge branch 'patch-1' of https://github.com/gloit042/fio
Jens Axboe [Thu, 5 Nov 2020 22:16:50 +0000 (15:16 -0700)]
Merge branch 'patch-1' of https://github.com/gloit042/fio

* 'patch-1' of https://github.com/gloit042/fio:
  goptions: correct postfix

2 weeks agoengines/hdfs: swap fio_hdfsio_init and fio_hdfsio_io_u_init
Albert Chang [Thu, 5 Nov 2020 18:30:22 +0000 (13:30 -0500)]
engines/hdfs: swap fio_hdfsio_init and fio_hdfsio_io_u_init

Commit 08dc3bd50 initialized the io engine before the io_u buffers. The
unintended consequence for the libhdfs io engine was that the HDFS
connection was not established before attempting to open HDFS file
handles. This caused a NPE in the java layer on start-up preventing the
libhdfs io engine from being able to be used.

Signed-off-by: Albert Chang <albert.chang@hitachivantara.com>
2 weeks agoMakefile: fix usage of JAVA_HOME environmental variable
Albert Chang [Thu, 5 Nov 2020 18:29:01 +0000 (13:29 -0500)]
Makefile: fix usage of JAVA_HOME environmental variable

Signed-off-by: Albert Chang <albert.chang@hitachivantara.com>
2 weeks agoWindows: update dobuild.cmd to run the configure/make
Rebecca Cran [Tue, 3 Nov 2020 02:47:41 +0000 (02:47 +0000)]
Windows: update dobuild.cmd to run the configure/make

Add a new shell script, _domake.sh which runs under Cygwin. This
is invoked via dobuild.cmd to run the configure and make steps of
the build. In this way, a single command, run from a Developer Command
Prompt, can be used to go from a clean workspace to a final installer
binary.

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agogoptions: correct postfix
gloit042 [Wed, 4 Nov 2020 04:46:24 +0000 (12:46 +0800)]
goptions: correct postfix

Signed-off-by: Jiahao Li <gloit042@gmail.com>
3 weeks agofio: fix dynamic engines soname definition
Yigal Korman [Mon, 5 Oct 2020 17:08:54 +0000 (20:08 +0300)]
fio: fix dynamic engines soname definition

The SONAME of the engines should not match the name of the dependent
library. Otherwise it confuses the dynamic loader into thinking the
dependency is already resolved.

Prefixing the name with fio make more sense here.

Signed-off-by: Yigal Korman <ykorman@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agot/zbd: Fix test target size of test case #48
Shin'ichiro Kawasaki [Fri, 30 Oct 2020 06:57:40 +0000 (15:57 +0900)]
t/zbd: Fix test target size of test case #48

Option --size was not specified to the fio command of test case #48. It
resulted in write operations to all available sequential write required
zones and relaxed zone locking test condition. Specify the option to
limit test target to 16 zones so that zone locking is tested with
expected condition.

Fixes: 3bd2078bdd1c ("zbd: add test for stressing zone locking")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agozbd: Avoid excessive zone resets
Shin'ichiro Kawasaki [Fri, 30 Oct 2020 06:57:39 +0000 (15:57 +0900)]
zbd: Avoid excessive zone resets

When zbd_reset_zone() is called for a zone repeatedly, reset zone command
is issued multiple times to a single zone even though its status is
empty. This is excessive and meaningless. Especially when zones are reset
at file set up with multiple jobs, zone reset is repeated for each job
and delays fio work load start.

To avoid the repeated zone resets, check the write pointer of the zone
before issuing zone reset command. If the write pointer is at the zone
start, do not reset the zone and just return.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 weeks agoMerge branch 'o_dsync' of https://github.com/anarazel/fio
Jens Axboe [Wed, 28 Oct 2020 22:20:43 +0000 (16:20 -0600)]
Merge branch 'o_dsync' of https://github.com/anarazel/fio

* 'o_dsync' of https://github.com/anarazel/fio:
  extend --sync to allow {sync,dsync,0,1}, to support O_DSYNC

3 weeks agoextend --sync to allow {sync,dsync,0,1}, to support O_DSYNC
Andres Freund [Fri, 4 Sep 2020 00:47:10 +0000 (17:47 -0700)]
extend --sync to allow {sync,dsync,0,1}, to support O_DSYNC

Fixes: https://github.com/axboe/fio/issues/1033

5 weeks agoDisallow offload IO mode for engines marked with FIO_NO_OFFLOAD
Jens Axboe [Thu, 15 Oct 2020 02:11:56 +0000 (20:11 -0600)]
Disallow offload IO mode for engines marked with FIO_NO_OFFLOAD

Previous commits did this for all async engines, this can potentially
break existing job files. There are only certain cases where it fails,
for now at least mark io_uring as one of those.

Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoMerge branch 'patch-1' of https://github.com/chengli-rutgers/fio into master
Jens Axboe [Tue, 13 Oct 2020 19:07:37 +0000 (13:07 -0600)]
Merge branch 'patch-1' of https://github.com/chengli-rutgers/fio into master

* 'patch-1' of https://github.com/chengli-rutgers/fio:
  getopt_long: avoid variable global initialization

6 weeks agogetopt_long: avoid variable global initialization
Cheng Li [Tue, 13 Oct 2020 17:03:12 +0000 (10:03 -0700)]
getopt_long: avoid variable global initialization

Issue #1100 shows that an address sanitizer(ASAN) complains about a few
global variables that are initialized globally in the getopt_long.c file.
We look into these variables and found they do not need to be initialized
globally. This patch fixes the issue by cleaning up the global
initialization for these variables.

Fixes https://github.com/axboe/fio/issues/1100
Signed-off-by: Cheng Li <chenglii@cs.rutgers.edu>
6 weeks agoMerge branch 'master' of https://github.com/bvanassche/fio into master
Jens Axboe [Sun, 11 Oct 2020 02:30:52 +0000 (20:30 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio into master

* 'master' of https://github.com/bvanassche/fio:
  helper_thread: Increase timer accuracy
  helper_thread: Rework the interval timer code
  Change the return value of two functions from 'void' into 'int'
  helper_thread: Introduce the wait_for_action() function
  helper_thread: Introduce a function for blocking Unix signals
  gettime: Introduce rel_time_since()
  gettime: Simplify get_cycles_per_msec()
  gettime: Introduce fio_get_mono_time()
  configure: Remove the CLOCK_MONOTONIC_PRECISE probe

6 weeks agosimplify MB/s, IOPS interactive printout code
Alexey Dobriyan [Sat, 10 Oct 2020 14:35:10 +0000 (17:35 +0300)]
simplify MB/s, IOPS interactive printout code

Use loops(!) instead of hardcoded if-else pairs.

Make it easier to add DDIR_APPEND.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoflow: use unsigned long for the counter
Jens Axboe [Sat, 10 Oct 2020 15:16:51 +0000 (09:16 -0600)]
flow: use unsigned long for the counter

Same size on 64-bit, it'll be 32-bit on 32-bit archs. But we can't
reliably use 64-bit atomics on 32-bit archs.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoflow: avoid holes in struct fio_flow
Jens Axboe [Sat, 10 Oct 2020 15:06:02 +0000 (09:06 -0600)]
flow: avoid holes in struct fio_flow

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 weeks agoMerge branch 'master' of https://github.com/chengli-rutgers/fio into master
Jens Axboe [Fri, 9 Oct 2020 13:28:55 +0000 (07:28 -0600)]
Merge branch 'master' of https://github.com/chengli-rutgers/fio into master

* 'master' of https://github.com/chengli-rutgers/fio:
  td_var: avoid arithmetic on a pointer to void cast (#1096)

6 weeks agotd_var: avoid arithmetic on a pointer to void cast (#1096)
Cheng Li [Thu, 8 Oct 2020 18:22:34 +0000 (11:22 -0700)]
td_var: avoid arithmetic on a pointer to void cast (#1096)

Issue 1096 shows that a compiler complains an `arithmetic on a pointer to
void` error when `-Wpointer-arith` is turned on in Makefile. Although
there are many violating instances, we only fix the one in `parse.h`
because it is shared across many core source code files. This patch fixes
the `arithmetic on a pointer to void` issue by casting it to a uintptr_t
type first and perform manipulation, and then cast it back to void* type.

Signed-off-by: Cheng Li <chenglii@cs.rutgers.edu>
8 weeks agohelper_thread: Increase timer accuracy
Bart Van Assche [Sat, 12 Sep 2020 14:58:00 +0000 (07:58 -0700)]
helper_thread: Increase timer accuracy

The helper_thread uses select() to wait for the next event. select() rounds
up its timeout argument to 1 / HZ. Increase timer accuracy by using a
high-res timer if possible. This patch has been tested by running the
following fio command and by verifying that the next_log and
msec_to_next_event were always above 100 ms:

./fio --debug=helperthread --bandwidth-log --write_bw_log=log --log_avg_msec=1000 --ioengine=null --name=null --size=1G --rw=randwrite --loops=999999 --runtime=60

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
8 weeks agohelper_thread: Rework the interval timer code
Bart Van Assche [Wed, 9 Sep 2020 02:34:45 +0000 (19:34 -0700)]
helper_thread: Rework the interval timer code

Instead of adding the number of milliseconds that have elapsed when
a timer fires, add the timer interval. Reduce code duplication by
introducing a helper function that verifies whether or not a timer
has expired.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoChange the return value of two functions from 'void' into 'int'
Bart Van Assche [Sun, 20 Sep 2020 23:27:58 +0000 (16:27 -0700)]
Change the return value of two functions from 'void' into 'int'

This patch does not change any functionality but makes the next patch in
this series easier to read.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agohelper_thread: Introduce the wait_for_action() function
Bart Van Assche [Wed, 9 Sep 2020 02:32:02 +0000 (19:32 -0700)]
helper_thread: Introduce the wait_for_action() function

This patch makes helper_thread_main() easier to read.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agohelper_thread: Introduce a function for blocking Unix signals
Bart Van Assche [Sat, 12 Sep 2020 14:43:56 +0000 (07:43 -0700)]
helper_thread: Introduce a function for blocking Unix signals

This patch reworks commit c31092b8ac06 ("Block signals for the helper
thread") but does not change any functionality.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agogettime: Introduce rel_time_since()
Bart Van Assche [Fri, 4 Sep 2020 23:18:53 +0000 (16:18 -0700)]
gettime: Introduce rel_time_since()

rel_time_since() subtracts two timespecs and returns the result as a
signed integer. Instead of using the implementation from mtime_since(),
use an implementation that only performs one division instead of two.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agogettime: Simplify get_cycles_per_msec()
Bart Van Assche [Mon, 7 Sep 2020 22:55:27 +0000 (15:55 -0700)]
gettime: Simplify get_cycles_per_msec()

Call fio_get_mono_time() directly instead of setting fio_clock_source and
calling __fio_gettime().

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agogettime: Introduce fio_get_mono_time()
Bart Van Assche [Sun, 20 Sep 2020 18:40:26 +0000 (11:40 -0700)]
gettime: Introduce fio_get_mono_time()

Introduce a new function for querying the monotonic clock, something that
is necessary in every context where relative time is measured and where
wall clock time jumps should have no effect. Remove fill_clock_gettime()
since the only contexts where CLOCK_MONOTONIC_RAW are used are
get_cycles_per_msec() and --clocksource=clock_gettime. I think both contexts
should use CLOCK_MONOTONIC instead of its raw variant such that the values
read from the clock are frequency adjusted (a computer clock crystal can
deviate up to 500 ppm from its nominal frequency).

This patch improves accuracy of the helper_thread code on Darwin. Darwin
supports CLOCK_MONOTONIC but not pthread_condattr_setclock(). In other
words, this patch causes the helper thread code to switch from the
real-time clock to the monotonic clock on Darwin.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoconfigure: Remove the CLOCK_MONOTONIC_PRECISE probe
Bart Van Assche [Mon, 7 Sep 2020 23:03:12 +0000 (16:03 -0700)]
configure: Remove the CLOCK_MONOTONIC_PRECISE probe

CLOCK_MONOTONIC_PRECISE is not used in the fio source code. Hence use the
probe for that constant. The removed test was introduced by commit
67bf982340d9 ("Add configure script").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoMerge branch 'msys2' of https://github.com/sitsofe/fio into master
Jens Axboe [Tue, 15 Sep 2020 01:43:39 +0000 (19:43 -0600)]
Merge branch 'msys2' of https://github.com/sitsofe/fio into master

* 'msys2' of https://github.com/sitsofe/fio:
  travis: cleanup build script
  configure: cleanup lex/yacc tests
  memlock: avoid type confusion in format string
  Makefile: introduce FIO_CFLAGS
  appveyor: cleanup and add separate install script
  Makefile/ci: Don't pass CFLAGS when linking
  configure/Makefile: add option to generate pdb symbols
  configure: be explicit about "XP" Windows API version
  windows: fix DWORD format string complaints
  windows: fix wrong format strings
  net: coerce the result of htonl before printing
  configure: pass non-null pointer to (v)asprintf

2 months agoMerge branch 'backend' of https://github.com/bvanassche/fio into master
Jens Axboe [Tue, 15 Sep 2020 01:42:23 +0000 (19:42 -0600)]
Merge branch 'backend' of https://github.com/bvanassche/fio into master

* 'backend' of https://github.com/bvanassche/fio:
  backend: Use asprintf() instead of strlen() + sprintf()
  backend: Remove two superfluous casts

2 months agotravis: cleanup build script
Sitsofe Wheeler [Thu, 3 Sep 2020 22:02:31 +0000 (23:02 +0100)]
travis: cleanup build script

- Standardise on spaces for indentation
- Exit on error or using variables before they are defined
- Fix up shellcheck complaints by exporting some variables
- Drop printing of python 2 information as we don't use it
- Use type -p rather than which

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agoconfigure: cleanup lex/yacc tests
Sitsofe Wheeler [Mon, 31 Aug 2020 13:59:59 +0000 (14:59 +0100)]
configure: cleanup lex/yacc tests

- Switch to prebuilt function for finding executable
- Search for the fl library from flex too
- Test lex command output directly

This allows lexing to be more likely to be found (e.g. MSYS2 but sadly
not MSYS2 Mingw-w64).

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agomemlock: avoid type confusion in format string
Sitsofe Wheeler [Mon, 31 Aug 2020 10:44:11 +0000 (11:44 +0100)]
memlock: avoid type confusion in format string

Windows GCC has the followng grumble:

t/memlock.c: In function 'worker':
t/memlock.c:25:26: error: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'size_t' {aka 'long long unsigned int'} [-Werror=format=]
   25 |    printf("loop%d: did %lu MiB\n", i+1, size/(1024UL*1024UL));
      |                        ~~^              ~~~~~~~~~~~~~~~~~~~~
      |                          |                  |
      |                          long unsigned int  size_t {aka long long unsigned int}
      |                        %llu

Fix it up by just using the original variable that was already in
megabytes.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agoMakefile: introduce FIO_CFLAGS
Sitsofe Wheeler [Sat, 22 Aug 2020 17:01:31 +0000 (18:01 +0100)]
Makefile: introduce FIO_CFLAGS

Build on the work  done in 4c0b3d98f2d05ddd3f16262c466dcedb22158065
("configure/Makefile: don't override user CFLAGS") by builing up all the
internal flags in a temporary variable.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agoappveyor: cleanup and add separate install script
Sitsofe Wheeler [Wed, 19 Aug 2020 19:21:22 +0000 (20:21 +0100)]
appveyor: cleanup and add separate install script

- Rename PLATFORM environment variable to ARCHITECTURE to avoid clash
  with Appveyor variable
  (https://www.appveyor.com/docs/environment-variables/ )
- Introduce support for using MSYS2
- Perform clang builds that also package PDB symbols
- Drop PACKAGE_ARCH variable since it can be derived
- Switch some lines to just call native binaries directly

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agoMakefile/ci: Don't pass CFLAGS when linking
Sitsofe Wheeler [Thu, 20 Aug 2020 08:05:03 +0000 (09:05 +0100)]
Makefile/ci: Don't pass CFLAGS when linking

There was an issue where a lld (the linker from LLVM) could fail because
a C compiler flag it didn't understand was passed to it. Fix this by no
longer passing CFLAGS to the linker.

As a result of the above, the travis "cross compiled" 32 bit Linux build
was changed to set LDFLAGS (so the linker can know about the cross
compiling).

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agoconfigure/Makefile: add option to generate pdb symbols
Sitsofe Wheeler [Fri, 21 Aug 2020 19:35:03 +0000 (20:35 +0100)]
configure/Makefile: add option to generate pdb symbols

Recent versions of Windows clang and LLVM's lld have the ability to
build Windows PDB symbols for debugging so add an explicit option to
test for support and use it so.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agoconfigure: be explicit about "XP" Windows API version
Sitsofe Wheeler [Sat, 22 Aug 2020 19:40:58 +0000 (20:40 +0100)]
configure: be explicit about "XP" Windows API version

With a Cygwin Mingw-w64 environment, when an explicit Windows API
version define isn't set the Windows Server 2003 API is targeted
(https://github.com/mirror/mingw-w64/blob/a25dc933f3d3ed6c8529fb24b7ae393f8792ca69/mingw-w64-headers/include/sdkddkver.h#L166
). This means the XP build was actually misnamed and have never been
fully compatible with plain XP for many years (I suspect such binaries
will likely still work on XP bar trying to use the net ioengine)...

With an MSYS2 Mingw-w64 environment, the default Windows API version is
Windows 7
(https://github.com/msys2/MINGW-packages/blob/e9f271c61a39076ab5de56299f7fd86020c280ae/mingw-w64-headers-git/PKGBUILD#L55
) which breaks an "XP" build because we were were relying on the default
being 2003. Fix this by making the "XP" build explicitly target the
Windows Server 2003 API.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agowindows: fix DWORD format string complaints
Sitsofe Wheeler [Wed, 19 Aug 2020 22:58:35 +0000 (23:58 +0100)]
windows: fix DWORD format string complaints

Printing DWORD variables causes Windows clang to grumble about types:

os/windows/cpu-affinity.c:263:5: error: format specifies type 'int' but the argument has type 'DWORD' (aka 'unsigned long') [-Werror,-Wformat]
                         GetLastError());
                         ^~~~~~~~~~~~~~

Since a DWORD is defined to be an unsigned 32 bit integer and is
declared to be an unsigned long
(https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/262627d8-3418-4627-9218-4ffe110850b2
) just use long as the format specifier. When printing the cpu mask in
hex on legacy Windows builds, cast to unsigned long long which should be
safe because there won't be more than 64 CPUs with old versions of
Windows.

(As some format strings are longer than the wrapping width, make an
exception and avoid splitting/preserve them to make greping easier)

Suggested-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agowindows: fix wrong format strings
Sitsofe Wheeler [Thu, 20 Aug 2020 06:08:46 +0000 (07:08 +0100)]
windows: fix wrong format strings

Found by Windows clang which mysteriously seemed able to ignore
-Wno-format...

(As some format strings are longer than the wrapping width, make an
exception and avoid splitting/preserve them to make greping easier)

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agonet: coerce the result of htonl before printing
Sitsofe Wheeler [Wed, 19 Aug 2020 20:57:05 +0000 (21:57 +0100)]
net: coerce the result of htonl before printing

Technically htonl() returns a long on Windows
(https://docs.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-htonl
) which upsets clang.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agoconfigure: pass non-null pointer to (v)asprintf
Sitsofe Wheeler [Wed, 19 Aug 2020 22:02:48 +0000 (23:02 +0100)]
configure: pass non-null pointer to (v)asprintf

Windows clang warns about redefinition of (v)asprintf when targeting
Windows 7 because the configure test failed due to passing NULL when to
those functions when they tag key parameters with the nonnull attribute.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2 months agobackend: Use asprintf() instead of strlen() + sprintf()
Bart Van Assche [Fri, 4 Sep 2020 23:05:37 +0000 (16:05 -0700)]
backend: Use asprintf() instead of strlen() + sprintf()

This patch does not change any functionality but makes exec_string() easier
to read.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agobackend: Remove two superfluous casts
Bart Van Assche [Fri, 4 Sep 2020 23:06:07 +0000 (16:06 -0700)]
backend: Remove two superfluous casts

Casting the type of a string constant to (const char *) is not necessary.
These casts were introduced by commit ce4864950d2c ("backend: Logging
exec_{pre|post}run").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoAllow offload with FAKEIO engines
Jens Axboe [Fri, 11 Sep 2020 15:58:15 +0000 (09:58 -0600)]
Allow offload with FAKEIO engines

The last check was too restrictive, we can allow it with the fake IO
engines.

Fixes: abfd235a0533 ("Disable io_submit_mode=offload with async engines")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoDisable io_submit_mode=offload with async engines
Jens Axboe [Fri, 11 Sep 2020 15:18:09 +0000 (09:18 -0600)]
Disable io_submit_mode=offload with async engines

We have various cases that aren't handled correctly with async engines,
or cannot work with async engines. Disable it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoengines/io_uring: mark as not compatible with io_submit_mode=offload
Jens Axboe [Fri, 11 Sep 2020 14:52:51 +0000 (08:52 -0600)]
engines/io_uring: mark as not compatible with io_submit_mode=offload

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'evelu-enghelp' of https://github.com/ErwanAliasr1/fio into master
Jens Axboe [Wed, 9 Sep 2020 16:48:27 +0000 (10:48 -0600)]
Merge branch 'evelu-enghelp' of https://github.com/ErwanAliasr1/fio into master

* 'evelu-enghelp' of https://github.com/ErwanAliasr1/fio:
  init: exiting with fio_show_ioengine_help return code

2 months agoinit: exiting with fio_show_ioengine_help return code
Erwan Velu [Wed, 9 Sep 2020 16:31:25 +0000 (18:31 +0200)]
init: exiting with fio_show_ioengine_help return code

In the current code, calling --enghelp=<engine> always return 0.
But fio_show_ioengine_help() implemented a return code to inform when an engine is present or not.

This commit makes fio returning fio_show_ioengine_help() value when calling --enghelp=<engine>

A typical use case is detecting which engine is available on a given fio build.
It is possible to use the return code to determine if a given engine is available or not :

[user@host]# fio --enghelp=libaio
userspace_reap          : Use alternative user-space reap implementation
[user@host]# echo $?
0

[user@host]#fio --enghelp=io_uring
fio: engine io_uring not loadable
IO engine io_uring not found
[user@host]# echo $?
1

Signed-off-by: Erwan Velu <e.velu@criteo.com>
2 months agoMerge branch 'zbd' of https://github.com/bvanassche/fio into master
Jens Axboe [Tue, 8 Sep 2020 16:25:48 +0000 (10:25 -0600)]
Merge branch 'zbd' of https://github.com/bvanassche/fio into master

* 'zbd' of https://github.com/bvanassche/fio:
  zbd: Add a missing pthread_mutex_unlock() call

2 months agoengines/io_uring: allow setting of IOSQE_ASYNC
Jens Axboe [Tue, 8 Sep 2020 15:57:03 +0000 (09:57 -0600)]
engines/io_uring: allow setting of IOSQE_ASYNC

Add engine option 'force_async'. If that is set to N, then every N
requests will have IOSQE_ASYNC set to force async offload.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agozbd: Add a missing pthread_mutex_unlock() call
Bart Van Assche [Sat, 30 May 2020 22:34:00 +0000 (15:34 -0700)]
zbd: Add a missing pthread_mutex_unlock() call

This patch fixes the following Coverity complaint:

CID 292491 (#1 of 1): Missing unlock (LOCK)
missing_unlock: Returning without unlocking zb->mutex.

Fixes: bfbdd35b3e8f ("Add support for zoned block devices")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2 months agoKill off old GUASI IO engine
Jens Axboe [Mon, 7 Sep 2020 19:38:03 +0000 (13:38 -0600)]
Kill off old GUASI IO engine

This library never saw widespread use, and it hasn't been updated in
more than a decade. On top of that, the fio engine was broken for a
long time and nobody noticed.

Time to take it out behind the barn.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agofio: cap io_size=N% at 100%, update man page
Alexey Dobriyan [Mon, 7 Sep 2020 17:59:23 +0000 (20:59 +0300)]
fio: cap io_size=N% at 100%, update man page

Passing io_size > 100% results in semi-infinite loop, anyway.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'guasi_fixes' of https://github.com/sitsofe/fio into master
Jens Axboe [Sun, 6 Sep 2020 22:14:58 +0000 (16:14 -0600)]
Merge branch 'guasi_fixes' of https://github.com/sitsofe/fio into master

* 'guasi_fixes' of https://github.com/sitsofe/fio:
  Makefile/configure: fix guasi build

2 months agoMakefile/configure: fix guasi build
Sitsofe Wheeler [Sun, 26 Jul 2020 16:22:09 +0000 (17:22 +0100)]
Makefile/configure: fix guasi build

2 months agoFio 3.23 fio-3.23
Jens Axboe [Sun, 6 Sep 2020 16:45:11 +0000 (10:45 -0600)]
Fio 3.23

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agofio: support io_size=N% (N <= 100)
Alexey Dobriyan [Thu, 3 Sep 2020 16:54:02 +0000 (19:54 +0300)]
fio: support io_size=N% (N <= 100)

Parse "io_size=N%".

Semantics is "multiply whatever size= calculations result in".

Example #1:

size=50%
io_size=50%

will do 25% of a file.

Example #2:

size=1G
io_size=50%

will do 512M I/O.

As side effect, fix a bug with essentially infinite loop if both size=N%
and io_size=M% are given: io_size is set to 2^64-... in this case (a lot!).

Note: only values under 100% work currently.
Going for io_size=150% requires resetting workload generator state
which is whole separate endeavour.

Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoengines/io_uring: use the atomic load acquire instead of a barrier
Jens Axboe [Thu, 3 Sep 2020 14:49:51 +0000 (08:49 -0600)]
engines/io_uring: use the atomic load acquire instead of a barrier

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/io_uring: allow setting fixed files/buffers as arguments
Jens Axboe [Thu, 3 Sep 2020 14:49:19 +0000 (08:49 -0600)]
t/io_uring: allow setting fixed files/buffers as arguments

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoengines/io_uring: move sqe clear out of hot path
Jens Axboe [Thu, 3 Sep 2020 14:23:21 +0000 (08:23 -0600)]
engines/io_uring: move sqe clear out of hot path

We just need to ensure we set the stuff we need, then we can avoid a
full memset of the sqe in the hot path.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/zbd: Remove unnecessary option for zbc_reset_zone
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:06 +0000 (17:20 +0900)]
t/zbd: Remove unnecessary option for zbc_reset_zone

When -all option is specified, zbc_reset_zone command does not take
offset option. Remove it.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/zbd: Reset all zones before test when max open zones is specified
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:05 +0000 (17:20 +0900)]
t/zbd: Reset all zones before test when max open zones is specified

When the test target device has maximum open zones limit, the zones in
test target region may not be opened up to the limit, because the zones
out of the test target region may have open zones. To ensure that the
test target zones can be opened up to the limit, reset all zones of the
test target device before the test cases with write work load starts.
Introduce the helper function prep_write() to check if all zone reset is
required and do the reset.

Also remove unnecessary reset_zone calls for test case #29 and #48. These
are no longer required by virtue of the improvement in zbd_setup_files()
to set up zones to meet max_open_zones limit.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/zbd: Add -o option to t/zbd/test-zoned-support
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:04 +0000 (17:20 +0900)]
t/zbd: Add -o option to t/zbd/test-zoned-support

To specify maximum open zones of the test target device, add -o option
to t/zbd/test-zoned-support. When this option is specified, add
max_open_zones option to all fio commands in the test script. The option
ensures the number of zones opened by fio is within the limit. This is
useful when the test target device has maximum open zones limit.

When the fio command does not have multiple jobs and the test case does
not specify max_open_zones, add single max_open_zones option to the fio
command line.

When the fio command takes multiple jobs, add max_open_zones option to
each job. Introduce job_var_opts variable to keep options to be added to
each job. To distinguish it from global options for all jobs, rename
var_opts variable to global_var_opts. When a test case with multiple jobs
always specifies max_open_zones option, exclude the max_open_zones option
from job_var_opts, using job_var_opts_exclude() helper function.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agot/zbd: Improve usage message of test-zbd-support script
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:03 +0000 (17:20 +0900)]
t/zbd: Improve usage message of test-zbd-support script

The usage message of t/zbd/test-zbd-support does not explain meaning of
each option. Elaborate meaning of the options.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agozbd: Initialize open zones list referring zone status at fio start
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:02 +0000 (17:20 +0900)]
zbd: Initialize open zones list referring zone status at fio start

When fio starts write workloads to zones with open status, fio does not
reflect the zone status to open zone list. This results in inconsistent
open zone accounting.

To avoid this inconsistency, initialize the open zone list referring the
zone status before workload start. If a zone in target range have open
status, call zbd_open_zone() function to build up the open zone list.
Since io_u is not available at that timing, make the function independent
from io_u. If the number of open zones exceeds maximum open zones limits,
reset the exceeding open zone by calling zbd_reset_zone().

Of note is that fio counts open zones within the write target range.
Open zones out of the range are not counted or checked.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agooslib/linux-blkzoned: Allow reset zone before file set up
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:01 +0000 (17:20 +0900)]
oslib/linux-blkzoned: Allow reset zone before file set up

To reset zones before file set up completion, modify the helper function
blkzoned_reset_wp() to work even when fio_file struct does not have valid
file descriptor yet. If the file descriptor is not available, open and
close the reset target file within the function. Otherwise use the
available descriptor to avoid file open/close overhead.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agozbd: Decrement open zones count at write command completion
Shin'ichiro Kawasaki [Tue, 1 Sep 2020 08:20:00 +0000 (17:20 +0900)]
zbd: Decrement open zones count at write command completion

When max_open_zones or job_max_open_zones option is provided, fio counts
number of open zones. This open zone accounting is done during submission
of write commands. When write command is submitted to a zone for the
first time, the zone is counted as open. When a write command which will
fill the zone gets submitted, the zone is counted as closed. However,
this count at write command submission may open zones more than the
limit. When writes to zones more than the limit are submitted with high
queue depth, those writes in-flight open zones more than the limit.

To avoid such writes beyond max open zones limit, decrement number of
open zones count not at write command submission but at write command
completion. By doing this, the number of zones with in-flight write
commands are kept within the limit with accuracy. Introduce the helper
function zbd_end_zone_io() for this decrement and zbd_close_zone() call.

The zbd_close_zone() function requires thread_data argument. To refer
thread_data at write command completion, add the argument to zbd_put_io()
and zbd_queue_io() callbacks. Also add a loop to the zbd_close_zone()
function which converts zone_info array index to open_zones array index
to avoid loop code duplication.

Even when io_u points to an open zone, the zone may not be valid for
write since in-flight write commands may make the zone full. Check this
in zbd_open_zone() to handle such zones as in full status.

Because of the zone close timing change, there might be no open zone when
zbd_convert_to_open_zone() is called. Do not handle such case as an
error and open other zones.

When zbd_convert_to_open_zone() finds all open zones can not be used for
next write, it opens other zones. This zone open fails when the number of
open zones hits one of the limits: 1) number of zones in the fio write
target region, 2) max_open_zones option or 3) job_max_open_zones option.
To avoid the zone open failure, wait for writes in-flight completes and
open zones get closed before opening other zones.

Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agooptions: flow should parse as FIO_OPT_INT
Jens Axboe [Mon, 31 Aug 2020 15:07:12 +0000 (09:07 -0600)]
options: flow should parse as FIO_OPT_INT

It's an unsigned int, if we parse it as FIO_OPT_ULL then we clobber
the variable around it in the options struct. This is apparent when
running the examples/flow.fio job file, as flow_sleep=1000 is set
before flow, but gets clobbered when flow is set. This results in a
flow_sleep value of 0 instead of 1000.

Fixes: d4e74fda98b6 ("flow: add ability for weight-based flow control on multiple jobs")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'multi_job_flow' of https://github.com/bardavid/fio into master
Jens Axboe [Mon, 31 Aug 2020 14:52:25 +0000 (08:52 -0600)]
Merge branch 'multi_job_flow' of https://github.com/bardavid/fio into master

* 'multi_job_flow' of https://github.com/bardavid/fio:
  flow: add ability for weight-based flow control on multiple jobs
  flow: reclaim flow when job is reaped

2 months agoflow: add ability for weight-based flow control on multiple jobs
David, Bar [Sun, 30 Aug 2020 14:29:24 +0000 (17:29 +0300)]
flow: add ability for weight-based flow control on multiple jobs

Fixes: #741 - fio how to run mix workload all 4 patterns in one job file

Previously, 'flow' was used to regulate activity between 2 or more jobs with
certain weight.  However, the implementation was ineffective when regulating
more than 2 jobs with different weights, since it relied on a single,
shared counter, and was missing logic to track individual job's activity
and regulate it according to it's requested weight in proportion to other jobs'
requested weight and activity.

This commit modifies the 'flow' functionality to track each job's
activity and the total activity belonging to a 'flow_id' group.
Using this new information, the 'flow' logic is then modified to
regulate a job's activity in proportion to other jobs in the
same group.  Similar to previous behavior, the regulation is done by
stalling a job's activity.

New jobs joining an existing 'flow_id' group or jobs repead from said
group will cause re-calibration of the weights and alter the activity as a
result.  For example, 3 jobs (j1,j2,j3) with weights (1:2:3)
respectively will maintain activity ratio of 1:2:3.  Thus, j1 will have
activity of 1/6, j2 - 2/6 and j3 - 1/2.

If j3 exits prematurely, the proportion is re-calibrated
such that j1 will strive to have activity of 1/3 and j2 - 2/3.

Re-calibration phase may cause a hiccup in the activity of one or more
jobs since the new desired proportion might not be achieved quickly
enough.

Signed-off-by: David, Bar <bardavvid@gmail.com>
2 months agoflow: reclaim flow when job is reaped
David, Bar [Thu, 13 Aug 2020 09:40:55 +0000 (12:40 +0300)]
flow: reclaim flow when job is reaped

Signed-off-by: David, Bar <bardavvid@gmail.com>
2 months agoUpdate the year to 2020 in os/windows/eula.rtf
Rebecca Cran [Sat, 29 Aug 2020 23:31:23 +0000 (23:31 +0000)]
Update the year to 2020 in os/windows/eula.rtf

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoUpdate os/windows/dobuild.cmd to support signing binaries/installer
Rebecca Cran [Sat, 29 Aug 2020 23:25:50 +0000 (23:25 +0000)]
Update os/windows/dobuild.cmd to support signing binaries/installer

Since organizations are increasingly requiring software to be digitally
signed, or have heuristics that penalize unsigned software, add support
to do build.cmd to sign both the .exe binaries and the .msi installer.

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoUse fallthrough attribute
Jens Axboe [Fri, 28 Aug 2020 15:14:38 +0000 (09:14 -0600)]
Use fallthrough attribute

Fio currently triggers a bunch of fall through errors on clang 10,
since it doesn't work with the /* fall through */ method of
indicating fallthrough.

Normalize this a bit and use the correct compiler attribute for this.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2 months agoMerge branch 'master' of https://github.com/donny372/fio into master
Jens Axboe [Fri, 28 Aug 2020 14:04:15 +0000 (08:04 -0600)]
Merge branch 'master' of https://github.com/donny372/fio into master

* 'master' of https://github.com/donny372/fio:
  Avoid multiple instance read iolog from stdin.

2 months agozbd: don't read past the WP on a read only workload with verify
Aravind Ramesh [Thu, 13 Aug 2020 15:01:48 +0000 (17:01 +0200)]
zbd: don't read past the WP on a read only workload with verify

When running a read only workload with verify option enabled,
fio continues to read beyond the write pointer, and thus
reads a deallocated block, which contains data of a predefined
pattern, thus causing the verify to fail.

Thus, we cannot simply jump to accept for a read only workload
with verify option enabled, we must let the regular flow of
execution to continue, so that the check if offset > wp is
performed, so that we will return the address to a new zone,
rather than an address that is past the wp.

Reading past the wp is not a recent bug, it is possible to
reproduce this bug even before zone capacity was introduced.

However, with the introduction of zone capacity, filling a
zone now only writes up to zone capacity of a zone, making
it much easier to reproduce this bug.

Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com>
Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agot/io_uring: cleanup vectored vs non-vectored
Jens Axboe [Sat, 22 Aug 2020 17:26:39 +0000 (11:26 -0600)]
t/io_uring: cleanup vectored vs non-vectored

No functional changes, just makes it a bit more logical.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
3 months agot/io_uring: use non-vectored reads if available
Jens Axboe [Sat, 22 Aug 2020 16:58:14 +0000 (10:58 -0600)]
t/io_uring: use non-vectored reads if available

Signed-off-by: Jens Axboe <axboe@kernel.dk>