Bart Van Assche [Tue, 17 Apr 2018 23:08:54 +0000 (16:08 -0700)]
Declare stat_calc_lat_nu() static
This was suggested by the sparse static analyzer.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Wed, 18 Apr 2018 16:52:00 +0000 (10:52 -0600)]
Remove verifysort/verifysort_nr from documentation
They have been deprecated.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 18 Apr 2018 14:12:07 +0000 (08:12 -0600)]
iolog: update stale comment
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 18 Apr 2018 03:50:55 +0000 (21:50 -0600)]
Deprecate verifysort and verifysort_nr
It was an optimization to read back verifies in a sorted order,
for rotational storage. But I don't think the option makes much
sense, and I've never heard of anyone using it. Mark it as
deprecated, and always verify in the same order that IO was
written.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 18 Apr 2018 03:49:51 +0000 (21:49 -0600)]
parse: add support for soft deprecated options
The currently deprecated option types will fail parsing. But we
can also have options where we don't care, log an error for those
but continue executing the job.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 17 Apr 2018 23:48:43 +0000 (17:48 -0600)]
init: ensure that read/write use the same random seed for verify
If we are verifying, then reads need to use the same blockrange
random trim as writes. Otherwise doing a deferred verify of
a write job may not yield the same block size ranges, causing
verify to fail.
Example of failed job:
$ cat job.fio
[global]
ioengine=sync
verify=md5
bsrange=1k-16k
rw=randwrite
randseed=50
size=10m
filename=foo
[write]
rw=randwrite
do_verify=0
write_iolog=wlog
[read]
rw=randread
do_verify=1
verify_fatal=1
verify_dump=1
write_iolog=rlog
$ fio --section=write job.fio
$ fio --section=read job.fio
Fixes: https://github.com/axboe/fio/issues/582
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 17 Apr 2018 01:40:46 +0000 (19:40 -0600)]
Merge branch 'nvml-to-pmdk' of https://github.com/sscargal/fio
* 'nvml-to-pmdk' of https://github.com/sscargal/fio:
NVML references renamed to PMDK
sscargal [Mon, 16 Apr 2018 22:47:47 +0000 (16:47 -0600)]
NVML references renamed to PMDK
Jens Axboe [Mon, 16 Apr 2018 15:22:37 +0000 (09:22 -0600)]
Fio 3.6
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 14 Apr 2018 22:26:17 +0000 (16:26 -0600)]
cconv: add conversion for 'replay_time_scale'
This was forgotten in the previous commit.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 14 Apr 2018 22:23:33 +0000 (16:23 -0600)]
Add 'replay_time_scale' option
This allows the user to scale the replay speed of an IO trace. It
defaults to 100, meaning run at 100% the original rate. If set to
50, fio will replay at 50% the original IO rate. If set to 400,
fio will replay at 4x the original rate. And so on.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 14 Apr 2018 21:22:10 +0000 (15:22 -0600)]
iolog: fix issue with replay rate
We attempt to keep the replay rate accurate, but we mistakenly use
the 'delay' value which we have decremented along the way. Store
the original and use that for the stall detection.
This fixes replays going quicker than they should.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 13 Apr 2018 23:25:35 +0000 (17:25 -0600)]
Merge branch 'proc_group' of https://github.com/sitsofe/fio
* 'proc_group' of https://github.com/sitsofe/fio:
doc: add cpus_allowed reference to log_compression_cpus
appveyor: make 32 bit build target XP + minor fixes
configure/Makefile: make Cygwin force less
doc: add Windows processor group behaviour and Windows target option
windows: target Windows 7 and add support for more than 64 CPUs
windows: prepare for Windows build split
windows: update EULA
Tomohiro Kusumi [Tue, 10 Apr 2018 15:35:32 +0000 (00:35 +0900)]
os/os-dragonfly: sync with header file changes in upstream
IOCTLTRIM was moved and renamed to a different header file,
so conditionally include previous or current header depending on
__DragonFly_version.
Some details in below commit messages.
In short, they have had IOCTLTRIM in a wrong header, and they
decided to fix/change it since fio was the only application using
IOCTLTRIM within their userspace package collections.
<sys/ioctl_compat.h>: Add a reminder comment about IOCTLTRIM.
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/
7226e3530be4df743a5cfe61f602b0c4bd436ee9
Move the IOCTLTRIM ioctl to a better header and rename it to DAIOCTRIM.
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/
32506cfa691f80faa47486d6f94a98ae12387eeb
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 9 Apr 2018 14:10:40 +0000 (08:10 -0600)]
stat: remove dead 'nr_uninit' assignment
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Mon, 9 Apr 2018 14:07:26 +0000 (08:07 -0600)]
init: fix memory leak in error handling
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:54:26 +0000 (15:54 -0600)]
steadystate: check for division by zero in mean calculation
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:50:45 +0000 (15:50 -0600)]
eta: fix dead variable assignments
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:48:35 +0000 (15:48 -0600)]
parse: fix dead 'org' assignment
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:47:55 +0000 (15:47 -0600)]
filesetup: fix dead assignment of 'ret'
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:46:17 +0000 (15:46 -0600)]
client: fix bad shadowing of 'ret'
Different scopes for ret, introduce 'wrote' to check the
return value of write(2).
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:46:07 +0000 (15:46 -0600)]
server: fix dead assignment of variable
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:42:50 +0000 (15:42 -0600)]
axmap: use calloc() for level alloc
We rely on it being set to NULL for error handling.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sun, 8 Apr 2018 21:42:20 +0000 (15:42 -0600)]
Remove binject engine
It was an experiment and never submitted upstream, let's just
drop it.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 6 Apr 2018 23:59:20 +0000 (17:59 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Fix floating point option range formatting
Bart Van Assche [Fri, 6 Apr 2018 22:49:44 +0000 (15:49 -0700)]
Fix floating point option range formatting
Ensure that floating point option ranges are properly formatted if
only one of the two boundaries is specified. A few examples of how
option ranges are formatted with this patch applied:
range: max=100.000000
range: min=0.000000, max=100.000000
range: min=0.000000
Reported-by: Sitsofe Wheeler <sitsofe@gmail.com>
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Rebecca Cran [Wed, 4 Apr 2018 23:18:42 +0000 (17:18 -0600)]
Fix return value checking of fread() in iolog.c
According to http://pubs.opengroup.org/onlinepubs/
7908799/xsh/fread.html
fread() returns a size_t, not ssize_t, and returns a value of 0 on
both eof and an error. Therefore, check both feof() and ferror() to
determine whether the call succeeded.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 4 Apr 2018 21:36:17 +0000 (15:36 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Ensure that .minfp and .maxfp are respected for FIO_OPT_FLOAT_LIST
option parsing: Mark arguments that are not modified as 'const'
parse.h: Remove a superfluous cast
Only populate the write buffer if necessary
Rename TD_F_VER_NONE into TD_F_DO_VERIFY
engines/sg: Make I/O error messages more informative
Bart Van Assche [Tue, 3 Apr 2018 18:23:11 +0000 (11:23 -0700)]
Ensure that .minfp and .maxfp are respected for FIO_OPT_FLOAT_LIST
option_init() overwrites the .minfp and .maxfp members defined in
fio_options[] which is wrong. Remove the code that overwrites these two
members and change the convention for no FIO_OPT_FLOAT_LIST limits from
(minfp == DBL_MIN && maxfp == DBL_MAX) into (minfp != 0 || maxfp != 0).
Fixes:
833491908a1a ("stats: Add a function to report completion latency percentiles")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 3 Apr 2018 20:04:02 +0000 (13:04 -0700)]
option parsing: Mark arguments that are not modified as 'const'
This patch does not modify any functionality but makes the option
parsing code slightly easier to read by marking arguments that are
not modified explicitly as 'const'. A new function has been added,
find_option_c(), that behaves identically to find_option(). Only
the function signature differs between these two functions.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 3 Apr 2018 20:04:25 +0000 (13:04 -0700)]
parse.h: Remove a superfluous cast
Adding an offset to a void pointer has the same effect as adding an
offset to a char pointer. Hence removing the (char *) cast from
td_var() does not change its behavior.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 26 Mar 2018 16:00:28 +0000 (09:00 -0700)]
Only populate the write buffer if necessary
This patch moves the populate_verify_io_u() call from inside
get_io_u() into all its callers except do_dry_run() and thereby skips
write buffer population for dry runs. This patch does not change the
behavior of fio but is necessary because the ZBC patch will insert
code in fill_io_u() after the get_io_u() call that may modify the
write offset.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 26 Mar 2018 15:50:09 +0000 (08:50 -0700)]
Rename TD_F_VER_NONE into TD_F_DO_VERIFY
Rename TD_F_VER_NONE into TD_F_DO_VERIFY to make it clear that this flag means
that data verification has to be performed. See also commit
d72be5454c8c ("Cache
layout improvements").
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 4 Apr 2018 20:45:54 +0000 (13:45 -0700)]
engines/sg: Make I/O error messages more informative
Ensure that if an I/O error is encountered that not only the error
number but also a textual error message and the engine name are
reported. Without this patch, SG error messages look like this:
fio: pid=12803, err=5/
With this patch applied SG I/O error messages change into:
fio: pid=12803, err=5/file:engines/sg.c:290, func=fio_sgio_doio, error=Input/output error
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Tue, 3 Apr 2018 14:19:38 +0000 (08:19 -0600)]
Merge branch 'fixbug_glfs' of https://github.com/simon-rock/fio
* 'fixbug_glfs' of https://github.com/simon-rock/fio:
glusterfs: always allocate io_u->engine_data
simon [Tue, 3 Apr 2018 12:16:36 +0000 (20:16 +0800)]
glusterfs: always allocate io_u->engine_data
When the iodepth is greater then 1 some io_us in fio_gf_io_u_init() can
be fooled into thinking their .engine_data has already been set when it
hasn't. This happens because .index and .engine_data are part of the
same union and .index can be set to non-zero values:
static int init_io_u(struct thread_data *td)
...
for (i = 0; i < max_units; i++) {
...
io_u->index = i;
...
if (td->io_ops->io_u_init) {
int ret = td->io_ops->io_u_init(td, io_u);
which means io_u->engine_data != 0 in fio_gf_io_u_init().
Fix the issue by removing the initial check of io_u->engine_data.
Signed-off-by: Simon Gao <simon29rock@gmail.com>
Jens Axboe [Fri, 30 Mar 2018 16:16:27 +0000 (10:16 -0600)]
Merge branch 'aarch64-crc32c' of https://github.com/sitsofe/fio
* 'aarch64-crc32c' of https://github.com/sitsofe/fio:
Minor style changes
aarch64: refactor HW ARM CRC32c detection
Sitsofe Wheeler [Wed, 28 Mar 2018 15:09:22 +0000 (16:09 +0100)]
Minor style changes
Change some #endif comments to exactly match the starting #ifdef and
in crc files and make the second posixaio configure test more
descriptive.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Wed, 28 Mar 2018 14:49:14 +0000 (15:49 +0100)]
aarch64: refactor HW ARM CRC32c detection
- Tweak how fio does the configure probe for hardware accelerated ARM
CRC32c
- Move the Linux specific runtime detection into os-linux.h making it
cleaner to implement runtime probing for OSes other than Linux in the
future
- Make the ARCH_HAVE_CRC_CRYPTO define go into the config-host.* files
the same way as most other configure defines
- Set/use only the ARCH_HAVE_CRC_CRYPTO define rather than also
creating/using ARCH_HAVE_ARM64_CRC_CRYPTO
This commit should silence the "Fails to compile: error: sys/auxv.h file
not found" compilation message that was seen on FreeBSD 12/arm64 (but
note a future commit will be required to implement runtime probing and
actually enable the accelerated path on that platform).
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Thu, 29 Mar 2018 16:02:25 +0000 (10:02 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
switch_ioscheduler(): only remove the last character if it's a newline
Make it clear to Coverity that the tmp buffer in switch_ioscheduler() is \0-terminated
Bart Van Assche [Thu, 29 Mar 2018 15:30:38 +0000 (08:30 -0700)]
switch_ioscheduler(): only remove the last character if it's a newline
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Thu, 29 Mar 2018 15:24:23 +0000 (08:24 -0700)]
Make it clear to Coverity that the tmp buffer in switch_ioscheduler() is \0-terminated
The 'tmp' buffer used by switch_ioscheduler() will be '\0'-terminated
because the I/O scheduler name read from sysfs has much fewer characters
than the buffer size (256). Make it clear to Coverity that it is
guaranteed that the 'tmp' buffer is '\0'-terminated. Additionally, fix
the order of the fread() arguments. This patch fixes Coverity ID #24132.
Fixes:
b44b9e45cf38 ("Clear sysfs path before reading current ioscheduler from sysfs")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Sitsofe Wheeler [Tue, 27 Mar 2018 16:45:33 +0000 (17:45 +0100)]
doc: add cpus_allowed reference to log_compression_cpus
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Mon, 26 Mar 2018 15:06:46 +0000 (16:06 +0100)]
appveyor: make 32 bit build target XP + minor fixes
To get some coverage of old XP paths make the 32 bit CI build target XP but
leave the 64 bit CI build targeting Windows 7.
Also fix Cygwin output being redirected into a file called NULL (rather
than going to NUL) while we're here.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Mon, 26 Mar 2018 14:50:03 +0000 (15:50 +0100)]
configure/Makefile: make Cygwin force less
Continue the work started in commit
ca205a752c3d6ebe7de74a3dfe81808e48a502e3 ("configure: Make Cygwin take
regular configure path") by removing the forceful setting of flags that
are automatically detected anyway.
Even CONFIG_64BIT_LLP64 is unneeded because despite their names
CONFIG_32BIT/CONFIG_64BIT are only used to control the size's of words
and the regular configure path correctly detects CONFIG_32BIT for 64 bit
Windows even though it's LLP64.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Tue, 27 Mar 2018 16:28:27 +0000 (17:28 +0100)]
doc: add Windows processor group behaviour and Windows target option
- Add information on targeting the build against different versions of
Windows to the README
- Stop listing -i686 packages in the 64 bit section of the README and
add a note to about them to the 32 bit build section
- Rejig cpus_allowed to be before cpumask in the documentation to make
people more likely to read that section first
- Make it clear that CPUs specified in the cpus_allowed section are
indexed from 0 (and not 1)
- Add information about how processor group awareness when setting CPUs
is only available when using a build targeted at Windows 7 (because
only then do we have the processor group APIs available to us) and how
selecting a CPU sets the processor group too (because we concatanate
the CPUs from all the processor groups together)
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 25 Mar 2018 19:56:07 +0000 (20:56 +0100)]
windows: target Windows 7 and add support for more than 64 CPUs
Introduce support for targeting the build to either Windows XP or
Windows 7 (the default). When targeting Windows 7 this allows us to use
the process group APIs which are required are required to support more
than 64 CPUs.
These changes mean if you want a Windows binary that supports versions
of Windows below Windows 7/Windows Server 2008 R2 you will need to
explicitly use the --target-win-ver=xp configure option at build time.
Such builds will lack features (such as the ability to access CPUs
beyond those in fio's default process group) that are dependent on
recent Windows APIs.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 25 Mar 2018 19:15:29 +0000 (20:15 +0100)]
windows: prepare for Windows build split
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Sun, 25 Mar 2018 17:50:53 +0000 (18:50 +0100)]
windows: update EULA
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Fri, 23 Mar 2018 15:58:51 +0000 (09:58 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
smalloc: Remove Valgrind instrumentation
Jens Axboe [Thu, 22 Mar 2018 17:29:25 +0000 (11:29 -0600)]
smalloc: Remove Valgrind instrumentation
Since it can happen that memory allocated with smalloc() is freed by
a child process and since Valgrind's memory tracking is per process
the smalloc()/sfree() instrumentation leads to incorrect Valgrind
reports. Hence remove that instrumentation again.
Fixes:
0ffccc21fcd6 ("Improve Valgrind instrumentation of memory allocations")
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Thu, 22 Mar 2018 17:29:25 +0000 (11:29 -0600)]
server: use scalloc() for sk_out allocation
This informs valgrind that it's properly initialized.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 17:02:52 +0000 (11:02 -0600)]
Merge branch 'gcc' of https://github.com/sitsofe/fio
* 'gcc' of https://github.com/sitsofe/fio:
compiler: set minimum compiler version to GCC 4.1.0
Sitsofe Wheeler [Wed, 21 Mar 2018 22:35:21 +0000 (22:35 +0000)]
compiler: set minimum compiler version to GCC 4.1.0
fio hasn't compiled under GCC 3 for a while due to unguarded calls to
GCC's __sync intrinsics in critical paths. Since those intrinsics didn't
come along until GCC 4.1.0 change fio to explicitly error out on older
compilers.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Thu, 22 Mar 2018 14:34:48 +0000 (08:34 -0600)]
smalloc: oom cleanups
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 02:09:36 +0000 (20:09 -0600)]
sg: fix sign extension
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 01:19:58 +0000 (19:19 -0600)]
server: fix error handling for shared memory handling
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Thu, 22 Mar 2018 01:01:33 +0000 (19:01 -0600)]
Merge branch 'include_refactor' of https://github.com/sitsofe/fio
* 'include_refactor' of https://github.com/sitsofe/fio:
Add include-what-you-use pragmas
oslib: make str* compat functions more uniform
Use POSIX path for poll.h and fcntl.h headers
Refactor #includes and headers
Jens Axboe [Thu, 22 Mar 2018 00:58:22 +0000 (18:58 -0600)]
server: handle shared mem pool allocation failures
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Sitsofe Wheeler [Tue, 20 Mar 2018 15:18:43 +0000 (15:18 +0000)]
Add include-what-you-use pragmas
Add IWYU pragmas to improve its analysis.
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Tue, 20 Mar 2018 10:46:32 +0000 (10:46 +0000)]
oslib: make str* compat functions more uniform
- Ensure there are CONFIG_ guards in the headers and .c files because
sometime utilities include oslib headers directly and reference oslib
.c files in the Makefile
- Ensure all the oslib/str*.h files have redefinition guards
- Shuffle some #include lines from the .h files to the .c ones
- Ensure the <name>.c files have #include "<name>.h" in them so the
compiler can check the declaration and definition match
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Mon, 19 Mar 2018 05:43:41 +0000 (05:43 +0000)]
Use POSIX path for poll.h and fcntl.h headers
musl complains when using include paths like <sys/poll.h> rather than
POSIX's <poll.h> so change the poll.h include path and update the
windows poll.h compatibility shim to match.
Allegedly sys/poll.h was needed for GLIBCs that were older than 2.3
according to
https://www.winehq.org/pipermail/wine-patches/2014-June/132877.html but
I can still see that path in an ancient Red Hat 5.0
glibc-devel-2.0.7-32.i386.rpm ...
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Sitsofe Wheeler [Mon, 19 Mar 2018 05:33:13 +0000 (05:33 +0000)]
Refactor #includes and headers
- Try and remove unneeded #include lines
- Try and add #include lines that would allow the files to be built in a
more standalone manner
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Jens Axboe [Wed, 21 Mar 2018 21:48:17 +0000 (15:48 -0600)]
server: process STOP/QUIT commands out-of-line
At the end of a job, the stats entries are queued up for
out-of-line transmission, but we immediately queue a client
STOP+QUIT command right after that. If the command handling
loop doesn't wakeup and process the stats commands before we
process the STOP+QUIT inline, then we lose the stats output
from those commands. The result is that the client won't see
the job output from that job, or the "All clients" output.
We should impose proper ordering for all commands, but this
is a quick fix to ensure that we don't hit the above mentioned
but. For other commands we don't care about ordering between
inline and out-of-line commands, so it's likely good enough
to carry us forward.
Reported-by: Jeff Furlong <jeff.furlong@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Wed, 21 Mar 2018 14:26:28 +0000 (08:26 -0600)]
Merge branch 'asprintf' of https://github.com/bvanassche/fio
* 'asprintf' of https://github.com/bvanassche/fio:
verify: Simplify dump_buf()
log: Modify the implementation such that it uses asprintf()
Add an asprintf() implementation
Jens Axboe [Wed, 21 Mar 2018 14:23:50 +0000 (08:23 -0600)]
debug: remove extra parens
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 20 Mar 2018 17:19:19 +0000 (11:19 -0600)]
optgroup: move debug code into function
The current global construct breaks on some compilers:
optgroup.c:208: error: expected identifier or '(' before 'do'
optgroup.c:208: error: expected identifier or '(' before 'while'
Reported-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
Fixes:
96344ff00349 ("optgroup: add check for optgroup bit numbers being within range")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 20 Mar 2018 14:27:43 +0000 (08:27 -0600)]
Fix whitespace issues in previous commit
Lots of trailing whitespace.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 20 Mar 2018 14:24:20 +0000 (08:24 -0600)]
Merge branch 'expand_fiohistparser' of https://github.com/shimrot/fio
* 'expand_fiohistparser' of https://github.com/shimrot/fio:
Expand fiologparser_hist operations with new options
krisd [Tue, 20 Mar 2018 05:42:26 +0000 (00:42 -0500)]
Expand fiologparser_hist operations with new options
New options are:
* noweight: don't weight samples across intervals
* percentiles: allow print of more percentiles than just 50,90,95%s
* Fix so works with newer 'ns' results, but add --usbin option
so can still be used with older results.
* directions=rwtm option to allow print of independent direction results
Jens Axboe [Mon, 19 Mar 2018 21:56:20 +0000 (15:56 -0600)]
optgroup: add check for optgroup bit numbers being within range
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Mon, 12 Mar 2018 22:29:34 +0000 (15:29 -0700)]
verify: Simplify dump_buf()
Use asprintf() to build the output file path instead of using a fixed
size buffer for that path.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 22:28:17 +0000 (15:28 -0700)]
log: Modify the implementation such that it uses asprintf()
This makes the logging implementation simpler and easier to read.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 22:38:28 +0000 (15:38 -0700)]
Add an asprintf() implementation
Since I would like to use the asprintf() function in the ZBC code and
since that function is not available on every platform supported by
fio, add an asprintf() implementation.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Mon, 19 Mar 2018 16:24:10 +0000 (10:24 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
Suppress uninteresting data race reports
gettime: Rework the clock thread starting mechanism
Improve Valgrind instrumentation of memory allocations
Rename fio_mutex into fio_sem
Split mutex.c and .h each into three files
Kris Davis [Mon, 19 Mar 2018 16:19:44 +0000 (10:19 -0600)]
sg: add read/write FUA options
Enable the user to ask for the FUA bits to be set on read and
write operations.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Sat, 17 Mar 2018 20:43:16 +0000 (14:43 -0600)]
Merge branch 'pthread-cond' of https://github.com/bvanassche/fio
* 'pthread-cond' of https://github.com/bvanassche/fio:
Signal td->free_cond with the associated mutex held
Make sure that assert() expressions do not have side effects
Bart Van Assche [Fri, 16 Mar 2018 15:38:34 +0000 (08:38 -0700)]
Signal td->free_cond with the associated mutex held
Calling pthread_cond_signal() or pthread_cond_broadcast() without
holding the associated mutex can lead to missed wakeups. Hence ensure
that td->io_u_lock is held around pthread_cond_signal(&td->free_cond)
calls. A quote from the POSIX spec
(http://pubs.opengroup.org/onlinepubs/
9699919799/functions/pthread_cond_broadcast.html):
"The pthread_cond_broadcast() or pthread_cond_signal() functions may be
called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait() or pthread_cond_timedwait() have
associated with the condition variable during their waits; however, if
predictable scheduling behavior is required, then that mutex shall be
locked by the thread calling pthread_cond_broadcast() or
pthread_cond_signal()."
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Fri, 16 Mar 2018 15:44:35 +0000 (08:44 -0700)]
Make sure that assert() expressions do not have side effects
Assert statements are compiled out if NDEBUG is defined. Hence make sure
that the expressions passed to assert do not have side effects.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 21:44:51 +0000 (14:44 -0700)]
Suppress uninteresting data race reports
Thread checkers like Helgrind and DRD report all concurrent
modifications of variables that are serialized by a recognized
synchronization primitive. Suppress reports about intentionally
unsynchronized accesses.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Tue, 13 Mar 2018 17:43:02 +0000 (10:43 -0700)]
gettime: Rework the clock thread starting mechanism
Replace two mutexes in struct clock_thread that are intended to make
the clock threads start simultaneously by a fio_mutex.
This patch avoids that drd reports the following:
==23831== Thread 2:
==23831== Mutex not locked by calling thread: mutex 0x6cb7bf8, recursion count 1, owner 1.
==23831== at 0x4C38EC3: pthread_mutex_unlock_intercept (drd_pthread_intercepts.c:978)
==23831== by 0x4C38EC3: pthread_mutex_unlock (drd_pthread_intercepts.c:991)
==23831== by 0x420974: clock_thread_fn (gettime.c:604)
==23831== by 0x4C34C44: vgDrd_thread_wrapper (drd_pthread_intercepts.c:444)
==23831== by 0x5E4159A: start_thread (in /lib64/libpthread-2.27.so)
==23831== by 0x6356A1E: clone (in /lib64/libc-2.27.so)
==23924== Mutex still locked at thread exit: mutex 0x6cb7bd0, recursion count 1, owner 2.
==23924== at 0x4C35BF2: pthread_join_intercept (drd_pthread_intercepts.c:711)
==23924== by 0x4C35BF2: pthread_join (drd_pthread_intercepts.c:718)
==23924== by 0x420D7F: fio_monotonic_clocktest (gettime.c:722)
==23924== by 0x420441: fio_clock_init (gettime.c:424)
==23924== by 0x437526: fio_time_init (time.c:141)
==23924== by 0x49BF41: main (fio.c:56)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Thu, 8 Mar 2018 21:41:36 +0000 (13:41 -0800)]
Improve Valgrind instrumentation of memory allocations
In smalloc, let Valgrind do detection of red zone modifications. In
synchr.c, let Valgrind perform use-after-free detection of
synchronization objects.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 14 Mar 2018 15:31:37 +0000 (08:31 -0700)]
Rename fio_mutex into fio_sem
Since fio_mutex implements a semaphore, rename fio_mutex into fio_sem.
Rename those fio_sem instances that have 'mutex' embedded in their
name. Rename the mutex.h and .c source files into fio_sem.h and .c
respectively.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Wed, 14 Mar 2018 15:26:15 +0000 (08:26 -0700)]
Split mutex.c and .h each into three files
Create separate header and source files for the struct fio_mutex
functions, for the struct fio_rwlock functions and for the helper
functions for initializing process shared mutexes and condition
variables. Replace an #include directive by a forward declaration
in fio.h. Minimize the #include directives in mutex.c, rwlock.c
and pshared.c. This patch does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Jens Axboe [Tue, 13 Mar 2018 17:49:55 +0000 (11:49 -0600)]
io_u: only rewind file position if it's non-zero
This:
bs=8k
rw=read:-4k
is supposed to read 0..8k, then add 8k and subtract 4k, ending
up with a next read at 4k..12k and so forth. But we rewind too
quickly, and the first IO fails as being out-of-bounds.
Fixes:
c22825bb537af ("Fix backwards reads with --size smaller than the file size")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 13 Mar 2018 14:41:37 +0000 (08:41 -0600)]
configure: don't disable lex on FreeBSD
Previous commit turned it off by mistake.
Fixes:
89556808ef14 ("configure: Disable lex on NetBSD")
Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Tue, 13 Mar 2018 00:13:10 +0000 (18:13 -0600)]
Merge branch 'master' of https://github.com/bvanassche/fio
* 'master' of https://github.com/bvanassche/fio:
parse: Fix two compiler warnings
stat: Fix a compiler warning in __show_run_stats()
Rename struct rb_node into struct fio_rb_node
configure: Disable lex on NetBSD
Bart Van Assche [Mon, 12 Mar 2018 21:24:27 +0000 (14:24 -0700)]
parse: Fix two compiler warnings
Avoid that gcc on NetBSD reports the following compiler warnings:
CC parse.o
parse.c: In function 'get_mult_time':
parse.c:175: warning: array subscript has type 'char'
parse.c: In function '__get_mult_bytes':
parse.c:221: warning: array subscript has type 'char'
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:16:56 +0000 (14:16 -0700)]
stat: Fix a compiler warning in __show_run_stats()
Avoid that gcc reports the following warning on NetBSD systems:
CC stat.o
stat.c: In function '__show_run_stats':
stat.c:1868: warning: dereferencing type-punned pointer will break strict-aliasing rules
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:13:38 +0000 (14:13 -0700)]
Rename struct rb_node into struct fio_rb_node
This patch avoids that building fio fails as follows on NetBSD systems:
CC crc/test.o
In file included from crc/../iolog.h:4,
from crc/../stat.h:4,
from crc/../thread_options.h:7,
from crc/../fio.h:18,
from crc/test.c:5:
crc/../lib/rbtree.h:101: error: redefinition of 'struct rb_node'
Makefile:336: recipe for target 'crc/test.o' failed
gmake: *** [crc/test.o] Error 1
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Mon, 12 Mar 2018 21:37:37 +0000 (14:37 -0700)]
configure: Disable lex on NetBSD
Since the NetBSD lex software does not support all features (yylval)
used by fio, disable lex on NetBSD.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Bart Van Assche [Sat, 10 Mar 2018 00:46:36 +0000 (16:46 -0800)]
filesetup: Initialize all members of struct fio_file
This patch avoids that Valgrind reports complaints like the following:
==10492== Conditional jump or move depends on uninitialised value(s)
==10492== at 0x43BF19: fio_file_reset (filesetup.c:1872)
==10492== by 0x43AB7F: alloc_new_file (filesetup.c:1496)
==10492== by 0x43AED4: add_file (filesetup.c:1581)
==10492== by 0x4528D3: str_filename_cb (options.c:1276)
==10492== by 0x44D83A: __handle_option (parse.c:722)
==10492== by 0x44E303: handle_option (parse.c:941)
==10492== by 0x44E68B: parse_cmd_option (parse.c:1061)
==10492== by 0x4541DD: fio_cmd_option_parse (options.c:4966)
==10492== by 0x427C6D: parse_cmd_line (init.c:2652)
==10492== by 0x428685: parse_options (init.c:2924)
==10492== by 0x4920AC: main (fio.c:47)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Sat, 10 Mar 2018 00:46:22 +0000 (16:46 -0800)]
helper_thread: Initialize all helper_data members before using it
This patch avoids that Valgrind reports the following:
==10492== Thread 2:
==10492== Conditional jump or move depends on uninitialised value(s)
==10492== at 0x480295: helper_thread_main (helper_thread.c:89)
==10492== by 0x57B961A: start_thread (pthread_create.c:465)
==10492== by 0x5CEC98E: clone (clone.S:95)
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Sat, 10 Mar 2018 00:46:02 +0000 (16:46 -0800)]
Declare debug_levels[] const
Since debug_levels[] is not modified by any code, declare it const.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 19:55:21 +0000 (12:55 -0700)]
filesetup: don't round/adjust size for size_percent == 100
For the full device, don't apply any rounding or bs based
adjustment. Just treat it like size_percent wasn't even set.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 9 Mar 2018 16:51:04 +0000 (08:51 -0800)]
Makefile: Rerun the configure script if it has been modified
Because of a typo in the Makefile ("config-host-mak") the configure
script is not rerun if the configure script is modified after
config-host.mak has been generated. Merge the two make goals for
rerunning configure into a single goal. This patch relies on the
following make behavior (see also https://www.gnu.org/software/make/manual/make.html#Remaking-Makefiles):
Sometimes makefiles can be remade from other files, such as RCS or SCCS
files. If a makefile can be remade from other files, you probably want
make to get an up-to-date version of the makefile to read in.
To this end, after reading in all makefiles, make will consider each as a
goal target and attempt to update it. If a makefile has a rule which says
how to update it (found either in that very makefile or in another one) or
if an implicit rule applies to it (see Using Implicit Rules), it will be
updated if necessary. After all makefiles have been checked, if any have
actually been changed, make starts with a clean slate and reads all the
makefiles over again. (It will also attempt to update each of them over
again, but normally this will not change them again, since they are
already up to date.)
This patch has been tested by verifying that the commands run by make
were as expected for the following invocations:
rm -f config-host.mak && make
make && touch configure && make
make
Note: fio already requires GNU make because of the use of "ifeq" etc.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 17:23:53 +0000 (10:23 -0700)]
io_u: 'is_random' can be a boolean
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 17:22:01 +0000 (10:22 -0700)]
io_u: kill get_next_{offset,buflen} wrappers
After the previous commit, they are just wrappers. Kill them.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Bart Van Assche [Fri, 9 Mar 2018 17:16:23 +0000 (09:16 -0800)]
Remove prof_io_ops.fill_io_u_off(), .fill_io_u_size() and .get_next_file()
Remove the struct prof_io_ops methods for which no implementations
have been defined. This patch does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Jens Axboe [Fri, 9 Mar 2018 14:59:03 +0000 (07:59 -0700)]
mutex: fix other locations where we are not waking within the lock
Our workqueue implementation had a few cases where it did not wake
while holding the lock, fix those up. Ditto for verify, where
async IO completion also needed the ordering switched.
Also see commit
e4ccf13bad6d.
Signed-off-by: Jens Axboe <axboe@kernel.dk>