AgeCommit message (Collapse)Author
2020-11-05Revert "Windows: update dobuild.cmd to run the configure/make"Jens Axboe
This reverts commit 38c2f9384db8dbd93f59d965d70ab0d3a53343fa. It's causing issues for CI, revert it for now. Signed-off-by: Jens Axboe <>
2020-11-05Merge branch 'master' of Axboe
* 'master' of engines/hdfs: swap fio_hdfsio_init and fio_hdfsio_io_u_init Makefile: fix usage of JAVA_HOME environmental variable
2020-11-05Merge branch 'patch-1' of Axboe
* 'patch-1' of goptions: correct postfix
2020-11-05engines/hdfs: swap fio_hdfsio_init and fio_hdfsio_io_u_initAlbert Chang
Commit 08dc3bd50 initialized the io engine before the io_u buffers. The unintended consequence for the libhdfs io engine was that the HDFS connection was not established before attempting to open HDFS file handles. This caused a NPE in the java layer on start-up preventing the libhdfs io engine from being able to be used. Signed-off-by: Albert Chang <>
2020-11-05Makefile: fix usage of JAVA_HOME environmental variableAlbert Chang
Signed-off-by: Albert Chang <>
2020-11-04Windows: update dobuild.cmd to run the configure/makeRebecca Cran
Add a new shell script, which runs under Cygwin. This is invoked via dobuild.cmd to run the configure and make steps of the build. In this way, a single command, run from a Developer Command Prompt, can be used to go from a clean workspace to a final installer binary. Signed-off-by: Rebecca Cran <> Signed-off-by: Jens Axboe <>
2020-11-04goptions: correct postfixgloit042
Signed-off-by: Jiahao Li <>
2020-11-01fio: fix dynamic engines soname definitionYigal Korman
The SONAME of the engines should not match the name of the dependent library. Otherwise it confuses the dynamic loader into thinking the dependency is already resolved. Prefixing the name with fio make more sense here. Signed-off-by: Yigal Korman <> Signed-off-by: Jens Axboe <>
2020-10-30t/zbd: Fix test target size of test case #48Shin'ichiro Kawasaki
Option --size was not specified to the fio command of test case #48. It resulted in write operations to all available sequential write required zones and relaxed zone locking test condition. Specify the option to limit test target to 16 zones so that zone locking is tested with expected condition. Fixes: 3bd2078bdd1c ("zbd: add test for stressing zone locking") Signed-off-by: Shin'ichiro Kawasaki <> Reviewed-by: Dmitry Fomichev <> Signed-off-by: Jens Axboe <>
2020-10-30zbd: Avoid excessive zone resetsShin'ichiro Kawasaki
When zbd_reset_zone() is called for a zone repeatedly, reset zone command is issued multiple times to a single zone even though its status is empty. This is excessive and meaningless. Especially when zones are reset at file set up with multiple jobs, zone reset is repeated for each job and delays fio work load start. To avoid the repeated zone resets, check the write pointer of the zone before issuing zone reset command. If the write pointer is at the zone start, do not reset the zone and just return. Signed-off-by: Shin'ichiro Kawasaki <> Reviewed-by: Dmitry Fomichev <> Signed-off-by: Jens Axboe <>
2020-10-28Merge branch 'o_dsync' of Axboe
* 'o_dsync' of extend --sync to allow {sync,dsync,0,1}, to support O_DSYNC
2020-10-28extend --sync to allow {sync,dsync,0,1}, to support O_DSYNCAndres Freund
2020-10-14Disallow offload IO mode for engines marked with FIO_NO_OFFLOADJens Axboe
Previous commits did this for all async engines, this can potentially break existing job files. There are only certain cases where it fails, for now at least mark io_uring as one of those. Reported-by: Jeff Furlong <> Signed-off-by: Jens Axboe <>
2020-10-13Merge branch 'patch-1' of into masterJens Axboe
* 'patch-1' of getopt_long: avoid variable global initialization
2020-10-13getopt_long: avoid variable global initializationCheng Li
Issue #1100 shows that an address sanitizer(ASAN) complains about a few global variables that are initialized globally in the getopt_long.c file. We look into these variables and found they do not need to be initialized globally. This patch fixes the issue by cleaning up the global initialization for these variables. Fixes Signed-off-by: Cheng Li <>
2020-10-10Merge branch 'master' of into masterJens Axboe
* 'master' of helper_thread: Increase timer accuracy helper_thread: Rework the interval timer code Change the return value of two functions from 'void' into 'int' helper_thread: Introduce the wait_for_action() function helper_thread: Introduce a function for blocking Unix signals gettime: Introduce rel_time_since() gettime: Simplify get_cycles_per_msec() gettime: Introduce fio_get_mono_time() configure: Remove the CLOCK_MONOTONIC_PRECISE probe
2020-10-10simplify MB/s, IOPS interactive printout codeAlexey Dobriyan
Use loops(!) instead of hardcoded if-else pairs. Make it easier to add DDIR_APPEND. Signed-off-by: Alexey Dobriyan (SK hynix) <> Signed-off-by: Jens Axboe <>
2020-10-10flow: use unsigned long for the counterJens Axboe
Same size on 64-bit, it'll be 32-bit on 32-bit archs. But we can't reliably use 64-bit atomics on 32-bit archs. Signed-off-by: Jens Axboe <>
2020-10-10flow: avoid holes in struct fio_flowJens Axboe
Signed-off-by: Jens Axboe <>
2020-10-09Merge branch 'master' of into masterJens Axboe
* 'master' of td_var: avoid arithmetic on a pointer to void cast (#1096)
2020-10-08td_var: avoid arithmetic on a pointer to void cast (#1096)Cheng Li
Issue 1096 shows that a compiler complains an `arithmetic on a pointer to void` error when `-Wpointer-arith` is turned on in Makefile. Although there are many violating instances, we only fix the one in `parse.h` because it is shared across many core source code files. This patch fixes the `arithmetic on a pointer to void` issue by casting it to a uintptr_t type first and perform manipulation, and then cast it back to void* type. Signed-off-by: Cheng Li <>
2020-09-25helper_thread: Increase timer accuracyBart Van Assche
The helper_thread uses select() to wait for the next event. select() rounds up its timeout argument to 1 / HZ. Increase timer accuracy by using a high-res timer if possible. This patch has been tested by running the following fio command and by verifying that the next_log and msec_to_next_event were always above 100 ms: ./fio --debug=helperthread --bandwidth-log --write_bw_log=log --log_avg_msec=1000 --ioengine=null --name=null --size=1G --rw=randwrite --loops=999999 --runtime=60 Signed-off-by: Bart Van Assche <>
2020-09-25helper_thread: Rework the interval timer codeBart Van Assche
Instead of adding the number of milliseconds that have elapsed when a timer fires, add the timer interval. Reduce code duplication by introducing a helper function that verifies whether or not a timer has expired. Signed-off-by: Bart Van Assche <>
2020-09-20Change the return value of two functions from 'void' into 'int'Bart Van Assche
This patch does not change any functionality but makes the next patch in this series easier to read. Signed-off-by: Bart Van Assche <>
2020-09-20helper_thread: Introduce the wait_for_action() functionBart Van Assche
This patch makes helper_thread_main() easier to read. Signed-off-by: Bart Van Assche <>
2020-09-20helper_thread: Introduce a function for blocking Unix signalsBart Van Assche
This patch reworks commit c31092b8ac06 ("Block signals for the helper thread") but does not change any functionality. Signed-off-by: Bart Van Assche <>
2020-09-20gettime: Introduce rel_time_since()Bart Van Assche
rel_time_since() subtracts two timespecs and returns the result as a signed integer. Instead of using the implementation from mtime_since(), use an implementation that only performs one division instead of two. Signed-off-by: Bart Van Assche <>
2020-09-20gettime: Simplify get_cycles_per_msec()Bart Van Assche
Call fio_get_mono_time() directly instead of setting fio_clock_source and calling __fio_gettime(). Signed-off-by: Bart Van Assche <>
2020-09-20gettime: Introduce fio_get_mono_time()Bart Van Assche
Introduce a new function for querying the monotonic clock, something that is necessary in every context where relative time is measured and where wall clock time jumps should have no effect. Remove fill_clock_gettime() since the only contexts where CLOCK_MONOTONIC_RAW are used are get_cycles_per_msec() and --clocksource=clock_gettime. I think both contexts should use CLOCK_MONOTONIC instead of its raw variant such that the values read from the clock are frequency adjusted (a computer clock crystal can deviate up to 500 ppm from its nominal frequency). This patch improves accuracy of the helper_thread code on Darwin. Darwin supports CLOCK_MONOTONIC but not pthread_condattr_setclock(). In other words, this patch causes the helper thread code to switch from the real-time clock to the monotonic clock on Darwin. Signed-off-by: Bart Van Assche <>
2020-09-20configure: Remove the CLOCK_MONOTONIC_PRECISE probeBart Van Assche
CLOCK_MONOTONIC_PRECISE is not used in the fio source code. Hence use the probe for that constant. The removed test was introduced by commit 67bf982340d9 ("Add configure script"). Signed-off-by: Bart Van Assche <>
2020-09-14Merge branch 'msys2' of into masterJens Axboe
* 'msys2' of travis: cleanup build script configure: cleanup lex/yacc tests memlock: avoid type confusion in format string Makefile: introduce FIO_CFLAGS appveyor: cleanup and add separate install script Makefile/ci: Don't pass CFLAGS when linking configure/Makefile: add option to generate pdb symbols configure: be explicit about "XP" Windows API version windows: fix DWORD format string complaints windows: fix wrong format strings net: coerce the result of htonl before printing configure: pass non-null pointer to (v)asprintf
2020-09-14Merge branch 'backend' of into masterJens Axboe
* 'backend' of backend: Use asprintf() instead of strlen() + sprintf() backend: Remove two superfluous casts
2020-09-12travis: cleanup build scriptSitsofe Wheeler
- Standardise on spaces for indentation - Exit on error or using variables before they are defined - Fix up shellcheck complaints by exporting some variables - Drop printing of python 2 information as we don't use it - Use type -p rather than which Signed-off-by: Sitsofe Wheeler <>
2020-09-12configure: cleanup lex/yacc testsSitsofe Wheeler
- Switch to prebuilt function for finding executable - Search for the fl library from flex too - Test lex command output directly This allows lexing to be more likely to be found (e.g. MSYS2 but sadly not MSYS2 Mingw-w64). Signed-off-by: Sitsofe Wheeler <>
2020-09-12memlock: avoid type confusion in format stringSitsofe Wheeler
Windows GCC has the followng grumble: t/memlock.c: In function 'worker': t/memlock.c:25:26: error: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'size_t' {aka 'long long unsigned int'} [-Werror=format=] 25 | printf("loop%d: did %lu MiB\n", i+1, size/(1024UL*1024UL)); | ~~^ ~~~~~~~~~~~~~~~~~~~~ | | | | long unsigned int size_t {aka long long unsigned int} | %llu Fix it up by just using the original variable that was already in megabytes. Signed-off-by: Sitsofe Wheeler <>
2020-09-12Makefile: introduce FIO_CFLAGSSitsofe Wheeler
Build on the work done in 4c0b3d98f2d05ddd3f16262c466dcedb22158065 ("configure/Makefile: don't override user CFLAGS") by builing up all the internal flags in a temporary variable. Signed-off-by: Sitsofe Wheeler <>
2020-09-12appveyor: cleanup and add separate install scriptSitsofe Wheeler
- Rename PLATFORM environment variable to ARCHITECTURE to avoid clash with Appveyor variable ( ) - Introduce support for using MSYS2 - Perform clang builds that also package PDB symbols - Drop PACKAGE_ARCH variable since it can be derived - Switch some lines to just call native binaries directly Signed-off-by: Sitsofe Wheeler <>
2020-09-12Makefile/ci: Don't pass CFLAGS when linkingSitsofe Wheeler
There was an issue where a lld (the linker from LLVM) could fail because a C compiler flag it didn't understand was passed to it. Fix this by no longer passing CFLAGS to the linker. As a result of the above, the travis "cross compiled" 32 bit Linux build was changed to set LDFLAGS (so the linker can know about the cross compiling). Signed-off-by: Sitsofe Wheeler <>
2020-09-12configure/Makefile: add option to generate pdb symbolsSitsofe Wheeler
Recent versions of Windows clang and LLVM's lld have the ability to build Windows PDB symbols for debugging so add an explicit option to test for support and use it so. Signed-off-by: Sitsofe Wheeler <>
2020-09-12configure: be explicit about "XP" Windows API versionSitsofe Wheeler
With a Cygwin Mingw-w64 environment, when an explicit Windows API version define isn't set the Windows Server 2003 API is targeted ( ). This means the XP build was actually misnamed and have never been fully compatible with plain XP for many years (I suspect such binaries will likely still work on XP bar trying to use the net ioengine)... With an MSYS2 Mingw-w64 environment, the default Windows API version is Windows 7 ( ) which breaks an "XP" build because we were were relying on the default being 2003. Fix this by making the "XP" build explicitly target the Windows Server 2003 API. Signed-off-by: Sitsofe Wheeler <>
2020-09-12windows: fix DWORD format string complaintsSitsofe Wheeler
Printing DWORD variables causes Windows clang to grumble about types: os/windows/cpu-affinity.c:263:5: error: format specifies type 'int' but the argument has type 'DWORD' (aka 'unsigned long') [-Werror,-Wformat] GetLastError()); ^~~~~~~~~~~~~~ Since a DWORD is defined to be an unsigned 32 bit integer and is declared to be an unsigned long ( ) just use long as the format specifier. When printing the cpu mask in hex on legacy Windows builds, cast to unsigned long long which should be safe because there won't be more than 64 CPUs with old versions of Windows. (As some format strings are longer than the wrapping width, make an exception and avoid splitting/preserve them to make greping easier) Suggested-by: Bart Van Assche <> Signed-off-by: Sitsofe Wheeler <>
2020-09-12windows: fix wrong format stringsSitsofe Wheeler
Found by Windows clang which mysteriously seemed able to ignore -Wno-format... (As some format strings are longer than the wrapping width, make an exception and avoid splitting/preserve them to make greping easier) Signed-off-by: Sitsofe Wheeler <>
2020-09-12net: coerce the result of htonl before printingSitsofe Wheeler
Technically htonl() returns a long on Windows ( ) which upsets clang. Signed-off-by: Sitsofe Wheeler <>
2020-09-12configure: pass non-null pointer to (v)asprintfSitsofe Wheeler
Windows clang warns about redefinition of (v)asprintf when targeting Windows 7 because the configure test failed due to passing NULL when to those functions when they tag key parameters with the nonnull attribute. Signed-off-by: Sitsofe Wheeler <>
2020-09-11backend: Use asprintf() instead of strlen() + sprintf()Bart Van Assche
This patch does not change any functionality but makes exec_string() easier to read. Signed-off-by: Bart Van Assche <>
2020-09-11backend: Remove two superfluous castsBart Van Assche
Casting the type of a string constant to (const char *) is not necessary. These casts were introduced by commit ce4864950d2c ("backend: Logging exec_{pre|post}run"). Signed-off-by: Bart Van Assche <>
2020-09-11Allow offload with FAKEIO enginesJens Axboe
The last check was too restrictive, we can allow it with the fake IO engines. Fixes: abfd235a0533 ("Disable io_submit_mode=offload with async engines") Signed-off-by: Jens Axboe <>
2020-09-11Disable io_submit_mode=offload with async enginesJens Axboe
We have various cases that aren't handled correctly with async engines, or cannot work with async engines. Disable it. Signed-off-by: Jens Axboe <>
2020-09-11engines/io_uring: mark as not compatible with io_submit_mode=offloadJens Axboe
Signed-off-by: Jens Axboe <>
2020-09-09Merge branch 'evelu-enghelp' of into masterJens Axboe
* 'evelu-enghelp' of init: exiting with fio_show_ioengine_help return code