summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
5 daysMerge branch 'msys2' of https://github.com/sitsofe/fio into masterHEADmasterJens Axboe
* 'msys2' of https://github.com/sitsofe/fio: travis: cleanup build script configure: cleanup lex/yacc tests memlock: avoid type confusion in format string Makefile: introduce FIO_CFLAGS appveyor: cleanup and add separate install script Makefile/ci: Don't pass CFLAGS when linking configure/Makefile: add option to generate pdb symbols configure: be explicit about "XP" Windows API version windows: fix DWORD format string complaints windows: fix wrong format strings net: coerce the result of htonl before printing configure: pass non-null pointer to (v)asprintf
5 daysMerge branch 'backend' of https://github.com/bvanassche/fio into masterJens Axboe
* 'backend' of https://github.com/bvanassche/fio: backend: Use asprintf() instead of strlen() + sprintf() backend: Remove two superfluous casts
8 daystravis: cleanup build scriptSitsofe Wheeler
- Standardise on spaces for indentation - Exit on error or using variables before they are defined - Fix up shellcheck complaints by exporting some variables - Drop printing of python 2 information as we don't use it - Use type -p rather than which Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysconfigure: cleanup lex/yacc testsSitsofe Wheeler
- Switch to prebuilt function for finding executable - Search for the fl library from flex too - Test lex command output directly This allows lexing to be more likely to be found (e.g. MSYS2 but sadly not MSYS2 Mingw-w64). Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysmemlock: avoid type confusion in format stringSitsofe Wheeler
Windows GCC has the followng grumble: t/memlock.c: In function 'worker': t/memlock.c:25:26: error: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'size_t' {aka 'long long unsigned int'} [-Werror=format=] 25 | printf("loop%d: did %lu MiB\n", i+1, size/(1024UL*1024UL)); | ~~^ ~~~~~~~~~~~~~~~~~~~~ | | | | long unsigned int size_t {aka long long unsigned int} | %llu Fix it up by just using the original variable that was already in megabytes. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysMakefile: introduce FIO_CFLAGSSitsofe Wheeler
Build on the work done in 4c0b3d98f2d05ddd3f16262c466dcedb22158065 ("configure/Makefile: don't override user CFLAGS") by builing up all the internal flags in a temporary variable. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysappveyor: cleanup and add separate install scriptSitsofe Wheeler
- Rename PLATFORM environment variable to ARCHITECTURE to avoid clash with Appveyor variable (https://www.appveyor.com/docs/environment-variables/ ) - Introduce support for using MSYS2 - Perform clang builds that also package PDB symbols - Drop PACKAGE_ARCH variable since it can be derived - Switch some lines to just call native binaries directly Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysMakefile/ci: Don't pass CFLAGS when linkingSitsofe Wheeler
There was an issue where a lld (the linker from LLVM) could fail because a C compiler flag it didn't understand was passed to it. Fix this by no longer passing CFLAGS to the linker. As a result of the above, the travis "cross compiled" 32 bit Linux build was changed to set LDFLAGS (so the linker can know about the cross compiling). Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysconfigure/Makefile: add option to generate pdb symbolsSitsofe Wheeler
Recent versions of Windows clang and LLVM's lld have the ability to build Windows PDB symbols for debugging so add an explicit option to test for support and use it so. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysconfigure: be explicit about "XP" Windows API versionSitsofe Wheeler
With a Cygwin Mingw-w64 environment, when an explicit Windows API version define isn't set the Windows Server 2003 API is targeted (https://github.com/mirror/mingw-w64/blob/a25dc933f3d3ed6c8529fb24b7ae393f8792ca69/mingw-w64-headers/include/sdkddkver.h#L166 ). This means the XP build was actually misnamed and have never been fully compatible with plain XP for many years (I suspect such binaries will likely still work on XP bar trying to use the net ioengine)... With an MSYS2 Mingw-w64 environment, the default Windows API version is Windows 7 (https://github.com/msys2/MINGW-packages/blob/e9f271c61a39076ab5de56299f7fd86020c280ae/mingw-w64-headers-git/PKGBUILD#L55 ) which breaks an "XP" build because we were were relying on the default being 2003. Fix this by making the "XP" build explicitly target the Windows Server 2003 API. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 dayswindows: fix DWORD format string complaintsSitsofe Wheeler
Printing DWORD variables causes Windows clang to grumble about types: os/windows/cpu-affinity.c:263:5: error: format specifies type 'int' but the argument has type 'DWORD' (aka 'unsigned long') [-Werror,-Wformat] GetLastError()); ^~~~~~~~~~~~~~ Since a DWORD is defined to be an unsigned 32 bit integer and is declared to be an unsigned long (https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/262627d8-3418-4627-9218-4ffe110850b2 ) just use long as the format specifier. When printing the cpu mask in hex on legacy Windows builds, cast to unsigned long long which should be safe because there won't be more than 64 CPUs with old versions of Windows. (As some format strings are longer than the wrapping width, make an exception and avoid splitting/preserve them to make greping easier) Suggested-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 dayswindows: fix wrong format stringsSitsofe Wheeler
Found by Windows clang which mysteriously seemed able to ignore -Wno-format... (As some format strings are longer than the wrapping width, make an exception and avoid splitting/preserve them to make greping easier) Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysnet: coerce the result of htonl before printingSitsofe Wheeler
Technically htonl() returns a long on Windows (https://docs.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-htonl ) which upsets clang. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysconfigure: pass non-null pointer to (v)asprintfSitsofe Wheeler
Windows clang warns about redefinition of (v)asprintf when targeting Windows 7 because the configure test failed due to passing NULL when to those functions when they tag key parameters with the nonnull attribute. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
8 daysbackend: Use asprintf() instead of strlen() + sprintf()Bart Van Assche
This patch does not change any functionality but makes exec_string() easier to read. Signed-off-by: Bart Van Assche <bvanassche@acm.org>
8 daysbackend: Remove two superfluous castsBart Van Assche
Casting the type of a string constant to (const char *) is not necessary. These casts were introduced by commit ce4864950d2c ("backend: Logging exec_{pre|post}run"). Signed-off-by: Bart Van Assche <bvanassche@acm.org>
9 daysAllow offload with FAKEIO enginesJens Axboe
The last check was too restrictive, we can allow it with the fake IO engines. Fixes: abfd235a0533 ("Disable io_submit_mode=offload with async engines") Signed-off-by: Jens Axboe <axboe@kernel.dk>
9 daysDisable io_submit_mode=offload with async enginesJens Axboe
We have various cases that aren't handled correctly with async engines, or cannot work with async engines. Disable it. Signed-off-by: Jens Axboe <axboe@kernel.dk>
9 daysengines/io_uring: mark as not compatible with io_submit_mode=offloadJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
11 daysMerge branch 'evelu-enghelp' of https://github.com/ErwanAliasr1/fio into masterJens Axboe
* 'evelu-enghelp' of https://github.com/ErwanAliasr1/fio: init: exiting with fio_show_ioengine_help return code
11 daysinit: exiting with fio_show_ioengine_help return codeErwan Velu
In the current code, calling --enghelp=<engine> always return 0. But fio_show_ioengine_help() implemented a return code to inform when an engine is present or not. This commit makes fio returning fio_show_ioengine_help() value when calling --enghelp=<engine> A typical use case is detecting which engine is available on a given fio build. It is possible to use the return code to determine if a given engine is available or not : [user@host]# fio --enghelp=libaio userspace_reap : Use alternative user-space reap implementation [user@host]# echo $? 0 [user@host]#fio --enghelp=io_uring fio: engine io_uring not loadable IO engine io_uring not found [user@host]# echo $? 1 Signed-off-by: Erwan Velu <e.velu@criteo.com>
12 daysMerge branch 'zbd' of https://github.com/bvanassche/fio into masterJens Axboe
* 'zbd' of https://github.com/bvanassche/fio: zbd: Add a missing pthread_mutex_unlock() call
12 daysengines/io_uring: allow setting of IOSQE_ASYNCJens Axboe
Add engine option 'force_async'. If that is set to N, then every N requests will have IOSQE_ASYNC set to force async offload. Signed-off-by: Jens Axboe <axboe@kernel.dk>
12 dayszbd: Add a missing pthread_mutex_unlock() callBart Van Assche
This patch fixes the following Coverity complaint: CID 292491 (#1 of 1): Missing unlock (LOCK) missing_unlock: Returning without unlocking zb->mutex. Fixes: bfbdd35b3e8f ("Add support for zoned block devices") Signed-off-by: Bart Van Assche <bvanassche@acm.org>
13 daysKill off old GUASI IO engineJens Axboe
This library never saw widespread use, and it hasn't been updated in more than a decade. On top of that, the fio engine was broken for a long time and nobody noticed. Time to take it out behind the barn. Signed-off-by: Jens Axboe <axboe@kernel.dk>
13 daysfio: cap io_size=N% at 100%, update man pageAlexey Dobriyan
Passing io_size > 100% results in semi-infinite loop, anyway. Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
14 daysMerge branch 'guasi_fixes' of https://github.com/sitsofe/fio into masterJens Axboe
* 'guasi_fixes' of https://github.com/sitsofe/fio: Makefile/configure: fix guasi build
14 daysMakefile/configure: fix guasi buildSitsofe Wheeler
14 daysFio 3.23fio-3.23Jens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-05fio: support io_size=N% (N <= 100)Alexey Dobriyan
Parse "io_size=N%". Semantics is "multiply whatever size= calculations result in". Example #1: size=50% io_size=50% will do 25% of a file. Example #2: size=1G io_size=50% will do 512M I/O. As side effect, fix a bug with essentially infinite loop if both size=N% and io_size=M% are given: io_size is set to 2^64-... in this case (a lot!). Note: only values under 100% work currently. Going for io_size=150% requires resetting workload generator state which is whole separate endeavour. Signed-off-by: Alexey Dobriyan (SK hynix) <adobriyan@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-03engines/io_uring: use the atomic load acquire instead of a barrierJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-03t/io_uring: allow setting fixed files/buffers as argumentsJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-03engines/io_uring: move sqe clear out of hot pathJens Axboe
We just need to ensure we set the stuff we need, then we can avoid a full memset of the sqe in the hot path. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-01t/zbd: Remove unnecessary option for zbc_reset_zoneShin'ichiro Kawasaki
When -all option is specified, zbc_reset_zone command does not take offset option. Remove it. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-01t/zbd: Reset all zones before test when max open zones is specifiedShin'ichiro Kawasaki
When the test target device has maximum open zones limit, the zones in test target region may not be opened up to the limit, because the zones out of the test target region may have open zones. To ensure that the test target zones can be opened up to the limit, reset all zones of the test target device before the test cases with write work load starts. Introduce the helper function prep_write() to check if all zone reset is required and do the reset. Also remove unnecessary reset_zone calls for test case #29 and #48. These are no longer required by virtue of the improvement in zbd_setup_files() to set up zones to meet max_open_zones limit. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-01t/zbd: Add -o option to t/zbd/test-zoned-supportShin'ichiro Kawasaki
To specify maximum open zones of the test target device, add -o option to t/zbd/test-zoned-support. When this option is specified, add max_open_zones option to all fio commands in the test script. The option ensures the number of zones opened by fio is within the limit. This is useful when the test target device has maximum open zones limit. When the fio command does not have multiple jobs and the test case does not specify max_open_zones, add single max_open_zones option to the fio command line. When the fio command takes multiple jobs, add max_open_zones option to each job. Introduce job_var_opts variable to keep options to be added to each job. To distinguish it from global options for all jobs, rename var_opts variable to global_var_opts. When a test case with multiple jobs always specifies max_open_zones option, exclude the max_open_zones option from job_var_opts, using job_var_opts_exclude() helper function. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-01t/zbd: Improve usage message of test-zbd-support scriptShin'ichiro Kawasaki
The usage message of t/zbd/test-zbd-support does not explain meaning of each option. Elaborate meaning of the options. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-01zbd: Initialize open zones list referring zone status at fio startShin'ichiro Kawasaki
When fio starts write workloads to zones with open status, fio does not reflect the zone status to open zone list. This results in inconsistent open zone accounting. To avoid this inconsistency, initialize the open zone list referring the zone status before workload start. If a zone in target range have open status, call zbd_open_zone() function to build up the open zone list. Since io_u is not available at that timing, make the function independent from io_u. If the number of open zones exceeds maximum open zones limits, reset the exceeding open zone by calling zbd_reset_zone(). Of note is that fio counts open zones within the write target range. Open zones out of the range are not counted or checked. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-01oslib/linux-blkzoned: Allow reset zone before file set upShin'ichiro Kawasaki
To reset zones before file set up completion, modify the helper function blkzoned_reset_wp() to work even when fio_file struct does not have valid file descriptor yet. If the file descriptor is not available, open and close the reset target file within the function. Otherwise use the available descriptor to avoid file open/close overhead. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-01zbd: Decrement open zones count at write command completionShin'ichiro Kawasaki
When max_open_zones or job_max_open_zones option is provided, fio counts number of open zones. This open zone accounting is done during submission of write commands. When write command is submitted to a zone for the first time, the zone is counted as open. When a write command which will fill the zone gets submitted, the zone is counted as closed. However, this count at write command submission may open zones more than the limit. When writes to zones more than the limit are submitted with high queue depth, those writes in-flight open zones more than the limit. To avoid such writes beyond max open zones limit, decrement number of open zones count not at write command submission but at write command completion. By doing this, the number of zones with in-flight write commands are kept within the limit with accuracy. Introduce the helper function zbd_end_zone_io() for this decrement and zbd_close_zone() call. The zbd_close_zone() function requires thread_data argument. To refer thread_data at write command completion, add the argument to zbd_put_io() and zbd_queue_io() callbacks. Also add a loop to the zbd_close_zone() function which converts zone_info array index to open_zones array index to avoid loop code duplication. Even when io_u points to an open zone, the zone may not be valid for write since in-flight write commands may make the zone full. Check this in zbd_open_zone() to handle such zones as in full status. Because of the zone close timing change, there might be no open zone when zbd_convert_to_open_zone() is called. Do not handle such case as an error and open other zones. When zbd_convert_to_open_zone() finds all open zones can not be used for next write, it opens other zones. This zone open fails when the number of open zones hits one of the limits: 1) number of zones in the fio write target region, 2) max_open_zones option or 3) job_max_open_zones option. To avoid the zone open failure, wait for writes in-flight completes and open zones get closed before opening other zones. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-08-31options: flow should parse as FIO_OPT_INTJens Axboe
It's an unsigned int, if we parse it as FIO_OPT_ULL then we clobber the variable around it in the options struct. This is apparent when running the examples/flow.fio job file, as flow_sleep=1000 is set before flow, but gets clobbered when flow is set. This results in a flow_sleep value of 0 instead of 1000. Fixes: d4e74fda98b6 ("flow: add ability for weight-based flow control on multiple jobs") Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-08-31Merge branch 'multi_job_flow' of https://github.com/bardavid/fio into masterJens Axboe
* 'multi_job_flow' of https://github.com/bardavid/fio: flow: add ability for weight-based flow control on multiple jobs flow: reclaim flow when job is reaped
2020-08-31flow: add ability for weight-based flow control on multiple jobsDavid, Bar
Fixes: #741 - fio how to run mix workload all 4 patterns in one job file Previously, 'flow' was used to regulate activity between 2 or more jobs with certain weight. However, the implementation was ineffective when regulating more than 2 jobs with different weights, since it relied on a single, shared counter, and was missing logic to track individual job's activity and regulate it according to it's requested weight in proportion to other jobs' requested weight and activity. This commit modifies the 'flow' functionality to track each job's activity and the total activity belonging to a 'flow_id' group. Using this new information, the 'flow' logic is then modified to regulate a job's activity in proportion to other jobs in the same group. Similar to previous behavior, the regulation is done by stalling a job's activity. New jobs joining an existing 'flow_id' group or jobs repead from said group will cause re-calibration of the weights and alter the activity as a result. For example, 3 jobs (j1,j2,j3) with weights (1:2:3) respectively will maintain activity ratio of 1:2:3. Thus, j1 will have activity of 1/6, j2 - 2/6 and j3 - 1/2. If j3 exits prematurely, the proportion is re-calibrated such that j1 will strive to have activity of 1/3 and j2 - 2/3. Re-calibration phase may cause a hiccup in the activity of one or more jobs since the new desired proportion might not be achieved quickly enough. Signed-off-by: David, Bar <bardavvid@gmail.com>
2020-08-30flow: reclaim flow when job is reapedDavid, Bar
Signed-off-by: David, Bar <bardavvid@gmail.com>
2020-08-29Update the year to 2020 in os/windows/eula.rtfRebecca Cran
Signed-off-by: Rebecca Cran <rebecca@bsdio.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-08-29Update os/windows/dobuild.cmd to support signing binaries/installerRebecca Cran
Since organizations are increasingly requiring software to be digitally signed, or have heuristics that penalize unsigned software, add support to do build.cmd to sign both the .exe binaries and the .msi installer. Signed-off-by: Rebecca Cran <rebecca@bsdio.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-08-28Use fallthrough attributeJens Axboe
Fio currently triggers a bunch of fall through errors on clang 10, since it doesn't work with the /* fall through */ method of indicating fallthrough. Normalize this a bit and use the correct compiler attribute for this. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-08-28Merge branch 'master' of https://github.com/donny372/fio into masterJens Axboe
* 'master' of https://github.com/donny372/fio: Avoid multiple instance read iolog from stdin.
2020-08-27zbd: don't read past the WP on a read only workload with verifyAravind Ramesh
When running a read only workload with verify option enabled, fio continues to read beyond the write pointer, and thus reads a deallocated block, which contains data of a predefined pattern, thus causing the verify to fail. Thus, we cannot simply jump to accept for a read only workload with verify option enabled, we must let the regular flow of execution to continue, so that the check if offset > wp is performed, so that we will return the address to a new zone, rather than an address that is past the wp. Reading past the wp is not a recent bug, it is possible to reproduce this bug even before zone capacity was introduced. However, with the introduction of zone capacity, filling a zone now only writes up to zone capacity of a zone, making it much easier to reproduce this bug. Reviewed-by: Dmitry Fomichev <dmitry.fomichev@wdc.com> Reviewed-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com> Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-08-22t/io_uring: cleanup vectored vs non-vectoredJens Axboe
No functional changes, just makes it a bit more logical. Signed-off-by: Jens Axboe <axboe@kernel.dk>