summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
31 hoursMerge branch 'github-issue-947' of https://github.com/vincentkfu/fioHEADmasterJens Axboe
* 'github-issue-947' of https://github.com/vincentkfu/fio: stat: eliminate extra log samples
36 hoursstat: eliminate extra log samplesVincent Fu
b2a432bfbb6d inadvertently added extra log samples. $ ./fio-canonical/fio --name=test --time_based --runtime=10s --write_lat_log=fio-07-b2a432 --log_avg_msec=1000 --size=1G --rw=rw test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.17-93-gb2a4 Starting 1 process ... $ cat fio-07-b2a432_clat.1.log 1000, 5851, 0, 0, 0 1000, 2551, 1, 0, 0 1000, 5028, 1, 0, 0 2000, 4175, 0, 0, 0 2000, 3214, 1, 0, 0 2000, 60619, 0, 0, 0 ... There should only be two lines at each timestamp (one for reads, one for writes), but the first two timestamps have three lines each. The cause is an inadvertent change in stat.c:add_log_sample() of __add_stat_to_log to _add_stat_to_log. Reverting to the two-underscore version resolves this issue. Fixes: https://github.com/axboe/fio/issues/947 Fixes: b2a432bfbb6d ("Per-command priority: Priority logging and libaio/io_uring cmdprio_percentage") Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
6 daysMerge branch 'jsonplus2csv' of https://github.com/vincentkfu/fioJens Axboe
* 'jsonplus2csv' of https://github.com/vincentkfu/fio: .travis.yml: remove pip line from xcode11.2 config t/jsonplus2csv_test.py: test script for tools/fio_jsonplus_clat2csv tools/fio_jsonplus2csv: accommodate multiple lat measurements
7 days.travis.yml: remove pip line from xcode11.2 configVincent Fu
travis-ci changed the xcode11.2 image and 'pip' is no longer available. So only run 'pip install scipy' for the default xcode image. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
7 dayst/jsonplus2csv_test.py: test script for tools/fio_jsonplus_clat2csvVincent Fu
Add a script to run a basic jsonplus to CSV conversion and then validate the conversion. Also integrate this test script with t/run-fio-tests.py and install the python package 'six' to support fio_jsonplus_clat2csv in the AppVeyor build/testing environment. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
7 daystools/fio_jsonplus2csv: accommodate multiple lat measurementsVincent Fu
Add some intelligence to this script so that it works for any of submission, completion, and total latency whenever they are present. The CSV data format is changed to accommodate this. While we're here also do the following: add a way to generate optional debug output add validate option that compares generated CSV data with the original json+ data fix style issues identified by pylint3 update documentation Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-03-17zbd: add test for stressing zone lockingNaohiro Aota
Add a test to stress zone locking mechanism by having a large number of threads with a small number of max_open_zones. Run 30 seconds time-based fio under the timeout command. After 45 seconds, "timeout" kill -KILL the fio process. If a zone lock deadlocks, fio is killed by the timeout command, and this test fails. If not, fio runs to the end and this test success. Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com> Tested-by: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-17io_u: ensure io_u_quiesce() to process all the IOsNaohiro Aota
Currently, when IO have an error io_u_quiesce() stops processing in-flight IOs there and leaves other IOs non-completed. This is not a desired behavior for io_u_quiesce(). Fix it by continuing even on error. Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com> Tested-by: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-17backend: always clean up pending aiosNaohiro Aota
cleanup_pending_aios() is called when a thread exits with error, so all the call site of this function is under "if (td->error)". However, commit d28174f0189c ("workqueue: ensure we see deferred error for IOs"), for some reason, added "if (td->error) return" at the head of this function, making this function practically void. Revert this part to ensure cleaning up pending aios. Besides, cleanup_pending_aios() should not return even when io_u_queued_complete() failed. Because, it keeps in-flight aios left. Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com> Tested-by: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-17zbd: use zone_lock to lock a zoneNaohiro Aota
commit 6f0c608564c3 ("zbd: Avoid async I/O multi-job workload deadlock") introduced io_u_quiesce() when it failed to lock a zone to avoid deadlock. This situation can happen on the other locking place like zbd_convert_to_open_zone(). Thus, introduce common helper "zone_lock" to lock a zone. Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com> Tested-by: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-17zbd: reset one zone at a timeNaohiro Aota
zbd_rest_zones() currently scans over device zones and try to reset as much zones as possible at a time. However, this routine takes all the lock on the range and causes a lot of lock contentions with other threads. This commit change the behavior to hold the lock and examine one zone at a time. While it will increase the number of ioctl() call when it need to reset contiguous, the overhead of increased number of ioctl()s are anyway amortized by device side's reset performance. Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com> Tested-by: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-17zbd: avoid initializing swd when unnecessaryNaohiro Aota
When enable_check_swd == false, there is no use to initialize swd. Just disable it in this case. Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com> Tested-by: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-17gauss.c: correct the stddev initializtionFeng Tang
Signed-off-by: Feng Tang <feng.tang@intel.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-16t/io_uring: don't use *rand48_r() variantsJens Axboe
Not available on all platforms. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-15Merge branch 'patch-1' of https://github.com/neheb/fioJens Axboe
* 'patch-1' of https://github.com/neheb/fio: configure: fix vasprintf check under musl
2020-03-15configure: fix vasprintf check under muslRosen Penev
It errors when passing NULL or 0. Passing an empty va_list works.
2020-03-12Fio 3.19fio-3.19Jens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-12engines/io_uring: delete fio_option_is_set() calls when submitting sqesXiaoguang Wang
The fio_option_is_set() call in fio_ioring_prep() is time-consuming, which will reduce sqe's submit rate drastically. To fix this issue, add two new variables to record whether ioprio_class or ioprio_set is set. I use a simple fio job to evaluate the performance: fio -name=fiotest -filename=/dev/nvme0n1 -iodepth=4 -thread -rw=read -ioengine=io_uring -hipri=0 -sqthread_poll=0 -direct=1 -bs=4k -size=10G -numjobs=1 -time_based -runtime=120 Before this patch: READ: bw=969MiB/s (1016MB/s), 969MiB/s-969MiB/s (1016MB/s-1016MB/s), io=114GiB (122GB), run=120001-120001msec With this patch: READ: bw=1259MiB/s (1320MB/s), 1259MiB/s-1259MiB/s (1320MB/s-1320MB/s), io=148GiB (158GB), run=120001-120001msec Signed-off-by: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-03-03Merge branch 'patch-1' of https://github.com/felixonmars/fioJens Axboe
* 'patch-1' of https://github.com/felixonmars/fio: Correct multiple typos in engines/libhdfs.c
2020-03-03Correct multiple typos in engines/libhdfs.cFelix Yan
2020-03-02Merge branch 'filestat3' of https://github.com/kusumi/fioJens Axboe
* 'filestat3' of https://github.com/kusumi/fio: engines/filestat: add statx(2) syscall support
2020-03-03engines/filestat: add statx(2) syscall supportTomohiro Kusumi
This commit 1) tests existence of statx(2) and libc support on ./confiugre. 2) adds oslib/statx.c and implements statx(2) using above result. 3) adds statx(2) support in filestat ioengine. Confirmed compilation on Fedora31, FreeBSD, and NetBSD. Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
2020-03-01Merge branch 'genfio-bash' of https://github.com/sitsofe/fioJens Axboe
* 'genfio-bash' of https://github.com/sitsofe/fio: genfio: use /bin/bash hashbang
2020-03-01Merge branch 'fix-win-raw' of https://github.com/sitsofe/fioJens Axboe
* 'fix-win-raw' of https://github.com/sitsofe/fio: filesetup: fix win raw disk access and improve dir creation failure msg
2020-03-01Merge branch 'clean1' of https://github.com/kusumi/fioJens Axboe
* 'clean1' of https://github.com/kusumi/fio: Makefile: don't fail to remove conditionally compiled binaries on clean
2020-03-01filesetup: fix win raw disk access and improve dir creation failure msgSitsofe Wheeler
The commit df18600fd06258b96ae6f6b530ecdff541c2a82d ("filesetup: fix directory creation issues") broke Windows raw/physical disk access because Windows doesn't consider a path that only consists of a namespace (such as the device namespace "\\.\" - see https://docs.microsoft.com/en-gb/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#namespaces for information on Windows' namespaces) to exist as a directory(!). Workaround the issue for raw devices by explicitly considering the device namespace subcomponent to always be valid. Further, intermediate path components in UNC paths or paths starting with a namespace might also not "exist" so introduce backwards scanning for the longest pre-existing directory to sidestep this. The function doing this is made available for non-windows platforms so a similar code path is used everywhere. Tests done: Windows: > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename 'fio.tmp' > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\\?\C\:\Windows\Temp\fio\fio.tmp' > Clear-Disk 1 -RemoveData -Confirm:$false # Destroys partition data! > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\\.\PhysicalDrive1' > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\fio.tmp' > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\\LOCALHOST\Users\User\fio\fio.tmp' macOS: $ rm -rf /tmp/fio $ ./fio --name=dtest --size=16k --filename /tmp/fio/fio.tmp \ --rw=write Finally, change the directory creation error message to give a human error message rather than just an errno. Fixes: https://github.com/axboe/fio/issues/916 Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2020-03-01genfio: use /bin/bash hashbangSitsofe Wheeler
Not all distros have done usrmerge - Debian https://salsa.debian.org/debian/fio/-/blob/5a608e04b947aed0d3b49d7ca9a85ed9afe9ec56/debian/patches/genfio-interpreter and SUSE (https://build.opensuse.org/request/show/541207 ) have been carrying a patch changing the interpreter for genfio back to /bin/bash and that's before you get to other OSes such as macOS... Since /bin/bash works everywhere let's do this upstream too. Fixes: https://github.com/axboe/fio/pull/883 Inspired-by: Changcheng Liu <changcheng.liu@aliyun.com> Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2020-02-26io_uring: we should not need two write barriers for SQ updatesJens Axboe
Just after updating the tail should be fine. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-26fio: remove duplicate global definition of tsc_reliableEric Sandeen
Newer gcc v10 complains about globals being defined twice due to a new default -fno-common flag. This causes build failures: LINK t/fio-dedupe /usr/bin/ld: t/arch.o:(.bss+0x4): multiple definition of `tsc_reliable'; gettime.o:(.bss+0x4): first defined here collect2: error: ld returned 1 exit status make: *** [Makefile:499: t/fio-dedupe] Error 1 Each of the tests which call arch_init() to initialize tsc_reliable: File Function Line 1 t/dedupe.c main 536 arch_init(argv); 2 t/lfsr-test.c main 37 arch_init(argv); 3 t/stest.c main 86 arch_init(argv); already link with gettime.o which defines tsc_reliable, so there is no need to provide it from t/arch.c Signed-off-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-24Merge branch 'div-by-zero' of https://github.com/vincentkfu/fioJens Axboe
* 'div-by-zero' of https://github.com/vincentkfu/fio: t/latency_percentiles: avoid division by zero
2020-02-24t/latency_percentiles: avoid division by zeroVincent Fu
For the smallest latency durations, the actual and approximate values should be exactly the same. Use this property to avoid a division by zero when the actual latency value is zero. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-14Makefile: don't fail to remove conditionally compiled binaries on cleanTomohiro Kusumi
`rm -f $(T_PROGS) $(T_TEST_PROGS)` in `clean` target is intended to remove all binaries under t directory, but it actually fails to remove `t/{fio-btrace2fio,io_uring,read-to-pipe-async}` files for not evaluating platform specific variables on `make clean`. # uname -r 5.4.17-200.fc31.x86_64 # make clean # file t/* | grep ELF | awk '{print $1,$2,$3,$4,$5}' t/fio-btrace2fio: ELF 64-bit LSB executable, t/io_uring: ELF 64-bit LSB executable, t/read-to-pipe-async: ELF 64-bit LSB executable, The easiest way to fix this is to hard-code these conditionally compiled binaries. Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
2020-02-06Merge branch 'master' of https://github.com/vincentkfu/fioJens Axboe
* 'master' of https://github.com/vincentkfu/fio: t/run-fio-tests: fix style issues gfio: add high/low priority latency results .gitignore: add some test programs stat: summary statistics for both high/low priority latencies
2020-02-06Merge branch 'master' of https://github.com/bvanassche/fioJens Axboe
* 'master' of https://github.com/bvanassche/fio: Make the JSON code easier to analyze
2020-02-06Unify architecture io_uring syscall numbersJens Axboe
Only Alpha is the odd one out, apart from that all architectures use the same system call number. So pull the code out arch headers, and into the generic arch header. Fixes: https://github.com/axboe/fio/issues/923 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-05Make the JSON code easier to analyzeBart Van Assche
Coverity reports the following false positive for the JSON code in stat.c: CID 284826 (#1 of 1): Resource leak (RESOURCE_LEAK) 17. leaked_storage: Variable percentile_object going out of scope leaks the storage it points to. Suppress this false positive by making the JSON code easier to analyze. The changes in this patch are as follows: - Change the macros in json.h into inline functions such that the compiler performs argument type verification at compile time. - Pass a struct json_value pointer to json_*_add_value_type() instead of using varargs. Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2020-02-05t/run-fio-tests: fix style issuesVincent Fu
Many style issues were identified by pylint3. Fix most of them. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-05gfio: add high/low priority latency resultsVincent Fu
For the cmdprio_percentage option, add high/low priority summary statistics and latency percentiles. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-05.gitignore: add some test programsVincent Fu
t/io_uring, t/memlock, and t/read-to-pipe-async are now routinely built. Add them to .gitignore to avoid cluttering up the output of 'git status'. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-05stat: summary statistics for both high/low priority latenciesVincent Fu
For the sake of completeness, include summary statistics for both high and low priority latencies in the normal output when cmdprio_percentage is used. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-05Fio 3.18fio-3.18Jens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-04Merge branch 'latency-rebase-again' of https://github.com/vincentkfu/fioJens Axboe
* 'latency-rebase-again' of https://github.com/vincentkfu/fio: t/run-fio-tests: increase time allowed for tests to pass t/run-fio-tests: add latency_percentiles.py t/latency_percentiles: test latency percentile reporting stat: fix high/low prio unified rw bug fio: better distinguish between high and low priority stat: make priority summary statistics consistent with percentiles docs: updates for slat, clat, lat percentile reporting gfio: display slat, clat, and lat percentiles fio: report percentiles for slat, clat, lat fio: groundwork for adding slat, lat percentiles
2020-02-04Merge branch 'filestat2' of https://github.com/kusumi/fioJens Axboe
* 'filestat2' of https://github.com/kusumi/fio: engines/filestat: change "lstat" bool option to "stat_type" str option
2020-02-05engines/filestat: change "lstat" bool option to "stat_type" str optionTomohiro Kusumi
Per suggestion from Jens, change a bool option to str option to better support stat(2) variants (at this point before 3.18). https://github.com/axboe/fio/pull/912#issuecomment-577814885 Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
2020-02-04t/run-fio-tests: increase time allowed for tests to passVincent Fu
With a 300s timeout I occasionally see tests fail when they usually pass. Bump the timeout to 600s to give tests more of an opportunity to finish successfully. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04t/run-fio-tests: add latency_percentiles.pyVincent Fu
Add the latency percentiles test script to the general test script. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04t/latency_percentiles: test latency percentile reportingVincent Fu
This script runs fio jobs exercising the slat_percentiles, clat_percentiles, and lat_percentiles options under various conditions. It carries out several tests comparing write_lat_log data with percentiles reported by fio. It also includes limited tests of high/low priority latency reporting with the cmdprio_percentage option. Only Linux supports the cmdprio_percentage option. Skips these tests on other platforms. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04stat: fix high/low prio unified rw bugVincent Fu
With unified_rw_reporting enabled, high/low priority summary statistics were not correctly reported. This patch fixes the problem. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04fio: better distinguish between high and low priorityVincent Fu
Change the variable names for the low priority latency variables to explicity distinguish them from the high priority latency variables. Also rename the JSON objects to do the same. This reduces the mental effort required to read the code and output. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04stat: make priority summary statistics consistent with percentilesVincent Fu
When lat_percentiles=1, reported high/low priority percentiles will describe total latency values. Make the summary statistics (min, max, mean, stddev) also describe total latency. Otherwise there will be a mismatch in the JSON output with completion latency summary statistics combined with total latency percentile values. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
generated by cgit v1.2.1-26-g8fc0 (git 2.23.0) at 2020-04-01 23:44:57 +0000