summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2019-11-06t/jobs: fixup t0002 test jobsVincent Fu
- add .fio extension to match other job files - change line endings from DOS to UNIX - use local file instead of device
2019-11-06t/stest: non-zero exit value on failureVincent Fu
To facilitate automated testing, return a non-zero exit value when a smalloc failure occurs.
2019-11-06t/sgunmap-test.py: drop six.moves dependencyVincent Fu
range was never actually used in this script.
2019-11-06t/steadystate_tests: better support automated testingVincent Fu
To better support using this script for automated testing, do the following: - Use a more portable python interpreter reference - Drop six.moves dependency - Eliminate two potential divide-by-zero runtime errors - Count and report the number of passed and failed tests - Keep the output of the test jobs for inspection in case of failure - Exit code 0 if all tests pass and non-zero otherwise
2019-11-06t/strided.py: change LFSR testsVincent Fu
LFSR init fails on occasion with small zoneranges. This patch increases the zonerange for LFSR tests to eliminate these failures. Also fix a typo in the last test job.
2019-11-06t/readonly: replace shell script with python scriptVincent Fu
The new script better supports automated testing than the previous one. Also get rid of the t/jobs/readonly-?.fio job files. Using jobs files with the --readonly parameter produces the same behavior as when --readonly precedes --rw=XXX on the command line.
2019-11-06t/iee754: add return valueVincent Fu
To facilitate automated testing, return 0 when all tests succeed and a non-zero value otherwise. Also add one more test value.
2019-11-05arch-arm: Consider armv7ve arch as wellKhem Raj
Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-11-03Merge branch 'master' of https://github.com/aphreet/fioJens Axboe
* 'master' of https://github.com/aphreet/fio: Enable io_uring engine on powerpc arch
2019-11-03Enable io_uring engine on powerpc archMikhail Malygin
Signed-off-by: Mikhail Malygin <mikhail@malygin.me>
2019-11-02engines/libaio.c: remove unused 'hipri' settingJens Axboe
Leftover from when we had aio-ring with polling, now dead. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-29Merge branch 'patch-1' of https://github.com/hannesweisbach/fioJens Axboe
* 'patch-1' of https://github.com/hannesweisbach/fio: Fix output redirection of exec_prerun/_postrun
2019-10-29Merge branch '1029_tet' of https://github.com/optimistyzy/fioJens Axboe
* '1029_tet' of https://github.com/optimistyzy/fio: backend: fix the memory leak if fio_memalign fails,
2019-10-29backend: fix the memory leak if fio_memalign fails,Ziye Yang
We should not use break, we should use return 1 and explictly fails this, then the cleanup routine will work. If not, this function may still return 0. Signed-off-by: Gang Cao <cao.gang@intel.com> Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2019-10-29Fix output redirection of exec_prerun/_postrunhannesweisbach
The previous version used the bashism "&>" to redirect stdout and stderr of the prerun/postrun commands to an output file. Since system() executes the command using "/bin/sh sh -c …". If /bin/sh is not bash, this wont work. Use the "posixly-correct" way of redirecting stdout to the output file and then redirect stderr to stdout. Signed-off-by: Hannes Weisbach <hannes.weisbach@gmail.com>
2019-10-24Merge branch 'doc_fixes' of https://github.com/sitsofe/fioJens Axboe
* 'doc_fixes' of https://github.com/sitsofe/fio: doc: delete repeated word "will" man: don't use non-breaking minuses when they're not necessary
2019-10-24doc: delete repeated word "will"Sitsofe Wheeler
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-24man: don't use non-breaking minuses when they're not necessarySitsofe Wheeler
Technically you only need to escape a dash in troff when you 1. Want it to be a typographical minus 2. Need to prevent the typesetter potentially doing a line break at it (e.g. to prevent a -option being split apart) so convert lots of "\-" to just "-" . Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-21Merge branch 'android-log-fix' of https://github.com/kdrag0n/fioJens Axboe
* 'android-log-fix' of https://github.com/kdrag0n/fio: Makefile: Link to the system logging library on Android
2019-10-20Makefile: Link to the system logging library on AndroidDanny Lin
Android replaces syslog calls with calls to the Android system logging library, liblog. Link to liblog when compiling for Android to fix the following undefined reference errors: aarch64-linux-android-ld: log.o: in function `android_polyfill_vsyslog': /usr/include/syslog.h:185: undefined reference to `__android_log_print' aarch64-linux-android-ld: /usr/include/syslog.h:182: undefined reference to `__android_log_vprint' Signed-off-by: Danny Lin <danny@kdrag0n.dev>
2019-10-15Merge branch 'fix-fsync-on-close' of https://github.com/sitsofe/fioJens Axboe
* 'fix-fsync-on-close' of https://github.com/sitsofe/fio: backend: fix final fsync behaviour
2019-10-15testing: add test script for zonemode=stridedVincent Fu
Python script to run jobs with randommap, with LFSR, and with nonrandommap. Uses null ioengine or a real file. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-15filesetup: use zonerange for map and LFSR with zonemode=stridedVincent Fu
When a random map is enabled, cover only the current zone when run with zonemode=strided. This ensures that when we reach the end of the zone we wrap around back into the current zone instead of accessing blocks outside of the zone. In addition, when an LFSR is used, constrain the blocks generated to the current zone. Previously, the LFSR random_generator would ignore the strided zonemode setting. Fixes: https://github.com/axboe/fio/issues/809 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-15io_u: skip to the next zone when zoneskip is set to zeroVincent Fu
If zoneskip is not set or set to 0 in zonemode=strided mode, all IO occurs in a single zone. If zoneskip is non-zero, the next zone begins zoneskip bytes beyond the end of the current zone. Thus, it's not possible to access contiguous zones since zoneskip bytes will separate the end of one zone and the beginning of the next zone. This patch allows fio to move to the next zone when zoneskip is explicitly set to 0, making it possible to issue IO to contiguous zones. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-14parse: improve detection of bad input stringJens Axboe
If we attempt to do number conversion, and strtoll() returns 0 AND the end pointer is the same as the starting string, then fail the conversion. Fixes: https://github.com/axboe/fio/issues/841 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-13Merge branch 'fix-corrupt-hist-log' of https://github.com/sitsofe/fioJens Axboe
* 'fix-corrupt-hist-log' of https://github.com/sitsofe/fio: stat: fix corruption in histogram logs
2019-10-13stat: fix corruption in histogram logsSitsofe Wheeler
Commit 6cc0e5aa9eddd2487dfa9ac80be1264151058409 ("Fix overflow of counters incremented on each I/O operation") changed various structures from [unsigned] int -> [u]int64_t but the size used when memcpy-ing histogram data was still the older (smaller) size. This led to un-copied values and usage of uninitialised data. An example job that demonstrates the issue is: $ valgrind ./fio --size=256k --thread=1 --rate_iops=5 --direct=1 \ --number_ios=15 --name=hist_test --filename=fio.test \ --log_hist_msec=1000 --write_hist_log=fio.test --log_hist_coarseness=6 ==1743== Memcheck, a memory error detector [...] fio-3.16-8-gf951 Starting 1 thread ==1743== Thread 3: IOPS: [R(1)][23.1%][r=20KiB/s][r=5 IOPS][eta 00m:10s] ==1743== Use of uninitialised value of size 8 ==1743== at 0x609D86B: _itoa_word (_itoa.c:179) ==1743== by 0x60A0F0D: vfprintf (vfprintf.c:1642) ==1743== by 0x60A9E53: fprintf (fprintf.c:32) ==1743== by 0x47FA6C: flush_hist_samples (iolog.c:869) ==1743== by 0x47F637: flush_log (iolog.c:1212) ==1743== by 0x4813C5: finish_log (iolog.c:1238) ==1743== by 0x4812CD: __write_log (iolog.c:1571) ==1743== by 0x481202: write_clat_hist_log (iolog.c:1625) ==1743== by 0x480045: td_writeout_logs (iolog.c:1721) ==1743== by 0x485C7A: thread_main (backend.c:1889) ==1743== by 0x5C296DA: start_thread (pthread_create.c:463) ==1743== by 0x616688E: clone (clone.S:95) This commit fixes the issue by using the new size in the memcpy. Fixes: https://github.com/axboe/fio/issues/827 Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-08Merge branch 'bumpflocks' of https://github.com/earlephilhower/fioJens Axboe
* 'bumpflocks' of https://github.com/earlephilhower/fio: Increase MAX_FILELOCKS for highly parallel IO test
2019-10-08Increase MAX_FILELOCKS for highly parallel IO testEarle F. Philhower, III
When running hundreds of threads and collecting CLAT, SLAT, IOS on a per-thread basis, FIO can need a lot of file locks to complete. When it can't get them, it gives the cryptic error message: > fio: filelock.c:182: __fio_lock_file: Assertion `!trylock' failed. Bump the maximum from 128 to 1024 to give headroom for high core count, high thread count testing.
2019-10-08backend: fix final fsync behaviourSitsofe Wheeler
Previously, fsync_on_close's "final" fsync was done after fio's accounting had finished thus causing the bandwidth reported to be too high (due to the reported time being too low). An example job that demonstrates the issue (on machines with a few gigabytes of RAM and non-fast disks) is the following: ./fio --gtod_reduce=1 --filename=fio.tmp --size=5G --bs=2M --rw=write \ --stonewall --name=end_fsync --end_fsync=1 --name=fsync_on_close \ --fsync_on_close=1 end_fsync: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fsync_on_close: (g=1): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 [...] Run status group 0 (all jobs): WRITE: bw=381MiB/s (400MB/s), 381MiB/s-381MiB/s (400MB/s-400MB/s), io=5120MiB (5369MB), run=13424-13424msec Run status group 1 (all jobs): WRITE: bw=1726MiB/s (1810MB/s), 1726MiB/s-1726MiB/s (1810MB/s-1810MB/s), io=5120MiB (5369MB), run=2966-2966msec This patch fixes the issue by doing an fsync for fsync_on_close at the same point as it does for end_fsync. Technically fsync_on_close will go and do another fsync after this point too but this should be harmless as there is no new write data. This patch also fixes the assert seen with the following job ./fio --size=8k --io_limit=32k --filename=fio.tmp --rw=write \ --end_fsync=1 --name=extended_end_fsync [...] fio: filesetup.c:1682: void get_file(struct fio_file *): Assertion `fio_file_open(f)' failed. by preventing auto-close on last I/O when it is being submitted as a final fsync (due to end_fsync, fsync_on_close). Finally it also fixes the strange IO depths seen with when using end_fsync: ./fio --size=4k --filename=fio.tmp --rw=write --end_fsync=1 \ --name=end_fsync_test end_fsync_test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 [...] IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% by no longer including such fsyncs in accounting. Fixes: https://github.com/axboe/fio/issues/831 Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-07Merge branch 'replay-blktrace-fixes' of https://github.com/shimrot/fioJens Axboe
* 'replay-blktrace-fixes' of https://github.com/shimrot/fio: Fix assert error on blktrace replay containing trims
2019-10-06Merge branch 'error-on-implicit-decl' of https://github.com/sitsofe/fioJens Axboe
* 'error-on-implicit-decl' of https://github.com/sitsofe/fio: configure: stop enabling fdatasync on OSX
2019-10-06configure: stop enabling fdatasync on OSXSitsofe Wheeler
Change configure compile probes to fail on implicit declarations. This (correctly) stops us from enabling fdatasync on OSX which was problematic because we were actually calling a syscall stub with arguments different to what was declared in its prototype (see https://github.com/gbrault/picoc/issues/145#issuecomment-89734655 and https://gitlab.freedesktop.org/xdg/shared-mime-info/issues/7 ). Fixes: https://github.com/axboe/fio/issues/834 ("Drop macOS support for fdatasync") Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-02Windows: Update URLs to https, and remove mention of WiX versionRebecca Cran
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-01Update the email and web address for Windows binaries.Rebecca Cran
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-27configure: add --enable-libaio-uring parameterJens Axboe
If set, this attempts to link with liburing instead of libaio, and use the libaio wrapper there. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-26t/zbd: Avoid magic number of test case countShin'ichiro Kawasaki
In the test script t/zbd/test-zbd-support, the number of test cases is defined as a raw constant. Every time a new test case is added, need to update the constant but this chore tends to be overlooked. When test case 47 was added recently, the number was not incremented from 46. Then the test script does not run the test case 47. To avoid the constant increment chore, automate the test case number counting. List bash functions which implement test cases and convert the list to test case numbers. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-25filesetup: Extend file size for 'null' and 'filecreate' ioenginesShin'ichiro Kawasaki
It was reported the fio command with following options fails because of file access beyond the file size specified with size option. ./fio --name=test --rw=randread --runtime=10s --offset=90% --time_based --ioengine=null --size=1T --norandommap --randrepeat=0 One of the options specifies null ioengine which has FIO_DISKLESSIO and FIO_FAKEIO flags. When FIO_DISKLESSIO flag is set, current fio code does not extend the file size even when io_size + file_offset is larger than real_file_size. With this, random offsets are generated within io_size + file_offset range and fail comparison with real_file_size. This failure does not happen with other engines such as libaio since the target file is extended to ensure real_file_size is larger than io_size + file_offset. To avoid the failure, extend file size for ioengines with FIO_DISKLESSIO and FIO_FAKEIO. This changes behavior of two ioengines only: null ioengine and filecreate engine. It does not change behavior of all other ioengines including 12 ioengines with FIO_DISKLESSIO flag, such as nbd, glusterfs, pmemblk, etc. Signed-off-by: Masato Suzuki <masato.suzuki@wdc.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-25Revert "filesetup: honor the offset option"Shin'ichiro Kawasaki
This reverts commit ad46c5715ffc5d4e70d2e941225b1f504f23e409. The commit tried to fix a failure observed with --offset=90%, --size=1T and --ioengine=null options. However, it affected fio behavior not only for null ioengine but also for other all ioengines. It changed io_size, then I/O read or written bytes are not same as that specified with command line 'size' option. This caused t/zbd/test-zbd-support test script failures for zoned block devices: test cases #17, #28 and #38 failed. To prepare for other fix with limited ioengine impacts, revert the fix which affects all ioengines. Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-24verify: check that the block size is big enoughJens Axboe
Fixes: https://github.com/axboe/fio/issues/822 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-23Merge branch 'master' of https://github.com/anatol/fioJens Axboe
* 'master' of https://github.com/anatol/fio: Fix compilation error with gfio
2019-09-23Fix compilation error with gfioAnatol Pomozov
Commit 36833fb04 replaced many usages of strncpy with snprintf. But there is one place where new arguments were added but the function name has not been changed. It leads to the following compilation error: gclient.c:333:2: error: too many arguments to function ‘strncpy’ 333 | strncpy(message, sizeof(message), "%s", status_message); | ^~~~~~~ In file included from /usr/include/features.h:450, from /usr/include/bits/libc-header-start.h:33, from /usr/include/stdlib.h:25, from gclient.c:1: /usr/include/bits/string_fortified.h:103:1: note: declared here 103 | __NTH (strncpy (char *__restrict __dest, const char *__restrict __src, | ^~~~~ Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
2019-09-19Fio 3.16fio-3.16Jens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-19engines/io_uring: remove debug printfJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-09-13Merge branch 'fio_reset_sqe' of https://github.com/anarazel/fioJens Axboe
* 'fio_reset_sqe' of https://github.com/anarazel/fio: engines/io_uring: Fully clear out previous SQE contents.
2019-09-13engines/io_uring: Fully clear out previous SQE contents.Andres Freund
Without this change SQEs can contain data set in previous submissions. E.g. a WRITE following an fdatasync would have still have IORING_FSYNC_DATASYNC set in sync_flags, which shares storage with the WRITE's rw_flags. Which was not reset, causing all writes to be synchronous. Similarly, an fsync following a READ/WRITE would not reset off/addr/len, which causes errors, because the kernel's io_prep_fsync returns an error if e.g. addr is not 0. While this could also be fixed by resetting only the unused fields in the respective branches, it seems less failure prone to start with a zeroed out sqe.
2019-09-12Merge branch 'ioring_add_sync_file_range' of https://github.com/anarazel/fioJens Axboe
* 'ioring_add_sync_file_range' of https://github.com/anarazel/fio: engines/io_uring: Add support for sync_file_range.
2019-09-12Merge branch 'fix_iouring_eintr' of https://github.com/anarazel/fioJens Axboe
* 'fix_iouring_eintr' of https://github.com/anarazel/fio: engines/io_uring: Handle EINTR.
2019-09-12engines/io_uring: Add support for sync_file_range.Andres Freund
Previously sync_file_range() requests were just dropped to the floor. Signed-off-by: Andres Freund <andres@anarazel.de>
2019-09-12engines/io_uring: Handle EINTR.Andres Freund
Several paths in io_uring_enter can trigger EINTR, but it was not handled, leading to fio failing with spurious error messages. An easy way to trigger EINTR is to just strace a running fio using the io_uring engine and detach again. Signed-off-by: Andres Freund <andres@anarazel.de>