summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2019-12-05Revert "filesetup: don't call create_work_dirs() for ioengine with ↵issue-870Jens Axboe
FIO_DISKLESSIO" This reverts commit 95af8dd57a7a9a564f99492bb1e04bb6d13b95b1.
2019-11-27Merge branch 'engine-rados-fix-first' of https://github.com/aclamk/fioJens Axboe
* 'engine-rados-fix-first' of https://github.com/aclamk/fio: engines/rados: fix error with getting last instead of first element from list
2019-11-27engines/rados: fix error with getting last instead of first element from listAdam Kupczyk
A thinko has popped in implementation. Consumer function `getevent()` did pick last (newest) instead of first (oldest). Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-11-26Merge branch 'fallocate-truncate' of https://github.com/tripped/fioJens Axboe
* 'fallocate-truncate' of https://github.com/tripped/fio: filesetup: add fallocate=truncate option.
2019-11-25Merge branch 'rados-now-use-completion-callbacks' of ↵Jens Axboe
https://github.com/aclamk/fio * 'rados-now-use-completion-callbacks' of https://github.com/aclamk/fio: engines/rados: changed polling to completion callbacks methodology
2019-11-25Merge branch 'const1' of https://github.com/kusumi/fioJens Axboe
* 'const1' of https://github.com/kusumi/fio: parse: Silence discard-const warning on OpenBSD
2019-11-25engines/rados: changed polling to completion callbacks methodologyAdam Kupczyk
Previously, getevent() function polled operations that have been submitted to check if some of them have already finished. Now, each rados operation invokes completion callback. This callback adds to list of completed operations. In this version getevent() only peeks if completed list has some operations. There are two benefits: 1) small one - this works faster then previous version 2) big one - when there is a huge amount of operations in fly, getevent() might be more overloaded then producer. In this scenario some old operations are finished, but not picked up and properly noticed. This caused absurdely long execution times, when actually everything was working properly. Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-11-26parse: Silence discard-const warning on OpenBSDTomohiro Kusumi
This might be overkill just to silence the warnings, but having a common function for const and non-const version makes it discard const pointer at some point. -- CC parse.o parse.c: In function 'find_option_c': parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type parse.c: In function 'get_option': parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type parse.c: In function 'parse_cmd_option': parse.c:1051: warning: passing argument 1 of 'find_option' discards qualifiers from pointer target type Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
2019-11-14t/run-fio-tests: improve error handlingVincent Fu
More gracefully handle errors when running a test. Also change the EXAMPLE to refer to fio's canonical git repository. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-11-14io_u: move to next zone even if zoneskip is unsetVincent Fu
This patch makes fio's behavior under zonemode=strided conform to the documentation: I/O happens in a single zone until zonesize bytes have been transferred. After that number of bytes has been transferred processing of the next zone starts. With zonemode=strided, before commit 35f561eb, fio would only move to the next zone when zoneskip > 0 and zonesize bytes were written. There would always be zoneskip bytes between the end of one zone and the beginning of the next zone. If zoneskip was not set or set to 0, all IO would happen in the first zone. Commit 35f561eb changed this so that fio would move to the next zone upon writing zonesize bytes if zoneskip was explicitly set to a value >= 0. This option made it possible for zones to be contiguous. The documentation was not updated to reflect the new behavior. I originally intended to submit a patch to update fio's documentation, but upon further reflection it seems better to change fio's behavior and have a clean user interface than to change the documentation to note that zoneskip must be explciitly set in order for fio to move to the next zone. This patch also updates t/strided.py to reflect the new behavior. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-11-14filesetup: improve LFSR init failure error messageVincent Fu
Especially with small sample spaces, the LFSR random generator occasionally fails to initialize successfully. When this occurs, the error message refers to problems allocating a random map. Change the error message to explicitly mention the LFSR failure. OLD BEHAVIOR $ ./fio --name=test --ioengine=null --size=4k --random_generator=lfsr --rw=randread --randrepeat=0 test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1 fio-3.16-13-g998b-dirty Starting 1 process fio: failed allocating random map. If running a large number of jobs, try the 'norandommap' option or set 'softrandommap'. Or give a larger --alloc-size to fio. NEW BEHAVIOR $ ./fio --name=test --ioengine=null --size=4k --random_generator=lfsr --rw=randread --randrepeat=0 test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1 fio-3.16-37-g65ee Starting 1 process fio: failed initializing LFSR Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-11-12filesetup: add fallocate=truncate option.Trip Volpe
Fixes #833. Provides the ability to initially "layout" the file by ftruncating it to the desired size before performing IO. This is mainly useful on Windows, which serializes all writes that extend the size of a file. Using this option with a suitable iodepth allows fio to emulate the behavior of Windows Explorer file copy, which always truncates to the expected size before issuing writes for performance reasons.
2019-11-06Merge branch 'testing' of https://github.com/vincentkfu/fioJens Axboe
* 'testing' of https://github.com/vincentkfu/fio: t/run-fio-tests: a script to automate running fio tests t/jobs: fix t0011 syntax error t/jobs: clean up t0009 and use only 4 CPUs t/jobs: drop time_based in t0007 t/jobs: use current directory for test file for t0003 and t0004 t/jobs: fixup t0002 test jobs t/stest: non-zero exit value on failure t/sgunmap-test.py: drop six.moves dependency t/steadystate_tests: better support automated testing t/strided.py: change LFSR tests t/readonly: replace shell script with python script t/iee754: add return value
2019-11-06t/run-fio-tests: a script to automate running fio testsVincent Fu
fio actually has a substantial test infrastructure but lacks a convenient way to run existing tests in an automated fashion. This script provides the means to run almost all of fio's existing tests with a single command. A full run finishes in less than 10 minutes. This script can be used to catch bugs before patches are submitted. It can also be used to test new fio builds as patches are committed to the repository.
2019-11-06t/jobs: fix t0011 syntax errorVincent Fu
2019-11-06t/jobs: clean up t0009 and use only 4 CPUsVincent Fu
There are two runtime= lines in t0009. Comment out the one that is ignored. Also reduce the number of CPUs required from 6 to 4 so that this can be run on more systems. The failure still occurs on systems with four CPUs.
2019-11-06t/jobs: drop time_based in t0007Vincent Fu
This is an extraneous option that produces a warning message which complicates using this job file for automated testing.
2019-11-06t/jobs: use current directory for test file for t0003 and t0004Vincent Fu
This makes it easier to investigate problems when these job files are used for automated testing.
2019-11-06t/jobs: fixup t0002 test jobsVincent Fu
- add .fio extension to match other job files - change line endings from DOS to UNIX - use local file instead of device
2019-11-06t/stest: non-zero exit value on failureVincent Fu
To facilitate automated testing, return a non-zero exit value when a smalloc failure occurs.
2019-11-06t/sgunmap-test.py: drop six.moves dependencyVincent Fu
range was never actually used in this script.
2019-11-06t/steadystate_tests: better support automated testingVincent Fu
To better support using this script for automated testing, do the following: - Use a more portable python interpreter reference - Drop six.moves dependency - Eliminate two potential divide-by-zero runtime errors - Count and report the number of passed and failed tests - Keep the output of the test jobs for inspection in case of failure - Exit code 0 if all tests pass and non-zero otherwise
2019-11-06t/strided.py: change LFSR testsVincent Fu
LFSR init fails on occasion with small zoneranges. This patch increases the zonerange for LFSR tests to eliminate these failures. Also fix a typo in the last test job.
2019-11-06t/readonly: replace shell script with python scriptVincent Fu
The new script better supports automated testing than the previous one. Also get rid of the t/jobs/readonly-?.fio job files. Using jobs files with the --readonly parameter produces the same behavior as when --readonly precedes --rw=XXX on the command line.
2019-11-06t/iee754: add return valueVincent Fu
To facilitate automated testing, return 0 when all tests succeed and a non-zero value otherwise. Also add one more test value.
2019-11-05arch-arm: Consider armv7ve arch as wellKhem Raj
Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-11-03Merge branch 'master' of https://github.com/aphreet/fioJens Axboe
* 'master' of https://github.com/aphreet/fio: Enable io_uring engine on powerpc arch
2019-11-03Enable io_uring engine on powerpc archMikhail Malygin
Signed-off-by: Mikhail Malygin <mikhail@malygin.me>
2019-11-02engines/libaio.c: remove unused 'hipri' settingJens Axboe
Leftover from when we had aio-ring with polling, now dead. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-29Merge branch 'patch-1' of https://github.com/hannesweisbach/fioJens Axboe
* 'patch-1' of https://github.com/hannesweisbach/fio: Fix output redirection of exec_prerun/_postrun
2019-10-29Merge branch '1029_tet' of https://github.com/optimistyzy/fioJens Axboe
* '1029_tet' of https://github.com/optimistyzy/fio: backend: fix the memory leak if fio_memalign fails,
2019-10-29backend: fix the memory leak if fio_memalign fails,Ziye Yang
We should not use break, we should use return 1 and explictly fails this, then the cleanup routine will work. If not, this function may still return 0. Signed-off-by: Gang Cao <cao.gang@intel.com> Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2019-10-29Fix output redirection of exec_prerun/_postrunhannesweisbach
The previous version used the bashism "&>" to redirect stdout and stderr of the prerun/postrun commands to an output file. Since system() executes the command using "/bin/sh sh -c …". If /bin/sh is not bash, this wont work. Use the "posixly-correct" way of redirecting stdout to the output file and then redirect stderr to stdout. Signed-off-by: Hannes Weisbach <hannes.weisbach@gmail.com>
2019-10-24Merge branch 'doc_fixes' of https://github.com/sitsofe/fioJens Axboe
* 'doc_fixes' of https://github.com/sitsofe/fio: doc: delete repeated word "will" man: don't use non-breaking minuses when they're not necessary
2019-10-24doc: delete repeated word "will"Sitsofe Wheeler
Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-24man: don't use non-breaking minuses when they're not necessarySitsofe Wheeler
Technically you only need to escape a dash in troff when you 1. Want it to be a typographical minus 2. Need to prevent the typesetter potentially doing a line break at it (e.g. to prevent a -option being split apart) so convert lots of "\-" to just "-" . Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-21Merge branch 'android-log-fix' of https://github.com/kdrag0n/fioJens Axboe
* 'android-log-fix' of https://github.com/kdrag0n/fio: Makefile: Link to the system logging library on Android
2019-10-20Makefile: Link to the system logging library on AndroidDanny Lin
Android replaces syslog calls with calls to the Android system logging library, liblog. Link to liblog when compiling for Android to fix the following undefined reference errors: aarch64-linux-android-ld: log.o: in function `android_polyfill_vsyslog': /usr/include/syslog.h:185: undefined reference to `__android_log_print' aarch64-linux-android-ld: /usr/include/syslog.h:182: undefined reference to `__android_log_vprint' Signed-off-by: Danny Lin <danny@kdrag0n.dev>
2019-10-15Merge branch 'fix-fsync-on-close' of https://github.com/sitsofe/fioJens Axboe
* 'fix-fsync-on-close' of https://github.com/sitsofe/fio: backend: fix final fsync behaviour
2019-10-15testing: add test script for zonemode=stridedVincent Fu
Python script to run jobs with randommap, with LFSR, and with nonrandommap. Uses null ioengine or a real file. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-15filesetup: use zonerange for map and LFSR with zonemode=stridedVincent Fu
When a random map is enabled, cover only the current zone when run with zonemode=strided. This ensures that when we reach the end of the zone we wrap around back into the current zone instead of accessing blocks outside of the zone. In addition, when an LFSR is used, constrain the blocks generated to the current zone. Previously, the LFSR random_generator would ignore the strided zonemode setting. Fixes: https://github.com/axboe/fio/issues/809 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-15io_u: skip to the next zone when zoneskip is set to zeroVincent Fu
If zoneskip is not set or set to 0 in zonemode=strided mode, all IO occurs in a single zone. If zoneskip is non-zero, the next zone begins zoneskip bytes beyond the end of the current zone. Thus, it's not possible to access contiguous zones since zoneskip bytes will separate the end of one zone and the beginning of the next zone. This patch allows fio to move to the next zone when zoneskip is explicitly set to 0, making it possible to issue IO to contiguous zones. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-14parse: improve detection of bad input stringJens Axboe
If we attempt to do number conversion, and strtoll() returns 0 AND the end pointer is the same as the starting string, then fail the conversion. Fixes: https://github.com/axboe/fio/issues/841 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-10-13Merge branch 'fix-corrupt-hist-log' of https://github.com/sitsofe/fioJens Axboe
* 'fix-corrupt-hist-log' of https://github.com/sitsofe/fio: stat: fix corruption in histogram logs
2019-10-13stat: fix corruption in histogram logsSitsofe Wheeler
Commit 6cc0e5aa9eddd2487dfa9ac80be1264151058409 ("Fix overflow of counters incremented on each I/O operation") changed various structures from [unsigned] int -> [u]int64_t but the size used when memcpy-ing histogram data was still the older (smaller) size. This led to un-copied values and usage of uninitialised data. An example job that demonstrates the issue is: $ valgrind ./fio --size=256k --thread=1 --rate_iops=5 --direct=1 \ --number_ios=15 --name=hist_test --filename=fio.test \ --log_hist_msec=1000 --write_hist_log=fio.test --log_hist_coarseness=6 ==1743== Memcheck, a memory error detector [...] fio-3.16-8-gf951 Starting 1 thread ==1743== Thread 3: IOPS: [R(1)][23.1%][r=20KiB/s][r=5 IOPS][eta 00m:10s] ==1743== Use of uninitialised value of size 8 ==1743== at 0x609D86B: _itoa_word (_itoa.c:179) ==1743== by 0x60A0F0D: vfprintf (vfprintf.c:1642) ==1743== by 0x60A9E53: fprintf (fprintf.c:32) ==1743== by 0x47FA6C: flush_hist_samples (iolog.c:869) ==1743== by 0x47F637: flush_log (iolog.c:1212) ==1743== by 0x4813C5: finish_log (iolog.c:1238) ==1743== by 0x4812CD: __write_log (iolog.c:1571) ==1743== by 0x481202: write_clat_hist_log (iolog.c:1625) ==1743== by 0x480045: td_writeout_logs (iolog.c:1721) ==1743== by 0x485C7A: thread_main (backend.c:1889) ==1743== by 0x5C296DA: start_thread (pthread_create.c:463) ==1743== by 0x616688E: clone (clone.S:95) This commit fixes the issue by using the new size in the memcpy. Fixes: https://github.com/axboe/fio/issues/827 Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-08Merge branch 'bumpflocks' of https://github.com/earlephilhower/fioJens Axboe
* 'bumpflocks' of https://github.com/earlephilhower/fio: Increase MAX_FILELOCKS for highly parallel IO test
2019-10-08Increase MAX_FILELOCKS for highly parallel IO testEarle F. Philhower, III
When running hundreds of threads and collecting CLAT, SLAT, IOS on a per-thread basis, FIO can need a lot of file locks to complete. When it can't get them, it gives the cryptic error message: > fio: filelock.c:182: __fio_lock_file: Assertion `!trylock' failed. Bump the maximum from 128 to 1024 to give headroom for high core count, high thread count testing.
2019-10-08backend: fix final fsync behaviourSitsofe Wheeler
Previously, fsync_on_close's "final" fsync was done after fio's accounting had finished thus causing the bandwidth reported to be too high (due to the reported time being too low). An example job that demonstrates the issue (on machines with a few gigabytes of RAM and non-fast disks) is the following: ./fio --gtod_reduce=1 --filename=fio.tmp --size=5G --bs=2M --rw=write \ --stonewall --name=end_fsync --end_fsync=1 --name=fsync_on_close \ --fsync_on_close=1 end_fsync: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fsync_on_close: (g=1): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 [...] Run status group 0 (all jobs): WRITE: bw=381MiB/s (400MB/s), 381MiB/s-381MiB/s (400MB/s-400MB/s), io=5120MiB (5369MB), run=13424-13424msec Run status group 1 (all jobs): WRITE: bw=1726MiB/s (1810MB/s), 1726MiB/s-1726MiB/s (1810MB/s-1810MB/s), io=5120MiB (5369MB), run=2966-2966msec This patch fixes the issue by doing an fsync for fsync_on_close at the same point as it does for end_fsync. Technically fsync_on_close will go and do another fsync after this point too but this should be harmless as there is no new write data. This patch also fixes the assert seen with the following job ./fio --size=8k --io_limit=32k --filename=fio.tmp --rw=write \ --end_fsync=1 --name=extended_end_fsync [...] fio: filesetup.c:1682: void get_file(struct fio_file *): Assertion `fio_file_open(f)' failed. by preventing auto-close on last I/O when it is being submitted as a final fsync (due to end_fsync, fsync_on_close). Finally it also fixes the strange IO depths seen with when using end_fsync: ./fio --size=4k --filename=fio.tmp --rw=write --end_fsync=1 \ --name=end_fsync_test end_fsync_test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 [...] IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% by no longer including such fsyncs in accounting. Fixes: https://github.com/axboe/fio/issues/831 Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2019-10-07Merge branch 'replay-blktrace-fixes' of https://github.com/shimrot/fioJens Axboe
* 'replay-blktrace-fixes' of https://github.com/shimrot/fio: Fix assert error on blktrace replay containing trims
2019-10-06Merge branch 'error-on-implicit-decl' of https://github.com/sitsofe/fioJens Axboe
* 'error-on-implicit-decl' of https://github.com/sitsofe/fio: configure: stop enabling fdatasync on OSX