path: root/init.c
AgeCommit message (Collapse)Author
2020-02-04fio: better distinguish between high and low priorityVincent Fu
Change the variable names for the low priority latency variables to explicity distinguish them from the high priority latency variables. Also rename the JSON objects to do the same. This reduces the mental effort required to read the code and output. Signed-off-by: Vincent Fu <>
2020-02-04fio: report percentiles for slat, clat, latVincent Fu
Currently, fio can report percentiles for either completion latencies or total latencies. This patch adds the ability to simultaneously report percentiles for submission, completion, and total latencies. - simplify the JSON lat percentile reporting code - parsing the JSON output will be unaffected, although now setting lat_percentiles=1 no longer suppresses clat percentiles - this may break parsing of the normal output since more latency percentile blocks may be displayed - to maintain backward compatibility in the terse output, total latencies are reported if lat_percentiles=1 even when clat percentiles have also been computed - change the gtod_reduce callback to only change option values when gtod_reduce=1 - for the new cmdprio_percentage latencies, if lat_percentiles=1, *total* latency percentiles will be tracked. Otherwise, *completion* latency percentiles will be tracked. - also make sure we skip the high/low priority latency code if we are producing output for sync latencies. - make add_slat_sample use an unsigned long long for the time to be consistent with other functions Signed-off-by: Vincent Fu <>
2020-01-22Per-command priority: Priority logging and libaio/io_uring cmdprio_percentagePhillip Chen
Add cmdprio_percentage option to libaio and io_uring engines to set ioprio on a per-command basis. Add tracking of high priority commands to be displayed separately in human readable and JSON outputs.
2020-01-06Include "oslib/asprintf.h" where necessaryBart Van Assche
The header file "oslib/asprintf.h" must be included by every source file that calls asprintf(). Signed-off-by: Bart Van Assche <>
2019-11-12filesetup: add fallocate=truncate option.Trip Volpe
Fixes #833. Provides the ability to initially "layout" the file by ftruncating it to the desired size before performing IO. This is mainly useful on Windows, which serializes all writes that extend the size of a file. Using this option with a suitable iodepth allows fio to emulate the behavior of Windows Explorer file copy, which always truncates to the expected size before issuing writes for performance reasons.
2019-08-14Optimize the code that copies stringsBart Van Assche
Using strncpy() to copy strings is suboptimal because strncpy writes a bunch of additional unnecessary null bytes. Use snprintf() instead of strncpy(). An additional advantage of snprintf() is that it guarantees that the output string is '\0'-terminated. This patch is an improvement for commit 32e31c8c5f7b ("Fix string copy compilation warnings"). Cc: Damien Le Moal <> Signed-off-by: Bart Van Assche <> Signed-off-by: Jens Axboe <>
2019-07-01Merge branch 'optlenmax' of Axboe
* 'optlenmax' of Increase maximum length of line in jobs file to 8192
2019-07-01Increase maximum length of line in jobs file to 8192Nick Principe
Increase OPT_LEN_MAX to 8192, use OPT_LEN_MAX for string in __parse_jobs_ini(), and document line length limit for job files in man page. Signed-off-by: Nick Principe
2019-06-04Fix string copy compilation warningsDamien Le Moal
Fix the many warnings that gcc 9 spits out. Signed-off-by: Damien Le Moal <> Signed-off-by: Jens Axboe <>
2019-04-17rand: fix truncated rand_seed on WindowsMing-Hung Tsai
The long data type is 32-bit on LLP64 platforms
2019-02-11Document switch fall-through casesJens Axboe
Signed-off-by: Jens Axboe <>
2018-10-15init: loosen serialize_overlap restrictionsVincent Fu
When io_submit_mode=offload, allow overlap checking including for jobs where no data is being changed and when the iodepth is one. These cases can occur when checking for overlap across different jobs.
2018-10-04Bool conversionsJens Axboe
Just various low hanging fruits in terms of local or global variables that are only 0/1, switch them to boolean. Signed-off-by: Jens Axboe <>
2018-09-20blktrace: add support to interleave blktrace filesDennis Zhou
Running concurrent workloads via multiple jobs can lead to nondeterministic results as we are at the scheduler's mercy. While the actual performance of the workload may vary regardless, this makes the order of events consistent. This patch introduces two flags: --merge_blktrace_file=<file> and --merge-blktrace-only. When the first is specified, files that are ':' separated in --read_iolog are passed to a merge function wich then produces a single sorted trace file. This file is then passed on in place of the sorted list or fio exists if --merge-blktrace-only is specified. During merging, events are filtered based on what fio cares about and the pdu is discarded as well. Signed-off-by: Dennis Zhou <> Signed-off-by: Jens Axboe <>
2018-09-07log: remember to free output buffer when doneJens Axboe
We need to ensure we call buf_output_free() after using the temp storage, otherwise we leaked the memory that we allocated while filling the buffer. Signed-off-by: Jens Axboe <>
2018-09-07init: use __log_buf() if we know buf != NULLJens Axboe
Signed-off-by: Jens Axboe <>
2018-09-07Collect startup output before logging itJens Axboe
We currently do the startup logging in separate chunks. This is inefficient in terms of network trafic for client/server, and it also causes output to be potentially mingled together: <x1> Starting <x1> Starting 1 process1 process Fix this by logging to a buffer, then transmitting that buffer in one piece. Signed-off-by: Jens Axboe <>
2018-09-05init: move log name generation into helperJens Axboe
Signed-off-by: Jens Axboe <>
2018-09-04Log files names start with _ when write_XX_log= keys in config file have ↵Sam Zaydel
empty value(s)
2018-08-24Add support for zoned block devicesBart Van Assche
This patch adds support for zoned block devices as follows: - After the file size has been determined, check whether the file name refers to a block device. If so, check whether that block device is a host-managed block device. If that is the case, read the zone information using the BLKREPORTZONE ioctl. That ioctl is supported by the Linux kernel since kernel version v4.10. - After all command-line options have been processed and all job files have been read, verify whether these refer to a zoned block device and also whether the specified options are compatible with a zoned block device. Complain if that is not the case. - After each get_next_block() call, verify whether the block is appropriate for a zoned block device. When writing data to a sequential zone, adjust the write offset to the zone write pointer. When reading from a sequential zone, avoid reading past the write pointer. - After I/O submission, update the variable that represents the write pointer. - When writing data and with data verification enabled, reset a zone before writing any data into a zone. Otherwise reset a zone before issuing a write if that zone is full. - Translate trim into zone resets. Zoned block devices namely do not have to support any of the SCSI commands that are used by the kernel to implement the discard ioctl (UNMAP / WRITE SAME). This work started from a patch from Masato Suzuki <>. Some of the ideas in this patch come from Phillip Chen <>. Signed-off-by: Bart Van Assche <> Signed-off-by: Jens Axboe <>
2018-08-24Add the zonemode job optionBart Van Assche
Fio's zone support makes fio perform I/O inside a zone before it skips to the next zone. That behavior is the opposite of the behavior needed for zoned block devices, namely to consider all zones when performing random I/O. Hence introduce a new job option that allows users to choose between fio's traditional zone mode and the behavior needed for zoned block devices. This patch makes fio behave identically with --zonemode=none and --zonemode=zbd. A later patch will implement new behavior for --zonemode=zbd. Signed-off-by: Bart Van Assche <> Signed-off-by: Jens Axboe <>
2018-08-24Add the 'zbd' debug levelBart Van Assche
This debug level will be used in the next patch. Signed-off-by: Bart Van Assche <> Signed-off-by: Jens Axboe <>
2018-08-23init: add semantics for all types of backends runningLuis Chamberlain
Commit 67a176fc7 ("Fix segfault with client/server and minimal output") addressed a segfault but it wasn't very clear *why* the fix was needed. The reason it was needed is that the disk util and its respective semaphore are only initialized when setup_disk_util() is called, this happens upon fio_backend() calls. That is, either we have a dedicated backend or have initiated a backend for the localhost due to some local work. And show_thread_status() is currently called from a complete client setup -- handle_ts() calls show_thread_status(), and a client does not collect any local disk data, it receives this from the backend. As such, the semaphore won't be setup in a client setup and this is why we segfault here. We can enable show_thread_status() then only if any type of backend is running, however there is are no semantics currently which enable us to query for this. Add such semantics and replace the previous check with a check for if fio_backend() was ever called. This will make it clearer. Signed-off-by: Luis Chamberlain <>
2018-08-05client: support --status-interval option in client/server modeTomohiro Kusumi
This fixes "FIO client/server periodic stats" for non JSON output (the reporter is using JSON output). Mark -L/--status-interval option with FIO_CLIENT_FLAG, so that the command line string is sent to the server side. The client can now control the server to periodically show status, and each status is sent to the client via fio_server_text_output() (FIO_NET_CMD_TEXT socket event) in log_info_buf(). Reported-by: Mark Beierl <> Signed-off-by: Tomohiro Kusumi <> Signed-off-by: Jens Axboe <>
2018-07-24init: unify 't' time periodJens Axboe
Signed-off-by: Jens Axboe <>
2018-07-23Add support for >= 4G block sizesJeff Furlong
For trims, it's useful to be able to support larger than (or equal to) 4GB. This extends the block sizes support for that. Change from Jeff, various little fixups from me. Signed-off-by: Jens Axboe <>
2018-06-18init: abort write and trim jobs when --readonly option is presentVincent Fu
Check for conflicts between --readonly and the job's data direction in fixup_options(). rw_verify() in options.c only detects this conflict if --readonly precedes --readwrite on the command line.
2018-06-18init: ensure that fatal errors in fixup_options are always propogated to callerVincent Fu
When warnings_fatal = 0, fatal problems detected in fixup_options may not always be propogated back to the caller. This patch ensures that the caller will always be notified about fatal problems.
2018-06-12init: use o-> instead of td->oJens Axboe
We already have the options pointer, just use it. Signed-off-by: Jens Axboe <>
2018-06-12init: kill get_rand_start_delay()Jens Axboe
It's now useless, just call rand_between(). Signed-off-by: Jens Axboe <>
2018-06-11Fix start delay being the same across threadsJens Axboe
Two issues here: 1) We copy the start_delay into the original thread, which ends up offsetting job 2..N off the thread 1 start delay 2) We don't initialize the start delay random seed prior to generating the offsets Signed-off-by: Jens Axboe <>
2018-04-18Introduce enum n2s_unitBart Van Assche
This patch does not change any functionality but makes num2str() slightly easier to read. Signed-off-by: Bart Van Assche <>
2018-04-17init: ensure that read/write use the same random seed for verifyJens Axboe
If we are verifying, then reads need to use the same blockrange random trim as writes. Otherwise doing a deferred verify of a write job may not yield the same block size ranges, causing verify to fail. Example of failed job: $ cat job.fio [global] ioengine=sync verify=md5 bsrange=1k-16k rw=randwrite randseed=50 size=10m filename=foo [write] rw=randwrite do_verify=0 write_iolog=wlog [read] rw=randread do_verify=1 verify_fatal=1 verify_dump=1 write_iolog=rlog $ fio --section=write job.fio $ fio --section=read job.fio Fixes: Signed-off-by: Jens Axboe <>
2018-04-09init: fix memory leak in error handlingJens Axboe
Signed-off-by: Jens Axboe <>
2018-04-04Rename TD_F_VER_NONE into TD_F_DO_VERIFYBart Van Assche
Rename TD_F_VER_NONE into TD_F_DO_VERIFY to make it clear that this flag means that data verification has to be performed. See also commit d72be5454c8c ("Cache layout improvements"). Signed-off-by: Bart Van Assche <>
2018-03-21Refactor #includes and headersSitsofe Wheeler
- Try and remove unneeded #include lines - Try and add #include lines that would allow the files to be built in a more standalone manner Signed-off-by: Sitsofe Wheeler <>
2018-03-15Suppress uninteresting data race reportsBart Van Assche
Thread checkers like Helgrind and DRD report all concurrent modifications of variables that are serialized by a recognized synchronization primitive. Suppress reports about intentionally unsynchronized accesses. Signed-off-by: Bart Van Assche <>
2018-03-15Rename fio_mutex into fio_semBart Van Assche
Since fio_mutex implements a semaphore, rename fio_mutex into fio_sem. Rename those fio_sem instances that have 'mutex' embedded in their name. Rename the mutex.h and .c source files into fio_sem.h and .c respectively. Signed-off-by: Bart Van Assche <>
2018-03-09Declare debug_levels[] constBart Van Assche
Since debug_levels[] is not modified by any code, declare it const. Signed-off-by: Bart Van Assche <> Signed-off-by: Jens Axboe <>
2018-02-12init: add global 'warned' stateJens Axboe
In various places we spew a warning if some static variable isn't set, but this still means it can happen once per job. Add some global state to allow to track this globally instead. Signed-off-by: Jens Axboe <>
2018-02-10init: fixup some bad style in previous commitJens Axboe
Make it follow the fio coding style. Signed-off-by: Jens Axboe <>
2018-02-10init: fix broken verify_intervalDamian Yurzola
The operands seems to have been inverted which in turn created the situation whereby the interval was always changed to match the min_bs Fixes: Signed-off-by: Damian Yurzola <>
2018-01-25Track fsync/fdatasync/sync_file_range issue countJens Axboe
Also fix an issue where we didn't initialize the sync minimum latency correctly, so it was always reported as 0. Signed-off-by: Jens Axboe <>
2018-01-05Fix client/server "all clients" reportingJeff Furlong
In commit e883cb3501de0f99297282c18842d50aaa7fa531, the default significant figures was added. The default value was preserved for local jobs and when client/server was used if jobs=1, but not when client/server was used if jobs > 1. This patch restores some reporting outputs ("all clients" output for IOPS and BW) when client/server is used if > 1 jobs, for example: OLD: All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan 3 14:30:13 2018 read: IOPS=0, BW=0 (0)(0B/5001msec) <--ZERO VALUES slat (nsec): min=762, max=108579, avg=1058.97, stdev=221.30 clat (nsec): min=254, max=4544.2k, avg=11297.64, stdev=8284.14 lat (usec): min=9, max=4545, avg=12.41, stdev= 8.30 bw ( KiB/s): min=290192, max=347160, per=49.35%, avg=303613.33, stdev=11582.87, samples=18 iops : min=72548, max=86790, avg=75903.33, stdev=2895.72, samples=18 NEW: All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan 3 15:08:33 2018 read: IOPS=147k, BW=575Mi (603M)(2876MiB/5001msec) <--CORRECT VALUES slat (nsec): min=841, max=61945, avg=1304.83, stdev=232.75 clat (nsec): min=544, max=4487.2k, avg=11736.68, stdev=9123.64 lat (usec): min=10, max=4489, avg=13.10, stdev= 9.13 bw ( KiB/s): min=260945, max=305976, per=49.99%, avg=294350.22, stdev=9192.53, samples=18 iops : min=65236, max=76494, avg=73587.56, stdev=2298.13, samples=18 Regards, Jeff Signed-off-by: Jens Axboe <>
2017-12-31init: disable percentiles when latency gathering is disabledSitsofe Wheeler
When disable_clat/disable_lat is set also disable the corresponding clat_percentiles/lat_percentiles option. Signed-off-by: Sitsofe Wheeler <>
2017-12-15init: fix missing dlhandle reference putJens Axboe
If we keep using the same engine, put the dlhandle reference that we now hold twice. Signed-off-by: Jens Axboe <>
2017-12-15ioengines: improve "is this the same IO engine" checkJens Axboe
We can't just compare the name, that assumes that name and filename match for an IO engine. While that's generally the case for the engines that fio ships with, it's not a requirement, and it's definitely not true for external engines. Fixup the check by re-loading the engine and checking the OPS instead. That should be bullet proof. Fixes: 800334d ("Correctly detect whether ioengine_load can exit early") Signed-off-by: Jens Axboe <>
2017-12-14Allow configurable ETA intervalsJens Axboe
By default, fio prints ETA output every second. For some client/server setups, it's desirable to allow a much longer interval, to avoid spending too much time getting and printing ETA time. Takes a normal time input, allowing usec/msec/sec etc postfixes. Signed-off-by: Jens Axboe <>
2017-12-08Remove old emailsJens Axboe
Just use my private email, it's the one I prefer using anyway. Remove the old fusionio/oracle emails, they just cause confusion and people tend to just write to all of them at once when reporting an issue. Signed-off-by: Jens Axboe <>
2017-12-01Add basic memcpy testJens Axboe
Research is ongoing in how to improve (and make deterministic) the memcpy speed, since it's especially applicable to the persistent memory engines. Not documented yet. Signed-off-by: Jens Axboe <>