summaryrefslogtreecommitdiff
path: root/cconv.c
AgeCommit message (Collapse)Author
2019-08-28options: allow offset_increment to understand percentagesVincent Fu
Add the ability for the offset_increment option to understand percentages. With this patch offset_increment=10% will, for example, increase the start offset by 10% of the file size for additional jobs created by numjobs. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-08-14Optimize the code that copies stringsBart Van Assche
Using strncpy() to copy strings is suboptimal because strncpy writes a bunch of additional unnecessary null bytes. Use snprintf() instead of strncpy(). An additional advantage of snprintf() is that it guarantees that the output string is '\0'-terminated. This patch is an improvement for commit 32e31c8c5f7b ("Fix string copy compilation warnings"). Cc: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-09-20blktrace: add option to iterate over a trace multiple timesDennis Zhou
Scaling a particular trace may result in different runtimes among the merging traces. By knowing the approximate length of each trace as a user, the overall runtime of each can be tuned to line up by letting certain traces loop multiple times. First, the last timestamp of a trace is recorded at the end of the first iteration to denote the length of a trace. This value is then used to offset subsequent iterations of a trace. Next, the "--merge_blktrace_iters" option is introduced to let the user specify the number of times to loop over each specific trace. This is done by passing a comma separated list that index-wise pairs with the passed files in "--read_iolog". Iteration counts are introduced as well as keeping track of the length of each trace. In an example, given two traces, A and B, each 60s long. If we want to see the impact of trace A issuing IOs twice as fast, the --merge_blktrace_scalars="50:100" can be set and then --merge_blktrace_iters="2:1". This runs trace A at 2x the speed twice for approximately the same runtime as a single run of trace B. Signed-off-by: Dennis Zhou <dennis@kernel.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-09-20blktrace: add option to scale a traceDennis Zhou
As we explore stacking traces, it is nice to be able to scale a trace to understand how the traces end up interacting. This patch adds scaling by letting the user pass in percentages to scale a trace by. When passed '--merge_blktrace_scalars="100"', the trace is ran at 100% speed. If passed 50%, this will halve the trace timestamps. The new option takes in a comma separated list that index-wise pairs with the passed files in "--read_iolog". This option differs from "--replay_time_scale" which scales the trace during runtime and will not change the output unlike this option. Signed-off-by: Dennis Zhou <dennis@kernel.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-09-20blktrace: add support to interleave blktrace filesDennis Zhou
Running concurrent workloads via multiple jobs can lead to nondeterministic results as we are at the scheduler's mercy. While the actual performance of the workload may vary regardless, this makes the order of events consistent. This patch introduces two flags: --merge_blktrace_file=<file> and --merge-blktrace-only. When the first is specified, files that are ':' separated in --read_iolog are passed to a merge function wich then produces a single sorted trace file. This file is then passed on in place of the sorted list or fio exists if --merge-blktrace-only is specified. During merging, events are filtered based on what fio cares about and the pdu is discarded as well. Signed-off-by: Dennis Zhou <dennis@kernel.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-08-24Add the zonemode job optionBart Van Assche
Fio's zone support makes fio perform I/O inside a zone before it skips to the next zone. That behavior is the opposite of the behavior needed for zoned block devices, namely to consider all zones when performing random I/O. Hence introduce a new job option that allows users to choose between fio's traditional zone mode and the behavior needed for zoned block devices. This patch makes fio behave identically with --zonemode=none and --zonemode=zbd. A later patch will implement new behavior for --zonemode=zbd. Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-07-23Add support for >= 4G block sizesJeff Furlong
For trims, it's useful to be able to support larger than (or equal to) 4GB. This extends the block sizes support for that. Change from Jeff, various little fixups from me. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-04-26blktrace: add 'reply_skip' optionJens Axboe
Allows the user to skip certain IO types. Useful for replaying workloads with trim on a device that doesn't support it, for instance. Or just replaying parts of a workload (eg just writes, or just reads). Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-04-17Deprecate verifysort and verifysort_nrJens Axboe
It was an optimization to read back verifies in a sorted order, for rotational storage. But I don't think the option makes much sense, and I've never heard of anyone using it. Mark it as deprecated, and always verify in the same order that IO was written. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-04-14cconv: add conversion for 'replay_time_scale'Jens Axboe
This was forgotten in the previous commit. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-21Refactor #includes and headersSitsofe Wheeler
- Try and remove unneeded #include lines - Try and add #include lines that would allow the files to be built in a more standalone manner Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2017-12-06Add option to ignore thinktime for rated IOJens Axboe
By default, fio will ignore thinktime when calculating the next time to issue and IO, if rated IO is specified. This leads to fio entering a catch-up type of mode after doing the specified sleep. For some workloads, that may not be useful. If someone asks for a specific amount of IOPS and sets a thinktime, they may want to exclude the sleep time. Fixes: https://github.com/axboe/fio/issues/497 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-11-29options: make max_latency a 64-bit variableJens Axboe
Also removes one pad in the thread options. This should bump the protocol version, but we just did that, so let it slide. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-11-22add significant_figures parameterJeff Furlong
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-26Add offset_align optionJeff Furlong
This allows us to align the start offsets, if percentage offsets are used. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-09-13Add support for doing total latency percentilesJens Axboe
By default, fio does completion latency percentiles. Sometimes what you want is total IO latency percentiles, including the submission part as well. If that's your thing, then set lat_percentiles=1 and get that instead of the default completion latency percentiles. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-08-14fio: add serialize_overlap optionSitsofe Wheeler
If this isn't set (the default) fio can submit write I/Os that overlap other in-flight I/Os leading to potential data races. For example the following job frequently fails at the verification stage: ./fio --random_distribution=zipf:1.6 --direct=1 --filename \ /tmp/fiofile --ioengine=posixaio --iodepth=32 --size=20M --bs=4k \ --rw=randwrite --verify=crc32c --name=verifyoverlap When serialize_overlap=1 fio avoids creating such races. Thanks to Rachel Lunnon (StorMagic) for helping me debug the initial version of this! Fixes: https://github.com/axboe/fio/issues/335 v2: Fix merge conflict and add missing conversion. v3: Add man page, fix serialize_overlap disabling, improve commit message. Tested-by: Jeff Furlong <jeff.furlong@wdc.com> Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2017-08-07move skip_bad= option to engines/mtd.cTomohiro Kusumi
This is ioengine specific, thus should be defined in engines/mtd.c. This option is useless unless engines/mtd.c is enabled. It could be conditionally compiled using CONFIG_MTD, but all other ioengines define their own option structure in their code. The only functional difference is that now .category is changed to FIO_OPT_C_ENGINE. This is also something done by all other ioengine specific options, though ->category isn't really used for anything meaningfull. Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-06-19Update write_hint mechanism to latest APIJens Axboe
Kill the fadvise stream stuff, we never used it. Also dump the pvsync2 options. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-06-05allow a percent value for the offset parameterBrantley West
2017-04-26gpu: ensure that we convert gpu_dev_id optionsJens Axboe
Fixes: 03553853 ("GPUDirect RDMA support") Signed-off-by: Jens Axboe <axboe@fb.com>
2017-03-16Add 'stats' optionJens Axboe
This option controls whether or not we collect runtime stats and show stat output for a given job. Defaults to on. Signed-off-by: Jens Axboe <axboe@fb.com>
2017-02-19Rename thread_options' ->io_limit to io_sizeTomohiro Kusumi
This is less confusing since io_size is the option name itself, and not much reason to use io_limit which is an alias of io_size. Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2017-01-02Fix unit_base kb_base mixup in thread option conversion functionsRobert Elliott
Don't set unit_base, which is 1 or 8, to kb_base, which is 1000 or 1024, during the convert functions used by the initial cconv test. Signed-off-by: Jens Axboe <axboe@fb.com>
2016-12-13Merge git://git.kernel.dk/fio into steady-stateVincent Fu
2016-12-08steadystate: rename options->ss to options->ss_state since ss is used ↵Vincent Fu
elsewhere to refer to struct steadystate_data
2016-12-06steadystate: get rid of ->ss_pct and encode this information in ->state via ↵Vincent Fu
__FIO_SS_PCT This leaves a 32-bit filler in thread_options_pack that can be removed when it is possible to maintain required byte alignment
2016-11-27fio: Fix (unsigned) integer overflow issuesSitsofe Wheeler
This patch tries to address issues found when compiling fio with clang's undefined behaviour sanitizer on 32 bit Linux. While this generated many false positives it did identify a number of subtle issues when rates/amounts/time calculations went above 32 bit values. The following job was used: [global] time_based runtime=180m bs=256k rw=randrw norandommap rate=512M ioengine=null direct=1 iodepth=8 thread numjobs=10 ramp_time=20m [null] size=1P It should fix https://github.com/axboe/fio/issues/270 because various 32 bit ETA variables were overflowing due to the microsecond arithmetic being done on them (even on 64 bit systems). This patch should also fix https://github.com/axboe/fio/issues/194 and https://github.com/axboe/fio/issues/266 because 64 bit Windows is a LLP64 platform so ints and longs still default to 32 bits. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2016-10-11iolog: make write_*_log prefix optionalOmar Sandoval
Providing write_bw_log=jobname for every job in a jobfile gets pretty tedious. This way, we can just do, e.g., [global] write_bw_log and all jobs will be logged. The documentation seems to suggest that this is the intended behavior, and it looks like that used to be the case. Fixes: 203160d52b86 ("options: get rid of more .cb option usage we don't need") Signed-off-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2016-08-18Option for changing log files to use Unix epoch instead of beingKarl Cronburg
zero-based (when fio starts) epoch. This makes it easier to analyze the data in the context of other benchmarking tools running at the same time as fio. Signed-off-by: Karl Cronburg <kcronbur@redhat.com>
2016-08-15steady-state: convert options across the wireJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2016-08-08Add 'unlink_each_loop' optionmrturtledev
If this option is set, fio will unlink the files after each loop of a job has completed. Original patch by Martin, various tweaks and updates from me. Signed-off-by: Jens Axboe <axboe@fb.com>
2016-08-07server: bump protocol versionJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2016-08-06This commit / feature adds completion latency histogram output to fio, ↵Karl Cronburg
piggybacking on the existing histograms recorded by stat.c and adding the following command line options: - log_hist_msec operates like log_avg_msec does, but produces files named *_clat_hist* - write_hist_log has the same semantics as write_lat_log but for histograms. - log_hist_coarseness changes how many bins are logged. The more substantial contribution is a post-processor for digesting these log files, located in fio/tools/hist/fiologparser_hist.py, which computes min/max/avg/percentile statistics. To compare the accuracy of the statistics produced by fiologparser_hist.py, a reference implementation is located here: https://github.com/cronburg/fio/blob/histograms/tools/hist/fiologparser_numpy.py for computing the same statistics from the *_clat.* files fio already produces. A comparison can be viewed here: https://cronburg.com/fio/demo/latency.html The reference implementation is excluded from this commit, as I'm still working with Mark (markhpc) on getting it into the branch where he's working on fiologparser. Also I'd eventually like to make FIO_IO_U_PLAT_GROUP_NR in stat.h configurable (allowing for a larger maximum histogram latency value than 17 seconds), but haven't added it to keep the changes to fio minimal for now. A more detailed description of the motivation behind this feature, and explanation of the design process and algorithms used in the post-processor can be found here: https://cronburg.com/fio/cloud-latency-problem-measurement/
2016-05-23cconv: wire up conversion of unique_filenameJens Axboe
Also bump server version, since thread_options changed. Signed-off-by: Jens Axboe <axboe@fb.com>
2016-03-04Add support for zones of random IO, with varying frequency of accessJens Axboe
Let's say you want to ensure that 50% of the IO falls in the first 5% of the flie, with the remaining 50% over the last 95%. Now that's possible with random_distribution=zoned. For this particular case, you would do: random_distribution=zoned:50/5:50/95 Up to 64 ranges can be specified, and multiple data directions can be given as well. The above would apply to both reads, writes, and trims. If you wanted to have 50% of the writes fall in the first 10%, 25% in the next 10%, and the last 25% over the remaining 80%, you could extend the above ala: random_distribution=zoned:50/5:50/95,50/10:25/10:25/80 Signed-off-by: Jens Axboe <axboe@fb.com>
2016-02-04log: add support for logging max instead of averagesJens Axboe
If we log with lov_avg_msec, fio will log the average over that period of time to the log. Add log_max_value to change this to logging maximums over the windowed period instead. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-12-22Add support for waiting for other jobs by nameAndrey Kuzmin
We have stonewall to barrier against previous jobs before a new job runs. This adds support for 'wait_for', which allows the user to explicitly wait for a given job before starting. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-12-10Add exitall_on_error optionJens Axboe
Like exitall, but only if a job exits due to erroring out. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-12-08options: add log_compression_cpus optionJens Axboe
We generally don't want online log compression to interfere with the IO jobs. Add an option that allows us to contain the log compression to a specific CPU, or set of CPUs. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-11-09Rename rate_poisson to rate_processJens Axboe
This will more accurately follow what other options could adopt, for instance if we want to do a dynamic IO depth. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-11-09Fixups for poisson rateJens Axboe
- Change name to poisson_rate, we could add other uses for this functionality (like depth, etc). - Add "rate" as option parent. - Ensure we convert the option for client/server, bump protocol version. - Add a separate rand state for poisson, so it becomes randomly controllable (through randrepeat). - Add random 0..1 helper. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-10-01Introduce new option: iodepth_batch_complete_maxRoman Pen
By now there is an option which is called 'iodepth_batch_complete'. This option provides a possibility to read exact amount of events of completed IOs, e.g. by default it is set to 1, so only 1 event will be read on each io_getevents system call: 7945 io_getevents(139876985688064, 1, 1, {...}NULL) = 1 This option can be set to any desired amount, say the whole iodepth size can be used, e.g. 128: 7952 io_getevents(140303500259328, 128, 128, {...}NULL) = 128 We will not exit the system call till the whole queue will be completed. Sounds not so efficient. In this patch I add the 'iodepth_batch_complete_max' option and rename 'iodepth_batch_complete' to 'iodepth_batch_complete_min' to have a possibility to specify a range of events which I want to read. (of course because of compatibility's sake 'iodepth_batch_complete_min' is an alias for 'iodepth_batch_complete'). So finally if options are set to: # or iodepth_batch_complete_min=1 iodepth_batch_complete=1 # take the iodepth value iodepth_batch_complete_max=128 The io_getevents call will look like: 7961 io_getevents(140173245206528, 1, 128, {...}NULL) = 73 and we will exit the system call with any amount of completed IOs >= 1. What are the benefits? Fetching events from the queue can be done more efficiently on some testing on specific configurations, e.g. stacked block devices based on null_blk devices, where completion happens immediately and events should be retried from the queue ASAP. Let's take a simple example. BASE CONFIG: [global] fadvise_hint=0 rw=randrw:2 direct=1 size=256M ioengine=libaio iodepth=128 time_based=1 runtime=60 ramp_time=10 [job1] filename=/dev/nullb0 [job2] filename=/dev/nullb1 [job3] filename=/dev/nullb2 [job4] filename=/dev/nullb3 Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-22cconv: add allow_mounted_writeJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-12Add 'allow_file_create' optionJens Axboe
For running certain jobs, it's convenient to tell fio that you never want it to create files. On Linux, this prevents filling up /dev with data for cases where the specified block device isn't available. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-11Add 'per_job_logs' optionJens Axboe
If set, this generates bw/clat/iops log with per file private filenames. If not set, jobs with identical names will share the log filename. Default: true. Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-06cconv: add missing conversions of block_error_hist and skip_badJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-06blktrace: add support for scaling and aligning replaysJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-04-15First cut at supporting IO offloadJens Axboe
rate_iops=x io_submit_mode=offload Signed-off-by: Jens Axboe <axboe@fb.com>
2015-04-08Unify gauss and zipf/pareto input valuesJens Axboe
Signed-off-by: Jens Axboe <axboe@fb.com>