fio.git
6 years agogenfio: Splitting gen_template in 2 parts
Erwan Velu [Mon, 15 Jul 2013 08:44:58 +0000 (10:44 +0200)]
genfio: Splitting gen_template in 2 parts

gen_template can be split in to two parts:
- the static headers
- the dynamic ones linked to the arguments from cmdline

The static ones are now in gen_template() call before parsing the
cmdline.

The dynamic ones are now call in finish_template() after parsing the
cmdline.

That also open the ability of feeding the fio header during the parsing
removing the need of specific variables to remember the values.

6 years agonet: close socket on error
Shawn Bohrer [Fri, 19 Jul 2013 18:24:10 +0000 (13:24 -0500)]
net: close socket on error

Signed-off-by: Shawn Bohrer <sbohrer@rgmadvisors.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agonet: fix recvfrom error message
Shawn Bohrer [Fri, 19 Jul 2013 18:24:09 +0000 (13:24 -0500)]
net: fix recvfrom error message

The error is from recvfrom not sendto.

Signed-off-by: Shawn Bohrer <sbohrer@rgmadvisors.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoHOWTO: update exec_{pre,post}run information
Jens Axboe [Thu, 18 Jul 2013 04:15:09 +0000 (22:15 -0600)]
HOWTO: update exec_{pre,post}run information

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix warning with newer gcc in str_numa_mpol_cb()
Jens Axboe [Wed, 17 Jul 2013 22:15:31 +0000 (16:15 -0600)]
Fix warning with newer gcc in str_numa_mpol_cb()

GCC complains about potentially accessing the array beyond
the end. Ensure that we include a NULL at the end.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agobackend: Logging exec_{pre|post}run
Erwan Velu [Wed, 17 Jul 2013 21:04:46 +0000 (23:04 +0200)]
backend: Logging exec_{pre|post}run

When using the exec_{pre|post}run option in fio jobs, it's very
important being able to read the output of thoses commands later.

As the output could be very verbose, let's log all the output of the
command in separate file.

As an example, when running two jobs called "write" and "read" the
exec_prerun option is used. During fio's execution, the following
message will be shown :
write : Saving output of prerun in write.prerun.txt
read : Saving output of prerun in read.prerun.txt

The output filename is so composed of the job name appended with the pre
or post run name and a final .txt extension.

That's pretty simple and shall do the job.

6 years agobackend: Get rid of sh -c for exec_string
Erwan Velu [Wed, 17 Jul 2013 20:29:27 +0000 (22:29 +0200)]
backend: Get rid of sh -c for exec_string

Running commands in a 'sh -c' does kill the existing environment values.
i.e that does kill ssh-agent.

It sounds more 'direct' to execute the command in the same environment
as fio.

6 years agogenfio: Enforce time_based if runtime > 0
Erwan Velu [Fri, 12 Jul 2013 15:13:59 +0000 (17:13 +0200)]
genfio: Enforce time_based if runtime > 0

If the device is short and the bandwidth pretty high, we can exit
_before_ runtime is achieved. To avoid this phenomena, let's use
time_based directive that will ensure that no run will be shorten.

6 years agogenfio: Adding iodepth support
Erwan Velu [Fri, 12 Jul 2013 15:01:31 +0000 (17:01 +0200)]
genfio: Adding iodepth support

It's useful to be able to specify the iodepth while running fio with
libaio.

By default, we keep iodepth=1 to keep the same as fio's default.

6 years agogenfio: Warn users if read occurs before write
Erwan Velu [Fri, 12 Jul 2013 13:21:25 +0000 (15:21 +0200)]
genfio: Warn users if read occurs before write

The default mode was to generate read pattern _before_ the write ones.
On some storage device that handle read on untouch sectors by answering
zero from the cache, the result is totally invalid and cheat about the
real read capabilities of the device.

This patch does change the default mode to emit writes before reads.
It also add a warning message if user is setting up some modes by itself
and generate a read_before_write pattern.

That's only a warning message that last a few seconds but could avoid
running a wrong benchmark.

This commit occurs after a real case of invalid performances due to read
emitted to an untouched block volume.

Typical output will be:
 root@vm1-1: genfio -d vdb -s -b 4k,128k,4m -m read,randread
 ###############################################################
 # Warning : You are reading data while never wrote them before#
 # On some storage devices, this could lead to invalid results #
 #                                                             #
 # Press Ctrl-C to adjust pattern order if you have doubts     #
 # Or Wait 5 seconds before the file will be created           #
 ###############################################################
 Generating vm1-1-4k,128k,4m-sequential-read,randread-vdb.fio
 Estimated Time = 1800 seconds : 0 hour 30 minutes

6 years agofio2gnuplot: Adjusting y axis plot (2D)
Erwan Velu [Fri, 12 Jul 2013 12:27:12 +0000 (14:27 +0200)]
fio2gnuplot: Adjusting y axis plot (2D)

It's better let gnuplot adjusting the tics on the Y axis unless that
becomes unreadble. Too many ticks where present on simple graphs.

This was a useless option, shouldn't have been here from the beginning.

6 years agofio2gnuplot: Don't graph 3D if not enough data
Erwan Velu [Fri, 12 Jul 2013 12:24:26 +0000 (14:24 +0200)]
fio2gnuplot: Don't graph 3D if not enough data

If the matching pattern doesn't get more than a single file, 3D plotting
doesn't have any meaning and confuse gnuplot.

So let's disable the 3D plotting if a single file matched the pattern.

6 years agofio2gnuplot: Adding sample files
Erwan Velu [Fri, 12 Jul 2013 09:36:43 +0000 (11:36 +0200)]
fio2gnuplot: Adding sample files

This commit does add some sample log files and makefile to use
fio2gnuplot. This is very useful for understanding, developping or testing fio2gnuplot.

A simple "make" will do the job for you.

6 years agofio2gnuplot: Adding 2D graphs for each fio trace
Erwan Velu [Fri, 12 Jul 2013 09:12:10 +0000 (11:12 +0200)]
fio2gnuplot: Adding 2D graphs for each fio trace

When tracing multiple files, the 3D graph is useful to understand the
behavior of a group of disks but makes difficult to understand how a
particular disk behave during the test.

This patch does add three 2D plots for each fio trace :
- the 2D Raw tracing
This plot is the exact plotting of the raw values reported by fio.
It's very precise but hard to read when having many many traces

- the 2D Smooth tracing
This plot is a smooth version of the raw tracing. It tries to keep as
much as possible a tracing close to the raw one but makes it more
readable by aggregating points.

- the 2D Trend tracing
This plot is trying to setup a trend of the global performance. It
aggregates many points and hide peaks. It only consider at giving a very
easy to understand plotting on how the bench was performing.
It's much more easier to read but could be misleading if raw or smooth
tracing didn't got read before.

6 years agogenfio: Fixing empty ETA while runtime > 0
Erwan Velu [Thu, 11 Jul 2013 08:16:05 +0000 (10:16 +0200)]
genfio: Fixing empty ETA while runtime > 0

Commit 920e0b66756814f7d01bcbbaf5cf0fd05213628d introduced the runtime=0
support but did considered the amount of minutes in addition of rounded
hours to consider if the ETA message can be shown or not.

This is perfectly wrong since it should have been the total amount of
seconds to consider. If runtime=0, the amount of seconds to run cannot
be computed as = 0.

The present code was wrong as if runtime was rounded on hours it did
show the message that ETA cannot be computed.

6 years agogenfio: Adding prefix to fio filename
Erwan Velu [Thu, 11 Jul 2013 07:56:28 +0000 (09:56 +0200)]
genfio: Adding prefix to fio filename

This patch add a prefix (-x) option to add context to fio jobs.
When running many many jobs it's pretty hard to remember which job was
made for.

Adding a prefix to the filename can give some interesting information
like the tested hardware or the day we run it, or a option you test
etc...

If the prefix owns a / in it, it will be considered as a directory and
will create it. The fio jobs will be stored into this directory.

6 years agogenfio: Claryfing default options
Erwan Velu [Wed, 10 Jul 2013 14:50:20 +0000 (16:50 +0200)]
genfio: Claryfing default options

Let's show the help if no disks are given as parameter.

This patch does also clarify the default values of each parameter.
That clearly helps understanding what's the default settings if no
option used.

6 years agogenfio: Adding cached_based option
Erwan Velu [Wed, 10 Jul 2013 14:44:24 +0000 (16:44 +0200)]
genfio: Adding cached_based option

By default fio is still run in direct mode which is safer for many
benchmarking approach. In some case, you might need to enable the
cached-based IOs.

The "-c" option is now here for enabling this.

6 years agogenfio: Adding runtime=0 support
Erwan Velu [Wed, 10 Jul 2013 14:38:33 +0000 (16:38 +0200)]
genfio: Adding runtime=0 support

If the runtime is set to 0, it will ask fio to get until the end of the
device. This could be useful in some cases.

6 years agoAdding fio2gnuplot tool
Erwan Velu [Tue, 25 Jun 2013 16:06:59 +0000 (18:06 +0200)]
Adding fio2gnuplot tool

fio2gnuplot aims at plotting fio output files into gnuplot traces.
It does work pretty differently from fio_generate_plots as :
- it plots simultaneous traces in 3D graphs
- it plots various histogram to represent min/max/average/stddev of the
  series
- it uses a pattern matching mecanism to select the input files
- it have a post-mortem analysis mode to compute global values across
  multiple log files
- it saves metadata to report what traces was related to what file
- it generates automatically picture's topic

fio2gnuplot aims at plotting a huge number of simultaneous fio traces in
a understable way. It got tested with up to 197 simultaneous disks traces.

The 3D rendering helps at understanding the "surface" of the benchmark
and answer is fraction of seconds about the homogenity of the results.

The histograms rendering helps at understanding the average values, the
standard deviations between hosts or storage device. By using the
pattern matching, it's really easy to select a set of files to generate
a dedicated graph.

Example:
fio2gnuplot.py -p 'm[1-3]sw[1-3]-*read*para-*iops*' -g
 Selected m1sw1-128k-sdb-read-para.results_iops.log
 Selected m1sw1-128k-sdc-read-para.results_iops.log
 Selected m1sw1-128k-sdd-read-para.results_iops.log
 Selected m1sw1-128k-sde-read-para.results_iops.log
 Selected m2sw1-128k-sdb-read-para.results_iops.log
 Selected m2sw1-128k-sdc-read-para.results_iops.log
 Selected m2sw1-128k-sdd-read-para.results_iops.log
 Selected m2sw1-128k-sde-read-para.results_iops.log
 Running gnuplot Rendering

The previous example shows how to match some iops log file from fio.
In this example, thoses files were generated by genfio.

As a result, the following files will be generated by gnuplot :
m[1-3]sw[1-3]-read-para-iops-3D.png
m[1-3]sw[1-3]-read-para-iops.max.png
m[1-3]sw[1-3]-read-para-iops.png
m[1-3]sw[1-3]-read-para-iops.average.png
m[1-3]sw[1-3]-read-para-iops.min.png
m[1-3]sw[1-3]-read-para-iops.stddev.png

Thoses files will graph all the selected files (traces) at the same time
with various goals (min/max/stddev/average/3D)

6 years agoAdding genfio tool : an fio job creator
Erwan Velu [Tue, 25 Jun 2013 15:47:01 +0000 (17:47 +0200)]
Adding genfio tool : an fio job creator

Some fio users need to create multiple fio jobs to compare storage
device's performance on different workload.

genfio is made for creating series of fio jobs based on the following
parameters:
- a disk serie
- a block size serie
- the time each job shall run (default is 300 seconds)
- the expected IO patterns to perform (read,write,randread,randwrite)
- a parameter to define if disks shall be tested one after the other or
  simultaneously

Based on all this parameters, a single fio script file will be computed
to process all this workloads.

Example:
[user@localhost]: genfio -d sdb,sdc,sdd,sde -a -b 4k,128k,1m -r 100 -a
Generating localhost-4k,128k,1m-all-read,write,randread,randwrite-sdb,sdc,sdd,sde.fio
Estimated Time = 6000 seconds : 1 hour 40 minutes

If the user run "fio
localhost-4k,128k,1m-all-read,write,randread,randwrite-sdb,sdc,sdd,sde.fio",
it will run:
- a sequential bench on /dev/sdb /dev/sdc /dev/sdd /dev/sde
    block size = 4k with read,write,randread,randwrite tests
    ETA ~ 4 tests * 4 disks * 100 seconds

- a sequential bench on /dev/sdb /dev/sdc /dev/sdd /dev/sde
    block size = 128k with read,write,randread,randwrite tests
    ETA ~ 4 tests * 4 disks * 100 seconds

- a sequential bench on /dev/sdb /dev/sdc /dev/sdd /dev/sde
    block size = 1m with read,write,randread,randwrite tests
    ETA ~ 4 tests * 4 disks * 100 seconds

- a parallel bench on /dev/sdb /dev/sdc /dev/sdd /dev/sde
    block size = 4k with read,write,randread,randwrite tests
    ETA ~ 4 tests * 100 seconds

- a parallel bench on /dev/sdb /dev/sdc /dev/sdd /dev/sde
    block size = 128k with read,write,randread,randwrite tests
    ETA ~ 4 tests * 100 seconds

- a parallel bench on /dev/sdb /dev/sdc /dev/sdd /dev/sde
    block size = 1m with read,write,randread,randwrite tests
    ETA ~ 4 tests * 100 seconds

For each job of this fio script, bandwidth & iops logs file will be
generated allow further investigations or plotting.

6 years agoInit buflen for our dummy buffer
Josef Bacik [Tue, 9 Jul 2013 00:35:01 +0000 (20:35 -0400)]
Init buflen for our dummy buffer

If we fail to verify our buffer we will allocate a dummy buf to fill in so we
have something to compare our bad file to.  Unfortunately we don't set the
buflen of the dummy buffer which means we will sometimes write well past the end
of our buffer and this make glibc complain loudly.  Thanks,

Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMake sure io_u->buflen is aligned to the verify_interval
Josef Bacik [Tue, 9 Jul 2013 00:32:50 +0000 (20:32 -0400)]
Make sure io_u->buflen is aligned to the verify_interval

If you specify verify and bs_unaligned you can make fio blow up in interesting
ways.  This is because the verify code assumes that the buflens will be bs
aligned and therefore aligned to the verify_interval.  However this isn't the
case for bs_unaligned.  So to fix this make sure we are always aligned to
verify_interval.  This keeps us from reading off the end of our io buf, or worse
writing past the end of it and makes this fio job pass properly instead of
blowing up.  Thanks,

Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFio 2.1.1 fio-2.1.1
Jens Axboe [Wed, 5 Jun 2013 14:55:53 +0000 (16:55 +0200)]
Fio 2.1.1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMakefile: don't run configure for make clean
Jens Axboe [Wed, 29 May 2013 19:49:21 +0000 (21:49 +0200)]
Makefile: don't run configure for make clean

Normally we auto-run the configure part, if config-host.mak
doesn't exist. But we don't need to do that for make clean.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoWindows: include winsock2.h instead of windows.h to avoid a warning.
Bruce Cran [Wed, 29 May 2013 09:21:04 +0000 (10:21 +0100)]
Windows: include winsock2.h instead of windows.h to avoid a warning.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoposix_fadvise is now provided in helpers.c: remove it from the Windows posix.c
Bruce Cran [Wed, 29 May 2013 09:12:27 +0000 (10:12 +0100)]
posix_fadvise is now provided in helpers.c: remove it from the Windows posix.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate windowsaio for changes to the thread structure
Bruce Cran [Wed, 29 May 2013 10:06:24 +0000 (12:06 +0200)]
Update windowsaio for changes to the thread structure

The free/busy/requeue list has been replaced with a FIFO + ring: update
windowsaio.c to work with the new code.

Modified by Jens to check for IO_U_F_FLIGHT before doing engine
private checks.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix bad merge of write_iops_log= option for fio 2.1
Jens Axboe [Wed, 29 May 2013 08:45:16 +0000 (10:45 +0200)]
Fix bad merge of write_iops_log= option for fio 2.1

We need to store the string.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoBump IO engine version number
Jens Axboe [Tue, 28 May 2013 12:17:50 +0000 (14:17 +0200)]
Bump IO engine version number

We changed the layout of io_u, which isn't binary
compatible with pre-compiled external engines.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoReplace list based free/busy/requeue list with FIFO + ring
Jens Axboe [Tue, 28 May 2013 12:16:55 +0000 (14:16 +0200)]
Replace list based free/busy/requeue list with FIFO + ring

Cache friendliness of the list is pretty low. This has
provably lower overhead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoCatch NULL td->o.ioengine
Jens Axboe [Mon, 27 May 2013 07:50:31 +0000 (09:50 +0200)]
Catch NULL td->o.ioengine

This should not happen, except if there's a bug in the
option parsing (or profile, see commit 774a99b5). But
lets be cautious and catch it instead of seg faulting.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Fri, 24 May 2013 18:11:02 +0000 (20:11 +0200)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoact: add slice support and pass/fail end criteria
Jens Axboe [Fri, 24 May 2013 07:09:39 +0000 (09:09 +0200)]
act: add slice support and pass/fail end criteria

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Fri, 24 May 2013 05:57:00 +0000 (07:57 +0200)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoRework fio_generate_plots script
Louwrentius [Fri, 24 May 2013 05:56:38 +0000 (07:56 +0200)]
Rework fio_generate_plots script

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agotiobench: ensure that ioengine is being specified
Jens Axboe [Thu, 23 May 2013 19:31:35 +0000 (21:31 +0200)]
tiobench: ensure that ioengine is being specified

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAlways exit if do_exit is set after parse
Jens Axboe [Thu, 23 May 2013 19:27:54 +0000 (21:27 +0200)]
Always exit if do_exit is set after parse

It need not be an error condition, it could be just after
doing --cmdhelp or similar.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoparse: if profile is set, only show options relevant to the profile
Jens Axboe [Thu, 23 May 2013 19:14:03 +0000 (21:14 +0200)]
parse: if profile is set, only show options relevant to the profile

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMake group_reporting be a string-set option again
Jens Axboe [Thu, 23 May 2013 18:24:12 +0000 (20:24 +0200)]
Make group_reporting be a string-set option again

Otherwise it'll break job files between 2.1 and 2.0.x and earlier.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoBetter catch arguments with required options and none passed
Jens Axboe [Thu, 23 May 2013 18:00:35 +0000 (20:00 +0200)]
Better catch arguments with required options and none passed

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix segfault is bool option is used on command line and fails parsing
Jens Axboe [Thu, 23 May 2013 17:54:43 +0000 (19:54 +0200)]
Fix segfault is bool option is used on command line and fails parsing

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoact: basic reporting
Jens Axboe [Thu, 23 May 2013 10:33:08 +0000 (12:33 +0200)]
act: basic reporting

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoact: bring closer to real ACT
Jens Axboe [Thu, 23 May 2013 07:32:31 +0000 (09:32 +0200)]
act: bring closer to real ACT

- Use the right write sizes and scale IOPS

- Use regular read(2)/write(2) and use read threads

- Add real prep phase

- Add error handling for running out of option space

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoact: fixes
Jens Axboe [Wed, 22 May 2013 22:29:38 +0000 (00:29 +0200)]
act: fixes

- Use libaio to be able to drive higher loads.

- Fix error in latency calculation.

- Fix 5ms vs 8ms typo.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoStart of ACT like benchmark profile
Jens Axboe [Wed, 22 May 2013 20:21:29 +0000 (22:21 +0200)]
Start of ACT like benchmark profile

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agohelpers: fixup bad fadvise check
Jens Axboe [Wed, 22 May 2013 10:17:43 +0000 (12:17 +0200)]
helpers: fixup bad fadvise check

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofusion atomic: retrieve all capabilities and support larger writes
Chandra Mallarapu [Tue, 21 May 2013 06:08:57 +0000 (08:08 +0200)]
fusion atomic: retrieve all capabilities and support larger writes

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofusion atomic: support for new SDK API
Santhosh Koundinya [Sat, 18 May 2013 06:44:01 +0000 (08:44 +0200)]
fusion atomic: support for new SDK API

vsl_vectored_write() is dead, long live nvm_atomic_write().

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoflow: quiesce pending IO before sleep
Jens Axboe [Fri, 17 May 2013 19:52:08 +0000 (21:52 +0200)]
flow: quiesce pending IO before sleep

Similar potential latency problem as was fixed for thinktime
in commit 002e7183 and for rated IO in commit c3a1b740.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate HOWTO and man page for thinktime blocks impact on queue depth
Jens Axboe [Fri, 17 May 2013 10:47:11 +0000 (12:47 +0200)]
Update HOWTO and man page for thinktime blocks impact on queue depth

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Fri, 17 May 2013 10:41:22 +0000 (12:41 +0200)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoEnsure that we have no IO pending when sleeping
Jens Axboe [Fri, 17 May 2013 10:39:53 +0000 (12:39 +0200)]
Ensure that we have no IO pending when sleeping

For rate limiting or thinktime handling, ensure that we have
no busy IO when we do spin or sleep. Otherwise we potentially
skew the latencies a lot, since events could have been reaped.

Already working for rate iops, just abstracted it out and
ensure that we do the same for thinktime. Only a problem for
certain workloads, those with queuedepth > 1.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agosync engine: guard pvsync engine register/unregister
Jens Axboe [Thu, 16 May 2013 18:50:26 +0000 (20:50 +0200)]
sync engine: guard pvsync engine register/unregister

Need to be behind CONFIG_PWRITEV check.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoDocument pvsync engine
Jens Axboe [Thu, 16 May 2013 18:38:09 +0000 (20:38 +0200)]
Document pvsync engine

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agosync engine: add support for pwritev/preadv
Jens Axboe [Thu, 16 May 2013 18:36:57 +0000 (20:36 +0200)]
sync engine: add support for pwritev/preadv

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFio 2.1 fio-2.1
Jens Axboe [Wed, 15 May 2013 06:42:26 +0000 (08:42 +0200)]
Fio 2.1

Now with the initial gfio code merged in.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoHOWTO: fix typo
Jens Axboe [Thu, 9 May 2013 19:10:58 +0000 (21:10 +0200)]
HOWTO: fix typo

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoHonor random/zero/compressible settings for filling an initial file
Jens Axboe [Fri, 3 May 2013 13:12:33 +0000 (15:12 +0200)]
Honor random/zero/compressible settings for filling an initial file

Right now we only do that for when the job writes, do it for the
initial setup as well. Otherwise you just get zero filled files
initially, which might not be what you want.

Also fixes a bug with compression if compress_chunk isn't set.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoandroid: use ashmem
Akers, Jason B [Wed, 1 May 2013 19:28:49 +0000 (21:28 +0200)]
android: use ashmem

I wrapped the ashmem calls with the existing shm* function definitions.
It's a little ugly, but makes for an easy patch touching only
os/os-android.h.

Tested on both arm and x86 android phones.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoconfigure: remove argument for --build-32bit-win
Jens Axboe [Wed, 1 May 2013 11:00:26 +0000 (13:00 +0200)]
configure: remove argument for --build-32bit-win

It's an enable type option, just remove the argument.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd support for dumping the status on Windows.
Bruce Cran [Sun, 28 Apr 2013 15:34:15 +0000 (17:34 +0200)]
Add support for dumping the status on Windows.

Windows uses a per-user temporary directory, with the path stored in the TEMP
environment variable. Update check_status_file to check both the Unix TMPDIR
and Windows TEMP variables before falling back to /tmp.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd distclean target
Jens Axboe [Sat, 27 Apr 2013 21:51:25 +0000 (15:51 -0600)]
Add distclean target

And only clean cscope.out output when doing that. I'm getting
tired of having to re-do cscope for no real reason.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMove __NR_shm* syscall numbers to x86-64 where they belong
Jens Axboe [Sat, 27 Apr 2013 02:24:59 +0000 (20:24 -0600)]
Move __NR_shm* syscall numbers to x86-64 where they belong

The definitions are only correct for 64-bit Linux, not for 32-bit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoEnsure that stat outputs are serialized
Jens Axboe [Fri, 26 Apr 2013 23:05:57 +0000 (17:05 -0600)]
Ensure that stat outputs are serialized

Two issues:

- We don't want potentially two stat outputs running at the same time.
  This could potentially happen if USR1 or interval stats race with
  the thread exit.

- We could potentially crash if the interval or USR1 signaled stat
  output trigger when fio is tearing down. Let the main thread wait
  for any in-progress output in that case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd log_info_flush()
Vincent Kang Fu [Fri, 26 Apr 2013 22:56:01 +0000 (16:56 -0600)]
Add log_info_flush()

If fio dumps status through -USR1 or similar and output is
being stored in a file, the buffering on the file descriptor
by the library can cause is not to see all the output. Ensure
that it gets flushed after dumping stats.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoGet rid of td->o.perc_seq
Jens Axboe [Fri, 26 Apr 2013 18:06:06 +0000 (12:06 -0600)]
Get rid of td->o.perc_seq

No point in tracking both perc_seq and perc_rand, when we know
that perc_seq + perc_rand == 100.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd support for modifying the randomness of a workload
Jens Axboe [Fri, 26 Apr 2013 14:56:17 +0000 (08:56 -0600)]
Add support for modifying the randomness of a workload

Fio supports random or sequential IO, or random IO with a number of
sequential IOs in between. The percentage_random options allows
more fine grained control over this. It defaults to 100, which would
make any random workload still purely random. If set below 100,
there will be some percentage of sequential IOs. This happens randomly,
but at the specified percentages.

The percentage_sequential option is just the complement of that, it
is guaranteed that:

percentage_sequential + percentage_random == 100%

The later setting has priority, each will adjust the other.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoCatch the case where size= is less than the minimum block size
Jens Axboe [Thu, 25 Apr 2013 16:11:41 +0000 (10:11 -0600)]
Catch the case where size= is less than the minimum block size

Fio will currently just attempt to start and immediately exit,
with no clue as to what went wrong. The behaviour is correct
in the sense that there's nothing to output, but it can be a
little confusing.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoparse: add debug output when enabled
Jens Axboe [Thu, 25 Apr 2013 16:06:32 +0000 (10:06 -0600)]
parse: add debug output when enabled

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix filling verify pattern for byte sizes of 3, 5, 7, ...
Jens Axboe [Thu, 25 Apr 2013 05:07:17 +0000 (23:07 -0600)]
Fix filling verify pattern for byte sizes of 3, 5, 7, ...

If we can't double up to the max pattern size, we leave unfilled
space. This causes verification failures. Fix this by properly
filling up to the max.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoEnsure we stop parsing and exit if a command line option fails
Jens Axboe [Thu, 25 Apr 2013 04:25:22 +0000 (22:25 -0600)]
Ensure we stop parsing and exit if a command line option fails

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate man page with new command line options
Jens Axboe [Thu, 25 Apr 2013 03:11:35 +0000 (21:11 -0600)]
Update man page with new command line options

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd support for other ways of triggering intermediate result outputs
Jens Axboe [Thu, 25 Apr 2013 02:38:54 +0000 (20:38 -0600)]
Add support for other ways of triggering intermediate result outputs

Fio has support for using the USR1 signal to dump the current
results while continuing to run. Add a --status-interval parameter
to allow this to be configured to automatically happen every X
seconds.

There's also support for signaling fio through the file system. To
do that, simply touch /tmp/fio-dump-status. When fio sees this file,
it will unlink and dump the current results.

A small tweak is needed for the file approach to work in Windows.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofix memory allocation in the cases we may need to align
peter chang [Wed, 24 Apr 2013 22:32:25 +0000 (16:32 -0600)]
fix memory allocation in the cases we may need to align

when using the sg backend we crash w/ a SEGV in fio_sgio_getevents()
because of memory corruption from the IO. when we allocate the td's
backing store there's a possible post-allocation adjustment to page
align the buffers. however, the extra memory that this alignment may
need is not taken into account when allocating the buffer.

Tested:
  - the problematic fio file looks like
  fdha347:~# cat foo.fio
  [global]
  bs=64k
  thread
  rw=read
  size=100g
  time_based=1
  runtime=60
  iodepth=1

  [sdc]
  ioengine=sg
  filename=/dev/sg2
  numjobs=1

Signed-off-by: peter chang <dpf@google.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoReally suppress ICE61 when building the Windows installer.
Bruce Cran [Tue, 23 Apr 2013 14:42:40 +0000 (08:42 -0600)]
Really suppress ICE61 when building the Windows installer.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix Windows build warnings.
Bruce Cran [Tue, 23 Apr 2013 14:27:23 +0000 (15:27 +0100)]
Fix Windows build warnings.

winsock2.h needs to be included before windows.h.
Separate declarations and code.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agogerror: mixed code/data section
Jens Axboe [Thu, 18 Apr 2013 12:22:37 +0000 (05:22 -0700)]
gerror: mixed code/data section

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoAdd -Wdeclaration-after-statement to compiler flags
Jens Axboe [Wed, 17 Apr 2013 17:54:53 +0000 (10:54 -0700)]
Add -Wdeclaration-after-statement to compiler flags

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoGet rid of mixed code/data sections
Jens Axboe [Wed, 17 Apr 2013 17:53:52 +0000 (10:53 -0700)]
Get rid of mixed code/data sections

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoioengine: kill unneeded thread_data forward declaration
Jens Axboe [Wed, 17 Apr 2013 17:33:41 +0000 (19:33 +0200)]
ioengine: kill unneeded thread_data forward declaration

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoKil unneeded forward declaration
Jens Axboe [Wed, 17 Apr 2013 17:30:43 +0000 (19:30 +0200)]
Kil unneeded forward declaration

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMove unused and other attributes to the compiler section
Jens Axboe [Wed, 17 Apr 2013 17:29:37 +0000 (19:29 +0200)]
Move unused and other attributes to the compiler section

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoSplit out td error handling code
Jens Axboe [Wed, 17 Apr 2013 17:27:46 +0000 (19:27 +0200)]
Split out td error handling code

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agonull: 'td' isn't actually unused, unmark it
Jens Axboe [Wed, 17 Apr 2013 17:21:14 +0000 (19:21 +0200)]
null: 'td' isn't actually unused, unmark it

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoPropagate io engine error back to 'td', if not already done
Jens Axboe [Wed, 17 Apr 2013 17:20:28 +0000 (19:20 +0200)]
Propagate io engine error back to 'td', if not already done

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agogoptions: fix bad log_err() type
Jens Axboe [Tue, 16 Apr 2013 19:10:46 +0000 (21:10 +0200)]
goptions: fix bad log_err() type

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoUpdate the examples filenames in the Windows installer.
Bruce Cran [Mon, 15 Apr 2013 12:39:58 +0000 (14:39 +0200)]
Update the examples filenames in the Windows installer.

The examples now have a ".fio" suffix - update the Windows installer.
Also, allow same version upgrades to avoid installing over the top of
an existing installation.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFixup bad logging types
Jens Axboe [Mon, 15 Apr 2013 09:40:57 +0000 (11:40 +0200)]
Fixup bad logging types

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agolog: add printf() attribute to log_*() functions
Jens Axboe [Mon, 15 Apr 2013 09:40:30 +0000 (11:40 +0200)]
log: add printf() attribute to log_*() functions

Similarly to how it's done for dprint(), so we can catch
format errors.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofio: fix segfault on 32bits platforms
Dmitry Monakhov [Mon, 15 Apr 2013 09:23:25 +0000 (11:23 +0200)]
fio: fix segfault on 32bits platforms

Uses wrong type in log_info().

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoeta: improve output for threads sitting in TD_SETTING_UP
Jens Axboe [Mon, 15 Apr 2013 08:21:56 +0000 (10:21 +0200)]
eta: improve output for threads sitting in TD_SETTING_UP

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoARM: Use generic assembly nop and barrier code for armv6
Otavio Salvador [Mon, 15 Apr 2013 07:36:24 +0000 (09:36 +0200)]
ARM: Use generic assembly nop and barrier code for armv6

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoREADME: quick note on how to cross compile
Aaron Carroll [Mon, 15 Apr 2013 06:41:41 +0000 (08:41 +0200)]
README: quick note on how to cross compile

Signed-off-by: Aaron Carroll <aaronc@cse.unsw.edu.au>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agofilesetup: set TD_SETTING_UP runstate when laying out files
Jens Axboe [Sat, 13 Apr 2013 18:40:53 +0000 (20:40 +0200)]
filesetup: set TD_SETTING_UP runstate when laying out files

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoRevert change that set group_reporting to 1
Jens Axboe [Sat, 13 Apr 2013 18:26:23 +0000 (20:26 +0200)]
Revert change that set group_reporting to 1

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoFix warning on platforms without disk util support
Jens Axboe [Fri, 12 Apr 2013 14:19:03 +0000 (16:19 +0200)]
Fix warning on platforms without disk util support

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agoMerge branch 'master' of ssh://git.kernel.dk/data/git/fio
Jens Axboe [Fri, 12 Apr 2013 13:40:50 +0000 (15:40 +0200)]
Merge branch 'master' of ssh://git.kernel.dk/data/git/fio

6 years agoserver: include uio.h for writev()
Bruce Cran [Fri, 12 Apr 2013 13:40:21 +0000 (15:40 +0200)]
server: include uio.h for writev()

Signed-off-by: Jens Axboe <axboe@kernel.dk>
6 years agogclient: update to new probe command format
Jens Axboe [Fri, 12 Apr 2013 12:13:15 +0000 (14:13 +0200)]
gclient: update to new probe command format

Signed-off-by: Jens Axboe <axboe@kernel.dk>