fio.git
3 years agoFio 2.2.12 fio-2.2.12
Jens Axboe [Wed, 25 Nov 2015 04:13:34 +0000 (21:13 -0700)]
Fio 2.2.12

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoA few min/max cleanups
Jens Axboe [Mon, 23 Nov 2015 22:04:56 +0000 (15:04 -0700)]
A few min/max cleanups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: set 'refill_buffers' for compress_percentage != 0
Jens Axboe [Mon, 23 Nov 2015 19:28:11 +0000 (12:28 -0700)]
options: set 'refill_buffers' for compress_percentage != 0

Only do it if the user hasn't set it.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoio_u: properly split buffer prep for compression
Jens Axboe [Mon, 23 Nov 2015 17:33:34 +0000 (10:33 -0700)]
io_u: properly split buffer prep for compression

Matthew reports:

I noticed that sometimes using --buffer_compress_percentage would not
result in the desired compression.  After running some tests I found
that only some block sizes are affected.

512b, 4K, 8K, 16K, and 32K seem to work as expected.

64K, 128K, 256K, 512K, and 1024K exhibited the bug.  I have not tested
beyond 1024K.

Here is the test script I used only changing block size for each test.

for i in {0..100}; do
  fio --name=test --rw=write --bs=128k --ioengine=libaio --direct=1
--iodepth=32 --size=512m --refill_buffers
--buffer_compress_percentage=$i --filename=testfile.$i --eta=never
--output=/dev/null
done

for i in {0..100}; do
  gzip -v testfile.$i &>> gzip.txt
done

rm *.gz

Below are compression results for 4K and then 128K.  For 128K, you can
see compression stops matching at around 50%.

bs=4k

testfile.0:      0.3% -- replaced with testfile.0.gz
testfile.1:      1.2% -- replaced with testfile.1.gz
testfile.2:      2.2% -- replaced with testfile.2.gz
testfile.3:      3.2% -- replaced with testfile.3.gz
testfile.4:      4.1% -- replaced with testfile.4.gz
testfile.5:      5.1% -- replaced with testfile.5.gz
testfile.6:      6.1% -- replaced with testfile.6.gz
testfile.7:      7.1% -- replaced with testfile.7.gz
testfile.8:      8.1% -- replaced with testfile.8.gz
testfile.9:      9.1% -- replaced with testfile.9.gz
testfile.10:     10.1% -- replaced with testfile.10.gz
testfile.11:     11.1% -- replaced with testfile.11.gz
testfile.12:     12.1% -- replaced with testfile.12.gz
testfile.13:     13.1% -- replaced with testfile.13.gz
testfile.14:     14.1% -- replaced with testfile.14.gz
testfile.15:     15.1% -- replaced with testfile.15.gz
testfile.16:     16.0% -- replaced with testfile.16.gz
testfile.17:     17.0% -- replaced with testfile.17.gz
testfile.18:     18.0% -- replaced with testfile.18.gz
testfile.19:     19.0% -- replaced with testfile.19.gz
testfile.20:     20.0% -- replaced with testfile.20.gz
testfile.21:     21.0% -- replaced with testfile.21.gz
testfile.22:     22.0% -- replaced with testfile.22.gz
testfile.23:     23.0% -- replaced with testfile.23.gz
testfile.24:     24.0% -- replaced with testfile.24.gz
testfile.25:     25.0% -- replaced with testfile.25.gz
testfile.26:     25.9% -- replaced with testfile.26.gz
testfile.27:     26.9% -- replaced with testfile.27.gz
testfile.28:     27.9% -- replaced with testfile.28.gz
testfile.29:     28.9% -- replaced with testfile.29.gz
testfile.30:     29.9% -- replaced with testfile.30.gz
testfile.31:     30.9% -- replaced with testfile.31.gz
testfile.32:     31.9% -- replaced with testfile.32.gz
testfile.33:     32.9% -- replaced with testfile.33.gz
testfile.34:     33.9% -- replaced with testfile.34.gz
testfile.35:     34.9% -- replaced with testfile.35.gz
testfile.36:     35.9% -- replaced with testfile.36.gz
testfile.37:     36.9% -- replaced with testfile.37.gz
testfile.38:     37.9% -- replaced with testfile.38.gz
testfile.39:     38.9% -- replaced with testfile.39.gz
testfile.40:     39.9% -- replaced with testfile.40.gz
testfile.41:     40.8% -- replaced with testfile.41.gz
testfile.42:     41.8% -- replaced with testfile.42.gz
testfile.43:     42.8% -- replaced with testfile.43.gz
testfile.44:     43.8% -- replaced with testfile.44.gz
testfile.45:     44.8% -- replaced with testfile.45.gz
testfile.46:     45.8% -- replaced with testfile.46.gz
testfile.47:     46.8% -- replaced with testfile.47.gz
testfile.48:     47.8% -- replaced with testfile.48.gz
testfile.49:     48.8% -- replaced with testfile.49.gz
testfile.50:     49.8% -- replaced with testfile.50.gz
testfile.51:     50.3% -- replaced with testfile.51.gz
testfile.52:     51.2% -- replaced with testfile.52.gz
testfile.53:     52.3% -- replaced with testfile.53.gz
testfile.54:     53.2% -- replaced with testfile.54.gz
testfile.55:     54.2% -- replaced with testfile.55.gz
testfile.56:     55.3% -- replaced with testfile.56.gz
testfile.57:     56.1% -- replaced with testfile.57.gz
testfile.58:     57.2% -- replaced with testfile.58.gz
testfile.59:     58.2% -- replaced with testfile.59.gz
testfile.60:     59.2% -- replaced with testfile.60.gz
testfile.61:     60.2% -- replaced with testfile.61.gz
testfile.62:     61.2% -- replaced with testfile.62.gz
testfile.63:     62.2% -- replaced with testfile.63.gz
testfile.64:     63.2% -- replaced with testfile.64.gz
testfile.65:     64.1% -- replaced with testfile.65.gz
testfile.66:     65.2% -- replaced with testfile.66.gz
testfile.67:     66.2% -- replaced with testfile.67.gz
testfile.68:     67.2% -- replaced with testfile.68.gz
testfile.69:     68.2% -- replaced with testfile.69.gz
testfile.70:     69.2% -- replaced with testfile.70.gz
testfile.71:     70.2% -- replaced with testfile.71.gz
testfile.72:     71.2% -- replaced with testfile.72.gz
testfile.73:     72.2% -- replaced with testfile.73.gz
testfile.74:     73.2% -- replaced with testfile.74.gz
testfile.75:     74.3% -- replaced with testfile.75.gz
testfile.76:     75.2% -- replaced with testfile.76.gz
testfile.77:     76.1% -- replaced with testfile.77.gz
testfile.78:     77.2% -- replaced with testfile.78.gz
testfile.79:     78.2% -- replaced with testfile.79.gz
testfile.80:     79.1% -- replaced with testfile.80.gz
testfile.81:     80.1% -- replaced with testfile.81.gz
testfile.82:     81.2% -- replaced with testfile.82.gz
testfile.83:     82.2% -- replaced with testfile.83.gz
testfile.84:     83.1% -- replaced with testfile.84.gz
testfile.85:     84.2% -- replaced with testfile.85.gz
testfile.86:     85.1% -- replaced with testfile.86.gz
testfile.87:     86.1% -- replaced with testfile.87.gz
testfile.88:     87.2% -- replaced with testfile.88.gz
testfile.89:     88.2% -- replaced with testfile.89.gz
testfile.90:     89.2% -- replaced with testfile.90.gz
testfile.91:     90.1% -- replaced with testfile.91.gz
testfile.92:     91.1% -- replaced with testfile.92.gz
testfile.93:     92.0% -- replaced with testfile.93.gz
testfile.94:     93.2% -- replaced with testfile.94.gz
testfile.95:     94.1% -- replaced with testfile.95.gz
testfile.96:     95.0% -- replaced with testfile.96.gz
testfile.97:     96.4% -- replaced with testfile.97.gz
testfile.98:     97.3% -- replaced with testfile.98.gz
testfile.99:     98.5% -- replaced with testfile.99.gz
testfile.100:     99.9% -- replaced with testfile.100.gz

bs=128k

testfile.0:      0.3% -- replaced with testfile.0.gz
testfile.1:      1.3% -- replaced with testfile.1.gz
testfile.2:      2.3% -- replaced with testfile.2.gz
testfile.3:      3.3% -- replaced with testfile.3.gz
testfile.4:      4.2% -- replaced with testfile.4.gz
testfile.5:      5.2% -- replaced with testfile.5.gz
testfile.6:      6.2% -- replaced with testfile.6.gz
testfile.7:      7.2% -- replaced with testfile.7.gz
testfile.8:      8.2% -- replaced with testfile.8.gz
testfile.9:      9.2% -- replaced with testfile.9.gz
testfile.10:     10.2% -- replaced with testfile.10.gz
testfile.11:     11.2% -- replaced with testfile.11.gz
testfile.12:     12.2% -- replaced with testfile.12.gz
testfile.13:     13.2% -- replaced with testfile.13.gz
testfile.14:     14.2% -- replaced with testfile.14.gz
testfile.15:     15.2% -- replaced with testfile.15.gz
testfile.16:     16.1% -- replaced with testfile.16.gz
testfile.17:     17.1% -- replaced with testfile.17.gz
testfile.18:     18.1% -- replaced with testfile.18.gz
testfile.19:     19.1% -- replaced with testfile.19.gz
testfile.20:     20.1% -- replaced with testfile.20.gz
testfile.21:     21.1% -- replaced with testfile.21.gz
testfile.22:     22.1% -- replaced with testfile.22.gz
testfile.23:     23.1% -- replaced with testfile.23.gz
testfile.24:     24.1% -- replaced with testfile.24.gz
testfile.25:     25.1% -- replaced with testfile.25.gz
testfile.26:     26.1% -- replaced with testfile.26.gz
testfile.27:     27.1% -- replaced with testfile.27.gz
testfile.28:     28.1% -- replaced with testfile.28.gz
testfile.29:     29.1% -- replaced with testfile.29.gz
testfile.30:     30.0% -- replaced with testfile.30.gz
testfile.31:     31.0% -- replaced with testfile.31.gz
testfile.32:     32.0% -- replaced with testfile.32.gz
testfile.33:     33.0% -- replaced with testfile.33.gz
testfile.34:     34.0% -- replaced with testfile.34.gz
testfile.35:     35.0% -- replaced with testfile.35.gz
testfile.36:     36.0% -- replaced with testfile.36.gz
testfile.37:     37.0% -- replaced with testfile.37.gz
testfile.38:     38.0% -- replaced with testfile.38.gz
testfile.39:     39.0% -- replaced with testfile.39.gz
testfile.40:     40.0% -- replaced with testfile.40.gz
testfile.41:     41.0% -- replaced with testfile.41.gz
testfile.42:     42.0% -- replaced with testfile.42.gz
testfile.43:     43.0% -- replaced with testfile.43.gz
testfile.44:     44.0% -- replaced with testfile.44.gz
testfile.45:     45.0% -- replaced with testfile.45.gz
testfile.46:     46.0% -- replaced with testfile.46.gz
testfile.47:     47.0% -- replaced with testfile.47.gz
testfile.48:     47.9% -- replaced with testfile.48.gz
testfile.49:     48.9% -- replaced with testfile.49.gz
testfile.50:     49.9% -- replaced with testfile.50.gz
testfile.51:     48.9% -- replaced with testfile.51.gz
testfile.52:     47.9% -- replaced with testfile.52.gz
testfile.53:     47.0% -- replaced with testfile.53.gz
testfile.54:     46.0% -- replaced with testfile.54.gz
testfile.55:     45.0% -- replaced with testfile.55.gz
testfile.56:     44.0% -- replaced with testfile.56.gz
testfile.57:     43.0% -- replaced with testfile.57.gz
testfile.58:     42.0% -- replaced with testfile.58.gz
testfile.59:     41.0% -- replaced with testfile.59.gz
testfile.60:     40.0% -- replaced with testfile.60.gz
testfile.61:     39.0% -- replaced with testfile.61.gz
testfile.62:     38.0% -- replaced with testfile.62.gz
testfile.63:     37.0% -- replaced with testfile.63.gz
testfile.64:     36.0% -- replaced with testfile.64.gz
testfile.65:     35.0% -- replaced with testfile.65.gz
testfile.66:     34.0% -- replaced with testfile.66.gz
testfile.67:     34.0% -- replaced with testfile.67.gz
testfile.68:     36.0% -- replaced with testfile.68.gz
testfile.69:     38.0% -- replaced with testfile.69.gz
testfile.70:     40.0% -- replaced with testfile.70.gz
testfile.71:     42.0% -- replaced with testfile.71.gz
testfile.72:     43.9% -- replaced with testfile.72.gz
testfile.73:     45.9% -- replaced with testfile.73.gz
testfile.74:     47.9% -- replaced with testfile.74.gz
testfile.75:     49.9% -- replaced with testfile.75.gz
testfile.76:     47.9% -- replaced with testfile.76.gz
testfile.77:     45.9% -- replaced with testfile.77.gz
testfile.78:     43.9% -- replaced with testfile.78.gz
testfile.79:     41.9% -- replaced with testfile.79.gz
testfile.80:     40.0% -- replaced with testfile.80.gz
testfile.81:     42.9% -- replaced with testfile.81.gz
testfile.82:     45.9% -- replaced with testfile.82.gz
testfile.83:     48.9% -- replaced with testfile.83.gz
testfile.84:     47.9% -- replaced with testfile.84.gz
testfile.85:     44.9% -- replaced with testfile.85.gz
testfile.86:     43.9% -- replaced with testfile.86.gz
testfile.87:     47.9% -- replaced with testfile.87.gz
testfile.88:     87.4% -- replaced with testfile.88.gz
testfile.89:     88.4% -- replaced with testfile.89.gz
testfile.90:     89.4% -- replaced with testfile.90.gz
testfile.91:     90.4% -- replaced with testfile.91.gz
testfile.92:     91.4% -- replaced with testfile.92.gz
testfile.93:     92.4% -- replaced with testfile.93.gz
testfile.94:     93.4% -- replaced with testfile.94.gz
testfile.95:     94.4% -- replaced with testfile.95.gz
testfile.96:     95.4% -- replaced with testfile.96.gz
testfile.97:     96.4% -- replaced with testfile.97.gz
testfile.98:     97.4% -- replaced with testfile.98.gz
testfile.99:     98.4% -- replaced with testfile.99.gz
testfile.100:     99.9% -- replaced with testfile.100.gz

The issue is that once we get above 32-64k, we slide out of the gzip
window for compression. Fixup the buffer_compress_chunk so that it
switches random states, hence doing the compression evenly over a
more limited range that normal compression will catch.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogettime: get rid of the (unecessary) 10x scaling factor
Jens Axboe [Sat, 21 Nov 2015 00:26:37 +0000 (17:26 -0700)]
gettime: get rid of the (unecessary) 10x scaling factor

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogettime: make cpuclock calibration more tolerant
Jens Axboe [Sat, 21 Nov 2015 00:20:06 +0000 (17:20 -0700)]
gettime: make cpuclock calibration more tolerant

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogettime: print first clock offset for CPU clock
Jens Axboe [Sat, 21 Nov 2015 00:01:16 +0000 (17:01 -0700)]
gettime: print first clock offset for CPU clock

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix ETA calculations
Jens Rosenboom [Thu, 19 Nov 2015 20:36:18 +0000 (13:36 -0700)]
Fix ETA calculations

There are a couple of issues in eta.c:

- The comparison for time- vs. rate-based ETA calculation is done the
  wrong way round.
- The elapsed ramp time should only be subtracted when the thread has
  indeed entered the ramp phase.
- Currently jobs that are stonewalled are not taken into account for ETA
  calculation. This patch at least fixes this for the case that there is
  only a single job per stonewall group.
---

This patch tries to fix some issues I have found, this may
also be related to the ones mentioned earlier in
http://www.spinics.net/lists/fio/msg04187.html

The part regarding stonewalled jobs may require some further work, but
at least it gives proper values now for my standard fio runs containing
just a sequence of singular jobs.

Also, I'm still wondering about the intention behind the way done_secs
is meant to work. It cumulates time spent in finished jobs globally, but
is being used in per-job ETA calculations. This seems to break at
least for the case of stonewalled jobs, too.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't output json headers for fio backend
Jens Axboe [Thu, 19 Nov 2015 18:26:21 +0000 (11:26 -0700)]
Don't output json headers for fio backend

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix group_id != groupid mixup
Jens Axboe [Mon, 16 Nov 2015 21:00:41 +0000 (14:00 -0700)]
Fix group_id != groupid mixup

Also make sure we don't overflow 'groupid'

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix error when compiled as c++
Casey Bodley [Wed, 4 Nov 2015 16:03:11 +0000 (11:03 -0500)]
iolog: fix error when compiled as c++

Given the main.cc source file:
  #include <fio.h>
  int main() { return 0; }

And the gcc command line:
  gcc main.cc -Ifio

In file included from fio/stat.h:4:0,
                 from fio/thread_options.h:7,
                 from fio/fio.h:18,
                 from main.cc:1:
fio/iolog.h: In function 'io_sample* __get_sample(void*, int,
uint64_t)':
fio/iolog.h:125:53: warning: pointer of type 'void *' used in arithmetic
[-Wpointer-arith]
  return samples + sample * __log_entry_sz(log_offset);
                                                     ^
fio/iolog.h:125:17: error: invalid conversion from 'void*' to
'io_sample*' [-fpermissive]
  return samples + sample * __log_entry_sz(log_offset);

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 years agojson output fixups
Vincent Fu [Wed, 11 Nov 2015 18:26:37 +0000 (11:26 -0700)]
json output fixups

- diskutil json+ fixup

- idletime missed a conversion for the multiple output formats

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix verification error with bsrange and split read/write phase
Jens Axboe [Wed, 11 Nov 2015 01:13:40 +0000 (18:13 -0700)]
Fix verification error with bsrange and split read/write phase

If you have separate phases for the verify writes and the verify reads,
and fio ends up having to align to the verify interval header size,
then we can end up trimming the buflen on the read side (where
do_verify=1 is set), but not on the write side. This causes a
verification failure. Fix this by always aligning, regardless of
whether do_verify is true or not.

Example job files:

-------------------------------------------------------------------------------
[global]
thread=1
blocksize_range=4k-256k
ioengine=libaio
verify_interval=512
iodepth=64
loops=1
verify_pattern=0x03715998
size=100MB
offset=0
verify_dump=1
filename=/tmp/disk.img

[write-phase]
rw=randwrite
fill_device=1
do_verify=0
-------------------------------------------------------------------------------
[global]
thread=1
blocksize_range=4k-256k
ioengine=libaio
verify_interval=512
iodepth=64
loops=1
verify_pattern=0x03715998
size=100MB
offset=0
verify_dump=1
filename=/tmp/disk.img

[verify-phase]
stonewall
rw=randread
create_serialize=0
do_verify=1
-------------------------------------------------------------------------------

Reported-by: Kevin Vajk <kmv@datera.io>
Fixes: a9f70b1f5087 ("Make sure io_u->buflen is aligned to the verify_interval")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosmalloc: get rid of global lock
Jens Axboe [Tue, 10 Nov 2015 02:38:15 +0000 (19:38 -0700)]
smalloc: get rid of global lock

We don't dynamically add pools anymore, so we don't need global
read/write locks.

Also drop the needless wrappers around pool locking.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoClarify spread/lambda of poisson
Jens Axboe [Tue, 10 Nov 2015 01:45:35 +0000 (18:45 -0700)]
Clarify spread/lambda of poisson

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRename rate_poisson to rate_process
Jens Axboe [Mon, 9 Nov 2015 23:37:33 +0000 (16:37 -0700)]
Rename rate_poisson to rate_process

This will more accurately follow what other options could adopt,
for instance if we want to do a dynamic IO depth.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoProvide some consistency in rate_* options
Jens Axboe [Mon, 9 Nov 2015 23:05:19 +0000 (16:05 -0700)]
Provide some consistency in rate_* options

ratemin and ratecycle are the only ones without an underscore.
Change them to rate_min and rate_cycle, but retain the old
names as an alias for compat reasons.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoIn fio.1 and HOWTO, add link to Poisson process in wikipedia
Song Liu [Mon, 9 Nov 2015 22:32:14 +0000 (14:32 -0800)]
In fio.1 and HOWTO, add link to Poisson process in wikipedia

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoadd example using --rate_poisson
Song Liu [Mon, 9 Nov 2015 22:32:13 +0000 (14:32 -0800)]
add example using --rate_poisson

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agomake sure __rand_0_1 does not return 0.0
Song Liu [Mon, 9 Nov 2015 22:32:12 +0000 (14:32 -0800)]
make sure __rand_0_1 does not return 0.0

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd poisson rate selection to FD_RATE output
Jens Axboe [Mon, 9 Nov 2015 22:07:14 +0000 (15:07 -0700)]
Add poisson rate selection to FD_RATE output

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFixups for poisson rate
Jens Axboe [Mon, 9 Nov 2015 19:54:37 +0000 (12:54 -0700)]
Fixups for poisson rate

- Change name to poisson_rate, we could add other uses for this
  functionality (like depth, etc).

- Add "rate" as option parent.

- Ensure we convert the option for client/server, bump protocol
  version.

- Add a separate rand state for poisson, so it becomes randomly
  controllable (through randrepeat).

- Add random 0..1 helper.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoEnable request flow under Poisson process
Song Liu [Mon, 9 Nov 2015 19:34:56 +0000 (12:34 -0700)]
Enable request flow under Poisson process

To better mimic real world wordload, where incoming requests follow
Poisson process, this patch enables poisson process request flow
for rate limited benchmarks.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosmalloc: only clear the bitmap, not the whole pool
Jens Axboe [Sun, 8 Nov 2015 00:33:38 +0000 (17:33 -0700)]
smalloc: only clear the bitmap, not the whole pool

This reduces startup time, and the memset() isn't useful nor
needed on the full range.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix spelling error in engines/rdma.c
Martin Steigerwald [Sun, 8 Nov 2015 00:30:22 +0000 (17:30 -0700)]
Fix spelling error in engines/rdma.c

> lintian -IE fio_2.2.10-1_amd64.changes
I: fio: spelling-error-in-binary usr/bin/fio
Recieve Receive

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd latency bin output to the json output format
Vincent Fu [Fri, 6 Nov 2015 15:46:44 +0000 (08:46 -0700)]
Add latency bin output to the json output format

We add a new format for this, json+, as it generates a lot of data.
As of now, the only difference between json and json+ is the full
dump of all latency bins.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.2.11 fio-2.2.11
Jens Axboe [Thu, 5 Nov 2015 16:05:15 +0000 (09:05 -0700)]
Fio 2.2.11

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooutput_buffer: only realloc once, and memset just what we need
Jens Axboe [Sat, 31 Oct 2015 01:34:21 +0000 (19:34 -0600)]
output_buffer: only realloc once, and memset just what we need

No need looping around a realloc(), just alloc what we need upfront.
Additionally, don't memset the parts we'll be copying into anyway.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoBump the client ETA timeout from 5s to 30s
Jens Axboe [Fri, 30 Oct 2015 02:52:23 +0000 (11:52 +0900)]
Bump the client ETA timeout from 5s to 30s

5 seconds is low enough that we sometimes falsely trigger the timeout.
It's meant to catch clients that have disappeared, so 5 seconds is
arguably way too low. Bump it to 30 seconds instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUpdate documentation for mmapshared (MMAP_SHARED)
Jens Axboe [Wed, 28 Oct 2015 05:04:24 +0000 (14:04 +0900)]
Update documentation for mmapshared (MMAP_SHARED)

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFixup bw/iops logging for short runs
Jens Axboe [Wed, 28 Oct 2015 00:29:33 +0000 (09:29 +0900)]
Fixup bw/iops logging for short runs

If an IO loop lasts less than the window for bw or iops logging,
we won't log any entries. Make that persist across loops.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'mmap_shared' of git://github.com/lsgunth/fio
Jens Axboe [Tue, 27 Oct 2015 21:27:41 +0000 (06:27 +0900)]
Merge branch 'mmap_shared' of git://github.com/lsgunth/fio

3 years agoAdd mmapshared option to use mmaped files with the MAP_SHARED flag.
Logan Gunthorpe [Tue, 27 Oct 2015 16:20:53 +0000 (10:20 -0600)]
Add mmapshared option to use mmaped files with the MAP_SHARED flag.

This is primarily useful for benchmarking DAX files. If they are mmaped
with the MAP_PRIVATE flag they will still have copy on write semantics
and the test will not actually run against the memory that backs the
DAX file. This memory option allows tests to be constructed that use the
MAP_SHARED flag which allows running directly with the memory backed
by a DAX file.

3 years agoFixup -Wshadow warnings
Jens Axboe [Tue, 27 Oct 2015 12:30:40 +0000 (21:30 +0900)]
Fixup -Wshadow warnings

Usually harmless, but some of them we can actually kill
variables.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient/backend: fix incomplete output_format checks
Jens Axboe [Tue, 27 Oct 2015 09:34:33 +0000 (18:34 +0900)]
client/backend: fix incomplete output_format checks

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: fix potential segfault for json output on network client
Jens Axboe [Tue, 27 Oct 2015 09:33:23 +0000 (18:33 +0900)]
stat: fix potential segfault for json output on network client

Fixes: b01af66b13f0 ("add eta and elapsed to root of json output")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: collapse buf output flush and free
Jens Axboe [Tue, 27 Oct 2015 09:16:12 +0000 (18:16 +0900)]
stat: collapse buf output flush and free

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUnbreak output buffer logging over the network
Jens Axboe [Tue, 27 Oct 2015 08:44:01 +0000 (17:44 +0900)]
Unbreak output buffer logging over the network

log_local_buf() is for the server to log locally, use
log_info_buf() to log over the specified channel.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: abstract out and use generic buffer logger
Jens Axboe [Tue, 27 Oct 2015 08:43:32 +0000 (17:43 +0900)]
log: abstract out and use generic buffer logger

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agojobs_eta alloc padding
Jens Axboe [Tue, 27 Oct 2015 07:47:35 +0000 (16:47 +0900)]
jobs_eta alloc padding

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't setup output buffers we are not going to use
Jens Axboe [Tue, 27 Oct 2015 07:43:44 +0000 (16:43 +0900)]
Don't setup output buffers we are not going to use

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoProper size return from output buffers
Jens Axboe [Tue, 27 Oct 2015 07:33:43 +0000 (16:33 +0900)]
Proper size return from output buffers

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRe-instate --append-terse
Jens Axboe [Tue, 27 Oct 2015 07:25:02 +0000 (16:25 +0900)]
Re-instate --append-terse

This wasn't well documented, but ensure that we don't people's
setup. The new way of doing multiple formats is to add them
to the output format. Which means that this:

fio --append-terse --output-format=normal

is now equivalent to

fio --output-format=normal,terse

Please use the latter.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUpdate fio man page
Jens Axboe [Tue, 27 Oct 2015 05:02:06 +0000 (14:02 +0900)]
Update fio man page

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoEnable the use of multiple output formats
Jens Axboe [Tue, 27 Oct 2015 04:45:21 +0000 (13:45 +0900)]
Enable the use of multiple output formats

You can now do

./fio --output-format=normal,json jobfile.fio

and get both types of output.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd support for multiple output formats
Jens Axboe [Sun, 25 Oct 2015 02:55:34 +0000 (11:55 +0900)]
Add support for multiple output formats

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient: remove duplicated code
Jens Axboe [Tue, 20 Oct 2015 15:07:31 +0000 (09:07 -0600)]
client: remove duplicated code

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostrlcat: fix header guard typo
Jens Axboe [Tue, 20 Oct 2015 04:37:39 +0000 (22:37 -0600)]
strlcat: fix header guard typo

FIO_STRLCAT_h -> FIO_STRLCAT_H

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: add faster 'memory is all zeroes' helper
Jens Axboe [Tue, 20 Oct 2015 03:53:13 +0000 (21:53 -0600)]
verify: add faster 'memory is all zeroes' helper

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoengines/rbd: potential re-use of active fio_rbd_iou
Jason Dillaman [Tue, 13 Oct 2015 19:40:47 +0000 (15:40 -0400)]
engines/rbd: potential re-use of active fio_rbd_iou

Running multiple concurrent RBD jobs can result in a segfault
attempting to retrieve the RBD AIO completion status from a
prematurely reused fio_rbd_iou object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
3 years agoFix Windows build by using lib/strcasestr.{c,h}
Bruce Cran [Tue, 6 Oct 2015 00:18:50 +0000 (18:18 -0600)]
Fix Windows build by using lib/strcasestr.{c,h}

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: bump protocol version
Jens Axboe [Thu, 1 Oct 2015 06:55:18 +0000 (08:55 +0200)]
server: bump protocol version

Commit 82407585a3b3 added options to the thread_options structures,
so we need to bump the client/server protocol version to signal
we are incompatible with previous versions.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoIntroduce new option: iodepth_batch_complete_max
Roman Pen [Sun, 27 Sep 2015 19:24:55 +0000 (21:24 +0200)]
Introduce new option: iodepth_batch_complete_max

By now there is an option which is called 'iodepth_batch_complete'.
This option provides a possibility to read exact amount of events
of completed IOs, e.g. by default it is set to 1, so only 1 event
will be read on each io_getevents system call:

 7945  io_getevents(139876985688064, 1, 1, {...}NULL) = 1

This option can be set to any desired amount, say the whole iodepth
size can be used, e.g. 128:

 7952  io_getevents(140303500259328, 128, 128, {...}NULL) = 128

We will not exit the system call till the whole queue will be completed.
Sounds not so efficient.

In this patch I add the 'iodepth_batch_complete_max' option and rename
'iodepth_batch_complete' to 'iodepth_batch_complete_min' to have a
possibility to specify a range of events which I want to read.
(of course because of compatibility's sake 'iodepth_batch_complete_min'
 is an alias for 'iodepth_batch_complete').

So finally if options are set to:

 # or iodepth_batch_complete_min=1
 iodepth_batch_complete=1
 # take the iodepth value
 iodepth_batch_complete_max=128

The io_getevents call will look like:

 7961  io_getevents(140173245206528, 1, 128, {...}NULL) = 73

and we will exit the system call with any amount of completed
IOs >= 1.

What are the benefits? Fetching events from the queue can be
done more efficiently on some testing on specific configurations,
e.g. stacked block devices based on null_blk devices, where
completion happens immediately and events should be retried from
the queue ASAP.

Let's take a simple example.

BASE CONFIG:

[global]
fadvise_hint=0
rw=randrw:2
direct=1
size=256M
ioengine=libaio
iodepth=128

time_based=1
runtime=60
ramp_time=10

[job1]
filename=/dev/nullb0

[job2]
filename=/dev/nullb1

[job3]
filename=/dev/nullb2

[job4]
filename=/dev/nullb3

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio.1,HOWTO: keep 'iodepth_batch' option in sync
Roman Pen [Sun, 27 Sep 2015 19:24:54 +0000 (21:24 +0200)]
fio.1,HOWTO: keep 'iodepth_batch' option in sync

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Cc: fio@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: fix short reads
Jens Axboe [Wed, 30 Sep 2015 16:31:12 +0000 (18:31 +0200)]
sg: fix short reads

We can't assume that if we read some data, we read all of it.
Check for this condition.

Don't error out if a read failed, but we already got valid
events.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse line buffering on stdout
Kris Davis [Wed, 30 Sep 2015 16:00:00 +0000 (18:00 +0200)]
Use line buffering on stdout

Line Buffer stdio to prevent multi-thread output from
mixing within a line

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosg: 16-byte cdb support and lots of fixes
Kris Davis [Wed, 30 Sep 2015 15:57:16 +0000 (17:57 +0200)]
sg: 16-byte cdb support and lots of fixes

-- Fixed prob with queue depths > 1 (usually would work at qd = 2,
   but had fundimental prob with retrieving multiple events)
   Only works up to QD 16 due to kernel limitation.

-- Fix problem with check and report scsi operation errors.
   There was no check, so op errors were being ignored.
   Added support to retrieve and print sense data to output.

-- Fix problem with op timeouts. No timeouts op time out was being set.
   Hardcoded to 30 seconds.

-- Support switch to 16 Byte operations when LBA range needed.

-- Set default size based on maxlba of device retrieved
   from ReadCapacity command.

Updated by Jens to:

- ->errdetails() must be reentrant. Return an allocated buffer, have
  the caller free it when done.

- Bump io engine version number, the ops have changed.

- Add strlcat() as a real helper, don't stuff it in sg.c

- Lots of style cleanups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix integer overflow in rate_iops
Jens Axboe [Fri, 25 Sep 2015 02:35:44 +0000 (20:35 -0600)]
Fix integer overflow in rate_iops

For the following job file:

[bla]
rw=randread
bs=1024m
rate_iops=100

We end up overflowing in multiplication, making the bps field 0.
Fio exits by reporting:

rate lower than supported

Fix this by casting to uint64_t, the type of the output.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUpdate fio2gnuplot
Thomas Findelkind [Thu, 17 Sep 2015 17:42:34 +0000 (19:42 +0200)]
Update fio2gnuplot

Should fix the "No log file..." with default pattern -b and -i and
 user generated a log file per thread/numjob with per_numjobs_log=1

4 years agoMerge branch 'patch-1' of git://github.com/Tfindelkind/fio
Jens Axboe [Wed, 16 Sep 2015 21:43:28 +0000 (15:43 -0600)]
Merge branch 'patch-1' of git://github.com/Tfindelkind/fio

4 years agoUpdate fio2gnuplot
Tfindelkind [Wed, 16 Sep 2015 21:38:12 +0000 (23:38 +0200)]
Update fio2gnuplot

4 years agoReduced minimum for option log_gz
Stephen Bates [Tue, 15 Sep 2015 21:37:53 +0000 (15:37 -0600)]
Reduced minimum for option log_gz

Reduced the permissable minimium for log_sz from 32MB to 1KB to avoid
a latency log issue associated with the reallocation of the buffer logs.

4 years agoFio 2.2.10 fio-2.2.10
Jens Axboe [Fri, 11 Sep 2015 21:13:37 +0000 (15:13 -0600)]
Fio 2.2.10

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoworkqueue: Ensure submission offload worker inherits parent's options
Andrey Kuzmin [Tue, 8 Sep 2015 15:10:55 +0000 (09:10 -0600)]
workqueue: Ensure submission offload worker inherits parent's options

Properly initialize sibling's eo, as otherwise it will miss parent's
options.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: move meta header to generic verify_header
Roman Pen [Sat, 5 Sep 2015 17:01:02 +0000 (19:01 +0200)]
verify: move meta header to generic verify_header

fio does not provide any possibility to verify checksum of a block with meta
information inside. You can create configuration for verifincation checksum
of random data either you can verify meta information with some pattern or
random data, but not both.

Why checksumming and meta together can be useful? Meta helps to figure out internally
on filesystem or storage what block was written in case of corruption, i.e. offset
of the block and block number explicitly tell us the virtual address of the block.
On the other hand checksum of random data helps to detect corruption. Using meta
and pattern together do not help a lot, since 'verify_interval' can be big enough
and same sequence of pattern bytes will be undistinguishable internally on filesystem
or storage.

Also, it seems to me that keeping meta header separately from generic verify header
does not make a lot of sense, since generic verify header can include all members
of meta header without any performance or other impact.

In this patch I move all members from vhdr_meta structure to generic verify_header,
always verifying meta with the possiblity to checksum the following data: random
or pattern.

You are allowed to specify verify_pattern=str with any of the possible verification
methods and have also meta verification, i.e.

   verify=md5
   verify_pattern=0xfe

 or

   verify=sha1
   verify_pattern=0xff

 etc.

To keep everything compatible with old configurations it is still possible to specify

   verify=meta

but this option marked and depricated and kept only for compatibility reasons.

Before that patch the verification layout according to the specified options looks
as the following, e.g.:

 #1
    --
    verify=meta
    verify_pattern=0xff
    --

    result layout of each block: [hdr|meta|pattern]

 #2
    --
    verify_pattern=0xff
    --

    result layout of each block: [hdr|pattern]

 #3
    --
    verify=pattern
    verify_pattern=0xff
    --

    result layout of each block: [pattern]

After applying of the patch 'vhdr_meta' is always embedded into 'verify_header' and layout
looks as the following, e.g.:

 #1
    --
    verify=meta
    verify_pattern=0xff
    --

    result layout of each block: [hdr+meta|pattern]
 #2
    --
    verify=md5|sha1|etc
    verify_pattern=0xff
    --

    result layout of each block: [hdr+meta|cksum|pattern]

 #3
    --
    verify_pattern=0xff
    --

    result layout of each block: [hdr+meta|pattern]

 #4
    --
    verify=pattern
    verify_pattern=0xff
    --

    result layout of each block: [pattern]

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Cc: fio@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofio.1: update description for 'buffer_pattern' and 'verify_pattern'
Roman Pen [Sat, 5 Sep 2015 17:01:01 +0000 (19:01 +0200)]
fio.1: update description for 'buffer_pattern' and 'verify_pattern'

Now those two support combined input and 'verify_pattern' supports '%o' format.
Update corresponding doc.

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Cc: fio@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agouse 'lib/pattern' to parse patterns and paste formats into buffers
Roman Pen [Wed, 19 Aug 2015 10:33:11 +0000 (12:33 +0200)]
use 'lib/pattern' to parse patterns and paste formats into buffers

Switch to new 'lib/pattern' helpers.

Now 'buffer_pattern' and 'verify_pattern' options support combined input
and strings like:

 buffer_pattern="1234"0xface"5678"
 verify_pattern=0xface"1234"-12+14

can be specified.

Also, 'verify_pattern' supports '%o' format, which means that buffer will
be patched on each iteration with real 'block offset' number.

So, f.e. 'verify_pattern' such combined input is valid:

 verify_pattern=%o

 or

 verify_pattern=%o"123"%o

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoadd FIELD_SIZE macro to calculate the size of the specified field
Roman Pen [Wed, 19 Aug 2015 10:33:10 +0000 (12:33 +0200)]
add FIELD_SIZE macro to calculate the size of the specified field

Will use this stuff later.

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: use 'cmp_pattern' from 'lib/pattern.c' to compare pattern and buffer
Roman Pen [Wed, 19 Aug 2015 10:33:09 +0000 (12:33 +0200)]
verify: use 'cmp_pattern' from 'lib/pattern.c' to compare pattern and buffer

'cmp_pattern' function should be faster, since it does not use loops
and while doing 'memcmp' the same CPU cache line is used.

Keep in mind, that now single-byte pattern becomes the string of 512 bytes.
In that case this optimization will not work as expected, since 'memcmp'
will be called twice for the whole pattern, but in next patches this
behaviour will be avoided and single-byte pattern will stay single-byte
pattern without any attempt to duplicate it.

Also print buffer and pattern bytes as unsigned chars.

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoreplace 'fill_pattern' with 'cpy_pattern' from 'lib/pattern.c'
Roman Pen [Wed, 19 Aug 2015 10:33:08 +0000 (12:33 +0200)]
replace 'fill_pattern' with 'cpy_pattern' from 'lib/pattern.c'

All pattern helpers are now located inside 'lib/pattern.c',
and 'cpy_pattern' should be sligtly faster, since it does
copy doubling the size on every iteration.

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolib/pattern: add set of functions to parse combined pattern input
Roman Pen [Wed, 19 Aug 2015 10:33:07 +0000 (12:33 +0200)]
lib/pattern: add set of functions to parse combined pattern input

The idea of this patch is to have several helpers to parse combined
pattern input, which can consists of strings, numbers and formats.

For example now pattern can be combined and look like this string:

 input : 0xdeadbeef"123"0xdeadface
 output: de ad be ef 31 32 33 de ad fa ce

  or

 input : -99"some string"0x66
 output: 9d ff ff ff 73 6f 6d 65 20 73 74 72 69 6e 67 66

  or with formats

 input : 0xdeadface0xffff%o
 output: de ad fa ce ff ff 00 00 00 00 00 00 00 00

 where %o - offset of the block, reserved 8 bytes

 Space for formats is reserved in output buffer.
 When buffer will be ready to be written to disk - 'paste' callback
 should be called for each pattern format. Each callback is responsible
 for writing data inside the reserved space in the output buffer.

Format array can be extended at any time, all you need is to provide
the name of the format and correct 'paste' callback.

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolib/strntol: add 'strntol' function
Roman Pen [Wed, 19 Aug 2015 10:33:06 +0000 (12:33 +0200)]
lib/strntol: add 'strntol' function

In future patches I will need it.

Signed-off-by: Roman Pen <r.peniaev@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRetry bdev cache invalidation for EAGAIN errors
Ken Raeburn [Fri, 4 Sep 2015 19:30:58 +0000 (13:30 -0600)]
Retry bdev cache invalidation for EAGAIN errors

I've hit a couple test failures where fio quickly died thusly:

    fio 2.0.7
    Starting 1 thread
    fio: pid=6196, err=11/file:filesetup.c:404, func=invalidate_cache, error=Resource temporarily unavailable

(Yeah, we're a little behind. But I think the relevant code is similar.)

We're testing against Linux (RHEL 6.6) multipath block devices, and it
appears that at the time the fio test starts, the multipath maps for
those devices may still be in flux. I poked around a bit with systemtap,
and it appears that while updating the maps, multipathd "suspends" its
multipath device for a few tens of milliseconds, and the kernel
multipath code rejects ioctl calls with EAGAIN if the device is
suspended. It's a small window, but we've managed to hit it multiple
times, though it's not reliably reproducible.

I know the current sources treat failure here as non-fatal, but if we're
using fio for performance tests, trying a little harder to do the
invalidation seems like a good idea. My approach is to add a retry loop
in __file_invalidate_cache; a patch is attached.

It could also be pushed down into blockdev_invalidate_cache, where it
could be local to the Linux (and Android?) implementation, since none of
the others actually do anything. (They return either EAGAIN or zero,
both of which are taken as success indicators.)

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: fix comparison of 'ret' pointer
Andrey Kuzmin [Fri, 4 Sep 2015 19:23:49 +0000 (13:23 -0600)]
backend: fix comparison of 'ret' pointer

Commit e9d512d8e7d2 abstracted out this code, but the assert()
was never changed.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix aux_path for verify state saving
Jens Axboe [Fri, 21 Aug 2015 18:09:03 +0000 (11:09 -0700)]
Fix aux_path for verify state saving

Botched the string copies.

Fixes: d264264a08 ("Add --aux-path support"
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoparse: only print option mismatch help, if the distance is close enough
Jens Axboe [Fri, 21 Aug 2015 18:01:29 +0000 (11:01 -0700)]
parse: only print option mismatch help, if the distance is close enough

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd --aux-path support
Jens Axboe [Thu, 20 Aug 2015 19:02:05 +0000 (12:02 -0700)]
Add --aux-path support

For certain files, fio just stores them in the current working
directory. This can create a bit of a mess. These files might
be verification dump files, or verification state saves.

Add --aux-path to enable storing these files in an arbitrary
directory instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoREADME: update for trigger options
Jens Axboe [Thu, 20 Aug 2015 18:59:08 +0000 (11:59 -0700)]
README: update for trigger options

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoconfigure: cleanup strsep() test case
Jens Axboe [Tue, 18 Aug 2015 18:26:00 +0000 (11:26 -0700)]
configure: cleanup strsep() test case

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoconfigure: fixup using wrong var for endmntent()
Jens Axboe [Tue, 18 Aug 2015 18:23:00 +0000 (11:23 -0700)]
configure: fixup using wrong var for endmntent()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'master' of git://github.com/DaveGlen/fio
Jens Axboe [Tue, 18 Aug 2015 17:19:50 +0000 (11:19 -0600)]
Merge branch 'master' of git://github.com/DaveGlen/fio

4 years agoCorrect handleing of rate_iops_min and ratemin
Dave [Tue, 18 Aug 2015 16:39:11 +0000 (10:39 -0600)]
Correct handleing of rate_iops_min and ratemin

change to allow ratemin to be set without rate disabled ratemin checking in __check_min_rate.

Separate from the change to ratemin parsing, rate_iops_min check did not result in "return 1;" so jobs would not terminate on a failure to maintain rate_iops_min.

4 years agomtd: expand configure check
Jens Axboe [Mon, 17 Aug 2015 21:47:42 +0000 (15:47 -0600)]
mtd: expand configure check

SLES 11.4 has enough of mtd to pass the configure checks, but not
to compile it. Expand the checks to hopefully be enough.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogettime: silence bogus gcc warnings
Jens Axboe [Mon, 17 Aug 2015 21:46:22 +0000 (15:46 -0600)]
gettime: silence bogus gcc warnings

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'master' of git://github.com/DaveGlen/fio
Jens Axboe [Mon, 17 Aug 2015 21:39:23 +0000 (15:39 -0600)]
Merge branch 'master' of git://github.com/DaveGlen/fio

4 years agoallow for ratemin to be used without setting rate
DaveGlen [Mon, 17 Aug 2015 16:04:09 +0000 (10:04 -0600)]
allow for ratemin to be used without setting rate

Only check if ratemin < rate when rate is set.

4 years agofilesetup: make random generator block auto-switch on huge files
Jens Axboe [Fri, 14 Aug 2015 18:25:20 +0000 (12:25 -0600)]
filesetup: make random generator block auto-switch on huge files

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoclient: make it explicit that we don't reuse 'eta' after freeing it
Jens Axboe [Fri, 14 Aug 2015 15:45:46 +0000 (09:45 -0600)]
client: make it explicit that we don't reuse 'eta' after freeing it

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoeta: ensure we include terminating 0 space in je->run_str[]
Jens Axboe [Fri, 14 Aug 2015 15:42:55 +0000 (09:42 -0600)]
eta: ensure we include terminating 0 space in je->run_str[]

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoarch-ppc.h: Add ilog2 implementation for ppc64
Chandan Rajendra [Tue, 11 Aug 2015 09:41:26 +0000 (15:11 +0530)]
arch-ppc.h: Add ilog2 implementation for ppc64

On a ppc64 machine, when fio is invoked with the following configuration file
(generated by fstests' generic/300 test), it loops indefinitely.

[global]
directory=/mnt/btrfs-xfstest-scratch
filesize=536870912
size=999G
continue_on_error=write
ignore_error=,ENOSPC
error_dump=0

create_on_open=1
fallocate=none
exitall=1

[direct_aio_raicer]
ioengine=libaio
iodepth=128*1
bs=128k
direct=1
numjobs=4
rw=randwrite
runtime=100*1
time_based
filename=racer

[falloc_raicer]
ioengine=falloc
runtime=100*1
iodepth=1
bssplit=128k/80:512k/10:32k/10
rw=randwrite
numjobs=1
filename=racer

[punch_hole_raicer]
ioengine=falloc
runtime=100*1
bs=4k
time_based=10
rw=randtrim
numjobs=2
filename=racer
time_based

[aio-dio-verifier]
ioengine=libaio
iodepth=128*1
numjobs=1
verify=crc32c-intel
verify_fatal=1
verify_dump=1
verify_backlog=1024
verify_async=4
verifysort=1
direct=1
bs=4k
rw=randwrite
filename=aio-dio-verifier

This is because arch_ffz() ends up invoking the 32-bit version of __ilog2()
which always returns a bit number in the range 0 - 31. This can cause
"overlap" variable in axmap_set_fn() to never becomes zero.

To fix this the commit adds a 64-bit version of __ilog2() (obtained from the
Linux kernel's arch/powerpc/include/asm/bitops.h).

Signed-off-by: Chandan Rajendra <chandan@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoWhitespace fixup
Jens Axboe [Thu, 13 Aug 2015 15:31:17 +0000 (09:31 -0600)]
Whitespace fixup

Introduced by commit 50a8ce864e2c.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoinitialize rate_io_issue_bytes
DaveGlen [Mon, 10 Aug 2015 19:39:31 +0000 (13:39 -0600)]
initialize rate_io_issue_bytes

4 years agoImplement new Rate Control
DaveGlen [Mon, 10 Aug 2015 18:47:53 +0000 (12:47 -0600)]
Implement new Rate Control

Current implementation of rate control has the potential for bursts and
stalls when iodepth>1 due to feedback delay while IO is pending.  more
description here:
https://docs.google.com/drawings/d/1EG-9eGlNvw-m9m0wMSb_C_lyJcaPlhEFbkHRsVpt4wo/edit?usp=sharing

This commit changes the rate control mechanisms to use feed forward io
issues for rate feedback. Moving the rate control on submissions
instead of completions eliminates feedback delay. More details on the
change here:
https://docs.google.com/drawings/d/1NphdZGjYGuOLWJzvXHv44zuy0nnSunvFrROCVfA67Y8/edit?usp=sharing

4 years agoserver: make the setsockopt() error output a bit more informative
Jens Axboe [Wed, 29 Jul 2015 15:00:03 +0000 (09:00 -0600)]
server: make the setsockopt() error output a bit more informative

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoconfigure: add --disable-optimizations
Jens Axboe [Mon, 27 Jul 2015 21:10:00 +0000 (15:10 -0600)]
configure: add --disable-optimizations

Sometimes we want to build without optimizations enabled for
debugging issues, make this possible without editing the
Makefile.

Also fixup --disable-shm to not output a symbol until after
we have initialized config-host.*.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix potential divide-by-zero in calc_iops()
Jens Axboe [Mon, 27 Jul 2015 18:29:55 +0000 (12:29 -0600)]
Fix potential divide-by-zero in calc_iops()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix potential divide-by-zero in calc_rate()
Jens Axboe [Mon, 27 Jul 2015 18:26:12 +0000 (12:26 -0600)]
Fix potential divide-by-zero in calc_rate()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoconfigure: make __sync_fetch_and_add() test use 64-bit types
Jens Axboe [Mon, 27 Jul 2015 18:22:14 +0000 (12:22 -0600)]
configure: make __sync_fetch_and_add() test use 64-bit types

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolibfio: add runstate names
Jens Axboe [Mon, 27 Jul 2015 18:08:33 +0000 (12:08 -0600)]
libfio: add runstate names

Enables --debug=process to show the actual transitions, instead of
numbers you have to look up in the source:

process  12113 pid=12123: runstate CREATED -> INITIALIZED

instead of

process  12113 pid=12123: runstate 1 -> 2

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRe-seed random generator correctly between loops
Dave [Thu, 16 Jul 2015 20:56:00 +0000 (14:56 -0600)]
Re-seed random generator correctly between loops

re-seeding random generator in this way means that random patterns will
precisely repeat between iterations.  if rand_repeatable is set, this is
desirable, otherwise it may produce unexpected IO patterns in random IO
testing.

Signed-off-by: Jens Axboe <axboe@fb.com>