fio.git
4 years agoMerge branch 'master' of https://github.com/davidzengxhsh/fio
Jens Axboe [Fri, 20 May 2016 17:22:37 +0000 (11:22 -0600)]
Merge branch 'master' of https://github.com/davidzengxhsh/fio

4 years agoiolog: remove dead define
Jens Axboe [Fri, 20 May 2016 17:17:37 +0000 (11:17 -0600)]
iolog: remove dead define

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: regrow log out-of-line
Jens Axboe [Fri, 20 May 2016 17:02:28 +0000 (11:02 -0600)]
iolog: regrow log out-of-line

From the completion side, when we run out of entries, store in a
temporary location that we size at log init time. Mark 'td' as
needing log regrow, and handle that on the submission side. We
can't easily quiesce and grow logs at completion time, since
it'll potentially recurse. On top of that, the IO engines rely
on the fact that we serialize getevents/completions, we can't
nest them.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoThe fixed CPU architecture in the Makefile will make failure on ppc64le.
David Zeng [Fri, 20 May 2016 09:41:14 +0000 (17:41 +0800)]
The fixed CPU architecture in the Makefile will make failure on ppc64le.

4 years agoiolog: fix duplicate handling of compression end
Jens Axboe [Thu, 19 May 2016 21:49:57 +0000 (15:49 -0600)]
iolog: fix duplicate handling of compression end

Also handle an error return from deflate() with Z_FINISH.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: fix bug with ret != Z_STREAM_END
Jens Axboe [Thu, 19 May 2016 21:35:18 +0000 (15:35 -0600)]
iolog: fix bug with ret != Z_STREAM_END

Before adding more chunks, adjust the previous chunk size. This
fixes an inflate issue, where we'd get Z_DATA_ERROR because the
middle chunk did not have the correct length.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: more compression debugging/fixes
Jens Axboe [Thu, 19 May 2016 21:00:21 +0000 (15:00 -0600)]
iolog: more compression debugging/fixes

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: sum last chunk length to total
Jens Axboe [Thu, 19 May 2016 20:36:21 +0000 (14:36 -0600)]
iolog: sum last chunk length to total

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: memset() zstream at init time
Jens Axboe [Thu, 19 May 2016 18:44:06 +0000 (12:44 -0600)]
iolog: memset() zstream at init time

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: fix missing new-line in inflate debug statement
Jens Axboe [Thu, 19 May 2016 18:43:19 +0000 (12:43 -0600)]
iolog: fix missing new-line in inflate debug statement

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: move iolog compression init before CPU affinity settings
Jens Axboe [Thu, 19 May 2016 17:41:28 +0000 (11:41 -0600)]
backend: move iolog compression init before CPU affinity settings

Previously we did this after we had set CPU affinity for the IO
worker. But if we do that, then we limit the log compression
to the exact CPUs that we don't want it to run on.

So let's do this earlier, which means the compression threads can
run anyway. We'll trust the OS scheduler to keep them out of our
way.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: don't quiesce on completion
Jens Axboe [Thu, 19 May 2016 17:24:30 +0000 (11:24 -0600)]
iolog: don't quiesce on completion

We can infinitely recurse if we do that, if log_compression= is set.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: switch to list based scheme
Jens Axboe [Thu, 19 May 2016 15:20:16 +0000 (09:20 -0600)]
iolog: switch to list based scheme

The previous iolog implementation had one big log, that we continually
resized when we ran out of space. The resize involved a realloc()
of the data, which could potentially be slow for big logs. This could
introduce long latency outliers for workloads, since fio did not
quiesce IO before resizing the log.

Rewrite the logging code to have a list of logs, and cap each of them
at a reasonable size. This means we never have to realloc data, and
that we allocate smaller chunks instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: dump state of stuck thread
Jens Axboe [Thu, 19 May 2016 15:19:56 +0000 (09:19 -0600)]
backend: dump state of stuck thread

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: mark the thread as finishing, when we are out of the IO loop
Jens Axboe [Thu, 19 May 2016 15:19:03 +0000 (09:19 -0600)]
backend: mark the thread as finishing, when we are out of the IO loop

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: only do forceful timeout exit if the job isn't actively finishing
Jens Axboe [Thu, 19 May 2016 14:49:31 +0000 (08:49 -0600)]
backend: only do forceful timeout exit if the job isn't actively finishing

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoflist: add flist_last_entry()
Jens Axboe [Thu, 19 May 2016 14:25:05 +0000 (08:25 -0600)]
flist: add flist_last_entry()

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofilesetup: align a size given as a percentage to the block size
Jens Axboe [Wed, 18 May 2016 21:06:05 +0000 (15:06 -0600)]
filesetup: align a size given as a percentage to the block size

We want it to be a nice multiple of (at least) our minimum block
size.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoadd -A option for better stats
Ben England [Tue, 17 May 2016 22:48:30 +0000 (18:48 -0400)]
add -A option for better stats

4 years agoinit: cleanup random inits
Jens Axboe [Tue, 17 May 2016 14:16:12 +0000 (08:16 -0600)]
init: cleanup random inits

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoWindows crash in ctime_r()
Michael Schoberg (mschoberg) [Mon, 16 May 2016 21:50:19 +0000 (21:50 +0000)]
Windows crash in ctime_r()

I think I found an issue in os\windows\posix.c that results in a FIO
crash (on Windows.)  I'm including a patch that resolves the crash for
us, but includes another (optional) fix.

Crash issue:  possix.c  - ctime_r() will reference a negative array
index on Sunday.  SYSTEMTIME states the days of the week as: "0=Sunday,
.. , 6=Saturday."  The "fix" can likely be dialed back to safely assume
the days/months will adhere to how they're documented.

Optional - StringCchPrintfA() calls should allow for the string plus a
NULL character.  Instead, the value getting passed in is for the entire
string size.

os/windows/posix.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agozipf/pareto/gauss: hash cleanup
Jens Axboe [Tue, 17 May 2016 01:18:09 +0000 (19:18 -0600)]
zipf/pareto/gauss: hash cleanup

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooptions: 0.00 is a valid gauss dev
Jens Axboe [Tue, 17 May 2016 01:13:15 +0000 (19:13 -0600)]
options: 0.00 is a valid gauss dev

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd support for non-uniformly random file service type
Jens Axboe [Tue, 17 May 2016 00:09:54 +0000 (18:09 -0600)]
Add support for non-uniformly random file service type

Similar options to random_distribution, this is just for the
selection of files. Like that option, you can do:

file_service_type=zipf:1.2

and get files selected through a zipfian distribution. This extends
the concept of hot and cold IO regions to have hotter and colder files
as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agozipf/pareto/gauss: add option to disable hashing
Jens Axboe [Tue, 17 May 2016 00:09:43 +0000 (18:09 -0600)]
zipf/pareto/gauss: add option to disable hashing

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agostat: add blocksize to averaged log, if it's consistent
Jens Axboe [Mon, 16 May 2016 16:53:44 +0000 (10:53 -0600)]
stat: add blocksize to averaged log, if it's consistent

If we average over time, fio doesn't add the blocksize used. The
reasoning behind that is that if we're using multiple block sizes,
there's no right answer. But if we are not, we can add the blocksize
that was used.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: unroll string copy
Jens Axboe [Wed, 11 May 2016 01:50:00 +0000 (19:50 -0600)]
verify: unroll string copy

Let's just turn the strcpy+replace into a hand rolled version.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: only escape 'name', not prefix+name
Jens Axboe [Wed, 11 May 2016 01:43:08 +0000 (19:43 -0600)]
verify: only escape 'name', not prefix+name

Fixes: e139c0c076e5 ("verify: escape '/' in state file name to '.'")
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: escape '/' in state file name to '.'
Jens Axboe [Wed, 11 May 2016 00:50:48 +0000 (18:50 -0600)]
verify: escape '/' in state file name to '.'

If the job name includes a front slash, then we get an error when
we try to create the state file name:

fio: open state file: No such file or directory

Change the '/' to '.' instead, so we generate a valid file name.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: increase state file name and log error on failure
Jens Axboe [Wed, 11 May 2016 00:23:37 +0000 (18:23 -0600)]
verify: increase state file name and log error on failure

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoioengines: cleanup
Jens Axboe [Tue, 10 May 2016 18:15:26 +0000 (12:15 -0600)]
ioengines: cleanup

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/pmemblk: more cleanups
Jens Axboe [Tue, 10 May 2016 15:26:08 +0000 (09:26 -0600)]
engines/pmemblk: more cleanups

No (intended) functional changes in this patch.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoChange default IO engine from sync to psync
Jens Axboe [Mon, 9 May 2016 19:35:09 +0000 (13:35 -0600)]
Change default IO engine from sync to psync

If the OS doesn't give us a default IO engine, then fio defaults to
sync. We should use psync instead, since we have that everywhere.
For fast storage, it avoids and lseek(), which can be a big deal.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoio_u: if we're doing backwards IO, wrap to end (not start)
Jens Axboe [Mon, 9 May 2016 19:31:01 +0000 (13:31 -0600)]
io_u: if we're doing backwards IO, wrap to end (not start)

If we have a job with a start file offset and we're doing holed
IO, we want to wrap to the start offset for forwards holes, but
to the end for backwards holes.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofilesetup: ensure that we align file starting offset
Jens Axboe [Mon, 9 May 2016 14:42:40 +0000 (08:42 -0600)]
filesetup: ensure that we align file starting offset

If --offset=xx is given, and xx isn't a multiple of the block size,
then align it if we're doing O_DIRECT IO.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMakefile: fix path to tools/fiologparser.py
Jens Axboe [Fri, 6 May 2016 23:10:33 +0000 (17:10 -0600)]
Makefile: fix path to tools/fiologparser.py

Fixes: 642483dbae99 ("Fix typo in tools/fiologparser.py")
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix typo in tools/fiologparser.py
Jens Axboe [Fri, 6 May 2016 21:40:55 +0000 (15:40 -0600)]
Fix typo in tools/fiologparser.py

Also change interpreter to /usr/bin/python, that's where it is on
most distros (not /bin/python).

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agodiskutil: adapt to new helper_thread functions
Jens Axboe [Fri, 6 May 2016 20:38:19 +0000 (14:38 -0600)]
diskutil: adapt to new helper_thread functions

We don't have helper_exit anymore. This caused the link to fail on
platforms where we don't have diskutil support.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoos/os-mac: kill unused code
Jens Axboe [Fri, 6 May 2016 20:32:03 +0000 (14:32 -0600)]
os/os-mac: kill unused code

We don't use the old timer functions anymore, kill the OSX
emulation of them.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agohelper_thread: split into separate file
Jens Axboe [Fri, 6 May 2016 20:17:14 +0000 (14:17 -0600)]
helper_thread: split into separate file

It's nicely contained now, split it into its own set of files.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'logging'
Jens Axboe [Fri, 6 May 2016 20:04:21 +0000 (14:04 -0600)]
Merge branch 'logging'

4 years agoWire up pmemblk
Jens Axboe [Fri, 6 May 2016 20:01:26 +0000 (14:01 -0600)]
Wire up pmemblk

Add it to our options, and to the documentation.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/pmemblk: get rid of CACHE_LOCK/UNLOCK defines
Jens Axboe [Fri, 6 May 2016 19:56:27 +0000 (13:56 -0600)]
engines/pmemblk: get rid of CACHE_LOCK/UNLOCK defines

Much clearer to just call the pthread mutex lock/unlock directly,
since you don't have to looku what this macro does.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoengines/pmeblk: fixup coding style
Jens Axboe [Fri, 6 May 2016 19:53:46 +0000 (13:53 -0600)]
engines/pmeblk: fixup coding style

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMerge branch 'libpmemblk' of https://github.com/bgbhpe/fio
Jens Axboe [Fri, 6 May 2016 19:51:53 +0000 (13:51 -0600)]
Merge branch 'libpmemblk' of https://github.com/bgbhpe/fio

4 years agoMakefile: add tools/fiologpaser.py
Jens Axboe [Fri, 6 May 2016 17:43:57 +0000 (11:43 -0600)]
Makefile: add tools/fiologpaser.py

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoadded fio log parser tool.
Mark Nelson [Fri, 6 May 2016 17:39:07 +0000 (12:39 -0500)]
added fio log parser tool.

Signed-off-by: Mark Nelson <mnelson@redhat.com>
4 years agostat: remove debug statement
Jens Axboe [Fri, 6 May 2016 17:19:55 +0000 (11:19 -0600)]
stat: remove debug statement

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoImprove logging accuracy
Jens Axboe [Fri, 6 May 2016 17:14:19 +0000 (11:14 -0600)]
Improve logging accuracy

If we do windowed logging (over a period of time), then we can
take the logging out of the IO worker fast path, and just have
our helper thread do it. This means we'll reduce the overhead
for the IO worker (which is important), and that we'll be able
to provide better accuracy of when we log. Before this change,
we'd log when we had exceeded the window of time at a completion
even. This means that if you had a logging time of 500 msec,
you'd always exceed that and even tend to skew it to being much
later. With this change, logging should be very accurate time
wise:

$ ./fio --inflate-log=fsrandW4-8-iopslog_iops.1.log.fz | head -n10
500, 470, 0, 0
1000, 492, 0, 0
1500, 484, 0, 0
2000, 492, 0, 0
2500, 454, 0, 0
3000, 462, 0, 0
3500, 480, 0, 0
4000, 462, 0, 0
4500, 472, 0, 0
5000, 496, 0, 0

and not exhibit any drift at all.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolibfio: clear iops/bw sample times on stats reset
Jens Axboe [Thu, 5 May 2016 16:55:47 +0000 (10:55 -0600)]
libfio: clear iops/bw sample times on stats reset

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: add define for default number of log entries
Jens Axboe [Thu, 5 May 2016 16:54:03 +0000 (10:54 -0600)]
iolog: add define for default number of log entries

We use two different ones, depending on gz or not...

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: free memory on gz init failure
Jens Axboe [Thu, 5 May 2016 16:51:23 +0000 (10:51 -0600)]
iolog: free memory on gz init failure

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: don't use the workqueue for sync work
Jens Axboe [Thu, 5 May 2016 16:49:14 +0000 (10:49 -0600)]
iolog: don't use the workqueue for sync work

Just run it inline. We'll need this going forward, since we can't
always rely on the proper shared memory from the helper thread.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoiolog: fix use-after-free of iolog_flush_data
Jens Axboe [Thu, 5 May 2016 14:48:09 +0000 (08:48 -0600)]
iolog: fix use-after-free of iolog_flush_data

If we run with compressed logs, drop_data_unlock() will free 'data'
before killing the condvar and mutex.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agohash: import Linux sparse hash fix
Jens Axboe [Tue, 3 May 2016 19:56:47 +0000 (13:56 -0600)]
hash: import Linux sparse hash fix

This is commit 689de1d6ca95 in the upstream Linux kernel tree.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUpdate RBD documentation
Jens Axboe [Mon, 2 May 2016 14:25:16 +0000 (08:25 -0600)]
Update RBD documentation

Sync them between HOWTO and fio.1 as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agorbd: add clustername option
Tianqing [Mon, 2 May 2016 14:21:53 +0000 (08:21 -0600)]
rbd: add clustername option

As the Ceph Jewel is released, ceph cluster name
may not be the default "ceph". So we should add
the clustername for rbd engine to support that.

Signed-off-by: Tianqing <tianqing@unitedstack.com>
Fixed up various coding style violations by me.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.9 fio-2.9
Jens Axboe [Thu, 28 Apr 2016 22:23:10 +0000 (16:23 -0600)]
Fio 2.9

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agooslib/getopt_long: allow (unique) short match
Jens Axboe [Fri, 22 Apr 2016 22:14:24 +0000 (18:14 -0400)]
oslib/getopt_long: allow (unique) short match

The GNU version allows short matches, if they are unique. Our version
does not, since it's weird, but let's at least make them have the
same behavior. Otherwise users will have different behavior for
command line parsing for edge cases, on different platforms.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoinclude sys/sysmacros.h for major/minor
Mike Frysinger [Wed, 20 Apr 2016 16:52:19 +0000 (12:52 -0400)]
include sys/sysmacros.h for major/minor

These functions have always been defined in sys/sysmacros.h under
Linux C libraries.  For some, including sys/types.h implicitly
includes that as well, but glibc wants to deprecate that, and some
others already have.  Include the header explicitly for the funcs.

The mtd change is already in upstream mtd-utils too.

Signed-off-by: Mike Frysinger <vapier@gentoo.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoDocumentation: it's 'log_max_value', not 'log_max'
Jens Axboe [Mon, 18 Apr 2016 20:54:12 +0000 (16:54 -0400)]
Documentation: it's 'log_max_value', not 'log_max'

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/fio-verify-state: show completions in order
Jens Axboe [Fri, 15 Apr 2016 15:03:34 +0000 (09:03 -0600)]
t/fio-verify-state: show completions in order

We currently show them in array order, but that's logged in
reverse. So show them from newest to latest.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agofilesetup: assign fileno at setup time
Jens Axboe [Thu, 14 Apr 2016 14:57:46 +0000 (08:57 -0600)]
filesetup: assign fileno at setup time

For duplicate jobs, we don't always get fileno assigned if we have
multiple files per job. Ensure that we do.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/fio-verify-state: allow multiple input files
Jens Axboe [Thu, 14 Apr 2016 14:21:18 +0000 (08:21 -0600)]
t/fio-verify-state: allow multiple input files

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoclient: bool conversion
Jens Axboe [Wed, 13 Apr 2016 21:53:06 +0000 (15:53 -0600)]
client: bool conversion

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/fio-verify-state: pretty up output a bit
Jens Axboe [Wed, 13 Apr 2016 21:13:55 +0000 (15:13 -0600)]
t/fio-verify-state: pretty up output a bit

Also parseable now.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoCleanup last write logging
Jens Axboe [Wed, 13 Apr 2016 18:47:24 +0000 (12:47 -0600)]
Cleanup last write logging

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix verify state for multiple files
Jens Axboe [Wed, 13 Apr 2016 18:25:52 +0000 (12:25 -0600)]
Fix verify state for multiple files

Fio currently only logs the completions on a per-job basis, but
that breaks down if the job uses multiple files and we rely
on the verify state save/load to check when to stop verifying.

This patch extends the verify state format to be version 3, and
removes support for v1/v2. The new version has support for
multiple files.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agopmemblk: don't use #defines for the pmemblk_* functions
Brian Boylston [Mon, 11 Apr 2016 15:24:57 +0000 (10:24 -0500)]
pmemblk: don't use #defines for the pmemblk_* functions

4 years agopmemblk: remove comments about an external engine
Brian Boylston [Mon, 11 Apr 2016 15:20:36 +0000 (10:20 -0500)]
pmemblk: remove comments about an external engine

4 years agoadd an example job file for pmemblk
Brian Boylston [Mon, 11 Apr 2016 15:17:17 +0000 (10:17 -0500)]
add an example job file for pmemblk

4 years agoModify RDMA engine to print strerror messages.
Logan Gunthorpe [Mon, 4 Apr 2016 16:06:03 +0000 (10:06 -0600)]
Modify RDMA engine to print strerror messages.

In order to ease debugging it would be nice to know why certain
rdma/ib functions fail. I've added %m messages to a large
number of log_err lines.

4 years agofio: register pvsync2 engine correctly
Jon Derrick [Wed, 30 Mar 2016 22:17:55 +0000 (16:17 -0600)]
fio: register pvsync2 engine correctly

I'm experimenting with the new syscalls and noticed fio's support was
incomplete. This patch fixes some of those issues.

Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoHOWTO/man: clarify that the usr/sys utilization numbers are averages
Jens Axboe [Tue, 29 Mar 2016 14:34:06 +0000 (08:34 -0600)]
HOWTO/man: clarify that the usr/sys utilization numbers are averages

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix crash with group_reporting and one group member that never started
Jens Axboe [Tue, 29 Mar 2016 14:28:56 +0000 (08:28 -0600)]
Fix crash with group_reporting and one group member that never started

If we have a job that nevers gets started, then we crash in
setting up the group stats, if group_reporting is enabled. This
can be reproduced like this:

fio --name=iwontstart --name=null --ioengine=null --size=1g --group_reporting=1
Starting 2 processes
iwontstart: you need to specify size=
fio: pid=0, err=22/file:filesetup.c:828, func=total_file_size, error=Invalid argument
Segmentation fault (core dumped)

Check for -1 groupid, and just skip the job if so.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoadd pmemblk engine
Brian Boylston [Sun, 27 Mar 2016 21:15:16 +0000 (16:15 -0500)]
add pmemblk engine

4 years agot/read-to-pipe-async: use gettimeofday() instead of clock_gettime()
Jens Axboe [Fri, 25 Mar 2016 15:40:41 +0000 (09:40 -0600)]
t/read-to-pipe-async: use gettimeofday() instead of clock_gettime()

We don't have clock_gettime() on all supported platforms, we can
just use gettimeofday() and convert to timespec.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/read-to-pipe-async: synchronization fixes
Jens Axboe [Fri, 25 Mar 2016 14:30:36 +0000 (08:30 -0600)]
t/read-to-pipe-async: synchronization fixes

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/memlock: sample utility to use X memory from Y threads
Jens Axboe [Thu, 24 Mar 2016 23:07:05 +0000 (17:07 -0600)]
t/memlock: sample utility to use X memory from Y threads

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMakefile: disable build of t/read-to-pipe-async by default
Jens Axboe [Thu, 24 Mar 2016 21:58:21 +0000 (15:58 -0600)]
Makefile: disable build of t/read-to-pipe-async by default

It's not part of fio, if you need it, you can just do:

$ make t/read-to-pipe-async

and it should build.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/read-to-pipe-async: needs time.h on some platforms
Jens Axboe [Thu, 24 Mar 2016 21:56:09 +0000 (15:56 -0600)]
t/read-to-pipe-async: needs time.h on some platforms

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/read-to-pipe-async: standalone test app
Jens Axboe [Thu, 24 Mar 2016 21:52:19 +0000 (15:52 -0600)]
t/read-to-pipe-async: standalone test app

Reads from a file through reads, spawning more threads if necessary
to keep the latency target. Writes the result to a pipe, and outputs
the latencies at the end.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agobackend: ensure that we run verification for short time based jobs
Jens Axboe [Thu, 24 Mar 2016 19:36:22 +0000 (13:36 -0600)]
backend: ensure that we run verification for short time based jobs

We need to break out of the main loop for the case where we did
exceed our bytes issued, if we have verification to run. For
time based jobs, we'll get back in this loop after verification.

Fixes c04e4661e4da ("time_based: Avoid restarting main I/O loop")
Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agotravis.yml: ensure we have libaio-dev and numa dev libs
Jens Axboe [Thu, 24 Mar 2016 03:38:59 +0000 (21:38 -0600)]
travis.yml: ensure we have libaio-dev and numa dev libs

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoMakefile: add empty 'test' target
Jens Axboe [Thu, 24 Mar 2016 03:36:09 +0000 (21:36 -0600)]
Makefile: add empty 'test' target

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd .travis.yml
Jens Axboe [Thu, 24 Mar 2016 03:31:48 +0000 (21:31 -0600)]
Add .travis.yml

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agogen-rand: fix dependency on strcasestr.o
Jens Axboe [Sun, 20 Mar 2016 15:38:57 +0000 (09:38 -0600)]
gen-rand: fix dependency on strcasestr.o

Fails on systems where we have to pull that in (eg Windows).

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoinit: seed repeatable jobs differently
Jens Axboe [Fri, 18 Mar 2016 16:07:16 +0000 (09:07 -0700)]
init: seed repeatable jobs differently

Factor in the job number for the stock random seeds, so we don't
generate the same patterns for identical jobs.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.8 fio-2.8
Jens Axboe [Tue, 15 Mar 2016 16:10:37 +0000 (09:10 -0700)]
Fio 2.8

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoverify: clear IO verify state all_io_list before writing
Jens Axboe [Tue, 15 Mar 2016 16:09:10 +0000 (09:09 -0700)]
verify: clear IO verify state all_io_list before writing

We only fill it partially sometimes, so ensure that we clear
the whole thing before writing it out. This fixes valgrind
complaining about writing unitialized memory.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoRevert "options: move pattern_fmt_desc where we need it"
Jens Axboe [Tue, 15 Mar 2016 15:48:25 +0000 (09:48 -0600)]
Revert "options: move pattern_fmt_desc where we need it"

This reverts commit 5f3bd0fd67254e3750681f0700a0cc162faee500.

We do need it to be globally reachable, otherwise we assign a
pointer to an on-stack structure. This causes the pattern filling
to read garbage, and segfault.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agolib/rand: make __init_randX() static
Jens Axboe [Thu, 10 Mar 2016 19:12:09 +0000 (12:12 -0700)]
lib/rand: make __init_randX() static

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUse 32-bit rand for parts that use rand_between()
Jens Axboe [Thu, 10 Mar 2016 19:09:14 +0000 (12:09 -0700)]
Use 32-bit rand for parts that use rand_between()

We don't need a full 32-bit space for these, and the 32-bit generator
is faster.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/gen-rand: use 32-bit random generator
Jens Axboe [Thu, 10 Mar 2016 18:08:19 +0000 (11:08 -0700)]
t/gen-rand: use 32-bit random generator

This was inadvertently changed.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFix compile of test programs on archs that use arch_flags at runtime
Jens Axboe [Thu, 10 Mar 2016 15:09:41 +0000 (08:09 -0700)]
Fix compile of test programs on archs that use arch_flags at runtime

SuperH compile currently fails with:

gettime.o: In function fio_gettime':
/home/thomas/projets/buildroot/output/build/fio-fio-2.7/gettime.c:163: undefined reference to arch_flags'
gettime.o: In function utime_since_now':
/home/thomas/projets/buildroot/output/build/fio-fio-2.7/gettime.c:164: undefined reference to arch_flags'
gettime.o: In function mtime_since_now':
/home/thomas/projets/buildroot/output/build/fio-fio-2.7/gettime.c:164: undefined reference to arch_flags'
gettime.o: In function time_since_now':
/home/thomas/projets/buildroot/output/build/fio-fio-2.7/gettime.c:164: undefined reference to arch_flags'
mutex.o: In function fio_mutex_up':
/home/thomas/projets/buildroot/output/build/fio-fio-2.7/mutex.c:189: undefined reference to arch_flags'
collect2: error: ld returned 1 exit status
Makefile:375: recipe for target 't/stest' failed
make[2]: *** [t/stest] Error 1
make[2]: *** Waiting for unfinished jobs....

Fix that by ensuring we have a stub arch.o with the necessary arch flags
for the standalone test programs.

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agot/gen-rand: remove compile warning on 32-bit
Jens Axboe [Wed, 9 Mar 2016 21:15:11 +0000 (14:15 -0700)]
t/gen-rand: remove compile warning on 32-bit

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoFio 2.7 fio-2.7
Jens Axboe [Wed, 9 Mar 2016 16:42:56 +0000 (09:42 -0700)]
Fio 2.7

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years ago.gitignore: ignore vim undo files
Jens Axboe [Tue, 8 Mar 2016 18:19:17 +0000 (11:19 -0700)]
.gitignore: ignore vim undo files

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoAdd t/gen-rand to test random generator
Jens Axboe [Tue, 8 Mar 2016 16:58:13 +0000 (09:58 -0700)]
Add t/gen-rand to test random generator

Signed-off-by: Jens Axboe <axboe@fb.com>
4 years agoUpdate documentation on log file formats
Jens Axboe [Tue, 8 Mar 2016 15:56:57 +0000 (08:56 -0700)]
Update documentation on log file formats

Signed-off-by: Jens Axboe <axboe@fb.com>