fio.git
3 years agoIgnore pre-read for character devices
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:57 +0000 (23:02 +0300)]
Ignore pre-read for character devices

pre_read_file() could ignore a chrdev due to its purpose of providing
unbuffered access to devices not limited to seekable disk devices,
while the purpose of this function (i.e. pre_read= option) is to
lseek(2) the given offset and page cache whatever read via read(2)
which is basically for filesystems and blkdevs.

(This commit directly goes on top of the one before previous one)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't proceed with error set when failed to pre-read files/devices
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:56 +0000 (23:02 +0300)]
Don't proceed with error set when failed to pre-read files/devices

pre_read_files() never fails (never returns negative), and because
of this behavior, fio threads can continue regardless of td_verror()
call while prereading, and results in statistics with error in it,
which is quite confusing as to how it should be interpreted for regular
users.

This commit makes pre_read_files() return -1 if preread fails, and
eventually aborts fio, which is also the case with some other options.
It also changes non error return from 1 to 0, since all it matters
is if it's <0 or not, and fio normally uses 0 for success.

Not sure what the proper way of fixing this is if it legitimately
failed for some error on prereading though. Some options just print
a message to indicate it's unsupported and continue normally withtout
any error set.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoIgnore pre-read for FIO_NOIO td
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:55 +0000 (23:02 +0300)]
Ignore pre-read for FIO_NOIO td

pre_read_file() should ignore FIO_NOIO (i.e. cpuio) in addition to
FIO_PIPEIO. It does/should fail anyway since FIO_NOIO td is supposed
to have -1 for fd for not supporting any I/O.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop redundant "ignore invalidate option" message from 21c1b29e
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:54 +0000 (23:02 +0300)]
Drop redundant "ignore invalidate option" message from 21c1b29e

The message added by 21c1b29e ("fio: ignore invalidate option...")
is correct in the sense that it resets invalidate= option, however
since invalidation is enabled by default, this message appears whenever
pre_read=1 is set (unless invalidate=0 is also explicitly set at the
same time, which is unliekly).

This is annoying and shouldn't have been added. 21c1b29e updating
HOWTO was enough, sorry.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: Mention niche detail of range format options
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:53 +0000 (23:02 +0300)]
HOWTO: Mention niche detail of range format options

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't malloc more than necessary on extending/prereading file
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:52 +0000 (23:02 +0300)]
Don't malloc more than necessary on extending/prereading file

If td->o.max_bs[DDIR_XXX] were larger than file size to extend/preread,
it only needs to malloc the file size.

Given that td->o.max_bs[DDIR_XXX] are tunable parameters, it's better
not to malloc whatever size specified if needed size is smaller than
that. Users could specify as large as maximum of uint.

I dropped "bs = td->o.max_bs[DDIR_WRITE];" inside the first while loop
since now it works the same with or without this.

(This commit directly goes on top of the previous one)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoTest malloc result when allocation size is tunable
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:51 +0000 (23:02 +0300)]
Test malloc result when allocation size is tunable

The allocation size td->o.max_bs[DDIR_XXX] can be specified by user,
so test the result although it can be over committed on some platforms.
In theory allocation size could be as large as maximum of uint.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix return value of make_filename() when no filename_format
Tomohiro Kusumi [Tue, 28 Mar 2017 20:02:49 +0000 (23:02 +0300)]
Fix return value of make_filename() when no filename_format

This is unlikely to happen in the first place since ->filename_format
has a default value, and option parser can also detect an empty string,
but it should return buf which is sprintf'd right before returning.

A caller expects this function to return filename string (which is
buf arg itself in this case), and it also doesn't handle NULL return.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: add a --disable-rdma flag to control rdma deps
Mike Frysinger [Tue, 21 Mar 2017 13:20:32 +0000 (07:20 -0600)]
configure: add a --disable-rdma flag to control rdma deps

Signed-off-by: Mike Frysinger <vapier@gentoo.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoTest fsync/fdatasync/sync_file_range for the next i/o only if should_fsync(td)
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:47 +0000 (23:28 +0200)]
Test fsync/fdatasync/sync_file_range for the next i/o only if should_fsync(td)

Checking an inline function should_fsync() first is better since
should_fsync() itself contains td_write() as a condition to return true,
and there's no need for non-write td to go through the rest of checks
when they result in false due to td->io_issues[DDIR_WRITE] being 0.

(This commit directly goes on top of the previous one)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoTest uint,int before division uint/int for the next i/o
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:46 +0000 (23:28 +0200)]
Test uint,int before division uint/int for the next i/o

When selecting ddir for the next i/o, test td->io_issues[DDIR_WRITE]
before division which is more expensive.

if (x && y && !(y % x))
rather than
if (x && !(y % x) && y)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix a comment after f227e2b6
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:45 +0000 (23:28 +0200)]
Fix a comment after f227e2b6

Add "for regular files" since it's true only on regfiles.
Also see below.
http://www.spinics.net/lists/fio/msg05646.html

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: Mention fsync=/fsyncdata= are set to 0 by default
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:44 +0000 (23:28 +0200)]
HOWTO: Mention fsync=/fsyncdata= are set to 0 by default

I think people could think sync ioengine and variants would somehow
fsync(2) for writes from what their name sound like, while they do
not unless these options are explicitly set (default 0).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: Mention cpuload= is mandatory for cpuio
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:43 +0000 (23:28 +0200)]
HOWTO: Mention cpuload= is mandatory for cpuio

fio jobs using cpuio fail without this option value specified,
so mention it's mandatory (or have default value for this ?).

fio: pid=9320, err=22/file:engines/cpu.c:75, func=cpuio, error=cpu thread needs rate (cpuload=)
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDefine struct sk_out in server.h (not server.c)
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:42 +0000 (23:28 +0200)]
Define struct sk_out in server.h (not server.c)

struct sk_out isn't local to server.c, so it should be visible
in its header (instead of having struct sk_out; in fio.h).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoReplace redundant TD_F_NOIO flag with td->io_ops_init
Tomohiro Kusumi [Mon, 20 Mar 2017 21:28:41 +0000 (23:28 +0200)]
Replace redundant TD_F_NOIO flag with td->io_ops_init

The only reason TD_F_NOIO exists in addition to FIO_NOIO is because
fio_running_or_pending_io_threads() needs to know whether td has
successfully initialized ioengine, whereas FIO_NOIO is statically
set regardless of ioengines's ->init() result.

This commit adds a new td field ->io_ops_init to inidicate ->init()
result, so that td needs no extra bit field for each ioengine like
TD_F_NOIO. It was rather odd that td was unable to tell ->init()
result afterward when ->init() failure (returning non zero) doesn't
mean aborting fio itself. This commit also changes TD_F_NOIO to
TD_F_RESERVED as it's no longer used.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'wip-fix-bs-title' of https://github.com/liupan1111/fio
Jens Axboe [Mon, 20 Mar 2017 01:49:04 +0000 (19:49 -0600)]
Merge branch 'wip-fix-bs-title' of https://github.com/liupan1111/fio

3 years agomake the bs info output clearer.
Pan Liu [Mon, 20 Mar 2017 01:42:07 +0000 (09:42 +0800)]
make the bs info output clearer.

3 years agois_power_of_2() should return bool
Jens Axboe [Fri, 17 Mar 2017 16:39:42 +0000 (10:39 -0600)]
is_power_of_2() should return bool

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoio_u: we don't need to set power_2 to false
Jens Axboe [Fri, 17 Mar 2017 16:33:37 +0000 (10:33 -0600)]
io_u: we don't need to set power_2 to false

We always set it before checking, no need for initializing it.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofixed the error=invalid argument when the lower bound of bsrange is not power of 2.
Pan Liu [Fri, 17 Mar 2017 16:24:09 +0000 (00:24 +0800)]
fixed the error=invalid argument when the lower bound of bsrange is not power of 2.

Signed-off-by: Pan Liu <liupan1111@gmail.com>
3 years agoAdd 'stats' option
Jens Axboe [Thu, 16 Mar 2017 20:43:37 +0000 (14:43 -0600)]
Add 'stats' option

This option controls whether or not we collect runtime stats
and show stat output for a given job. Defaults to on.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofilesetup: remove bogus message on forcing file size
Jens Axboe [Thu, 16 Mar 2017 20:33:14 +0000 (14:33 -0600)]
filesetup: remove bogus message on forcing file size

This trigger all the time when size isn't being set and we
just query it, it's pointless and confusing to log. Kill
it with fire.

Fixes: c4aa2d08 ("Fixup for a minor 0 byte file size case")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: add generic pshared mutex test
Sitsofe Wheeler [Sun, 26 Feb 2017 11:47:28 +0000 (11:47 +0000)]
configure: add generic pshared mutex test

Add a feature test to configure to determine if a platform has
interprocess synchronization primitives that work with threads and
remove the hardcoded enablement in the individual os/os-* headers. This
should work better for platforms where support was overlooked or didn't
exist at some point in the past but has since been added. Also update
the README to not single out FreeBSD for being thread only as this might
change...

This is based off work by the Boost project
(https://github.com/boostorg/interprocess/blob/boost-1.63.0/include/boost/interprocess/detail/workaround.hpp#L45
) that found that OSX 10.6 and Cygwin are buggy even though
_POSIX_THREAD_PROCESS_SHARED says this feature should be work on them.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoConditionally enable FIO_HAVE_PSHARED_MUTEX on FreeBSD
Tomohiro Kusumi [Mon, 13 Mar 2017 18:42:37 +0000 (20:42 +0200)]
Conditionally enable FIO_HAVE_PSHARED_MUTEX on FreeBSD

FreeBSD ports has had an outbox patch to conditionally enable
FIO_HAVE_PSHARED_MUTEX, which can be upstream'd.

This commit brings in below diff from FreeBSD ports.
https://github.com/freebsd/freebsd-ports/commit/271a84d2df7bc9b983fe2fabe1615d01202d429b

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agomanpage: Add URL links to HOWTO/README
Tomohiro Kusumi [Mon, 13 Mar 2017 18:15:02 +0000 (20:15 +0200)]
manpage: Add URL links to HOWTO/README

Add links to the official repository since whether HOWTO/README
get installed somewhere in fs is distributions/OS dependent.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix errval variable to be positive errno value
Tomohiro Kusumi [Mon, 13 Mar 2017 18:15:01 +0000 (20:15 +0200)]
Fix errval variable to be positive errno value

The local variable errval is used for strerror(3) which only works
with positive values (i.e. -EXXX doesn't return an expected string).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse ENOTSUP if OS doesn't support blkdev page cache invalidation
Tomohiro Kusumi [Mon, 13 Mar 2017 18:15:00 +0000 (20:15 +0200)]
Use ENOTSUP if OS doesn't support blkdev page cache invalidation

The return value of blockdev_invalidate_cache() has been platform
dependent (some pretend success, others return EINVAL) while Linux
is the only one actually supports this via Linux specific ioctl.

This commit makes all the rest return a consistent value ENOTSUP,
so __file_invalidate_cache() prints a blkdev invalidation failure
message that is consistent and makes sense.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMinor fixup for page cache invalidation debug prints
Tomohiro Kusumi [Mon, 13 Mar 2017 18:14:59 +0000 (20:14 +0200)]
Minor fixup for page cache invalidation debug prints

offset/length is used only for posix_fadvise(2), and each invalidation
method has its own meaning, so use different messages for them.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: Add note/exception on allow_mounted_write=
Tomohiro Kusumi [Mon, 13 Mar 2017 18:14:58 +0000 (20:14 +0200)]
HOWTO: Add note/exception on allow_mounted_write=

allow_mounted_write= is useful on Linux, but some platforms
just return some error on open(2) for mounted devices.

 # uname
 Linux
 # mkfs.ext4 /dev/loop0 > /dev/null
 mke2fs 1.43.3 (04-Sep-2016)
 # mount /dev/loop0 /mnt
 # mount | grep /dev/loop0
 /dev/loop0 on /mnt type ext4 (rw,relatime,seclabel,data=ordered)
 # fio --name=xxxxx --ioengine=sync --rw=write --bs=32k --size=1m --filename=/dev/loop0
 xxxxx: (g=0): rw=write, bs=32.0KiB-32.0KiB,32.0KiB-32.0KiB,32.0KiB-32.0KiB, ioengine=sync, iodepth=1
 fio-2.18-21-gca205
 fio: /dev/loop0 appears mounted, and 'allow_mounted_write' isn't set. Aborting.
 Run status group 0 (all jobs):
 # fio --name=xxxxx --ioengine=sync --rw=write --bs=32k --size=1m --filename=/dev/loop0 --allow_mounted_write=1 > /dev/null; echo $?
 0

 # uname
 FreeBSD
 # newfs /dev/da1 > /dev/null
 # mount /dev/da1 /mnt
 # mount | grep /dev/da1
 /dev/da1 on /mnt (ufs, local)
 # fio --name=xxxxx --ioengine=sync --rw=write --bs=32k --size=1m --filename=/dev/da1 --allow_mounted_write=1
 fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
 xxxxx: (g=0): rw=write, bs=32.0KiB-32.0KiB,32.0KiB-32.0KiB,32.0KiB-32.0KiB, ioengine=sync, iodepth=1
 fio-2.18-21-gca205
 Starting 1 thread
 fio: failed opening chardev /dev/da1 for size check
 file:filesetup.c:654, func=open(/dev/da1), error=Operation not permitted
 fio: pid=0, err=1/file:filesetup.c:654, func=open(/dev/da1), error=Operation not permitted
 Run status group 0 (all jobs)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMake check_mount_writes() test appropriate device types
Tomohiro Kusumi [Mon, 13 Mar 2017 18:14:57 +0000 (20:14 +0200)]
Make check_mount_writes() test appropriate device types

There are platforms that use chrdev (instead of blkdev) for block
devices, thus check_mount_writes() needs to test an appropriate file
type(s) for potential mounted devices.

fio has had a macro FIO_HAVE_CHARDEV_SIZE basically for those using
chrdev for block devices, and this macro can be used here as well.

In FreeBSD and several others, blkdev is no longer used as a file type,
so it's safe to only test FIO_TYPE_CHAR if FIO_HAVE_CHARDEV_SIZE is
defined, but this commit leaves FIO_TYPE_BLOCK test since some
platforms may have/use both for block devices (not sure if exist).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agorbd: don't log version to stdout
Jens Axboe [Mon, 13 Mar 2017 14:01:52 +0000 (08:01 -0600)]
rbd: don't log version to stdout

This screws up json/csv output for no useful reason.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMakefile: make test target use thread
Jens Axboe [Mon, 13 Mar 2017 03:04:45 +0000 (21:04 -0600)]
Makefile: make test target use thread

Some platforms don't support process shared mutexes. For the basic
Makefile test case, specify threads to avoid the warning.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoOnly enable arm64 CRC32 acceleration if the required header files are there
Theodore Ts'o [Sun, 12 Mar 2017 18:50:10 +0000 (12:50 -0600)]
Only enable arm64 CRC32 acceleration if the required header files are there

If the necessary header files are missing, then fio will bomb out when
building on arm64.  So disable the hardware acceleration option in
that case, since it's only performance improvement and not a
functional regression to turn it off.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: Make Cygwin take regular configure path
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:08 +0000 (22:13 +0200)]
configure: Make Cygwin take regular configure path

Cygwin (and only Cygwin) has been using customized configure process
by forcing hard-coded "known good options" without going through
regular compile_prog() based configure path. This was necessary due
to difference from other Unix likes, but it could be changed so that
Cygwin at least takes the regular configure path while continue to
use hard-coded configurations. This makes the configure script less
platform specific.

This commit basically gets rid of "exit 0" part from Cygwin case,
which enables Cygwin to go through the regular compile_prog() based
configure path. Removing "exit 0" requires some other small fixes
explained below so that Cygwin outputs consistent results (i.e. don't
ignore hard-coded configurations and lie on outputs).

Also note that this commit does nothing to Linux or other platforms.
It only affects Cygwin or any other platform that wants to override
or ignore regular configure path (i.e. compile_prog() results) in
the future. If there's any specific reason Cygwin has to workaround
the regular configure path, this shouldn't be applied, but there
seems to be no reason not to do so as it compiles and runs on Cygwin.

--
0. Remove "exit 0" in Cygwin case which is the main objective.

1. Change hard-coded output_sym() calls in Cygwin case to use shell
variables that have been used and initialized with "no" in the regular
compile_prog() based configure path.

2. Change the regular compile_prog() based configure path to check
if shell variables have already been initialized with "yes", and
initialize them with "no" only if they haven't been set to "yes",
so that shell variables statically initialized with "yes" in Cygwin
case don't get reset. This does nothing to Linux/etc because they
never preconfigure variables with "yes".

3. Get rid of output_sym "CONFIG_LITTLE_ENDIAN" call in Cygwin case.
This doesn't need to be here as endianness detection isn't anything
specific to Cygwin. Although x86 is the only supported arch, the regular
compile_prog() based configure path can detect it, and if it doesn't
then a compiler is probably broken.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse union for per file engine private data storage
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:07 +0000 (22:13 +0200)]
Use union for per file engine private data storage

fio_file::engine_data has been used by i/o engines to either keep
the offset or point to engine specific private data with casts.

This commit changes it to a union consists of void* and uint64_t
so get/set of offset/pointer can be done without casts (which is
a common technique used even within this same struct).

This may break external engines on compile time, but fio generally
doesn't care about breakage on external engines (no guarantees on
api/abi compatibility) anyway.

Also confirmed this compiles with pmemblk enabled.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDefine struct file_name as a file local structure
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:06 +0000 (22:13 +0200)]
Define struct file_name as a file local structure

This is just an one-off structure for filesetup.c to walk through
all the files (among all jobs), so it can be file local.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: Add some details for invalidate=
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:05 +0000 (22:13 +0200)]
HOWTO: Add some details for invalidate=

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMinor fixup for "Layint out IO file..." message
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:04 +0000 (22:13 +0200)]
Minor fixup for "Layint out IO file..." message

This is the same as what run_threads() does for threads/processes.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDon't set FIO_FILE_extend when create_on_open= option is set
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:03 +0000 (22:13 +0200)]
Don't set FIO_FILE_extend when create_on_open= option is set

FIO_FILE_extend gets set and later cleared only when create_on_open=
is disabled (which is default). If create_on_open=1 is set this flag
could be set for the entire runtime, but this flag is basically
designed to be enabled only during setup time.

(If there's no "Layint out IO file..." message, both FIO_FILE_extend
and need_extend probably aren't even needed, as files can be extended
at the moment when these flag/variable are set)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix a function name typo in debug print
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:02 +0000 (22:13 +0200)]
Fix a function name typo in debug print

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop fio_unused attribute from used variable
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:01 +0000 (22:13 +0200)]
Drop fio_unused attribute from used variable

This is actually used.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove unassigned fio_unused variable
Tomohiro Kusumi [Tue, 7 Mar 2017 20:13:00 +0000 (22:13 +0200)]
Remove unassigned fio_unused variable

This doesn't need to exist.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop redundant td_rw(td) tests
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:59 +0000 (22:12 +0200)]
Drop redundant td_rw(td) tests

Since TD_DDIR_READ is 1, TD_DDIR_WRITE is 1<<1, and td_rw() requires
both TD_DDIR_READ/WRITE bits set, testing (td_read(td) || td_rw(td))
turns out to be

if td_read(td) is false, td_rw(td) is always false,
if td_read(td) is true, td_rw(td) won't be tested,
thus the result equals that of td_read(td).

It's the same for td_write(td) as well.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoExplicitly check td_trim(td) to detect open(2) flag
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:58 +0000 (22:12 +0200)]
Explicitly check td_trim(td) to detect open(2) flag

This part is a bit complicated due to td's ddir being a bit flag.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFixup for a minor 0 byte file size case
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:57 +0000 (22:12 +0200)]
Fixup for a minor 0 byte file size case

When a file size calculated by (size / nrfiles) happens to be 0 in
setup_files(), it simply sets ->file_size to ->real_file_size which
means the size is set to the existing file size (if any) or 0.

Add log_info() to show that it's been set that way, and also 0 byte
file isn't likely to be created in the first place. Also add a check
to avoid negative file size.

This usually doesn't happen unless one tries to make it happen by
for e.g. using a small size= and large nrfiles= options.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix debug print format of file ->file_name
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:56 +0000 (22:12 +0200)]
Fix debug print format of file ->file_name

This should be %s.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAvoid irrelevant "offset extend ends" error message for chrdev
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:55 +0000 (22:12 +0200)]
Avoid irrelevant "offset extend ends" error message for chrdev

This commit expands 957c81b9 to chrdev as well when char_size()
is configured (i.e. FIO_HAVE_CHARDEV_SIZE disabled) to just return
0 with ->real_file_size set to -1.

Platforms without FIO_HAVE_CHARDEV_SIZE enabled basically means
block devices are blkdev as a file type, so having this change
or not most likely makes no difference. If an I/O engine needs
to use chrdev (e.g. sgio) it usually has its own .get_file_size.

Also add a dprint(FD_FILE) message since ->file_offset is normally
0 at this point regardless of offset option value, despite having
a conditional to check this against ->real_file_size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: Align help messages
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:54 +0000 (22:12 +0200)]
configure: Align help messages

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoHOWTO: Add platforms without fdatasync(2)
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:53 +0000 (22:12 +0200)]
HOWTO: Add platforms without fdatasync(2)

Added DragonFlyBSD which is a fork of FreeBSD.
FreeBSD seems to have fdatasync(2) in their upstream, but not yet
in the latest release version if I checked correctly.

Btw, having "Windows" here sounds a bit ambiguous.
Cygwin with gcc (not MinGW) compiles below taken from ./configure
by linking against POSIX compat dll.

--
 $ uname
 CYGWIN_NT-10.0
 $ cat ./fdatasync1.c
 #include <stdio.h>
 #include <unistd.h>
 int main(int argc, char **argv)
 {
    return fdatasync(0);
 }
 $ gcc -Wall -g ./fdatasync1.c ; echo $?
 0

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd runtime handlers for 97900ebf for FreeBSD/DragonFlyBSD
Tomohiro Kusumi [Tue, 7 Mar 2017 20:12:52 +0000 (22:12 +0200)]
Add runtime handlers for 97900ebf for FreeBSD/DragonFlyBSD

As 97900ebf itself explains, this is a runtime tunable on FreeBSD
and DragonFlyBSD, so they need to check the sysctl on runtime.

Not sure if OpenBSD has any tunable for this as the original commit
only says about >=5.1, but afaik recent versions of OpenBSD can't
even compile fio anyway. Also not sure how it works on NetBSD, and
the original commit said nothing about it too. Also see below url.

http://www.spinics.net/lists/fio/msg05586.html
http://www.spinics.net/lists/fio/msg05588.html

--
 # uname -r
 11.0-RELEASE-p1
 # cat ./test1.c
 #include <stdio.h>
 #include "./os/os-freebsd.h"
 int main(void) {
         printf("%d\n", shm_attach_to_open_removed());
         return 0;
 }
 # clang -Wall -g ./test1.c
 # sysctl kern.ipc.shm_allow_removed
 kern.ipc.shm_allow_removed: 1
 # ./a.out
 1
 # sysctl kern.ipc.shm_allow_removed=123
 kern.ipc.shm_allow_removed: 1 -> 123
 # ./a.out
 1
 # sysctl kern.ipc.shm_allow_removed=0
 kern.ipc.shm_allow_removed: 123 -> 0
 # ./a.out
 0

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: add support for the sha3 variants
Jens Axboe [Wed, 8 Mar 2017 16:13:14 +0000 (09:13 -0700)]
verify: add support for the sha3 variants

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agocrc: add support for sha3 variants
Jens Axboe [Wed, 8 Mar 2017 16:10:03 +0000 (09:10 -0700)]
crc: add support for sha3 variants

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoio_u: don't add slat samples if we are in ramp time
Jens Axboe [Tue, 7 Mar 2017 17:18:53 +0000 (10:18 -0700)]
io_u: don't add slat samples if we are in ramp time

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofio: fix overflow trying to use 'd' suffix
Sitsofe Wheeler [Fri, 24 Feb 2017 01:38:18 +0000 (01:38 +0000)]
fio: fix overflow trying to use 'd' suffix

Fix overflow that happened when using the 'd' time suffix on platforms
with 32 bit longs which led to a time of less than 10 minutes being
used.

3 years agoFio 2.18 fio-2.18
Jens Axboe [Thu, 23 Feb 2017 15:44:32 +0000 (08:44 -0700)]
Fio 2.18

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: disable compile time asserts for !opt and !static_assert
Jens Axboe [Wed, 22 Feb 2017 16:00:06 +0000 (09:00 -0700)]
configure: disable compile time asserts for !opt and !static_assert

If we don't have _Static_assert(), disable compile time
assertion checks if optimizations are disabled.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: use 'bool' for get_new_job()
Jens Axboe [Wed, 22 Feb 2017 03:53:48 +0000 (20:53 -0700)]
init: use 'bool' for get_new_job()

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: exit on failure to add all jobs
Jens Axboe [Tue, 21 Feb 2017 15:16:21 +0000 (08:16 -0700)]
init: exit on failure to add all jobs

It's too late to set do_exit here, so just exit manually if we
fail adding jobs.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoappveyor: add CI building of the Windows version of fio
Sitsofe Wheeler [Sun, 19 Feb 2017 08:52:28 +0000 (08:52 +0000)]
appveyor: add CI building of the Windows version of fio

Add AppVeyor continuous integration build of the Windows port. This also
builds the Windows installer and the resulting MSI can be downloaded.

Note that AppVeyor limits you to one build at a time.

3 years agoRevert "configure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin"
Tomohiro Kusumi [Sun, 19 Feb 2017 18:24:44 +0000 (20:24 +0200)]
Revert "configure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin"

This reverts commit d0f2828f6e49613d944a980e018cbfc02901264b
to unbreak on Cygwin.

As mentioned at below, it also needed to remove "exit 0",
but I'd rather like to revert this commit for now since there are
some other small niche to be fixed if we remove "exit 0".

https://github.com/axboe/fio/issues/314

The orginal commit was to get rid of redundant config setup on
endianness on Cygwin, but it turned out many of the unconditional
output_sym() calls in this CYGWIN*) case were done under Cygwin
specific way without going through regular configure procedure
(see "Forcing known good options on Windows" part for details),
although there are some unconditional output_sym() calls that need
to be there (e.g. CONFIG_WINDOWSAIO).

Since fixing endianness alone (by removing "exit 0") doesn't
have much meaning as a whole, it makes more sense to just revert
this for the time being.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoSilence Cygwin warning "'vsprintf_s' redeclared without dllimport..."
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:22 +0000 (20:22 +0200)]
Silence Cygwin warning "'vsprintf_s' redeclared without dllimport..."

Silence a compiler warning which used to didn't show up with some
old version of Cygwin. ad9c0fbc in 2012 added this, but it seems
this prototype should have been declared by an appropriate header
that's already been included here, so drop this prototype.
It also sounds what this message means according to Google.

Feel free to drop this if this is really needed for some reason.

  os/windows/posix.c:43:5: warning: 'vsprintf_s' redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
   int vsprintf_s(
       ^

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRemove irrelevant Cygwin config flag CONFIG_FADVISE
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:21 +0000 (20:22 +0200)]
Remove irrelevant Cygwin config flag CONFIG_FADVISE

This is unused and nothing happens regardless of this flag being
set on Cygwin. The one for POSIX fadvise is CONFIG_POSIX_FADVISE.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd details of file number/size related options to HOWTO
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:20 +0000 (20:22 +0200)]
Add details of file number/size related options to HOWTO

File number/size related options for each job/thread are difficult
to understand without looking at source.

fio(1) hasn't been updated (yet). There's already quite significant
amount of diff between these two, and people are basically only
updating HOWTO.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAvoid irrelevant "offset extend ends" error message
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:19 +0000 (20:22 +0200)]
Avoid irrelevant "offset extend ends" error message

When ->real_file_size is -1 due to unsupported file type/etc,
"if (f->file_offset > f->real_file_size)" conditional is always true
(unless ->file_offset is also negative which is an another problem),
and results in irrelevant error message. To avoid this, just return
1 without showing the irrelevant error message and then return 1.

It's still possible that char_size() returns with ->real_file_size
set to -1 (though platform dependent), and the irrelevant error
message be shown.

Also this message itself isn't likely to show any relevant info
due to the timing it's been called as mentioned in a comment.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRename thread_options' ->io_limit to io_size
Tomohiro Kusumi [Sun, 19 Feb 2017 18:22:18 +0000 (20:22 +0200)]
Rename thread_options' ->io_limit to io_size

This is less confusing since io_size is the option name itself,
and not much reason to use io_limit which is an alias of io_size.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRevert "Always set ->real_file_size to -1 when failed to get file size"
Jens Axboe [Fri, 17 Feb 2017 20:30:43 +0000 (13:30 -0700)]
Revert "Always set ->real_file_size to -1 when failed to get file size"

This reverts commit 34c9f82301a584899a9e65e4b9f56d98c38a52e7.

It breaks a very basic use case, ala:

./fio --ioengine=libaio --rw=randread --numjobs=1 --iodepth=256 --bs=1M --size=10G --direct=1 --filename=/tmp/test100 --name=test

if /tmp/test100 doesn't already exist.

3 years agoMerge branch 'travis_osx' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 16 Feb 2017 16:00:08 +0000 (09:00 -0700)]
Merge branch 'travis_osx' of https://github.com/sitsofe/fio

3 years agotravis: add OS X builds
Sitsofe Wheeler [Wed, 15 Feb 2017 18:48:10 +0000 (18:48 +0000)]
travis: add OS X builds

Add two enabled OS X builds (Travis' default OS X and macOS 10.12 with
Xcode 8.2) and one disabled OS X build (OS X 10.11 with the 10.12 SDK
but targeting 10.11).

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agotravis: prepare for additional builds
Sitsofe Wheeler [Wed, 15 Feb 2017 18:44:48 +0000 (18:44 +0000)]
travis: prepare for additional builds

Prepare the .travis.yml file to accept new builds by making the Linux
OS choice more explicit and protecting some Linux specific steps.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agotravis: run parallel make jobs
Sitsofe Wheeler [Mon, 6 Feb 2017 07:00:41 +0000 (07:00 +0000)]
travis: run parallel make jobs

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoMakefile: add test using the null ioengine
Sitsofe Wheeler [Sat, 11 Feb 2017 10:05:15 +0000 (10:05 +0000)]
Makefile: add test using the null ioengine

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoExplicitly check td_trim(td) for the direction of next io_u
Tomohiro Kusumi [Wed, 15 Feb 2017 20:40:55 +0000 (22:40 +0200)]
Explicitly check td_trim(td) for the direction of next io_u

This else case should explicitly check td_trim(td) since there is
a potential unused enum (e.g. DDIR_WAIT only used by iolog for now).

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop obsolete comment on a race condition
Tomohiro Kusumi [Wed, 15 Feb 2017 20:40:54 +0000 (22:40 +0200)]
Drop obsolete comment on a race condition

which is also mentioned in below posts.

http://www.spinics.net/lists/fio/msg05548.html
http://www.spinics.net/lists/fio/msg05552.html
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoUse 0 instead of DDIR_READ to iterate from 0 to DDIR_RWDIR_CNT
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:48 +0000 (17:19 +0200)]
Use 0 instead of DDIR_READ to iterate from 0 to DDIR_RWDIR_CNT

I think this is better since we want to count up DDIR_RWDIR_CNT,
but not specifically (count - something) even if something is 0,
and that's basically what DDIR_RWDIR_CNT is there for to equal
SYNC=3 after READ/WRITE/TRIM in io_ddir.h.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd missing "rand"/"trimwrite" strings to corresponding ddir slots
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:46 +0000 (17:19 +0200)]
Add missing "rand"/"trimwrite" strings to corresponding ddir slots

though "rand" alone is unused.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAlways set ->real_file_size to -1 when failed to get file size
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:45 +0000 (17:19 +0200)]
Always set ->real_file_size to -1 when failed to get file size

In some cases it's set to -1, but on others it's not,
while the existing code (e.g. get_fs_free_counts(), setup_files(), etc)
seem to expect ->real_file_size to be -1 in case of any error.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoDrop conditional declaration of disk_list
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:44 +0000 (17:19 +0200)]
Drop conditional declaration of disk_list

disk_list is used unconditionally whether it's Linux or not,
so leave the one in libfio.c, and remove the one in diskutil.c.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: Use x86 instead of i386 for $cpu for IA32
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:43 +0000 (17:19 +0200)]
configure: Use x86 instead of i386 for $cpu for IA32

fio internally uses "x86" for any sort of IA32, so use x86 for $cpu
as well which gets printed on ./configure.

This change should have been a part of e12f4ede in 2016.

 # grep i386 . -rI
 ./arch/arch.h:#if defined(__i386__)
 ./configure:elif check_define __i386__ ; then
 ./configure:  cpu="i386"
 ./configure:  i386|i486|i586|i686|i86pc|BePC)
 ./configure:    cpu="i386"
 ./crc/sha1.c:#if defined(__i386__) || defined(__x86_64__)
 ./crc/xxhash.c:#if defined(__ARM_FEATURE_UNALIGNED) || defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin
Tomohiro Kusumi [Tue, 14 Feb 2017 15:19:42 +0000 (17:19 +0200)]
configure: Drop default CONFIG_LITTLE_ENDIAN for Cygwin

It's a bit strange that Cygwin is assumed to be le by default when
no other platforms have such assumption even if they only target
a single or certain arch (e.g. non-Android Linux often runs on x86,
DragonFlyBSD only runs on x86, and so on..).

Endianness will be detected by pointer arithmetic anyway, and it's
better to rely on it to handle any sort of newly supported archs
in the future.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoWindows: re-enable the mmap ioengine and fix static asserts
Rebecca Cran [Thu, 9 Feb 2017 22:17:27 +0000 (22:17 +0000)]
Windows: re-enable the mmap ioengine and fix static asserts

Windows supports mmap functionality via the CreateFileMapping and
MapViewOfFile functions. Update posix.c and re-enable the mmap
ioengine in the Makefile.

Also, enable the built-in static asserts, without which the build
breaks when configured with --disable-optimizations.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix the return type of log_err and log_info functions
Rebecca Cran [Tue, 7 Feb 2017 16:13:19 +0000 (09:13 -0700)]
Fix the return type of log_err and log_info functions

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoWindows: Update the EULA year and add more examples to the installer
Rebecca Cran [Mon, 6 Feb 2017 16:17:24 +0000 (16:17 +0000)]
Windows: Update the EULA year and add more examples to the installer

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogfio: fix git location for fio
Jens Axboe [Mon, 13 Feb 2017 19:43:17 +0000 (12:43 -0700)]
gfio: fix git location for fio

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoBe more verbose on endianness detection failure
Tomohiro Kusumi [Mon, 13 Feb 2017 16:13:43 +0000 (18:13 +0200)]
Be more verbose on endianness detection failure

> I think this deserves and enum. That also enables the compiler to check
> if the below switch covers all the potential values.

Updated with enum ENDIAN_XXX, which you can change the naming if it
doesn't sound good.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agosteadystate: Use calloc(3)
Tomohiro Kusumi [Sat, 11 Feb 2017 14:28:54 +0000 (16:28 +0200)]
steadystate: Use calloc(3)

which guarantees zero clear.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd a comment to clarify 941bda94
Tomohiro Kusumi [Sat, 11 Feb 2017 14:28:53 +0000 (16:28 +0200)]
Add a comment to clarify 941bda94

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: try to disable weak linking on OSX
Sitsofe Wheeler [Sat, 11 Feb 2017 09:19:46 +0000 (09:19 +0000)]
configure: try to disable weak linking on OSX

macOS 10.12 introduced support for clock_gettime() but when compiling
with Xcode 8 (or later) and targeting 10.11 (or older) the clock_gettime
symbol is now found at configure/compile time but referenced weakly.
Running the created binary on 10.11 results in a "dyld: lazy symbol
binding failed: Symbol not found: _clock_gettime" error.

This drama has played out across other projects:
- https://github.com/Homebrew/homebrew-core/issues/2674
- https://github.com/curl/curl/issues/1069
- https://svn.filezilla-project.org/filezilla?view=revision&revision=7824OO
- https://github.com/mesonbuild/meson/pull/949
- https://github.com/Homebrew/homebrew-core/issues?utf8=%E2%9C%93&q=label%3Aclock_gettime%20

Options for working around this issue include:
1. Turn references to functions that might be weak into compilation
   errors (-Werror=partial-availability).
2. Disable visibility of functions that might be weak at compile link
   time (-Wl,-no_weak_imports).
3. Change configure tests to directly check the targeted OSX version and
   fail the test if it is too old to have the required function.
4. Make an empty function declaration marked with
   __attribute__((weak_import)) that is used if the symbol would
   otherwise be unavailable. It is then possible to check if the symbol
   is NULL at runtime to determine availability.

This commit does 2. when targeting the Darwin platform after checking if
the compiler/linker work with that option. It also changes the clockid_t
configure test to no longer use clock_gettime() because targeting OSX
10.11 when using the 10.12 SDK results in having the clockid_t typedef
without having the clock_gettime function. The combination of these
fixes should solve https://github.com/axboe/fio/issues/305 .

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agofnv: work with non-64-bit aligned chunks of data
Jens Axboe [Tue, 7 Feb 2017 22:11:37 +0000 (15:11 -0700)]
fnv: work with non-64-bit aligned chunks of data

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobloom: if we're not setting bits, break after first failed mask check
Jens Axboe [Tue, 7 Feb 2017 22:10:26 +0000 (15:10 -0700)]
bloom: if we're not setting bits, break after first failed mask check

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: don't crash server on failure to open output log
Jens Axboe [Tue, 7 Feb 2017 16:06:16 +0000 (09:06 -0700)]
init: don't crash server on failure to open output log

We don't use it on the server side anyway.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agodoc: minor HOWTO fixes
Sitsofe Wheeler [Sat, 4 Feb 2017 09:10:13 +0000 (09:10 +0000)]
doc: minor HOWTO fixes

- Add a note to to size/offset to see the other for constraining I/O to
  a range (requested in https://github.com/axboe/fio/issues/298 ).
- Minor grammar and typo fixes.
- Update example fio ETA status to match what modern fio would likely
  display.
- Explain f= is open files and note that trim may be displayed.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agofix to replay_align on iolog
Dylan Fairchild [Thu, 2 Feb 2017 18:37:00 +0000 (10:37 -0800)]
fix to replay_align on iolog
previous code was rejecting all positive alignment values and
only accepting align to 0 value, opposite of what it should be doing

3 years agodoc: document profiles, minor fixes
Sitsofe Wheeler [Sun, 29 Jan 2017 21:20:08 +0000 (21:20 +0000)]
doc: document profiles, minor fixes

- Add documentation for profile to the HOWTO.
- Explictly document the how parameters that take time are interpreted
  when the unit is omitted.
- Add disable_bw_measurement default alias.
- Minor sentence rewording.
- Change max_latency to say it returns ETIMEDOUT rather than ETIME
  to match the commit c5b3adc41e0a91be5c6accdbe6506522ebddfbdd .

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: minor consistency and spelling changes
Sitsofe Wheeler [Fri, 27 Jan 2017 09:43:12 +0000 (09:43 +0000)]
doc: minor consistency and spelling changes

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agodoc: minor documentation changes
Sitsofe Wheeler [Thu, 26 Jan 2017 17:02:47 +0000 (17:02 +0000)]
doc: minor documentation changes

- Change some parameter types to be time in the HOWTO.
- Put a space before the end of sentence but after links in the README.
- Update the Red Hat fio package information.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
3 years agoMerge branch 'shm_rm' of https://github.com/sitsofe/fio
Jens Axboe [Thu, 26 Jan 2017 17:07:48 +0000 (10:07 -0700)]
Merge branch 'shm_rm' of https://github.com/sitsofe/fio

3 years agoMerge branch 'sphinx-doc' of https://github.com/termim/fio
Jens Axboe [Thu, 26 Jan 2017 17:04:40 +0000 (10:04 -0700)]
Merge branch 'sphinx-doc' of https://github.com/termim/fio

3 years agoshm: have os remove shared memory if fio dies unexpectedly
Sitsofe Wheeler [Tue, 24 Jan 2017 06:31:58 +0000 (06:31 +0000)]
shm: have os remove shared memory if fio dies unexpectedly

When fio doesn't exit cleanly shared memory segments are left behind on
those platforms where shared memory is supported.

Due to how fio manages shared memory lifetime we can combat the above by
marking the shared memory segment as removed shortly after we attach to
it for the first time if the OS offers both the following (non-standard)
behaviours:

1. A shared memory segment marked for removal with IPC_RMID isn't
   destroyed until all processes have detached from it.
2. After 1. has occurred it is still possible to (re)attach to the
   marked-for-removal shared memory segment.

Linux has always done the above so enable the workaround there. OpenBSD
switched to having the above behaviours in 5.1 (see shmctl in
https://www.openbsd.org/plus51.html ) so add a comment about it in
os/os-openbsd.h . FreeBSD only has the above behaviours if
kern.ipc.shm_allow_removed (introduced in 5.2) is 1 and this only became
the default in FreeBSD 11
(https://svnweb.freebsd.org/base?view=revision&revision=289112 ) so add
a comment in os/os-freebsd.h . DragonFly also has the FreeBSD flag but
at the time of writing it defaults to 0 so add a comment to
os/os-dragonfly.h .

There's a nice (if dated) post discussing how IPC_RMID behaves on
various OSes in https://www.cygwin.com/ml/cygwin/2007-11/msg00078.html
("Re: cygwin stable and cvs snapshot - fork() bug").  While fio doesn't
support the Cygwin "OS", Cygwin itself has the
kern.ipc.shm_allow_removed option in a config file and defaults it to
no.

3 years agoRemove/Move Linux specific sysfs_root field from thread_data
Tomohiro Kusumi [Mon, 23 Jan 2017 15:13:11 +0000 (00:13 +0900)]
Remove/Move Linux specific sysfs_root field from thread_data

struct thread_data is better without having a Linux specific field
(even if sysfs_root is simply unused on non Linux environment) given
that other Linux specific code such as diskutil/cgroup/blktrace/etc
are compiled only on Linux.

This commit uses td->files[0]->du->sysfs_root instead of td->sysfs_root,
but results the same from the way diskutil is currently implemented.
Besides being platform independent, it's also better in terms of data
structure since sysfs_root isn't always per-td attribute as mentioned
below.

For example, when N(>1) files are specified via filename= option
(e.g. filename=/dev/sdb:/dev/sdc for N=2), td:files is 1:N, therefore
td:du and td:du->sysfs_root are also 1:N in case of /dev/sdb:/dev/sdc,
however td:td->sysfs_root is obviously 1:1 which in this case means
td->sysfs_root is sysfs root for /dev/sdb (regardless of other files)
from the way td->sysfs_root is set only if it hasn't yet been set.

In other words, if filename=/dev/sdb:/dev/sdc is given, ioscheduler=
option only works against /dev/sdb, which could be different from what
users would expect, but this commit just removes sysfs_root from
struct thread_data without any functional change.

Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com>
Signed-off-by: Jens Axboe <axboe@fb.com>