fio.git
3 years agoFio 2.3 fio-2.3
Jens Axboe [Wed, 23 Dec 2015 16:39:04 +0000 (09:39 -0700)]
Fio 2.3

Let's switch to a simpler numbering scheme. 2.3 follows 2.2.13, so
it should have been 2.2.14. But from now on, let's just use a simpler
MAJOR.MINOR release numbering scheme.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogfio: use ffz64 for 64-bit types
Jens Axboe [Wed, 23 Dec 2015 16:10:05 +0000 (09:10 -0700)]
gfio: use ffz64 for 64-bit types

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoarch: add ffs64 and ffz64
Jens Axboe [Wed, 23 Dec 2015 16:09:23 +0000 (09:09 -0700)]
arch: add ffs64 and ffz64

These just use the normal C software version, extended to 64-bit
even on 32-bit archs. The fast variant is still machine word native.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix runtime of terse output
shoichi.sawada@toshiba.co.jp [Tue, 22 Dec 2015 08:16:24 +0000 (08:16 +0000)]
Fix runtime of terse output

Hi,

When I used fio with option of '--status-interval=1 --minimal' and 'random_distribution=zipf:1.2',
'runtime' of 2nd line was 18446744073709551345.
This value is -271 in int_64.

This is because 'runtime' was subtracted in function '__show_running_run_stats'.
If 'td->io_bytes' is 0 when this function was called,
and 'td->io_bytes' is not 0 when '__show_run_stats' returned,
'runtime' is subtracted by 'rt' even though 'runtime' have not added.

I fixed the problem by applying this patch to fio-2.2.13.

Regards
Shoichi

----

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: continue on failed wait_for() check, not break
Jens Axboe [Tue, 22 Dec 2015 20:40:14 +0000 (13:40 -0700)]
backend: continue on failed wait_for() check, not break

If we have this job file:

[a]

[b]
wait_for=a

[c]
wait_for=a

[d]

then we expect a/d to run together, then b/c after a is done. However,
right now we break out of the job loop, so it acts like a stonewall.
That means that a will run first, then b/c/d. That isn't correct.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogfio: update for 64-bit option category/group
Jens Axboe [Tue, 22 Dec 2015 18:11:14 +0000 (11:11 -0700)]
gfio: update for 64-bit option category/group

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd support for waiting for other jobs by name
Andrey Kuzmin [Tue, 22 Dec 2015 16:21:25 +0000 (09:21 -0700)]
Add support for waiting for other jobs by name

We have stonewall to barrier against previous jobs before a new
job runs. This adds support for 'wait_for', which allows the
user to explicitly wait for a given job before starting.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoparse: ensure that option group/category hold 64-bit
Jens Axboe [Tue, 22 Dec 2015 16:04:27 +0000 (09:04 -0700)]
parse: ensure that option group/category hold 64-bit

options.c:3459:12: warning: large integer implicitly truncated to unsigned type [-Woverflow]
   .group = FIO_OPT_G_INVALID,

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agorewrote libhdfs engine
Fabrice Bacchella [Tue, 22 Dec 2015 11:26:51 +0000 (12:26 +0100)]
rewrote libhdfs engine

3 years agoMake options mask a 64-bit type
Jens Axboe [Mon, 21 Dec 2015 23:03:01 +0000 (16:03 -0700)]
Make options mask a 64-bit type

We're running out of bit numbers, double it up.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: only terminate multiple loops if we did no IO
Jens Axboe [Mon, 21 Dec 2015 17:01:40 +0000 (10:01 -0700)]
backend: only terminate multiple loops if we did no IO

Commit 095196b1e76b introduced a stop point if do_io() returned
nothing, but that only counts writes and trims. Change that to
count all types of IO.

Fixes: 095196b1e76b ("backend: terminate loop if we didn't do IO")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: count iterative bytes for progress
Jens Axboe [Sat, 19 Dec 2015 05:33:35 +0000 (22:33 -0700)]
backend: count iterative bytes for progress

Commit ac28d9053679 changed it so we don't clear ->bytes_done[]
necessarily for each loop. However, this introduced a busy loop
where we this we did IO and continue to loop, but we really
didn't.

Fixes: ac28d9053679 ("Fixup bw/iops logging for short runs")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: command tag cleanup
Jens Axboe [Fri, 18 Dec 2015 17:36:57 +0000 (10:36 -0700)]
server: command tag cleanup

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: don't store command tag on the stack
Jens Axboe [Fri, 18 Dec 2015 16:17:41 +0000 (09:17 -0700)]
server: don't store command tag on the stack

It'll go out-of-scope, making the server unhappy.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient: iolog cleanups
Jens Axboe [Fri, 18 Dec 2015 15:57:24 +0000 (08:57 -0700)]
client: iolog cleanups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: code cleanups
Jens Axboe [Thu, 17 Dec 2015 22:23:48 +0000 (15:23 -0700)]
server: code cleanups

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: remove leftover debug statement
Jens Axboe [Thu, 17 Dec 2015 21:59:49 +0000 (14:59 -0700)]
server: remove leftover debug statement

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: fix zlib typo
Jens Axboe [Thu, 17 Dec 2015 21:55:16 +0000 (14:55 -0700)]
configure: fix zlib typo

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient/server: transparent handling of storing compressed logs
Jens Axboe [Thu, 17 Dec 2015 21:54:15 +0000 (14:54 -0700)]
client/server: transparent handling of storing compressed logs

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix compile warning for !zlib
Jens Axboe [Thu, 17 Dec 2015 21:53:45 +0000 (14:53 -0700)]
Fix compile warning for !zlib

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient/server: various bug fixes
Jens Axboe [Thu, 17 Dec 2015 17:24:03 +0000 (10:24 -0700)]
client/server: various bug fixes

- Better handling of vectored commands
- Improve sk_out mutex handling (don't alloc separately)
- Add support for sync sending of network data
- Prep for network xmit of logs

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoEnsure that command line options also end up in json output
Jens Axboe [Thu, 17 Dec 2015 15:32:15 +0000 (08:32 -0700)]
Ensure that command line options also end up in json output

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agogclient: don't free pdu on iolog return
Jens Axboe [Wed, 16 Dec 2015 22:05:54 +0000 (15:05 -0700)]
gclient: don't free pdu on iolog return

The command is freed separately, don't free it twice.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient: fix double free of command on timeout
Jens Axboe [Wed, 16 Dec 2015 22:05:22 +0000 (15:05 -0700)]
client: fix double free of command on timeout

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: dump failed header, if verification fails
Jens Axboe [Wed, 16 Dec 2015 15:59:38 +0000 (08:59 -0700)]
verify: dump failed header, if verification fails

And verify_dump=1 is enabled.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoparse: fix segfault if posval option doesn't have a value
Jens Axboe [Wed, 16 Dec 2015 15:32:30 +0000 (08:32 -0700)]
parse: fix segfault if posval option doesn't have a value

If the user did

continue_on_error

in the job file without setting one of the possible values
for that option, we'd crash in string compare. Now fio will
exit in error, showing the valid options instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient/server: ensure we don't overrun memory for long option values
Jens Axboe [Wed, 16 Dec 2015 04:02:51 +0000 (21:02 -0700)]
client/server: ensure we don't overrun memory for long option values

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient: fix segfault for !json output
Jens Axboe [Wed, 16 Dec 2015 03:47:14 +0000 (20:47 -0700)]
client: fix segfault for !json output

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient/server: pass back job options to client
Jens Axboe [Wed, 16 Dec 2015 03:33:29 +0000 (20:33 -0700)]
client/server: pass back job options to client

Also fill in a few more missing pieces from the server based
json output, like time stamp.

Now the json output should match the local, on all accounts.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient: cleanup json output
Jens Axboe [Tue, 15 Dec 2015 20:48:32 +0000 (13:48 -0700)]
client: cleanup json output

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agostat: work around 'numjobs' oddity in dumping job output
Jens Axboe [Tue, 15 Dec 2015 17:38:05 +0000 (10:38 -0700)]
stat: work around 'numjobs' oddity in dumping job output

Inherit options between duped jobs, and don't include 'numjobs' in the
global option output.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFirst stab at adding job options to json output
Jens Axboe [Tue, 15 Dec 2015 17:30:57 +0000 (10:30 -0700)]
First stab at adding job options to json output

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoRework 'dump_cmdline'
Jens Axboe [Tue, 15 Dec 2015 16:53:22 +0000 (09:53 -0700)]
Rework 'dump_cmdline'

Store the options in &td->opt_list. Should be no change for
dump_cmdline, but it will prepare us for being able to jump
the job definition as part of the json output.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoMerge branch 'server'
Jens Axboe [Tue, 15 Dec 2015 15:42:13 +0000 (08:42 -0700)]
Merge branch 'server'

3 years agoserver: bump listen() backlog
Jens Axboe [Tue, 15 Dec 2015 02:26:18 +0000 (19:26 -0700)]
server: bump listen() backlog

With the default if 0, we trigger a lot of these:

[29055.817884] TCP: request_sock_TCP: Possible SYN flooding on port 8765. Sending cookies.  Check SNMP counters.

Set it to 4, even 2 seems to be enough to get rid of it.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoclient: make SEND_ETA timeout non-fatal
Jens Axboe [Tue, 15 Dec 2015 01:51:22 +0000 (18:51 -0700)]
client: make SEND_ETA timeout non-fatal

If we fail a single ETA, then just soldier on. If we fail 5 in a row,
then give up permanently.

Also bring the command timeout back to 5s per command.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFio 2.2.13 fio-2.2.13
Jens Axboe [Mon, 14 Dec 2015 21:53:35 +0000 (14:53 -0700)]
Fio 2.2.13

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: create sk_out_key earlier
Jens Axboe [Mon, 14 Dec 2015 21:46:32 +0000 (14:46 -0700)]
server: create sk_out_key earlier

The previous code worked on Linux, since we seem to get '0' as
the key always. It segfaults on other platforms, since we
call pthread_getspecific() before the key has been created.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: don't pass sk_out into accept loop
Jens Axboe [Mon, 14 Dec 2015 21:26:07 +0000 (14:26 -0700)]
server: don't pass sk_out into accept loop

Previous commit killed the use of it, just define it locally
in accept_loop().

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: cleanup exported functions
Jens Axboe [Mon, 14 Dec 2015 20:31:34 +0000 (13:31 -0700)]
server: cleanup exported functions

Lots of cruft left from previously.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: comments, and clear sk_out->sk when we close the 'sk'
Jens Axboe [Mon, 14 Dec 2015 17:55:24 +0000 (10:55 -0700)]
server: comments, and clear sk_out->sk when we close the 'sk'

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: ensure each connection sets up its own sk_out
Jens Axboe [Mon, 14 Dec 2015 17:48:29 +0000 (10:48 -0700)]
server: ensure each connection sets up its own sk_out

Otherwise multiple connections will screw each other up. This
makes multiple connections to a single backend work again.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: comment sk_entry struct and remove hole
Jens Axboe [Fri, 11 Dec 2015 22:13:41 +0000 (15:13 -0700)]
server: comment sk_entry struct and remove hole

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFixup and improve per-thread data
Jens Axboe [Fri, 11 Dec 2015 20:25:24 +0000 (13:25 -0700)]
Fixup and improve per-thread data

- Get rid of backend_data, just use sk_out
- Add reference counting to sk_out and proper assign/drop helpers
- Ensure we pass it to other threads created
- Fix leaks of sk_out

This should get us closer to the new model actually working as
intended.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: cleanup and proper error returns
Jens Axboe [Fri, 11 Dec 2015 00:46:05 +0000 (17:46 -0700)]
server: cleanup and proper error returns

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: make the io log transmit use the new infrastructure
Jens Axboe [Thu, 10 Dec 2015 23:07:32 +0000 (16:07 -0700)]
server: make the io log transmit use the new infrastructure

Add support for vectored commands, and pass them on the backend
pipe as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: rewrite message handling
Jens Axboe [Thu, 10 Dec 2015 22:32:15 +0000 (15:32 -0700)]
server: rewrite message handling

Currently we don't do any synchronization when we send out
network messages, this can cause a variety of issues. Instead
of adding serialization for transmits, let the messages queue
up in the network backend, and let the backend send them out.

Work in progress, not everything has been tested with this.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agolog: remove log_local_buf()
Jens Axboe [Thu, 10 Dec 2015 22:31:02 +0000 (15:31 -0700)]
log: remove log_local_buf()

Have the server return -1 if it can't log, then we can handle
it internally.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd backend related data
Jens Axboe [Thu, 10 Dec 2015 21:26:44 +0000 (14:26 -0700)]
Add backend related data

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agomutex: fixup fio_mutex_down_timeout()
Jens Axboe [Thu, 10 Dec 2015 21:22:43 +0000 (14:22 -0700)]
mutex: fixup fio_mutex_down_timeout()

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoAdd exitall_on_error option
Jens Axboe [Thu, 10 Dec 2015 18:15:52 +0000 (11:15 -0700)]
Add exitall_on_error option

Like exitall, but only if a job exits due to erroring out.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoeta: use bool for forcing jobs_eta return
Jens Axboe [Wed, 9 Dec 2015 19:44:09 +0000 (12:44 -0700)]
eta: use bool for forcing jobs_eta return

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: ensure we send the right sized jobs_eta when faking it
Jens Axboe [Wed, 9 Dec 2015 19:40:52 +0000 (12:40 -0700)]
server: ensure we send the right sized jobs_eta when faking it

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoserver: always end back ETA reply
Jens Axboe [Wed, 9 Dec 2015 19:37:05 +0000 (12:37 -0700)]
server: always end back ETA reply

Otherwise the client will time out waiting for a response. If we don't
have a local jobs_eta, then just send a zero filled one.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agothread_options: kill converted cpu masks
Jens Axboe [Wed, 9 Dec 2015 18:23:15 +0000 (11:23 -0700)]
thread_options: kill converted cpu masks

We can't currently convert them in cconv.c, so let's just disable
them as they eat a lot of space in the thread options.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: don't copy log samples for compression
Jens Axboe [Wed, 9 Dec 2015 16:16:13 +0000 (09:16 -0700)]
iolog: don't copy log samples for compression

For larger settings of log_compression, the allocation and copying of
the log will adversely affect performance and logged latencies. Instead
just reference the log in the compression code, while resetting the
current log to a smaller size. We pay a slightly higher cost in memory
until compression has completd, but it saves us from allocating and
copying tens (or hundreds) of megabytes inline.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoeta: fix comparison reversal for time based jobs
Jens Axboe [Wed, 9 Dec 2015 16:08:10 +0000 (09:08 -0700)]
eta: fix comparison reversal for time based jobs

If time_based=1 and runtime=x, then we know the job will take x
time to run. Fix a bug introduced in a previous commit.

Fixes: ("Fix ETA calculations")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix ramp time for io_submit_mode=offload
Jens Axboe [Wed, 9 Dec 2015 15:37:50 +0000 (08:37 -0700)]
Fix ramp time for io_submit_mode=offload

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: terminate loop if we didn't do IO
Jens Axboe [Wed, 9 Dec 2015 15:28:00 +0000 (08:28 -0700)]
backend: terminate loop if we didn't do IO

The below commit modified how we clear stats, and that can sometimes
cause fio to hang around when ramp + runtime has been exceeded. The
below job file showed the issue:

[test-job]
bs=4k
rw=read
direct=1
runtime=6s
ramp_time=5s
ioengine=null
iodepth=32
size=100g

Where it would do the ramp time, then count down the 6s of runtime, then
fio would just sit there busy looping.

Fixes: ac28d9053679 ("Fixup bw/iops logging for short runs")
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: add log_compression_cpus option
Jens Axboe [Tue, 8 Dec 2015 22:45:12 +0000 (15:45 -0700)]
options: add log_compression_cpus option

We generally don't want online log compression to interfere with the
IO jobs. Add an option that allows us to contain the log compression
to a specific CPU, or set of CPUs.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: check for dryrun in cpu mask setting
Jens Axboe [Tue, 8 Dec 2015 22:31:22 +0000 (15:31 -0700)]
options: check for dryrun in cpu mask setting

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: cleanup up data unref'ing
Jens Axboe [Tue, 8 Dec 2015 22:20:54 +0000 (15:20 -0700)]
iolog: cleanup up data unref'ing

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: cleanup log writing
Jens Axboe [Tue, 8 Dec 2015 22:17:27 +0000 (15:17 -0700)]
iolog: cleanup log writing

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix potential duplicate definition of compress init/exit
Jens Axboe [Tue, 8 Dec 2015 20:04:59 +0000 (13:04 -0700)]
iolog: fix potential duplicate definition of compress init/exit

Move the code inside the #else part of the zlib check.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: fix build with zlib not being installed
Jens Axboe [Tue, 8 Dec 2015 19:51:23 +0000 (12:51 -0700)]
iolog: fix build with zlib not being installed

backend.o: In function `thread_main':
/home/axboe/git/fio/backend.c:1553: undefined reference to `iolog_compress_init'
/home/axboe/git/fio/backend.c:1656: undefined reference to `iolog_compress_exit'
collect2: error: ld returned 1 exit status
make: *** [fio] Error 1

Provide dummy init/exit functions.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: cleanup data wait
Jens Axboe [Tue, 8 Dec 2015 19:43:40 +0000 (12:43 -0700)]
iolog: cleanup data wait

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: remember to destroy cond/lock when done
Jens Axboe [Tue, 8 Dec 2015 19:42:26 +0000 (12:42 -0700)]
iolog: remember to destroy cond/lock when done

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: make workqueue_exit() safe for multiple exit calls
Jens Axboe [Tue, 8 Dec 2015 19:37:30 +0000 (12:37 -0700)]
workqueue: make workqueue_exit() safe for multiple exit calls

This might happen if we fail init.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agorate-submit: clean up init/exit helpers
Jens Axboe [Tue, 8 Dec 2015 19:36:27 +0000 (12:36 -0700)]
rate-submit: clean up init/exit helpers

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: reinstate log compression nice level
Jens Axboe [Tue, 8 Dec 2015 19:31:38 +0000 (12:31 -0700)]
iolog: reinstate log compression nice level

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue/rate-submit: update comments
Jens Axboe [Tue, 8 Dec 2015 19:30:24 +0000 (12:30 -0700)]
workqueue/rate-submit: update comments

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: ensure that workqueue_enqueue() can't fail
Jens Axboe [Tue, 8 Dec 2015 19:25:07 +0000 (12:25 -0700)]
workqueue: ensure that workqueue_enqueue() can't fail

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: replace tp usage with workqueue
Jens Axboe [Tue, 8 Dec 2015 18:35:39 +0000 (11:35 -0700)]
iolog: replace tp usage with workqueue

Now that workqueue.c is generic enough to handle any kind of
work offloading, convert the online log compression to workqueue
usage and kill off lib/tp.*

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: add nice support
Jens Axboe [Tue, 8 Dec 2015 15:56:30 +0000 (08:56 -0700)]
workqueue: add nice support

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agobackend: move rated submit code to its own file
Jens Axboe [Tue, 8 Dec 2015 15:51:43 +0000 (08:51 -0700)]
backend: move rated submit code to its own file

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: move last bits of end accounting to caller
Jens Axboe [Tue, 8 Dec 2015 15:44:56 +0000 (08:44 -0700)]
workqueue: move last bits of end accounting to caller

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: move private accounting to caller
Jens Axboe [Tue, 8 Dec 2015 15:36:01 +0000 (08:36 -0700)]
workqueue: move private accounting to caller

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: move private exit code to caller
Jens Axboe [Tue, 8 Dec 2015 15:32:46 +0000 (08:32 -0700)]
workqueue: move private exit code to caller

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: move init worker private code to the caller
Jens Axboe [Tue, 8 Dec 2015 15:28:53 +0000 (08:28 -0700)]
workqueue: move init worker private code to the caller

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: move 'td' private data to the workqueue user
Jens Axboe [Tue, 8 Dec 2015 15:23:24 +0000 (08:23 -0700)]
workqueue: move 'td' private data to the workqueue user

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: remove knowledge of td queue state
Jens Axboe [Tue, 8 Dec 2015 05:35:31 +0000 (22:35 -0700)]
workqueue: remove knowledge of td queue state

Add pre sleep checker and worker ops, so we can push this to the
owner of the workqueue.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: add a workqueue_work type
Jens Axboe [Tue, 8 Dec 2015 05:20:02 +0000 (22:20 -0700)]
workqueue: add a workqueue_work type

Don't pass in a struct io_u, that's specific to the use case of
using this for queueing IO. Use a generic work type, so we can reuse
this helper code for other purposes.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: don't use ioengine return codes
Jens Axboe [Tue, 8 Dec 2015 05:13:22 +0000 (22:13 -0700)]
workqueue: don't use ioengine return codes

Use true/false as return, signaling ok/busy.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: remove knowledge of io issue (and others) stats
Jens Axboe [Tue, 8 Dec 2015 05:09:24 +0000 (22:09 -0700)]
workqueue: remove knowledge of io issue (and others) stats

This doesn't belong in the workqueue code, let the caller deal with
it instead.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoiolog: ensure we always store compressed, if log_store_compressed == 1
Jens Axboe [Tue, 8 Dec 2015 04:56:34 +0000 (21:56 -0700)]
iolog: ensure we always store compressed, if log_store_compressed == 1

Before this commit, we depend on log_compression= always being set.
But we might want the logs store compressed, even if we don't
continually compress chunks at runtime.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix latency logging if disable_slat and disable_clat is set
Jens Axboe [Tue, 8 Dec 2015 03:24:51 +0000 (20:24 -0700)]
Fix latency logging if disable_slat and disable_clat is set

We can still do total latencies.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoFix stat summing for unified_rw_reporting
Jens Axboe [Mon, 7 Dec 2015 21:30:34 +0000 (14:30 -0700)]
Fix stat summing for unified_rw_reporting

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoverify: fix header verification version check
Jens Axboe [Mon, 7 Dec 2015 17:45:57 +0000 (10:45 -0700)]
verify: fix header verification version check

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: grab sw->lock for flag manipulation
Jens Axboe [Mon, 7 Dec 2015 17:43:45 +0000 (10:43 -0700)]
workqueue: grab sw->lock for flag manipulation

For consistency, should not matter in the shutdown path.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoinit: have set_debug() check for NULL optarg
Jens Axboe [Mon, 7 Dec 2015 17:42:24 +0000 (10:42 -0700)]
init: have set_debug() check for NULL optarg

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agot/genzip: cast division to double
Jens Axboe [Mon, 7 Dec 2015 17:40:18 +0000 (10:40 -0700)]
t/genzip: cast division to double

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agooptions: don't throw away bssplit() return value
Jens Axboe [Mon, 7 Dec 2015 17:37:30 +0000 (10:37 -0700)]
options: don't throw away bssplit() return value

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agocrc/test: don't throw away results
Jens Axboe [Mon, 7 Dec 2015 17:34:25 +0000 (10:34 -0700)]
crc/test: don't throw away results

We don't want the compiler thinking that it can toss them.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: fix potential ABBA deadlock in stats summing
Jens Axboe [Fri, 4 Dec 2015 20:15:36 +0000 (13:15 -0700)]
workqueue: fix potential ABBA deadlock in stats summing

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agopowerpc: enable cpu clock for powerpc64
Li Zhong [Fri, 4 Dec 2015 05:50:06 +0000 (13:50 +0800)]
powerpc: enable cpu clock for powerpc64

This patch tries to enable cpu clock for powerpc64, the code is copied from
mftb() in kernel source.

The two instructions after mfspr are added in the kernel to solve an erratum on
Cell and fsl booke CPUs. On those CPUs, 64 bits mftb is not atomic, so it is
possible that the low order 32 bits are already reset to 0x00000000 but the
high order bits are not yet incremented by one.

Don't know how to tell whether it will be running on those CPUs or not, I just
keep the above fix for all ppc64 CPUs. Even if we have some method to check
whether we are on those CPUs or not at run time, I think the check won't cost
less than the two added instructions. Maybe we could use the similar fix up
code kernel uses to dynamically patch the instructions with nops if not needed.
But that would add much more complexity.

Signed-off-by: Li Zhong <zhong@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoworkqueue: properly account ->cur_depth
Jens Axboe [Fri, 4 Dec 2015 17:20:57 +0000 (10:20 -0700)]
workqueue: properly account ->cur_depth

Also handle edge cases better.

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoio_u: make io_u_quiesce() return how many IOs it completed
Jens Axboe [Fri, 4 Dec 2015 02:58:23 +0000 (19:58 -0700)]
io_u: make io_u_quiesce() return how many IOs it completed

Ditto for io_u_queued_complete().

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoposix_fadvise() returns positive error values
Jens Axboe [Thu, 3 Dec 2015 22:08:08 +0000 (15:08 -0700)]
posix_fadvise() returns positive error values

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agofilesetup: properly propagate error value from invalidate
Jens Axboe [Thu, 3 Dec 2015 21:52:27 +0000 (14:52 -0700)]
filesetup: properly propagate error value from invalidate

The IO engine ops ->invalidate() will return the error value, and
so does posix_fadvise().

Signed-off-by: Jens Axboe <axboe@fb.com>
3 years agoconfigure: add --disable-lex
Jens Axboe [Tue, 1 Dec 2015 16:25:08 +0000 (09:25 -0700)]
configure: add --disable-lex

On some centos 5 the lex -o option is broken. Add a way to just
disable lex/yacc for math support.

Signed-off-by: Jens Axboe <axboe@fb.com>