summaryrefslogtreecommitdiff
path: root/server.c
AgeCommit message (Collapse)Author
2020-02-04fio: better distinguish between high and low priorityVincent Fu
Change the variable names for the low priority latency variables to explicity distinguish them from the high priority latency variables. Also rename the JSON objects to do the same. This reduces the mental effort required to read the code and output. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04fio: report percentiles for slat, clat, latVincent Fu
Currently, fio can report percentiles for either completion latencies or total latencies. This patch adds the ability to simultaneously report percentiles for submission, completion, and total latencies. - simplify the JSON lat percentile reporting code - parsing the JSON output will be unaffected, although now setting lat_percentiles=1 no longer suppresses clat percentiles - this may break parsing of the normal output since more latency percentile blocks may be displayed - to maintain backward compatibility in the terse output, total latencies are reported if lat_percentiles=1 even when clat percentiles have also been computed - change the gtod_reduce callback to only change option values when gtod_reduce=1 - for the new cmdprio_percentage latencies, if lat_percentiles=1, *total* latency percentiles will be tracked. Otherwise, *completion* latency percentiles will be tracked. - also make sure we skip the high/low priority latency code if we are producing output for sync latencies. - make add_slat_sample use an unsigned long long for the time to be consistent with other functions Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-02-04fio: groundwork for adding slat, lat percentilesVincent Fu
Expand the data structure used for tabulating completion latency percentiles so that we will have the ability to tabulate percentiles for submission and total latency as well. This should not change fio's behavior. Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2020-01-22Per-command priority: Priority logging and libaio/io_uring cmdprio_percentagePhillip Chen
Add cmdprio_percentage option to libaio and io_uring engines to set ioprio on a per-command basis. Add tracking of high priority commands to be displayed separately in human readable and JSON outputs.
2020-01-22Whitespace standardizationPhillip Chen
Remove trailing whitespace.
2020-01-11server: Make it explicit that the setsockopt() return value is ignoredBart Van Assche
This patch fixes the following Coverity complaint: CID 169316 (#1 of 1): Unchecked return value from library (CHECKED_RETURN) 5. check_return: Calling setsockopt(sk, 1, 15, &opt, 4U) without checking return value. This library function may fail and return an error code. Signed-off-by: Bart Van Assche <bvanassche@acm.org>
2019-12-19Expand choices for exitallHannes Weisbach
Add exit_what thread_option with the following choices: - "group" (default) exit all jobs of the same group (equal to "exitall=1") - "stonewall", quit all jobs until the next stonewall - "all", quit *all* jobs Signed-off-by: Hannes Weisbach <hannes.weisbach@gmail.com>
2019-12-17client/server: add missing fsync data structuresVincent Fu
In client/server mode, fsync latencies were missing because a couple data structures were missed in the client/server send/receive thread stats code. This patch adds those data structures and now we are able to see the fsync latencies in client/server mode. Fixes: https://github.com/axboe/fio/issues/878 Signed-off-by: Vincent Fu <vincent.fu@wdc.com>
2019-08-14Optimize the code that copies stringsBart Van Assche
Using strncpy() to copy strings is suboptimal because strncpy writes a bunch of additional unnecessary null bytes. Use snprintf() instead of strncpy(). An additional advantage of snprintf() is that it guarantees that the output string is '\0'-terminated. This patch is an improvement for commit 32e31c8c5f7b ("Fix string copy compilation warnings"). Cc: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Bart Van Assche <bvanassche@acm.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04Fix string copy compilation warningsDamien Le Moal
Fix the many warnings that gcc 9 spits out. Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-12-14Add cache hit statsJens Axboe
With the aioring engine, we can get notified if a buffered read was a cache hit or if it hit media. Add that to the output stats for normal and json output. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-12-14client/server: convert nr_zone_resets on the wireJens Axboe
Fixes: fd5d733fa34 ("Collect and show zone reset statistics") Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-10-04Bool conversionsJens Axboe
Just various low hanging fruits in terms of local or global variables that are only 0/1, switch them to boolean. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-09-27server: be locally vocal about communication issuesJens Axboe
If we fail the CRC check or if we have a mismatched client/server, then we can't reliably transmit anything across the wire. For that case, in addition to attempting to relay this information across the wire, also print a local warning. This fixes a case where we silently exit when the client/server versions aren't properly matched. Fixes: https://github.com/axboe/fio/issues/620 Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-07-23Add support for >= 4G block sizesJeff Furlong
For trims, it's useful to be able to support larger than (or equal to) 4GB. This extends the block sizes support for that. Change from Jeff, various little fixups from me. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-06-05Fix variable shadowingJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-04-13Merge branch 'proc_group' of https://github.com/sitsofe/fioJens Axboe
* 'proc_group' of https://github.com/sitsofe/fio: doc: add cpus_allowed reference to log_compression_cpus appveyor: make 32 bit build target XP + minor fixes configure/Makefile: make Cygwin force less doc: add Windows processor group behaviour and Windows target option windows: target Windows 7 and add support for more than 64 CPUs windows: prepare for Windows build split windows: update EULA
2018-04-08server: fix dead assignment of variableJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-27windows: target Windows 7 and add support for more than 64 CPUsSitsofe Wheeler
Introduce support for targeting the build to either Windows XP or Windows 7 (the default). When targeting Windows 7 this allows us to use the process group APIs which are required are required to support more than 64 CPUs. These changes mean if you want a Windows binary that supports versions of Windows below Windows 7/Windows Server 2008 R2 you will need to explicitly use the --target-win-ver=xp configure option at build time. Such builds will lack features (such as the ability to access CPUs beyond those in fio's default process group) that are dependent on recent Windows APIs. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2018-03-22server: use scalloc() for sk_out allocationJens Axboe
This informs valgrind that it's properly initialized. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-21server: fix error handling for shared memory handlingJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-21Merge branch 'include_refactor' of https://github.com/sitsofe/fioJens Axboe
* 'include_refactor' of https://github.com/sitsofe/fio: Add include-what-you-use pragmas oslib: make str* compat functions more uniform Use POSIX path for poll.h and fcntl.h headers Refactor #includes and headers
2018-03-21server: handle shared mem pool allocation failuresJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-21Use POSIX path for poll.h and fcntl.h headersSitsofe Wheeler
musl complains when using include paths like <sys/poll.h> rather than POSIX's <poll.h> so change the poll.h include path and update the windows poll.h compatibility shim to match. Allegedly sys/poll.h was needed for GLIBCs that were older than 2.3 according to https://www.winehq.org/pipermail/wine-patches/2014-June/132877.html but I can still see that path in an ancient Red Hat 5.0 glibc-devel-2.0.7-32.i386.rpm ... Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2018-03-21Refactor #includes and headersSitsofe Wheeler
- Try and remove unneeded #include lines - Try and add #include lines that would allow the files to be built in a more standalone manner Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2018-03-21server: process STOP/QUIT commands out-of-lineJens Axboe
At the end of a job, the stats entries are queued up for out-of-line transmission, but we immediately queue a client STOP+QUIT command right after that. If the command handling loop doesn't wakeup and process the stats commands before we process the STOP+QUIT inline, then we lose the stats output from those commands. The result is that the client won't see the job output from that job, or the "All clients" output. We should impose proper ordering for all commands, but this is a quick fix to ensure that we don't hit the above mentioned but. For other commands we don't care about ordering between inline and out-of-line commands, so it's likely good enough to carry us forward. Reported-by: Jeff Furlong <jeff.furlong@wdc.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15Rename fio_mutex into fio_semBart Van Assche
Since fio_mutex implements a semaphore, rename fio_mutex into fio_sem. Rename those fio_sem instances that have 'mutex' embedded in their name. Rename the mutex.h and .c source files into fio_sem.h and .c respectively. Signed-off-by: Bart Van Assche <bart.vanassche@wdc.com>
2018-03-01Fix overflow of counters incremented on each I/O operationAlexander Larin
- In the thread_stat struct: uint32_t counters are updated to uint64_t. - In the io_u_plat_entry struct: unsigned int counters are updated to uint64_t. It fixes overflow of these counters. Signed-off-by: Alexander Larin <znahar@yandex-team.ru>
2018-01-05Fix client/server "all clients" reportingJeff Furlong
In commit e883cb3501de0f99297282c18842d50aaa7fa531, the default significant figures was added. The default value was preserved for local jobs and when client/server was used if jobs=1, but not when client/server was used if jobs > 1. This patch restores some reporting outputs ("all clients" output for IOPS and BW) when client/server is used if > 1 jobs, for example: OLD: All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan 3 14:30:13 2018 read: IOPS=0, BW=0 (0)(0B/5001msec) <--ZERO VALUES slat (nsec): min=762, max=108579, avg=1058.97, stdev=221.30 clat (nsec): min=254, max=4544.2k, avg=11297.64, stdev=8284.14 lat (usec): min=9, max=4545, avg=12.41, stdev= 8.30 bw ( KiB/s): min=290192, max=347160, per=49.35%, avg=303613.33, stdev=11582.87, samples=18 iops : min=72548, max=86790, avg=75903.33, stdev=2895.72, samples=18 NEW: All clients: (groupid=0, jobs=2): err= 0: pid=0: Wed Jan 3 15:08:33 2018 read: IOPS=147k, BW=575Mi (603M)(2876MiB/5001msec) <--CORRECT VALUES slat (nsec): min=841, max=61945, avg=1304.83, stdev=232.75 clat (nsec): min=544, max=4487.2k, avg=11736.68, stdev=9123.64 lat (usec): min=10, max=4489, avg=13.10, stdev= 9.13 bw ( KiB/s): min=260945, max=305976, per=49.99%, avg=294350.22, stdev=9192.53, samples=18 iops : min=65236, max=76494, avg=73587.56, stdev=2298.13, samples=18 Regards, Jeff Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-12-14server: convert more memset to on-stack initializationJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-12-14server: cleanup iolog pdu prepJens Axboe
We don't have to memset() the whole thing, we just need to ensure that the members we don't assign are cleared. Remove the memset and initialize the pdu on the stack. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-12-13server: initialize first iolog header properlyJens Axboe
We're sending potential stack garbage over the wire, init the whole pdu to zeroes before filling it in. This shuts up valgrind about using uninitialized memory. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-11-30steadystate: make flags conform to usual fio standardJens Axboe
__VAL are usually bit shift values, with VAL being the proper mask. For some reason, the steadystate code uses both as masks, fix that up. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-11-30server: process connection list before executing triggerJens Axboe
This shrinks the window a bit for allowing the backend to notice that jobs have gone away, and notify the client. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-11-22add significant_figures parameterJeff Furlong
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-09-18fix strncpy(3) copy lengthTomohiro Kusumi
0 terminate the last byte, and copy at most size-1. (or 0 terminate the last byte after copying upto size bytes) Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-09-13Add support for doing total latency percentilesJens Axboe
By default, fio does completion latency percentiles. Sometimes what you want is total IO latency percentiles, including the submission part as well. If that's your thing, then set lat_percentiles=1 and get that instead of the default completion latency percentiles. Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-08-26backend: verify-trigger fixesSitsofe Wheeler
Clear the trigger_timeout after we have found it has been exceeded in trigger_timedout() to stop us repeatedly firing the trigger again and again in the future. When --trigger-remote is unset, the server winds up with the remote trigger command represented by buf[] being zero length rather than buf being NULL in handle_trigger_cmd(). Cope with this by returning right away if we find the cmd string is NULL or empty in exec_trigger(). Make server mode fio terminate after sending the verify state back to the client so it goes on behave the same way as standalone fio. This stops the while loop in thread_main() restating and in turn prevents state being cleared away by clear_io_state(). Make keep_running() return false if td->terminate has been set Just In Case. The below would trigger the above problem before this patch: cat <<EOF > trigger.fio [global] ioengine=null time_based=1 runtime=10s size=1M bs=4k rate=4k [trigtest] EOF ./fio --server & ./fio --trigger-timeout=2s --trigger='hostname' --client=localhost \ --debug=net trigger.fio Fixes https://github.com/axboe/fio/issues/431 ("fio intermittently fails to perform IO when --trigger-timeout, --trigger is specified"). Reported-by: saurabhbpl <saurabh.shiva@gmail.com> Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
2017-06-23stat: Print one-line iops statAndreas Herrmann
This adds basic iops statistics (likewise to what we already have for bandwidth) to normal and json fio output. Example for normal output: bw ( KiB/s): min=42192, max=162068, per=0.10%, avg=93236.21, stdev=22180.02 iops : min=21090, max=81020, avg=46606.84, stdev=11088.40 lat (usec) : 2=55.13%, 4=21.11%, 10=0.36%, 20=0.03%, 50=23.14% Added handling for iops_stat to client/server and bump server version. Signed-off-by: Andreas Herrmann <aherrmann@suse.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-06-23server: don't use void* for pointer arithmetic (gcc)Tomohiro Kusumi
I think gcc extension should be avoided when it can be done by just changing a pointer type. Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-06-22client/server: bool conversionJens Axboe
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-06-21nanosecond: update completion latency recording and normal, json output to ↵Vincent Fu
use nanoseconds A new set of nsec bins (0-1, 2-3, 4-9, 10-19, ...) is created for tracking latencies. The number of io_u_plat[] latency bins is increased by about 50% to accommodate the greater resolution while retaining the same max latency value of ~9 seconds. Example output test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1 fio-2.20-39-g83b3 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=14.6GiB/s,w=0KiB/s][r=3832k,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=17621: Thu Jun 1 15:09:38 2017 read: IOPS=3784k, BW=14.4GiB/s (15.5GB/s)(144GiB/10000msec) clat (nsec): min=15, max=1685.2k, avg=26.92, stdev=433.62 lat (nsec): min=46, max=5989.9k, avg=67.31, stdev=1659.58 clat percentiles (nsec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 26], | 70.00th=[ 26], 80.00th=[ 27], 90.00th=[ 36], 95.00th=[ 41], | 99.00th=[ 46], 99.50th=[ 50], 99.90th=[ 104], 99.95th=[ 139], | 99.99th=[ 7840] bw ( MiB/s): min=13393, max=15647, per=0.10%, avg=14784.86, stdev=558.40 lat (nsec) : 20=20.87%, 50=78.57%, 100=0.43%, 250=0.10%, 500=0.01% lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=99.72%, sys=0.12%, ctx=321, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=37840231,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=14.4GiB/s (15.5GB/s), 14.4GiB/s-14.4GiB/s (15.5GB/s-15.5GB/s), io=144GiB (155GB), run=10000-10000msec
2017-06-21nanosecond: initial commit changing timeval to timespecVincent Fu
struct timeval has usec resolution whereas struct timespec has nsec resolution. This commit changes several dozen instances of struct timeval to struct timespec. All of the output is unchanged. The ticks to nsec conversion in this patch is broken but fixed in a later patch.
2017-06-02use correct syscall name in log_err()Tomohiro Kusumi
Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2017-05-03client/server: make sure that all elements in io_u_lat_m[] are transferred ↵Vincent Fu
and received FIO_IO_U_LAT_U_NR is 10 whereas FIO_IO_U_LAT_M_NR is 12. So we must iterate over io_u_lat_u and io_u_lat_m separately. Signed-off-by: Jens Axboe <axboe@fb.com>
2017-03-21Define struct sk_out in server.h (not server.c)Tomohiro Kusumi
struct sk_out isn't local to server.c, so it should be visible in its header (instead of having struct sk_out; in fio.h). Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2017-01-03Add missing trailing \n in log_err/info()Tomohiro Kusumi
Callers need trailing \n in case of fwrite(3) (if not syslog(3)). These are the ones that seem to need trailing \n not followed by another log function call as consequence. Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2017-01-02Convert group_run_stats to use bytes instead of KiB/KBRobert Elliott
In group_run_stats, convert these fields: * io_kb (bytes transferred) * agg (aggregate average bandwidth) * min_bw * max_bw from tracking either KiB or KB depending on kb_base to simply tracking bytes. Rename io_kb to iobytes to match this new meaning. This will simplify computing quantities and bandwidth rates with both IEC binary and SI decimal unit prefixes (e.g., GiB/s and GB/s) in future patches. Signed-off-by: Jens Axboe <axboe@fb.com>
2016-12-13Merge git://git.kernel.dk/fio into steady-stateVincent Fu
2016-12-07steadystate: instead of including ss_sum_y in thread_stat record whether ↵Vincent Fu
ss_sum_y is nonzero in ss_state via __FIO_SS_BUFFER_FULL