Alan D. Brunelle [Wed, 11 Feb 2009 18:40:09 +0000 (13:40 -0500)]
btt: fixed open in setup_ifile
Took my_open & my_fopen code from blktrace 2.0: needed to add in open
resource limit increasing stuff.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Alan D. Brunelle [Wed, 11 Feb 2009 18:23:21 +0000 (13:23 -0500)]
Synchronized trace gathering
Previously, each tracer thread would start gathering traces as soon as
it got going - which might slow down later thread start ups. This change
allows each thread to be ready to gather traces, and then the main
thread starts all the threads gathering at the same time.
Alan D. Brunelle [Wed, 11 Feb 2009 18:10:13 +0000 (13:10 -0500)]
Invoke gethostbyname once, handle errors better
Instead of invoking gethostbyname once per client, we only need to do it
once at initialization time. Plus: gethostbyname has a non-standard
errno reporting mechanism, handle this better.
Alan D. Brunelle [Wed, 11 Feb 2009 16:42:09 +0000 (11:42 -0500)]
Added accept as a system call needing resource increases
accept(2) opens a socket, and thus needs to handle EMFILE/ENFILE errors
like other system calls.
Alan D. Brunelle [Mon, 9 Feb 2009 20:11:49 +0000 (15:11 -0500)]
Rewrote blktrace to have a single thread per CPU
Massive changes: mostly around the notion of having much fewer threads
(instead of N(devs) X N(cpus) threads, we'll have just N(cpus)). This
is very important for larger systems (with lots of devices to
trace). A lot of the code was stolen from the original blktrace code,
major changes include:
o On the client side we only have a single thread per client CPU. Each
thread will then open all device files for that CPU, and use poll to
determine which file needs processing.
o For network client mode w/ sendfile, this means that a single socket
will carry all data to the remote network server. The network server
side will then distribute its reads off that one socket onto different
trace files.
o For network client mode w/out sendfile, we fall back to doing things
like piped mode: keep buffers of tracers read in, and then the main
thread will issue these on sockets to the server. In this case, the main
thread will still have a single socket per CPU.
o For networked mode we added an OPEN concept on the client side: as
soon as the connection to the server is set up, a "header" is sent
signifying that this connection will handle a <cpu, device> tuple. For
each socket opened on the client side, it will send a header per device
being managed. The server side will handle utilize opens to set up
appropriate data structures to handle incoming data streams.
o For both the OPEN and CLOSE headers the server will acknowledge with
a short write back to the client. This allows the client & server sides
to gracefully close socket connections.
o I also re-did the resource limitiation issue a bit differently: for
open calls (including socket) or for memory map/lock calls I have
provided a wrapper function that will try to increase specific limits as
needed. The previous method (attempting to do it at the beginning of the
run) fails for network server mode - you don't know at initialization
how many devices and CPUs will be handled.
o The standard output is slightly different in a few places, if this is
a problem w/ compatibility we can work to rectify that. The command line
argument handling is identical though.
o Using code stolen from Linux to manipulate doubly-linked lists. I've
found that this makes the code easier to read/write (but may be a bit of
overkill here...)
o The code passes valgrind quite well (at least for my tests so far).
The only nit has to do with inet_ntoa - but that is out of our control.
Thanks to Stefan Raspl <raspl@linux.vnet.ibm.com> for testing and
finding some issues and for providing suggestions.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Alan D. Brunelle [Fri, 23 Jan 2009 14:48:21 +0000 (09:48 -0500)]
Fix btt to handle large numbers of output files
Simply bump resource limits if file opens fail, and retry.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Alan D. Brunelle [Fri, 23 Jan 2009 10:04:42 +0000 (11:04 +0100)]
Increased limits to allow for large system runs
On 16-way w/ 104 disks and a 32-way w/ 96 disks, I was getting:
$ sudo blktrace -b 1024 -n 8 -I ../files
./cciss_c1d6.blktrace.10: Too many open files
Failed to start worker threads
Due to the nature of our N(cpus) X N(devices) order of file opens, and
our N(cpus) X N(devices) X N(buffers) X (buffer size) amount of mmaps()
going on we're exceeding both the RLIMIT_NOFILE and RLIMIT_MEMLOCK
limits.
This patch raises limits for RLIMIT_NOFILE and RLIMIT_MEMLOCK to
"infinity", and allows blktrace to handle the large(ish) systems. (If
these settings fail, we "guestimate" about how much we really need.)
There is still an underlying blktrace and/or kernel problem: The
directory /sys/kernel/debug/block/<DSF> where <DSF> is the device that
encountered the limit is left behind (not cleaned up correctly). This
stops blktrace from running a second time (even on another device):
$ ls /sys/kernel/debug/block
cciss_c1d6
$ sudo blktrace /dev/sda
BLKTRACESETUP: No such file or directory
Failed to start trace on /dev/sda
and requires a reboot. (Looking into that next, as this patch - whilst
stopping the original problem from happening - does not address the
secondary problem. And there may be some other ways for the secondary
problem to still occur...)
I also fixed a warning concerning ftruncate's return value being
ignored.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Signed-off-by: Jens Axboe <axboe@carl.(none)>
Martin Peschke [Wed, 21 Jan 2009 13:58:35 +0000 (14:58 +0100)]
A couple of min-counters weren't initialised correctly (thrput_r,
thrput_w).
We have got a perfectly working init function for this purpose.
Removing partially duplicated code.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Stefan Raspl <raspl@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Wed, 21 Jan 2009 13:58:35 +0000 (14:58 +0100)]
The git commit
11914a53d2ec2974a565311af327b8983d8c820d added __BLK_TA_ABORT
to blktrace_api.h. A corresponding addition to the blktrace tools repository
has been missing, breaking the API. Blkparse complained:
"Bad fs action
40010011"
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Mon, 12 Jan 2009 17:29:53 +0000 (18:29 +0100)]
Added no messages option to blkparse.c
Added a new option (-M, --no-msgs) option to blkparse: I have found that
the CFQ I/O scheduler sends a *tremendous* amount of messages, that
bloat the .bin file generated when using the -d option. The file sizes
can shrink by >50% when using the -M option in those case.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Wed, 12 Nov 2008 12:11:22 +0000 (07:11 -0500)]
gcc 4.3.2 has started to warn about:
gcc -Wall -W -O2 -g -I. -I.. -D_GNU_SOURCE -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -c -o output.o output.c
output.c: In function ‘output_section_hdr’:
output.c:57: warning: format not a string literal and no format
arguments
output.c: In function ‘__output_pip_avg’:
output.c:496: warning: format not a string literal and no format
arguments
output.c:496: warning: format not a string literal and no format
arguments
so this patch cleans this up.
Alan D. Brunelle [Tue, 11 Nov 2008 18:51:33 +0000 (13:51 -0500)]
Merge branch 'add-P'
Alan D. Brunelle [Tue, 11 Nov 2008 18:50:23 +0000 (13:50 -0500)]
Merge branch 'fix-m'
Alan D. Brunelle [Tue, 11 Nov 2008 18:46:13 +0000 (13:46 -0500)]
Added -P to create a data file w/ Q, D and C per line
Easy parsing for graph creation
Alan D. Brunelle [Tue, 11 Nov 2008 18:42:55 +0000 (13:42 -0500)]
Merge branch 'fix-m' into add-P
Alan D. Brunelle [Tue, 11 Nov 2008 18:41:05 +0000 (13:41 -0500)]
Fixed 'M' displays on per-io output and added in I/O separator
Alan D. Brunelle [Tue, 11 Nov 2008 18:40:10 +0000 (13:40 -0500)]
Fixed segfault in aqd.c : need to check for NULL (not requested)
Alan D. Brunelle [Mon, 10 Nov 2008 15:35:44 +0000 (10:35 -0500)]
Added in -z to provide running waiting-for-issue latencies
Alan D. Brunelle [Mon, 10 Nov 2008 15:34:02 +0000 (10:34 -0500)]
Merge branch 'master' of ssh://alanbrunelle@git.kernel.dk/data/git/blktrace
Jens Axboe [Thu, 30 Oct 2008 14:06:01 +0000 (15:06 +0100)]
Set release version 1.0.0
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Thu, 30 Oct 2008 14:04:20 +0000 (15:04 +0100)]
Update rbtree to version with unified parent + color
This saves 4-8 bytes per node, which can be quite a bit of memory
with blktrace.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Wed, 29 Oct 2008 19:04:03 +0000 (15:04 -0400)]
Moved btrecord/btreplay to version 1.0.0
Martin Peschke [Tue, 28 Oct 2008 16:08:12 +0000 (17:08 +0100)]
blkiomon: add through-put statistics
Add accounting of per-request throughput in bytes per millisecond
both for read ad write I/O.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Tue, 28 Oct 2008 16:08:10 +0000 (17:08 +0100)]
blkiomon: separate statistics for read and write requests
Split min/max/avg statistics for request sizes and dispatch-to-completion
latencies into separate statistics for read and write requests.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Tue, 28 Oct 2008 16:08:07 +0000 (17:08 +0100)]
blkiomon: fix some debug messages
Cleaning up error messages. Some perror()'s didn't make sense.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Tue, 28 Oct 2008 16:08:05 +0000 (17:08 +0100)]
blkiomon: fix trace debug output
Removed leftovers of trace tree and made debug code work by using trace
hash instead of trace tree.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Tue, 28 Oct 2008 16:08:02 +0000 (17:08 +0100)]
blkiomon: fix unit in histogram output
Fix unit of request sizes as printed in histogram (it's bytes not kilobytes).
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Tue, 28 Oct 2008 16:07:57 +0000 (17:07 +0100)]
blkiomon: fix cross-arch data analysis issue
This fixes cross-arch issues. Binary data gathered on System z could not
be read on x86.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Mon, 20 Oct 2008 16:14:21 +0000 (18:14 +0200)]
blkiomon: drv_data traces pass-through
This patch adds pass-through support for device driver specific traces
to blkiomon. This way we can aggregate block I/O statistics and device
driver specific statistics at the same time.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Fri, 17 Oct 2008 13:09:05 +0000 (15:09 +0200)]
blkparse: add hint for discarded drv_data traces
Display an informational message on blkparse exit to notify users that
additional data was available which would require to be dumped to binary
output.
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Thu, 16 Oct 2008 16:03:45 +0000 (12:03 -0400)]
Added in -L option - output periodic latency information
Alan D. Brunelle [Thu, 16 Oct 2008 14:53:07 +0000 (10:53 -0400)]
Added in -Q / --active-queue-depth option
This will output a data file containing the time stamp and number of
I/Os issued to underlying drivers per device. It will give you an idea
as to how many I/Os are being actively worked per device at any time
during the run.
Stefan Raspl [Thu, 16 Oct 2008 06:14:17 +0000 (08:14 +0200)]
Add driver data support
Adds a new type of action 'drv_data' for blktrace to handle binary
driver-specific data. Since the data is binary, blkparse will only put it in
a binary file, not in the regular human-readable output.
Signed-off-by: Stefan Raspl <raspl@linux.vnet.ibm.com>
Signed-off-by: Martin Peschke <mpeschke@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Thu, 16 Oct 2008 06:05:53 +0000 (08:05 +0200)]
blktrace: accept -v (lower case) for version info as well
Christof Schmitt <christof.schmitt@de.ibm.com> points out that the
documentation uses -v but blktrace supports only -V, so change
blktrace to accept both cases.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Martin Peschke [Thu, 16 Oct 2008 05:48:25 +0000 (07:48 +0200)]
blkiomon: I/O monitor
blkiomon periodicaly generates per devive request size and request latency
statistics from blktrace data. It provides histograms as well as data that
can be used to calculate min, max, average and variance. For this purpose,
it consumes D and C traces read from stdin.
There are options for binary output and human-readable output to files and
stdout. Output to a message queue is supported as well.
#blktrace /dev/sdw -a issue -a complete -w 200 -o - | blkiomon -I 8 -h -
time: Tue Sep 30 17:39:25 2008
device: 65,96
requests: read 62, write 40, bidir: 0
sizes: num 102, min 4096, max 430080, sum
13312000, squ
3102442782720,
avg 130509.8, var
13383296793.3
d2c: num 102, min 393, max 14261, sum 359441, squ
2830211755, avg 3523.9,
var
15329081.8
sizes histogram (in kB):
0: 0 1024: 0 2048: 0 4096: 6
8192: 0 16384: 15 32768: 4 65536: 24
131072: 11 262144: 30 524288: 12
1048576: 0
2097152: 0
4194304: 0
8388608: 0 >
8388608: 0
d2c histogram (in usec):
0: 0 8: 0 16: 0 32: 0
64: 0 128: 0 256: 0 512: 13
1024: 21 2048: 27 4096: 14 8192: 8
16384: 19 32768: 0 65536: 0 131072: 0
262144: 0 524288: 0
1048576: 0
2097152: 0
4194304: 0
8388608: 0
16777216: 0
33554432: 0
>
33554432: 0
time: Tue Sep 30 17:39:33 2008
device: 65,96
requests: read 312, write 47, bidir: 0
sizes: num 359, min 4096, max 430080, sum
13197312, squ
1575816790016,
avg 36761.3, var
3038067547.5
d2c: num 359, min 294, max 9211, sum 387134, squ
1262489694, avg 1078.4,
var
2353807.5
sizes histogram (in kB):
0: 0 1024: 0 2048: 0 4096: 32
8192: 17 16384: 133 32768: 87 65536: 59
131072: 9 262144: 18 524288: 4
1048576: 0
2097152: 0
4194304: 0
8388608: 0 >
8388608: 0
d2c histogram (in usec):
0: 0 8: 0 16: 0 32: 0
64: 0 128: 0 256: 0 512: 129
1024: 164 2048: 33 4096: 15 8192: 13
16384: 5 32768: 0 65536: 0 131072: 0
262144: 0 524288: 0
1048576: 0
2097152: 0
4194304: 0
8388608: 0
16777216: 0
33554432: 0
>
33554432: 0
Signed-off-by: Martin Peschke <mp3@de.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Fri, 10 Oct 2008 12:40:57 +0000 (08:40 -0400)]
Removed excessive amounts of seek modes (for random sets of I/Os)
When doing a random load, we'd get a LARGE amount of single-seek buckets,
this patch just notes that fact, without dumping all the data...
Alan D. Brunelle [Fri, 10 Oct 2008 12:40:21 +0000 (08:40 -0400)]
Merge branch 'master' of ssh://alanbrunelle@git.kernel.dk/data/git/blktrace
Nathan Scott [Fri, 26 Sep 2008 09:05:51 +0000 (11:05 +0200)]
spec file tweak
I found I needed this tweak to make the spec file usable on a RHEL5
system (I hit rpmbuild errors about unpackaged files without this).
Not sure if its correct though, maybe someone with stronger rpm-fu
than I have could take a closer look?
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Fri, 26 Sep 2008 09:05:16 +0000 (11:05 +0200)]
Merge branch 'master' of ssh://axboe@router.home.kernel.dk/data/git/blktrace
Nathan Scott [Fri, 26 Sep 2008 09:05:13 +0000 (11:05 +0200)]
man page typo
This fixes up a formatting problem when displaying the blktrace
man pages - just a typo (bold mode not correctly terminated, so
it flows on until the next bold-mode-termination a few lines on).
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Tue, 23 Sep 2008 12:06:41 +0000 (08:06 -0400)]
Added in %done for btt
David Woodhouse [Fri, 15 Aug 2008 09:12:20 +0000 (11:12 +0200)]
Add documentation of 'D' discard operation
Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
David Woodhouse [Fri, 15 Aug 2008 08:44:39 +0000 (10:44 +0200)]
blktrace: support discard requests
Add support for discard requests to blktrace userspace tools.
Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jeff Moyer [Tue, 1 Jul 2008 11:35:29 +0000 (13:35 +0200)]
spelling and grammar fixes for btreplay.tex
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Tue, 27 May 2008 14:39:14 +0000 (10:39 -0400)]
Put message notes from kernel into a separate file for easy tracking
Also made made_dev_hdr standard for all usages.
Alan D. Brunelle [Tue, 27 May 2008 12:34:13 +0000 (08:34 -0400)]
Added in new message updates to the documentation.
Alan D. Brunelle [Tue, 27 May 2008 12:19:57 +0000 (08:19 -0400)]
Added in handling of MESSAGE notes
Sample output:
8,16 1 691118 17.
417000000 0 C R
2660776 + 8 [0]
8,16 1 691119 17.
417000000 0 D R
2660792 + 8 [swapper]
8,16 1 691120 17.
417000000 4688 U N [dd] 42
8,16 1 0 17.
418000000 0 m N elv switch: noop
8,16 1 691121 17.
418000000 4638 C R
2660784 + 8 [0]
8,16 1 691122 17.
418000000 4638 D R
2660800 + 8 [bash]
8,16 1 691123 17.
418000000 4638 C R
2660792 + 8 [0]
Thanks to Carl Henrik Lunde <chlunde@ping.uio.no> for adding in sequence
printing & time-stamp correction.
Alan D. Brunelle [Wed, 21 May 2008 19:55:57 +0000 (15:55 -0400)]
Handled no difference in seek times
For some reason recent kernels (2.6.25.4, for example) we've lost a lot
of resolution in our blktrace times. This can result in lots of things
happening "simultaneously." This change at least tries to handle the
case where all the seeks happen at once.
Probably have other issues that need to be looked into...
Alan D. Brunelle [Wed, 21 May 2008 14:55:20 +0000 (10:55 -0400)]
Added in -m option, seeks-per-second
btt can now output data files containing seeks-per-second information.
Jens Axboe [Sun, 18 May 2008 18:55:25 +0000 (20:55 +0200)]
blkparse: cope with missing process notify event
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Mon, 12 May 2008 13:56:33 +0000 (09:56 -0400)]
Fixed percentage calculations for phases of an I/O
Alan D. Brunelle [Fri, 9 May 2008 17:46:47 +0000 (13:46 -0400)]
Added S2G times + fixed up -X output to include X2X
Including Q2Q, Q2G, S2G, G2I, Q2M, I2D, M2D, D2C, Q2C.
S2G is part of Q2G, and shows the number of times we had to sleep to
get a request.
Ignored 0-byte I/Os - coming from barrier I/Os...
Alan D. Brunelle [Thu, 8 May 2008 19:28:32 +0000 (15:28 -0400)]
Added -X option - generate easily parseable file
Writes a portion of the default output into a separate file that more
easily parsed.
Luis Useche [Mon, 5 May 2008 18:53:13 +0000 (20:53 +0200)]
Add -x accellerator option
This patch adds a new functionality to the btreplay tool, the -x option.
This parameter accelerate the replication by the factor specified. This
means that the stall time is divided by the number introduced.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Luis Useche [Mon, 5 May 2008 18:53:10 +0000 (20:53 +0200)]
Fix problem with -w option
This patch fixes the problem when the -w option is used in the file mode
(i.e., no fifo mode). It just consists of moving the checking of the
stopwatch_end after the time is updated with genesis. This also includes
the checking of the stopwatch_start.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Luis Useche [Mon, 5 May 2008 18:53:07 +0000 (20:53 +0200)]
eliminate check of empty -F format
This patch eliminates the checking of -F format when it is empty. I am
using this in order to blank out the events that I do not want for
certain act mask. Note that there is no real motivation to have this
checking.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Christof Schmitt [Wed, 2 Apr 2008 07:21:08 +0000 (09:21 +0200)]
blkparse: Introduce optional accounting of PC requests
The current blkparse lists only PC requests, but does not account for
number of PC requests and the size of the attached data. This patch
introduces optional accounting for PC requests. If any PC event was
encountered, the summary output will also include data from PC
requests.
For PC completion events the amount of transferred data is not
available, since the kernel uses the data_len field as residual bytes
count on event completion.
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Christof Schmitt [Wed, 2 Apr 2008 07:21:08 +0000 (09:21 +0200)]
blkparse: Add PC requests to depth trace
The PC requests use the same queue and also contribute to the
queue length, so they should be added to the queue depth trace.
Signed-off-by: Christof Schmitt <christof.schmitt@de.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Bas Zoetekouw [Fri, 22 Feb 2008 11:58:12 +0000 (12:58 +0100)]
Fix section of btrecord and btreplay man pages
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Bas Zoetekouw [Fri, 22 Feb 2008 11:51:42 +0000 (12:51 +0100)]
Don't like btrecord against libaio and librt, as it doesn't use any of their symbols
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Bas Zoetekouw [Fri, 22 Feb 2008 10:27:04 +0000 (11:27 +0100)]
Really commit the changes to btt.1
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Bas Zoetekouw [Fri, 22 Feb 2008 10:25:51 +0000 (11:25 +0100)]
Added man page for bno_plot and updated btt man page to refer to bno_plot
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Wed, 13 Feb 2008 16:46:22 +0000 (11:46 -0500)]
Cleanups: Fixed IOPs in btt left over at end of run
o Using valgrind, determined we had Q IOPs left over that weren't
used. Added "all" list, and then deleted these at end.
o Removed old debug stuff (COUNT_IOS, DEBUG, ...)
o Fixed a bunch of white space at end of lines.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Aaron Carroll [Wed, 13 Feb 2008 07:48:14 +0000 (18:48 +1100)]
btt: fix missing cleanup call
clean_args() exists but is never called, thus any files opened in args remain
open while the buffers backing those files are closed.
Signed-off-by: Aaron Carroll <aaronc@gelato.unsw.edu.au>
Aaron Carroll [Wed, 13 Feb 2008 07:46:40 +0000 (18:46 +1100)]
btt: fix iostat interval default
Change the default iostat interval from 1ns to 1s, which is the
default implied by btt.1 and common sense.
Signed-off-by: Aaron Carroll <aaronc@gelato.unsw.edu.au>
Alan D. Brunelle [Tue, 12 Feb 2008 13:58:25 +0000 (08:58 -0500)]
Added info about bno_plot.py and clean ups
o Added description of bno_plot.py, including sample screenshot
o Various spelling clean ups
o Updated sample btt output
o Cleaned up table of contents
Alan D. Brunelle [Fri, 8 Feb 2008 15:03:13 +0000 (10:03 -0500)]
Merge branch 'bno_plot.submit'
Alan D. Brunelle [Tue, 5 Feb 2008 22:31:21 +0000 (17:31 -0500)]
Added new IOs per unplug table
==================== Plug Information ====================
DEV | # Plugs # Timer Us | % Time Q Plugged
---------- | ---------- ---------- | ----------------
( 8, 0) | 1171( 123) | 0.
280946640%
( 8, 32) | 4( 0) | 0.
000325469%
---------- | ---------- ---------- | ----------------
Overall | # Plugs # Timer Us | % Time Q Plugged
Average | 587( 61) | 0.
140636055%
DEV | IOs/Unp IOs/Unp(to)
---------- | ---------- ----------
( 8, 0) | 9.2 8.8
( 8, 32) | 2.5 0.0
---------- | ---------- ----------
DEV | IOs/Unp IOs/Unp(to)
Overall | 9.2 8.8
Alan D. Brunelle [Tue, 5 Feb 2008 18:46:51 +0000 (13:46 -0500)]
Added bno_plot.py - generate interactive 3D plot of IO blocks and sizes
bno_plot.py
[ -h | --help ]
[ -K | --keys-below ]
[ -v | --verbose ]
[ <file...> ]
Utilizes gnuplot to generate a 3D plot of the block number output
from btt. If no <files> are specified, it will utilize all files
generated after btt was run with -B blknos (meaning: all files of the
form blknos*[rw].dat).
The -K option forces bno_plot.py to put the keys below the graph,
typicall all keys for input files are put in the upper right corner of
the graph. If the number of devices exceed 10, then bno_plot.py will
automatically push the keys under the graph.
To exit the plotter, enter 'quit' or ^D at the 'gnuplot> ' prompt.
Alan D. Brunelle [Thu, 31 Jan 2008 22:10:52 +0000 (17:10 -0500)]
UNPLUG does the timing stuff, UNPLUG TIMEOUT only does timeout
Each UNPLUG TIMEOUT should be followed by an UNPLUG, so we were getting
double information on time outs.
Alan D. Brunelle [Thu, 31 Jan 2008 18:02:24 +0000 (13:02 -0500)]
Fixed excess bucket information for unplug histograms.
Dave Boutcher [Thu, 3 Jan 2008 15:21:32 +0000 (10:21 -0500)]
Fix Q counts during requeue and merges.
It looks to me like btt doesn't correctly keep track
of the number of requests currently in the queue for
a device. n_act_q gets incremented in trace_queue
and decremented in trace_issue, but I think it also
needs to get updated in trace_merge and trace_requeue.
The one thing I'm not sure about is whether we want
r_iop->dip->n_qs++ in the new handle_requeue routine.
The following patch makes the "active requests at Q"
count a little more sane for me. This is against
git as of yesterday.
Signed-off-by: Alan D. Brunelle <Alan.Brunelle@hp.com>
Alan D. Brunelle [Mon, 10 Dec 2007 18:41:37 +0000 (13:41 -0500)]
Separated out g/i/m trace handling.
Also separated out DM-device calculations.
Bas Zoetekouw [Sun, 9 Dec 2007 18:43:16 +0000 (19:43 +0100)]
btreplay/btrecord man pages
Attached are manpages for the btreplay and btrecord utilities for
inclusion in the git repo. I'm about to upload anew version to Debian
which will include these manpages, too.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Thu, 6 Dec 2007 14:00:47 +0000 (09:00 -0500)]
Merge branch 'master' into add-q2d
Joshua Root [Thu, 6 Dec 2007 08:13:48 +0000 (09:13 +0100)]
Remove strange make dependency
Listing "-lpthread" as a dependency in a make rule doesn't make much sense.
Signed-off-by: Joshua Root <jmr@gelato.unsw.edu.au>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Wed, 5 Dec 2007 23:08:10 +0000 (18:08 -0500)]
Added in Q2D histograms (requires -A)
As an example, the buckets are in seconds:
==================== Q2D Histogram ====================
DEV | <.005 <.010 <.025 <.050 <.075 <.100 <.250 <.500 < 1.0 >=1.0
--------- | ===== ===== ===== ===== ===== ===== ===== ===== ===== =====
( 66, 80) | 61.2 7.9 12.1 7.9 3.0 1.4 1.5 0.2 0.0 4.6
( 65,192) | 42.3 5.0 8.7 30.0 8.9 3.0 1.8 0.1 0.0 0.1
( 65,128) | 34.3 5.3 8.9 32.0 9.7 3.7 5.3 0.6 0.0 0.1
( 66, 0) | 45.5 9.9 19.0 14.8 4.6 1.2 0.3 0.0 0.0 4.7
( 65, 48) | 50.2 4.6 7.2 27.6 6.9 2.1 1.1 0.0 0.0 0.1
( 66, 32) | 66.4 6.7 10.9 7.0 2.5 1.1 0.6 0.1 0.0 4.7
( 65,224) | 33.9 9.6 20.6 17.8 7.3 3.1 3.3 0.5 0.1 3.8
( 65,112) | 35.3 5.2 8.7 30.8 9.3 3.9 5.6 0.9 0.1 0.1
( 65, 80) | 57.9 4.6 6.6 25.2 4.7 0.8 0.2 0.0 0.0 0.1
( 65,240) | 62.6 8.1 13.1 7.7 2.3 0.6 0.5 0.0 0.0 5.1
( 66,112) | 59.6 8.0 13.1 8.6 3.3 1.5 1.2 0.1 0.0 4.6
( 65,144) | 45.8 5.1 7.9 29.3 7.8 2.5 1.4 0.0 0.0 0.1
( 65,176) | 43.2 5.2 8.9 30.8 8.3 2.5 1.0 0.0 0.0 0.1
( 66, 16) | 77.0 5.4 7.4 3.3 0.7 0.3 0.7 0.1 0.0 5.0
( 65, 96) | 54.6 4.9 7.2 26.6 5.3 1.0 0.2 0.0 0.0 0.1
( 66, 48) | 71.0 6.8 10.5 5.4 0.9 0.1 0.1 0.0 0.0 5.2
( 65,208) | 43.5 5.6 8.5 31.0 7.2 2.2 1.9 0.0 0.0 0.1
( 66,128) | 69.5 7.0 10.3 6.2 1.5 0.3 0.1 0.0 0.0 5.0
( 65, 64) | 59.9 4.2 6.0 24.6 4.2 0.8 0.1 0.0 0.0 0.1
( 66, 64) | 62.6 8.1 12.7 7.9 2.4 0.6 0.1 0.0 0.0 5.4
========== | ===== ===== ===== ===== ===== ===== ===== ===== ===== =====
AVG | 52.9 6.2 10.0 20.1 5.3 1.7 1.4 0.2 0.0 2.1
Jens Axboe [Wed, 14 Nov 2007 09:45:23 +0000 (10:45 +0100)]
memset() must be done after NULL check
Pointed out by Jan Blunck <jblunck@suse.de>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Tue, 13 Nov 2007 22:16:38 +0000 (17:16 -0500)]
Added active requests at Q information.
An important consideration when analyzing block IO schedulers is to know
how many requests the scheduler has to work with. The metric provided
in this section details how many requests (on average) were being held
by the IO scheduler when an incoming IO request was being handled. To
determine this, btt keeps track of how many Q requests came in, and
subtacts requests that have been issued (D).
Sample:
==================== Active Requests At Q Information ====================
DEV | Avg Reqs @ Q
---------- | -------------
( 65, 80) | 12.0
( 65,240) | 16.9
( 65,112) | 13.3
( 66, 64) | 32.8
( 66, 80) | 21.5
( 65, 96) | 8.6
( 66, 16) | 16.2
( 65, 64) | 20.4
( 66, 96) | 18.3
( 65,176) | 17.4
( 66, 32) | 13.6
( 65,144) | 13.4
( 66, 0) | 21.4
( 65,192) | 19.4
( 66,128) | 18.4
( 66,144) | 16.2
( 65,128) | 8.0
( 66,112) | 44.2
---------- | -------------
Overall | Avgs Reqs @ Q
Average | 17.4
Signed-off-by: Alan D. Brunelle <Alan.Brunelle@hp.com>
Alan D. Brunelle [Thu, 8 Nov 2007 13:34:44 +0000 (08:34 -0500)]
Fixed REMAP to update Q2A & fixed #Q calculations
Aneesh Kumar K.V [Mon, 29 Oct 2007 19:47:28 +0000 (20:47 +0100)]
blktrace segfault
Core was generated by `blktrace -d /dev/hdc'.
Program terminated with signal 11, Segmentation fault.
#0 0xb7e4cdec in ?? ()
(gdb) where
#0 0xb7e4cdec in ?? ()
#1 0xb7dbf000 in ?? ()
#2 0x00021000 in ?? ()
#3 0xb7dee6e8 in ?? ()
#4 0x0804ecf0 in ?? ()
#5 0x00000001 in ?? ()
#6 0x6c616367 in ?? ()
#7 0xbfee3f68 in ?? ()
#8 0xb7f51300 in ?? ()
#9 0x00000168 in ?? ()
#10 0x0804ecf0 in ?? ()
#11 0x00000001 in ?? ()
#12 0xbfee3f88 in ?? ()
#13 0xbfee3f68 in ?? ()
#14 0x080499dc in close_thread (tip=0xb7f1eff4) at blktrace.c:637
Backtrace stopped: frame did not save the PC
(gdb)
the below diff fix the same.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Wed, 10 Oct 2007 17:25:20 +0000 (13:25 -0400)]
Added O_NOATIME to replay file
Signed-off-by: Alan D. Brunelle <Alan.Brunelle@hp.com>
Joshua Root [Wed, 10 Oct 2007 11:17:56 +0000 (07:17 -0400)]
Fix segfault in replay_sub when verbose is 1
tip->vfp is only initialised when verbose > 1, so we must only use it
under the same circumstance.
Signed-off-by: Alan D. Brunelle <Alan.Brunelle@hp.com>
Alan D. Brunelle [Fri, 5 Oct 2007 22:30:27 +0000 (18:30 -0400)]
Merge branch 'master' of git://brick.kernel.dk/data/git/blktrace
Alan D. Brunelle [Fri, 5 Oct 2007 20:32:46 +0000 (16:32 -0400)]
Converted fatal to an inline function
Removes warning from gcc-4.2.1
Alan D. Brunelle [Wed, 3 Oct 2007 14:49:42 +0000 (10:49 -0400)]
Converted fatal from macro to inline
Hopefully this will fix a gcc-4.2.1 warning.
Signed-off-by: Alan D. Brunelle <adb@korik.(none)>
Alan D. Brunelle [Tue, 2 Oct 2007 16:35:07 +0000 (12:35 -0400)]
Add btrecord/btreplay capability
These facilities allow one to attempt to replay a stream of IOs
captured with blktrace. The general workflow is:
1. Initiate blktrace to capture traces
2. Do whatever to generate initial IO stream...
3. Stop blktrace
4. Run btrecord to convert traces into IO records
5. Run btreplay to replay IOs
The IO stream characteristics during replay will try to respect the
following characteristics of the original IO stream:
1. The IOs will target the same device(s) as originally seen. [One can
alter this behavior by specifyin the -M option to btreplay, which
allows one to remap IOs slated to one set of devices to a specified
other set of devices.]
2. IO direction: the IOs will follow the same read/write
(from-device/to-device) characteristics of the originating flow. [Note:
By default replay will /not/ do writes, one must specify the -W option
to do this. THis is a meager attempt to stop someone from shooting
themselves in the foot (with a very large-caliber weapon).]
3. IO offset & size are maintained.
4. CPU: IOs are submitted on the originating CPU whenever possible. [Note:
Since we are using asynchronous IO, IOs may be routed to another CPU
prior to being processed by the block IO layer.]
In order to try and replicate inter-IO timing as much as possible,
btrecord will combine IOs "close in time" into one set, or bunch, of
IOs. Then btreplay will replay all the IOs in one go (via asynchronous
direct IO - io_submit). The size of the bunches are configurable via
the -m flag to btrecord (which specifies the a time-based bunch size)
and/or the -M flag (which specifies the maximum amount of IOs to put
into a bunch). At the low-end, specifying '-M 1' instructs btrecord to
act like fio - replay each IO as an individual unit.
Besides the potential to remap devices (utilizing the -M option to replay,
as noted above), one can also limit the number of CPUs on the replay
machine - so if you have fewer CPUs on the replay machine you specify
the -c option to btreplay.
Lastly, one can specify the -N option to btreplay to instruct it to ignore
inter-IO (inter-bunch of IOs) timings. Thus, this instructs btreplay
to replay the bunches as fast as possible, ignoring the original delays
between original IOs.
The utilities include a write-up in the docs directory.
Signed-off-by: Alan D. Brunelle <Alan.Brunelle@hp.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Tue, 2 Oct 2007 16:08:34 +0000 (12:08 -0400)]
Added list_splice to btt/list.h
list_splice is needed for the addition of btrecord/btreplay.
Signed-off-by: Alan D. Brunelle <Alan.Brunelle@hp.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Bas Zoetekouw [Mon, 1 Oct 2007 06:33:37 +0000 (08:33 +0200)]
btt manpages
Attached is an update for the btt.1 man page that should bring the man
page in sync with the TeX documentation.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Bas Zoetekouw [Mon, 1 Oct 2007 06:32:29 +0000 (08:32 +0200)]
Fix compilation on m68k
Missing defines for store_barrier()
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Alan D. Brunelle [Mon, 10 Sep 2007 16:37:49 +0000 (18:37 +0200)]
Major revamping (ver 2.0)
After a lot of fighting with maintaining a tree-styled design (each trace
having it's own node), it was just getting too cumbersome to work in all
circumstances. Taking a clue from blkparse itself, I decided to just keep
track of IOs at queue time, and updating fields based upon later traces.
The attached (large) patch works much faster, handles larger test cases
with less failures, and is managing some pretty large jobs I'm working on
(large Oracle-based DB analysis - 32-way box w/ lots of storage).
I've also added a Q2Q seek distance feature - it's come in handy when
comparing results of IO scheduler choice: We can see what the incoming IO
seek distances are (at queue time), and then see how the scheduler itself
manages things (via merges & sorting) by looking at D2D seek distances
generated.
As noted in the subject, I arbitrarily bumped this to version 2.00 as the
innards are so different. The documentation (btt/doc/btt.tex) has been
updated to reflect some minor output changes. I also fixed a bug dealing
with process name notification: there was a problem that if a new PID came
up with a name that was previously seen, btt wouldn't keep track of it
right. [When running with Oracle, a lot of processes have the same name but
different PIDs of course.]
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Tue, 28 Aug 2007 13:03:59 +0000 (15:03 +0200)]
blktrace 0.99.3
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Tue, 28 Aug 2007 13:00:33 +0000 (15:00 +0200)]
blktrace man page: mention buffer size / number of buffers defaults.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Valerie Henson [Mon, 27 Aug 2007 19:50:57 +0000 (21:50 +0200)]
blktrace: fix indent typo
Fix minor indentation typo in blktrace man page.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jan Blunck [Sun, 26 Aug 2007 17:35:31 +0000 (19:35 +0200)]
Makefile fixes
Here is a patch with small fixes to the Makefiles which I use in the
openSUSE blktrace package.
Signed-off-by: Jan Blunck <jblunck@suse.de>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Mon, 20 Aug 2007 13:31:50 +0000 (15:31 +0200)]
Disregard generated doc files in .gitignore
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Vincent Legoll [Mon, 30 Jul 2007 13:23:43 +0000 (15:23 +0200)]
Fix Makefile buglets
- make install without having called make before
- create man directories if non pre-existent
Signed-off-by: Vincent Legoll <vincent.legoll@gmail.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Tue, 24 Jul 2007 14:02:28 +0000 (16:02 +0200)]
Add man pages
Thanks to Bas Zoetekouw <bas@debian.org> for providing these! And
shame on me for forgetting to add them to the git repo.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Wed, 18 Jul 2007 11:51:56 +0000 (13:51 +0200)]
git:// url location update
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Jens Axboe [Wed, 13 Jun 2007 14:16:00 +0000 (16:16 +0200)]
Update email address
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>