fio.git
13 years agoSpeed up verify fill
Jens Axboe [Fri, 27 Jul 2007 10:57:25 +0000 (12:57 +0200)]
Speed up verify fill

It's silly to fill an on-stack header, and then copy it to the io_u.
Just map the pointer to the buffer start.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoType in crc16 verify failure
Jens Axboe [Fri, 27 Jul 2007 07:53:14 +0000 (09:53 +0200)]
Type in crc16 verify failure

Should dump crc16, not crc32.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd crc7 verify type
Jens Axboe [Fri, 27 Jul 2007 07:52:40 +0000 (09:52 +0200)]
Add crc7 verify type

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd crc16 verify type
Jens Axboe [Fri, 27 Jul 2007 07:07:17 +0000 (09:07 +0200)]
Add crc16 verify type

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years ago[PATCH] Only verify_io_u on reads
Shawn Lewis [Fri, 27 Jul 2007 06:02:45 +0000 (08:02 +0200)]
[PATCH] Only verify_io_u on reads

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd data integrity example to netio sample job file
Jens Axboe [Thu, 26 Jul 2007 11:19:57 +0000 (13:19 +0200)]
Add data integrity example to netio sample job file

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoMissed FIO_UNIDIR io ops flags on net engine
Jens Axboe [Thu, 26 Jul 2007 11:13:49 +0000 (13:13 +0200)]
Missed FIO_UNIDIR io ops flags on net engine

Inadvertenly only set it on the netsplice engine, not net.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoIO verification improvements
Jens Axboe [Thu, 26 Jul 2007 10:22:30 +0000 (12:22 +0200)]
IO verification improvements

- Add a FIO_UNIDIR io ops flag. This like networked IO can't be read/write,
  so verify only makes sense for data summed by another writer.
- Allow a read workload to verify previously written data.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix verify random bytes filling
Jens Axboe [Thu, 26 Jul 2007 09:55:10 +0000 (11:55 +0200)]
Fix verify random bytes filling

Shawn Lewis <shawnlewis@google.com> reports:

---

verify.c: fill_random_bytes() doesn't work as intended.

drand48 on linux returns a double whose value is evenly distributed between
0.0 and 0.1. This is not the same as having an 8-byte value where each bit
has the same chance of being 0 as it does of being 1.

I'd fix this but it should probably be done in a cross-platformish way and
I'm not quite sure how to do it. Maybe just use smaller randoms? Have an
os_random_uint32?

---

I just got rid of os_random_double() and used os_random_long(),
truncating that to an unsigned integer.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years ago[PATCH] empty result
ljzhang,Yaxin Hu,Jianchao Tang [Thu, 26 Jul 2007 09:00:29 +0000 (11:00 +0200)]
[PATCH] empty result

1. The job file looks like this:
---------runtime------------
[global]
directory=./temp
nrfiles=1
rw=randread
size=32k
thread
[runtime]
description="Too tiny a mission using less than 1sec."
----------------------------

Most of the time,it seems like no io has been performed:
----------------------------
nonggia@nonggia-desktop:~/fio$ fio runtime
runtime: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 1 thread

runtime: (groupid=0, jobs=1): err= 0: pid=7155
  Description  : ["Too tiny a mission using less than 1sec."]
  cpu          : usr=0.00%, sys=0.00%, ctx=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     issued r/w: total=8/0, short=0/0
     lat (usec): 100=87.50%, 250=12.50%

Run status group 0 (all jobs):

Disk stats (read/write):
  sda: ios=8/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
nonggia@nonggia-desktop:~/fio$
----------------------------

2. Reason for the problem:
It is so tiny a job that it may cost less than one second to finish.And
the runtime is rounded down to 0.Just like the io never happened.
We agree that the problem is really trifle, but for testing purpose, we
think it maybe while the fixing.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years ago[PATCH] non-stop thread
ljzhang,Yaxin Hu,Jianchao Tang [Thu, 26 Jul 2007 08:59:25 +0000 (10:59 +0200)]
[PATCH] non-stop thread

1. The job file causing the problem:
-------------------rw-------------------
[global]
directory=./temp
nrfiles=1
rw=randread:8
norandommap
size=32k
bs=2k
thread
[rw]
description="Offset over real file size."
-----------------------------------------

The job file can keep fio running with the eta increasing and the
progress pausing like this:
----------------------------------------
rw: (g=0): rw=randread, bs=2K-2K/2K-2K, ioengine=sync, iodepth=1
Starting 1 thread
Jobs: 1 (f=1): [r] [92.3% done] [     0/     0 kb/s] [eta 00m:01s]]

----------------------------------------

If we force it to stop, it prints like this:
--------------------------------------------
fio: terminating on signal 2

rw: (groupid=0, jobs=1): err= 0: pid=6124
  Description  : ["Offset over real file size."]
  read : io=28KiB, bw=0KiB/s, iops=0, runt= 61900msec
    clat (usec): min=    5, max=  436, avg=87.79, stdev=123.71
  cpu          : usr=2.01%, sys=93.48%, ctx=13644
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     issued r/w: total=14/0, short=0/0
     lat (usec): 10=42.86%, 100=35.71%, 250=7.14%, 500=14.29%

Run status group 0 (all jobs):
   READ: io=28KiB, aggrb=0KiB/s, minb=0KiB/s, maxb=0KiB/s,
mint=61900msec, maxt=61900msec

Disk stats (read/write):
  sda: ios=8/45, merge=0/40, ticks=4/368, in_queue=372, util=0.27%
--------------------------------------------
>From above we can see an uncompleted job with a io 28KiB less than what
is specified by 'size'.

2. Reason for problem:
After looking into the source codes,We guess the problem comes from the
function get_next_offset().If the io reaches the end of the file, the
function fails, and the file will be marked as DONE.We wonder if that is
the planned behavior.Isn't it very common to reach the file's end when
doing a random io?

3. Fix suggestion:
Here is the patch:
-----------------------------------------------------
diff -Nraup fio-7.25/io_u.c fio-7.25-rw/io_u.c
--- fio-7.25/io_u.c     2007-07-25 14:25:05.000000000 +0800
+++ fio-7.25-rw/io_u.c 2007-07-26 09:55:22.000000000 +0800
@@ -162,10 +162,12 @@ static int get_next_offset(struct thread
if (get_next_rand_offset(td, f, ddir, &b))
return 1;
} else {
- if (f->last_pos >= f->real_file_size)
- return 1;
-
- b = f->last_pos / td->o.min_bs[ddir];
+ if (f->last_pos >= f->real_file_size) {
+ if (!td_random(td) || get_next_rand_offset(td, f, ddir, &b))
+ return 1;
+ } else {
+ b = f->last_pos / td->o.min_bs[ddir];
+ }
}

io_u->offset = (b * td->o.min_bs[ddir]) + f->file_offset;
------------------------------------------------------

After applying that, we got the job file run to the end normally:
-------------------------------------------------------
nonggia@nonggia-desktop:~/fio$ ./rw_fio rw
rw: (g=0): rw=randread, bs=2K-2K/2K-2K, ioengine=sync, iodepth=1
Starting 1 thread

rw: (groupid=0, jobs=1): err= 0: pid=6519
  Description  : ["Offset over real file size."]
  read : io=32KiB, bw=1560KiB/s, iops=761, runt=    21msec
    clat (usec): min=    5, max=14251, avg=1293.50, stdev=3510.37
  cpu          : usr=0.00%, sys=19.05%, ctx=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     issued r/w: total=16/0, short=0/0
     lat (usec): 10=37.50%, 20=12.50%, 250=12.50%, 750=6.25%, 1000=6.25%
     lat (msec): 2=18.75%, 20=6.25%

Run status group 0 (all jobs):
   READ: io=32KiB, aggrb=1560KiB/s, minb=1560KiB/s, maxb=1560KiB/s,
mint=21msec, maxt=21msec

Disk stats (read/write):
  sda: ios=8/0, merge=0/0, ticks=20/0, in_queue=20, util=13.89%
-------------------------------------------------------

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years ago[PATCH] Random map block coverage
ljzhang,Yaxin Hu,Jianchao Tang [Thu, 26 Jul 2007 08:51:46 +0000 (10:51 +0200)]
[PATCH] Random map block coverage

1. The job file causing the problem:
----------cover-------------------
[global]
directory=./temp
nrfiles=1
rw=randread
size=256K
bsrange=4k-16k
thread
[cover]
description="Not covering every block."
-----------------------------------------

According to HOWTO,the job file will makes fio to cover every block of
the 256k file.But running that, we sometimes got a io gross more than
256k:
----------------------------------------
nonggia@nonggia-desktop:~$ fio --version
fio 1.16.9
nonggia@nonggia-desktop:~/fio$ fio cover
cover: (g=0): rw=randread, bs=4K-16K/4K-4K, ioengine=sync, iodepth=1
Starting 1 thread

cover: (groupid=0, jobs=1): err= 0: pid=6969
  Description  : ["Not covering every block."]
  read : io=264KiB, bw=90112KiB/s, iops=7666, runt=     3msec
    clat (usec): min=   66, max=  438, avg=151.00, stdev=88.90
  cpu          : usr=0.00%, sys=133.33%, ctx=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     issued r/w: total=23/0, short=0/0
     lat (usec): 100=34.78%, 250=56.52%, 500=8.70%

Run status group 0 (all jobs):
   READ: io=264KiB, aggrb=90112KiB/s, minb=90112KiB/s, maxb=90112KiB/s,
mint=3msec, maxt=3msec

Disk stats (read/write):
  sda: ios=49/0, merge=0/0, ticks=4/0, in_queue=4, util=2.86%
--------------------------------------------

It seems fio doesn't cover every block.And using gdb we can see it in
detail:
--------------------------------------------
(gdb) where
from /lib/tls/i686/cmov/libpthread.so.0
(gdb) p td->files[0].num_maps
$1 = 2
(gdb) x/2w td->files[0].file_map
0x8073ff0:      0x00000000      0x00000000
(gdb) finish
Run till exit from #0  do_io (td=0xb7bec000) at fio.c:423
thread_main (data=0xb7bec000) at fio.c:875    0 kb/s] [eta 00m:00s]
875                     clear_state = 1;
(gdb) x/2w td->files[0].file_map
0x8073ff0:      0xffffeffd      0x7ce2df47
(gdb) c
Continuing.
[Thread -1212236896 (LWP 7099) exited]

cover: (groupid=0, jobs=1): err= 0: pid=7098
  Description  : ["Not covering every block."]
  read : io=264KiB, bw=7KiB/s, iops=0, runt= 37522msec
    clat (usec): min=   67, max=  347, avg=143.28, stdev=67.59
    bw (KiB/s) : min=    0, max=    0, per=0.00%, avg= 0.00, stdev= 0.00
  cpu          : usr=0.00%, sys=0.01%, ctx=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     issued r/w: total=29/0, short=0/0
     lat (usec): 100=31.03%, 250=62.07%, 500=6.90%

Run status group 0 (all jobs):
   READ: io=264KiB, aggrb=7KiB/s, minb=7KiB/s, maxb=7KiB/s,
mint=37522msec, maxt=37522msec

Disk stats (read/write):
  sda: ios=50/23, merge=0/32, ticks=0/160, in_queue=160, util=0.13%

Program exited normally.
--------------------------------------------------

2. Reason for problem:
We think when there is no 'norandommap' in the job file, and no ':int'
appended with 'rw',just like the job file above, fio is meant to use
only the blocks not covered yet.
But it seems in the function mark_random_map(), the condition before
calling random_map_free() will never be satisfied.

3. Fix suggestion:
Here is the patch:
-----------------------------------------------------
diff -Nraup fio-7.25/io_u.c fio-7.25-cover/io_u.c
--- fio-7.25/io_u.c     2007-07-25 14:25:05.000000000 +0800
+++ fio-7.25-cover/io_u.c       2007-07-25 20:43:58.000000000 +0800
@@ -55,7 +55,7 @@ static void mark_random_map(struct threa
                 * If we have a mixed random workload, we may
                 * encounter blocks we already did IO to.
                 */
-               if (!td->o.ddir_nr && !random_map_free(td, f, block))
+               if (td->o.ddir_nr==1 && !random_map_free(td, f, block))
                        break;

                idx = RAND_MAP_IDX(td, f, block);
------------------------------------------------------

After that, we check it with gdb:
----------------------------------------------------
(gdb) where
from /lib/tls/i686/cmov/libpthread.so.0
(gdb) p td->files[0].num_maps
$2 = 2
(gdb) x/2w td->files[0].file_map
0x8073ff0:      0x00000000      0x00000000
(gdb) finish
Run till exit from #0  do_io (td=0xb7c48000) at fio.c:423
thread_main (data=0xb7c48000) at fio.c:875    0 kb/s] [eta 00m:00s]
875                     clear_state = 1;
(gdb) x/2w td->files[0].file_map
0x8073ff0:      0xffffffff      0xffffffff
(gdb) c
Continuing.
[Thread -1212073056 (LWP 7443) exited] 5/     0 kb/s] [eta 00m:00s]

cover: (groupid=0, jobs=1): err= 0: pid=7442
  Description  : ["Not covering every block."]
  read : io=256KiB, bw=5KiB/s, iops=0, runt= 46612msec
    clat (usec): min=   62, max=  522, avg=119.50, stdev=88.07
    bw (KiB/s) : min=    0, max=    0, per=0.00%, avg= 0.00, stdev= 0.00
  cpu          : usr=0.00%, sys=0.01%, ctx=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     issued r/w: total=38/0, short=0/0
     lat (usec): 100=55.26%, 250=39.47%, 500=2.63%, 750=2.63%

Run status group 0 (all jobs):
   READ: io=256KiB, aggrb=5KiB/s, minb=5KiB/s, maxb=5KiB/s,
mint=46612msec, maxt=46612msec

Disk stats (read/write):
  sda: ios=64/45, merge=0/62, ticks=8/476, in_queue=484, util=0.19%

Program exited normally.
(gdb)
---------------------------------------------------------
Now we can see the expected '0xffffffff 0xffffffff' and the 'io=256KiB'.
By the way, we tested this both on a 32bit machine and a 64bit one.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix problem with io_u being beyond EOF
Jens Axboe [Thu, 26 Jul 2007 08:47:51 +0000 (10:47 +0200)]
Fix problem with io_u being beyond EOF

Bug report from "ljzhang,Yaxin Hu,Jianchao Tang" <nonggia@sjtu.edu.cn> :

1. The job file causing the problem:
----------ranmap-------------------
[global]
directory=./temp
nrfiles=1
rw=randread
size=64K
bsrange=1k-32k
thread
[ranmap]
description="Over-bounded in mark_random_map()."
-----------------------------------------

Running the job file, we have a high possibility to get a core dump like
this:
----------------------------------------
nonggia@nonggia-desktop:~/fio$ fio --version
fio 1.16.9
nonggia@nonggia-desktop:~/fio$ fio ranmap
ranmap: (g=0): rw=randread, bs=1K-32K/4K-4K, ioengine=sync, iodepth=1
Starting 1 thread
file:io_u.c:64, assert idx < f->num_maps failed
Segmentation fault (core dumped)
nonggia@nonggia-desktop:~/fio$ gdb fio core.6135
... ...
Core was generated by `fio ranmap'.
Program terminated with signal 11, Segmentation fault.
io_u.c:64
64                      fio_assert(td, idx < f->num_maps);
(gdb) bt
io_u.c:64
io_u.c:348
io_u.c:588
from /lib/tls/i686/cmov/libpthread.so.0
(gdb) p io_u->offset
$1 = 41984
(gdb) p io_u->buflen
$2 = 27648
(gdb) p io_u->file->real_file_size
$3 = 65536

2. Reason for the core dump:
The io_u->offset+io_u->buflen was too large to be map in the map file.
They just produced an idx out of the file_map's range.

The fix I produced made sure that get_next_buflen() doesn't go
beyond the real file size, by shrinking the buflen.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years ago[PATCH] Don't hide pthread_mutex_setpshared in os-linux.h
Shawn Lewis [Thu, 26 Jul 2007 05:36:14 +0000 (07:36 +0200)]
[PATCH] Don't hide pthread_mutex_setpshared in os-linux.h

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix headers for building with some older libs
Shawn Lewis [Wed, 25 Jul 2007 05:51:58 +0000 (07:51 +0200)]
Fix headers for building with some older libs

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd ->done variable to force exit when we know we are done
Jens Axboe [Mon, 23 Jul 2007 12:36:16 +0000 (14:36 +0200)]
Add ->done variable to force exit when we know we are done

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoOnly do the root warning once per thread
Jens Axboe [Mon, 23 Jul 2007 12:36:00 +0000 (14:36 +0200)]
Only do the root warning once per thread

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoMissing carriage return in latency print
Jens Axboe [Mon, 23 Jul 2007 12:21:17 +0000 (14:21 +0200)]
Missing carriage return in latency print

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoOnly warn once for blktrace log open failure
Jens Axboe [Mon, 23 Jul 2007 12:16:55 +0000 (14:16 +0200)]
Only warn once for blktrace log open failure

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoOne more fio_ddir fixup
Jens Axboe [Mon, 23 Jul 2007 09:26:27 +0000 (11:26 +0200)]
One more fio_ddir fixup

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoddir is of type enum fio_ddir
Jens Axboe [Mon, 23 Jul 2007 09:25:39 +0000 (11:25 +0200)]
ddir is of type enum fio_ddir

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoRemove gotos from read_iolog_get()
Jens Axboe [Mon, 23 Jul 2007 09:23:26 +0000 (11:23 +0200)]
Remove gotos from read_iolog_get()

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoIO log fixes
Jens Axboe [Mon, 23 Jul 2007 08:38:43 +0000 (10:38 +0200)]
IO log fixes

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd version 2 of the iolog format
Jens Axboe [Mon, 23 Jul 2007 06:56:43 +0000 (08:56 +0200)]
Add version 2 of the iolog format

The old format had some short comings - notably not supporting > 1
file workloads. So add a new format that defines adding/open/close
of files. Fio will still load the older format iologs, but the new
ones wont work on older fio versions.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.9 fio-1.16.9
Jens Axboe [Fri, 20 Jul 2007 14:08:28 +0000 (16:08 +0200)]
Fio 1.16.9

Lots of good bug reporting from Erwan Velu, so lets celebrate
those fixes with a new release.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoSet minimum value allowed for some options
Jens Axboe [Fri, 20 Jul 2007 12:25:31 +0000 (14:25 +0200)]
Set minimum value allowed for some options

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix disk zone test case
Jens Axboe [Fri, 20 Jul 2007 11:25:04 +0000 (13:25 +0200)]
Fix disk zone test case

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoMention what the [ xxx / xxx kb/s ] means in the status output
Jens Axboe [Fri, 20 Jul 2007 10:43:05 +0000 (12:43 +0200)]
Mention what the [ xxx / xxx kb/s ] means in the status output

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoDon't align memory unless using raw io
Jens Axboe [Fri, 20 Jul 2007 10:39:22 +0000 (12:39 +0200)]
Don't align memory unless using raw io

If it's not raw io, we didn't align the buffer size. So we cannot
align.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoDefault stdev to 0.0, not -1.0
Jens Axboe [Thu, 19 Jul 2007 19:38:35 +0000 (21:38 +0200)]
Default stdev to 0.0, not -1.0

If we only have a single sample, then the deviation is by definition 0.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.8 fio-1.16.8
Jens Axboe [Thu, 19 Jul 2007 13:08:54 +0000 (15:08 +0200)]
Fio 1.16.8

1.16.7 was rather short lived, as it had a stupid bug for
the ordinary malloc() backed buffers.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agomalloc memory still needs to be aligned, if we are doing direct IO
Jens Axboe [Thu, 19 Jul 2007 13:06:57 +0000 (15:06 +0200)]
malloc memory still needs to be aligned, if we are doing direct IO

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoTreat option outside of job section as fatal and quit
Jens Axboe [Thu, 19 Jul 2007 12:53:01 +0000 (14:53 +0200)]
Treat option outside of job section as fatal and quit

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix memory overflow bugs
Jens Axboe [Thu, 19 Jul 2007 12:50:05 +0000 (14:50 +0200)]
Fix memory overflow bugs

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.7 fio-1.16.7
Jens Axboe [Thu, 19 Jul 2007 12:22:50 +0000 (14:22 +0200)]
Fio 1.16.7

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoMake a note of Mandriva package availability
Jens Axboe [Thu, 19 Jul 2007 12:21:08 +0000 (14:21 +0200)]
Make a note of Mandriva package availability

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix bad alignment in buffer size allocation
Jens Axboe [Thu, 19 Jul 2007 08:17:09 +0000 (10:17 +0200)]
Fix bad alignment in buffer size allocation

We typically end up allocating a block more than we need. Usually
not a problem for normal memory, but when we are dealing with
scarce resources such as hugepages, it can be problematic. For many
threads the memory waste could also be a problem.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd clue for ENOMEM hugepage allocation
Jens Axboe [Thu, 19 Jul 2007 08:09:45 +0000 (10:09 +0200)]
Add clue for ENOMEM hugepage allocation

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix typo in shmhuge commit
Jens Axboe [Thu, 19 Jul 2007 08:01:03 +0000 (10:01 +0200)]
Fix typo in shmhuge commit

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoDetect and print when the OS doesn't support huge pages
Jens Axboe [Thu, 19 Jul 2007 08:00:05 +0000 (10:00 +0200)]
Detect and print when the OS doesn't support huge pages

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agogit:// url location change
Jens Axboe [Wed, 18 Jul 2007 11:51:19 +0000 (13:51 +0200)]
git:// url location change

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.6 fio-1.16.6
Jens Axboe [Fri, 6 Jul 2007 12:24:19 +0000 (14:24 +0200)]
Fio 1.16.6

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix bug with FIO_IO_U_M/U_NR being transposed
Jens Axboe [Fri, 6 Jul 2007 12:09:49 +0000 (14:09 +0200)]
Fix bug with FIO_IO_U_M/U_NR being transposed

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoGuard __must_check ifdef
Jens Axboe [Sun, 24 Jun 2007 20:41:27 +0000 (22:41 +0200)]
Guard __must_check ifdef

Some systems seem to pick up the kernel definitions, so check
if __must_check is defined before (potentially) redefining it.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agonet engine: cleanup the splice handling
Jens Axboe [Sun, 24 Jun 2007 19:41:46 +0000 (21:41 +0200)]
net engine: cleanup the splice handling

The splice and vmsplice bits were largely duplicated, so abstract
the stuff out so it's clearer to see what is going on.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agonet engine: fix listenfd/pipe fd leaks
Jens Axboe [Sun, 24 Jun 2007 19:28:39 +0000 (21:28 +0200)]
net engine: fix listenfd/pipe fd leaks

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agonetsplice: use zero-copy if possible
Jens Axboe [Fri, 22 Jun 2007 18:41:27 +0000 (20:41 +0200)]
netsplice: use zero-copy if possible

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agonetsplice: fix a few typos
Jens Axboe [Fri, 22 Jun 2007 18:40:10 +0000 (20:40 +0200)]
netsplice: fix a few typos

It actually works now.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoSupport for 'netsplice' engine
Jens Axboe [Fri, 22 Jun 2007 13:42:21 +0000 (15:42 +0200)]
Support for 'netsplice' engine

This IO engine does network transfers, but uses splice() to move data
to/from a socket to a pipe and vmsplice() to move that data to/from
user space.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoguasi engine: remove STFU_GCC()
Jens Axboe [Tue, 19 Jun 2007 12:25:46 +0000 (14:25 +0200)]
guasi engine: remove STFU_GCC()

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.5 fio-1.16.5
Jens Axboe [Tue, 19 Jun 2007 10:49:52 +0000 (12:49 +0200)]
Fio 1.16.5

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdaptive io latency distrobution "histogram"
Jens Axboe [Tue, 19 Jun 2007 10:48:41 +0000 (12:48 +0200)]
Adaptive io latency distrobution "histogram"

Log ranges all the way down to microseconds, and display the most
appropriate values for those ranges.

This makes the latency usable for sub-milisecond devices as well.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoHOWTO: note that clat/slat can be in micro or mili seconds
Jens Axboe [Tue, 19 Jun 2007 07:53:31 +0000 (09:53 +0200)]
HOWTO: note that clat/slat can be in micro or mili seconds

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoio_u_mark_depth: just set index, don't fall through
Jens Axboe [Tue, 19 Jun 2007 07:50:28 +0000 (09:50 +0200)]
io_u_mark_depth: just set index, don't fall through

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoCleanup msec <-> usec lat conversion
Jens Axboe [Tue, 19 Jun 2007 07:48:44 +0000 (09:48 +0200)]
Cleanup msec <-> usec lat conversion

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agolibaio engine: print warning for depth > 1 and buffered IO
Jens Axboe [Mon, 18 Jun 2007 07:48:57 +0000 (09:48 +0200)]
libaio engine: print warning for depth > 1 and buffered IO

Linux aio doesn't support queued buffered IO, it has to be
O_DIRECT. So print a warning to that effect, so that users
of fio don't get bad results by mistake.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoMake completion and submission latency use appropriate time base
Jens Axboe [Mon, 18 Jun 2007 07:41:23 +0000 (09:41 +0200)]
Make completion and submission latency use appropriate time base

It's hardcoded as msecs right now, switch the accounting to usecs
and print out the appropriate base determined by the value range.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd cpus_allowed option
Jens Axboe [Fri, 15 Jun 2007 08:33:49 +0000 (10:33 +0200)]
Add cpus_allowed option

cpumask can be cumbersome to use, since it requires you to calculate
a decimal mask to pass to fio. So add a cpus_allowed option that
takes a text list of allowed cpus, ala: cpus_allowed=1,3,8 will set
the affinity mask for CPUS 1, 3, and 8.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.4 fio-1.16.4
Jens Axboe [Thu, 14 Jun 2007 06:17:08 +0000 (08:17 +0200)]
Fio 1.16.4

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoHOWTO: expand the cpumask explanation a bit
Jens Axboe [Wed, 13 Jun 2007 19:00:46 +0000 (21:00 +0200)]
HOWTO: expand the cpumask explanation a bit

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoMerge branch 'master' of ssh://brick.kernel.dk/data/git/fio
Jens Axboe [Wed, 13 Jun 2007 18:58:09 +0000 (20:58 +0200)]
Merge branch 'master' of ssh://brick.kernel.dk/data/git/fio

13 years agoFix cpu mask setting
Gough, Corey D [Wed, 13 Jun 2007 18:58:01 +0000 (20:58 +0200)]
Fix cpu mask setting

Fio passed the mask by value, not by reference. So when
fill_cpu_mask() was done setting the valid CPUs, it was
only on the local value.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoMerge branch 'master' of ssh://router/data/git/fio
Jens Axboe [Wed, 13 Jun 2007 17:04:20 +0000 (19:04 +0200)]
Merge branch 'master' of ssh://router/data/git/fio

13 years agosplice engine: support for vmsplice to user space
Jens Axboe [Wed, 13 Jun 2007 17:04:16 +0000 (19:04 +0200)]
splice engine: support for vmsplice to user space

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agosplice flags are unsigned int, not unsigned long
Jens Axboe [Fri, 8 Jun 2007 09:24:58 +0000 (11:24 +0200)]
splice flags are unsigned int, not unsigned long

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoDocumentation update
Jens Axboe [Fri, 25 May 2007 07:26:05 +0000 (09:26 +0200)]
Documentation update

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.3 fio-1.16.3
Jens Axboe [Thu, 24 May 2007 08:34:30 +0000 (10:34 +0200)]
Fio 1.16.3

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoTweak meaning of size option
Jens Axboe [Thu, 24 May 2007 08:32:47 +0000 (10:32 +0200)]
Tweak meaning of size option

Consider it the full amount of IO to be done, even if the
files used are smaller. Then we'll just restart the workload
until we reach the desired number of bytes.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix bug with time_based run
Jens Axboe [Wed, 23 May 2007 07:34:42 +0000 (09:34 +0200)]
Fix bug with time_based run

We need to clear the FIO_FILE_DONE flag and the number of
files done, to avoid do_io() exiting quickly for a re-run.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFio 1.16.2 fio-1.16.2
Jens Axboe [Tue, 22 May 2007 15:07:44 +0000 (17:07 +0200)]
Fio 1.16.2

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAdd comment in os/os-linux.h wrt glibc-2.3.2
Jens Axboe [Tue, 22 May 2007 15:06:24 +0000 (17:06 +0200)]
Add comment in os/os-linux.h wrt glibc-2.3.2

It was a broken release wrt cpu affinity, they removed the
size argument from the set/get functions. If you want to
compile on such an ancient glibc, manually define GLIBC_2_3_2.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix build on older systems
Jens Axboe [Tue, 22 May 2007 13:51:50 +0000 (15:51 +0200)]
Fix build on older systems

Avoid conflict with MINOR/MAJOR.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix build/compile for non-blktrace platforms
Jens Axboe [Tue, 22 May 2007 11:27:30 +0000 (13:27 +0200)]
Fix build/compile for non-blktrace platforms

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoIO verification needs to open and grab a reference to a file
Jens Axboe [Tue, 22 May 2007 09:12:13 +0000 (11:12 +0200)]
IO verification needs to open and grab a reference to a file

Otherwise it can disappear in midair.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix bug with random IO and network connections
Jens Axboe [Tue, 22 May 2007 07:21:37 +0000 (09:21 +0200)]
Fix bug with random IO and network connections

It isn't a supported configuration.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoUpdate HOWTO on cpumask setting
Jens Axboe [Tue, 22 May 2007 07:13:31 +0000 (09:13 +0200)]
Update HOWTO on cpumask setting

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix cpumask setting/getting on older kernels
Jens Axboe [Tue, 22 May 2007 07:13:02 +0000 (09:13 +0200)]
Fix cpumask setting/getting on older kernels

Don't call into the affinity functions, unless the user has
specified a mask.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agommap engine: fix open-for-size check
Jens Axboe [Mon, 21 May 2007 12:52:43 +0000 (14:52 +0200)]
mmap engine: fix open-for-size check

Don't call into mmap() yet, since we didn't setup ->io_size.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAttempt to work-around possible time warp
Jens Axboe [Mon, 21 May 2007 12:27:32 +0000 (14:27 +0200)]
Attempt to work-around possible time warp

A theory - we see gettimeofday() going backwards on some
systems, try and correct for that by just returning 0
for time passed.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoImprove random pattern without norandommap
Jens Axboe [Mon, 21 May 2007 11:23:30 +0000 (13:23 +0200)]
Improve random pattern without norandommap

Using the random map, random workloads have a natural tendency to
get less random at the end of the run, due to failures to find a
new unused random offset.

Improve the logic to avoid this problem.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoRename ffz() function to fio_ffz() to avoid conflicts
Jens Axboe [Mon, 21 May 2007 09:42:33 +0000 (11:42 +0200)]
Rename ffz() function to fio_ffz() to avoid conflicts

Newer gcc/glibc include an ffz function. Lets just do the easy
thing and rename ours to fio_ffz(), then we can look into reusing
a provided one later if necessary.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support: major/minor fixups
Jens Axboe [Wed, 16 May 2007 16:28:47 +0000 (18:28 +0200)]
blktrace support: major/minor fixups

We get passed the kernel variant, so we need a bit of manual
support to untangle it (major() and minor() wont cut it).

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support: support for looking up a device
Jens Axboe [Wed, 16 May 2007 12:40:29 +0000 (14:40 +0200)]
blktrace support: support for looking up a device

We can get the device number out of the trace, turn that into
a major/minor pair and look it up in /dev.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support: discard notify messages
Jens Axboe [Wed, 16 May 2007 10:29:06 +0000 (12:29 +0200)]
blktrace support: discard notify messages

They are not interesting to us (they contain things like timestamps
and process information).

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support: fixups/cleanups/improvements
Jens Axboe [Wed, 16 May 2007 10:16:44 +0000 (12:16 +0200)]
blktrace support: fixups/cleanups/improvements

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agofifo updates
Jens Axboe [Wed, 16 May 2007 09:54:43 +0000 (11:54 +0200)]
fifo updates

13 years agoFix timeout on 64-bit BE archs
Jens Axboe [Wed, 16 May 2007 06:59:27 +0000 (08:59 +0200)]
Fix timeout on 64-bit BE archs

We stored the timeout in an unsigned long long pointer, but
the value was really unsigned long.

13 years agoFix fifo leak
Jens Axboe [Wed, 16 May 2007 07:26:23 +0000 (09:26 +0200)]
Fix fifo leak

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support: speedup reading of data
Jens Axboe [Wed, 16 May 2007 07:25:09 +0000 (09:25 +0200)]
blktrace support: speedup reading of data

We used to read in data in really small chunks (48 bytes at the time,
the size of the trace). This is really slow for large traces, so
add a fifo frontend to refill the cache in much larger sizes.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support: adjust largest io_u buffer size to match trace
Jens Axboe [Wed, 16 May 2007 06:47:46 +0000 (08:47 +0200)]
blktrace support: adjust largest io_u buffer size to match trace

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace replay: delay support
Jens Axboe [Tue, 15 May 2007 12:29:58 +0000 (14:29 +0200)]
blktrace replay: delay support

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace fixup + documentation
Jens Axboe [Tue, 15 May 2007 11:23:19 +0000 (13:23 +0200)]
blktrace fixup + documentation

Add a little documentation on using blktrace data files.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support now works
Jens Axboe [Tue, 15 May 2007 11:10:41 +0000 (13:10 +0200)]
blktrace support now works

One little buglet - the blktrace offset is the sector, not the
byte offset. For now it's hardcoded at 512, should really get
the value from the device.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoblktrace support update
Jens Axboe [Tue, 15 May 2007 09:54:21 +0000 (11:54 +0200)]
blktrace support update

This should essentially work, except that it will not delay between
io but just issue the entries as fast as it can. Missing bits:

- Utilize the ipo->delay to suitably delay issuing an io
- Extract device name from blktrace file
- Document it

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoiolog bug: TD_DDIR_READ should be TD_DDIR_WRITE
Jens Axboe [Tue, 15 May 2007 09:12:19 +0000 (11:12 +0200)]
iolog bug: TD_DDIR_READ should be TD_DDIR_WRITE

It's a typo, if we don't have reads stored but writes, then the
global direction should naturally be TD_DDIR_WRITE.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoFix typo in blktrace code
Jens Axboe [Tue, 15 May 2007 08:13:04 +0000 (10:13 +0200)]
Fix typo in blktrace code

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoSome more bits of blktrace support
Jens Axboe [Tue, 15 May 2007 08:12:26 +0000 (10:12 +0200)]
Some more bits of blktrace support

Basic read/write is there. Still need to extract device name from
the trace, and check time delays to wait appropriately.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoInitial blktrace support
Jens Axboe [Tue, 15 May 2007 06:44:04 +0000 (08:44 +0200)]
Initial blktrace support

This doesn't work yet, just committing what little bits I did as not
to lose them.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoLog error on io buffer allocation
Jens Axboe [Thu, 26 Apr 2007 13:24:20 +0000 (15:24 +0200)]
Log error on io buffer allocation

Currently it just exits silently.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
13 years agoAllow termination during io_u building
Jens Axboe [Thu, 26 Apr 2007 11:47:07 +0000 (13:47 +0200)]
Allow termination during io_u building

If we are setting up gigabytes of io buffers, it can take many seconds
for that operation to complete. If the user wishes to abort, fio wont
honor that request. Fix that.

Signed-off-by: Jens Axboe <jens.axboe@oracle.com>