verify: adjust fio_offset_overlap_risk to include randommap

Currently we are using a list to log I/O history if:
 * randommap is enabled and fio manages to allocate memory for it.
 * there are no offset modfiers with any jobs.
For any different scenario we use an RB tree to handle offset overlaps,
which disables header seed checks for them.

This commit expands fio_offset_overlap_risk() such that it covers
file_randommap() cases.
For random workload with this change these are the possible scenarios

 -----------------------------------------------------------------------
|                 |         norandommap=0              |  norandommap=1 |
|-----------------------------------------------------------------------|
| softrandommap=0 |        list (No change)            |    RB tree     |
|                 | (fio was able to allocate memory)  |  (No change)   |
|-----------------|------------------------------------|----------------|
|                 |      RB tree (Now always)          |    RB tree     |
| softrandommap=1 |Even if fio was able to allocate mem|  (No Change)   |
 -----------------------------------------------------------------------

With randommap enabled and softrandommap=1 we now always use an RB tree,
even when fio is able to allocate memory for random map. In this case
verify header seed check will be disabled. If users want to check header
seed they can either disable softrandommap or explicilty enable
verify_header_seed.

Effectively this changes randommap from being a per-file property to
per-job property.

This also fixes rand seed mismatch isues, that have been observed when
multiple files are used, such as for the below mentioned configuration.

[global]
do_verify=1
verify=md5
direct=1
[multi_file]
rw=readwrite
directory=.
nrfiles=2
size=32K

Here is the truncated log with debug=verify flag, and an extra log when
the seed gets generated as well as the mismatch.

verify   368109 file ./multi_file.0.1 seed 46386204153304124 offset=0, length=4096
verify   368109 file ./multi_file.0.0 seed 9852480210356360750 offset=0, length=4096
verify   368109 file ./multi_file.0.1 seed 4726550845720924880 offset=4096, length=4096
verify: bad header rand_seed 9852480210356360750, wanted 46386204153304124 at file ./multi_file.0.0 offset 0, length 4096 (requested block: offset=0, length=4096)

Earlier the I/O entries were getting logged in an RB tree, as we were
relying on file_randommap(), which was false for sequential workloads.
In RB tree, files are prioritized first and then the offset. Thus during
the verify phase the I/O entries are removed from tree in order of file
and then offset which is not how it was originally written. With the new
checks, for sequential workload we now store the entries in the list
instead of RB tree.
Even for sequential workload if the user fortuitously specified
norandommap or softrandommap, then I/Os will be stored in an RB tree.
However in this case header seed checks will be disabled.

fixes #740
fixes #746
fixes #844
fixes #1538

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Signed-off-by: Ankit Kumar <ankit.kumar@samsung.com>

diff --git a/fio.h b/fio.h
index c7c0d147c4dbb6b1591ab3645e21277f2fb9cf72..d642325813fd844859e1ff74f0ddc4752daa059f 100644 (file)
--- a/fio.h
+++ b/fio.h
@@ -802,7 +802,8 @@ extern void lat_target_reset(struct thread_data *);
 
 static inline bool fio_offset_overlap_risk(struct thread_data *td)
 {
-       if (td->o.ddir_seq_add || (td->o.ddir_seq_nr > 1))
+       if (td->o.norandommap || td->o.softrandommap ||
+           td->o.ddir_seq_add || (td->o.ddir_seq_nr > 1))
                return true;
 
        return false;

diff --git a/init.c b/init.c
index feaa3f28435b3a1d15aafc6935149c4a9239e87c..95f2179d6b4b189f9055c696f2e0ea1def2ec700 100644 (file)
--- a/init.c
+++ b/init.c
@@ -886,13 +886,12 @@ static int fixup_options(struct thread_data *td)
 
                /*
                 * Disable rand_seed check when we have verify_backlog,
-                * zone reset frequency for zonemode=zbd, norandommap, or
-                * offset modifiers.
+                * zone reset frequency for zonemode=zbd, or if we are using
+                * an RB tree for IO history logs.
                 * Unless we were explicitly asked to enable it.
                 */
                if (!td_write(td) || (td->flags & TD_F_VER_BACKLOG) ||
-                   o->zrf.u.f || o->norandommap ||
-                   fio_offset_overlap_risk(td)) {
+                   o->zrf.u.f || fio_offset_overlap_risk(td)) {
                        if (!fio_option_is_set(o, verify_header_seed))
                                o->verify_header_seed = 0;
                }

diff --git a/iolog.c b/iolog.c
index 6b5dc9abbae051f1aeb47365d19ea6ea3d67d763..dcf6083c7d38aac15227f414acf7e7cdc3a09050 100644 (file)
--- a/iolog.c
+++ b/iolog.c
@@ -306,7 +306,7 @@ void log_io_piece(struct thread_data *td, struct io_u *io_u)
         * the rb insert/lookup for handling. Sort writes if we have offset
         * modifier which can also create duplicate blocks.
         */
-       if (file_randommap(td, ipo->file) && !fio_offset_overlap_risk(td)) {
+       if (!fio_offset_overlap_risk(td)) {
                INIT_FLIST_HEAD(&ipo->list);
                flist_add_tail(&ipo->list, &td->io_hist_list);
                ipo->flags |= IP_F_ONLIST;