compiler: clang
osx_image: xcode8.3
env: BUILD_ARCH="x86_64"
+ - os: osx
+ compiler: clang
+ osx_image: xcode9.4
+ env: BUILD_ARCH="x86_64"
exclude:
- os: osx
compiler: gcc
.. option:: --aux-path=path
- Use this `path` for fio state generated files.
+ Use the directory specified by `path` for generated state files instead
+ of the current working directory.
Any parameters following the options will be assumed to be job files, unless
they match a job file parameter. Multiple job files can be listed and each job
assigned equally distributed to job clones created by :option:`numjobs` as
long as they are using generated filenames. If specific `filename(s)` are
set fio will use the first listed directory, and thereby matching the
- `filename` semantic which generates a file each clone if not specified, but
- let all clones use the same if set.
+ `filename` semantic (which generates a file for each clone if not
+ specified, but lets all clones use the same file if set).
See the :option:`filename` option for information on how to escape "``:``" and
"``\``" characters within the directory path itself.
+ Note: To control the directory fio will use for internal state files
+ use :option:`--aux-path`.
+
.. option:: filename=str
Fio normally makes up a `filename` based on the job name, thread number, and
Unlink job files after each iteration or loop. Default: false.
-.. option:: zonesize=int
+.. option:: zonerange=int
- Divide a file into zones of the specified size. See :option:`zoneskip`.
+ Size of a single zone in which I/O occurs. See also :option:`zonesize`
+ and :option:`zoneskip`.
-.. option:: zonerange=int
+.. option:: zonesize=int
- Give size of an I/O zone. See :option:`zoneskip`.
+ Number of bytes to transfer before skipping :option:`zoneskip`
+ bytes. If this parameter is smaller than :option:`zonerange` then only
+ a fraction of each zone with :option:`zonerange` bytes will be
+ accessed. If this parameter is larger than :option:`zonerange` then
+ each zone will be accessed multiple times before skipping
.. option:: zoneskip=int
- Skip the specified number of bytes when :option:`zonesize` data has been
- read. The two zone options can be used to only do I/O on zones of a file.
+ Skip the specified number of bytes when :option:`zonesize` data have
+ been transferred. The three zone options can be used to do strided I/O
+ on a file.
I/O type
(RBD) via librbd without the need to use the kernel rbd driver. This
ioengine defines engine specific options.
+ **http**
+ I/O engine supporting GET/PUT requests over HTTP(S) with libcurl to
+ a WebDAV or S3 endpoint. This ioengine defines engine specific options.
+
+ This engine only supports direct IO of iodepth=1; you need to scale this
+ via numjobs. blocksize defines the size of the objects to be created.
+
+ TRIM is translated to object deletion.
+
**gfapi**
Using GlusterFS libgfapi sync interface to direct access to
GlusterFS volumes without having to go through FUSE. This ioengine
transferred to the device. The writefua option is ignored with this
selection.
+.. option:: http_host=str : [http]
+
+ Hostname to connect to. For S3, this could be the bucket hostname.
+ Default is **localhost**
+
+.. option:: http_user=str : [http]
+
+ Username for HTTP authentication.
+
+.. option:: http_pass=str : [http]
+
+ Password for HTTP authentication.
+
+.. option:: https=str : [http]
+
+ Enable HTTPS instead of http. *on* enables HTTPS; *insecure*
+ will enable HTTPS, but disable SSL peer verification (use with
+ caution!). Default is **off**
+
+.. option:: http_mode=str : [http]
+
+ Which HTTP access mode to use: *webdav*, *swift*, or *s3*.
+ Default is **webdav**
+
+.. option:: http_s3_region=str : [http]
+
+ The S3 region/zone string.
+ Default is **us-east-1**
+
+.. option:: http_s3_key=str : [http]
+
+ The S3 secret key.
+
+.. option:: http_s3_keyid=str : [http]
+
+ The S3 key/access id.
+
+.. option:: http_swift_auth_token=str : [http]
+
+ The Swift auth token. See the example configuration file on how
+ to retrieve this.
+
+.. option:: http_verbose=int : [http]
+
+ Enable verbose requests from libcurl. Useful for debugging. 1
+ turns on verbose logging from libcurl, 2 additionally enables
+ HTTP IO tracing. Default is **0**
+
I/O depth
~~~~~~~~~
.. option:: write_iops_log=str
Same as :option:`write_bw_log`, but writes an IOPS file (e.g.
- :file:`name_iops.x.log`) instead. See :option:`write_bw_log` for
- details about the filename format and `Log File Formats`_ for how data
- is structured within the file.
+ :file:`name_iops.x.log`) instead. Because fio defaults to individual
+ I/O logging, the value entry in the IOPS log will be 1 unless windowed
+ logging (see :option:`log_avg_msec`) has been enabled. See
+ :option:`write_bw_log` for details about the filename format and `Log
+ File Formats`_ for how data is structured within the file.
.. option:: log_avg_msec=int
**2**
I/O is a TRIM
-The entry's *block size* is always in bytes. The *offset* is the offset, in bytes,
-from the start of the file, for that particular I/O. The logging of the offset can be
+The entry's *block size* is always in bytes. The *offset* is the position in bytes
+from the start of the file for that particular I/O. The logging of the offset can be
toggled with :option:`log_offset`.
-Fio defaults to logging every individual I/O. When IOPS are logged for individual
-I/Os the *value* entry will always be 1. If windowed logging is enabled through
-:option:`log_avg_msec`, fio logs the average values over the specified period of time.
-If windowed logging is enabled and :option:`log_max_value` is set, then fio logs
-maximum values in that window instead of averages. Since *data direction*, *block
-size* and *offset* are per-I/O values, if windowed logging is enabled they
-aren't applicable and will be 0.
+Fio defaults to logging every individual I/O but when windowed logging is set
+through :option:`log_avg_msec`, either the average (by default) or the maximum
+(:option:`log_max_value` is set) *value* seen over the specified period of time
+is recorded. Each *data direction* seen within the window period will aggregate
+its values in a separate row. Further, when using windowed logging the *block
+size* and *offset* entries will always contain 0.
Client/Server
-------------
gettime-thread.c helpers.c json.c idletime.c td_error.c \
profiles/tiobench.c profiles/act.c io_u_queue.c filelock.c \
workqueue.c rate-submit.c optgroup.c helper_thread.c \
- steadystate.c
+ steadystate.c zone-dist.c
ifdef CONFIG_LIBHDFS
HDFSFLAGS= -I $(JAVA_HOME)/include -I $(JAVA_HOME)/include/linux -I $(FIO_LIBHDFS_INCLUDE)
ifdef CONFIG_RBD
SOURCE += engines/rbd.c
endif
+ifdef CONFIG_HTTP
+ SOURCE += engines/http.c
+endif
SOURCE += oslib/asprintf.c
ifndef CONFIG_STRSEP
SOURCE += oslib/strsep.c
#if defined (__ARM_ARCH_4__) || defined (__ARM_ARCH_4T__) \
|| defined (__ARM_ARCH_5__) || defined (__ARM_ARCH_5T__) || defined (__ARM_ARCH_5E__)\
|| defined (__ARM_ARCH_5TE__) || defined (__ARM_ARCH_5TEJ__) \
- || defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || defined(__ARM_ARCH_6Z__) || defined(__ARM_ARCH_6ZK__)
+ || defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || defined(__ARM_ARCH_6Z__) || defined(__ARM_ARCH_6ZK__) \
+ || defined(__ARM_ARCH_6KZ__)
#define nop __asm__ __volatile__("mov\tr0,r0\t@ nop\n\t")
#define read_barrier() __asm__ __volatile__ ("" : : : "memory")
#define write_barrier() __asm__ __volatile__ ("" : : : "memory")
#include "rate-submit.h"
#include "helper_thread.h"
#include "pshared.h"
+#include "zone-dist.h"
static struct fio_sem *startup_sem;
static struct flist_head *cgroup_list;
static int init_io_u(struct thread_data *td)
{
struct io_u *io_u;
- unsigned long long max_bs, min_write;
int cl_align, i, max_units;
- int data_xfer = 1, err;
- char *p;
+ int err;
max_units = td->o.iodepth;
- max_bs = td_max_bs(td);
- min_write = td->o.min_bs[DDIR_WRITE];
- td->orig_buffer_size = (unsigned long long) max_bs
- * (unsigned long long) max_units;
-
- if (td_ioengine_flagged(td, FIO_NOIO) || !(td_read(td) || td_write(td)))
- data_xfer = 0;
err = 0;
err += !io_u_rinit(&td->io_u_requeues, td->o.iodepth);
return 1;
}
+ cl_align = os_cache_line_size();
+
+ for (i = 0; i < max_units; i++) {
+ void *ptr;
+
+ if (td->terminate)
+ return 1;
+
+ ptr = fio_memalign(cl_align, sizeof(*io_u));
+ if (!ptr) {
+ log_err("fio: unable to allocate aligned memory\n");
+ break;
+ }
+
+ io_u = ptr;
+ memset(io_u, 0, sizeof(*io_u));
+ INIT_FLIST_HEAD(&io_u->verify_list);
+ dprint(FD_MEM, "io_u alloc %p, index %u\n", io_u, i);
+
+ io_u->index = i;
+ io_u->flags = IO_U_F_FREE;
+ io_u_qpush(&td->io_u_freelist, io_u);
+
+ /*
+ * io_u never leaves this stack, used for iteration of all
+ * io_u buffers.
+ */
+ io_u_qpush(&td->io_u_all, io_u);
+
+ if (td->io_ops->io_u_init) {
+ int ret = td->io_ops->io_u_init(td, io_u);
+
+ if (ret) {
+ log_err("fio: failed to init engine data: %d\n", ret);
+ return 1;
+ }
+ }
+ }
+
+ init_io_u_buffers(td);
+
+ if (init_file_completion_logging(td, max_units))
+ return 1;
+
+ return 0;
+}
+
+int init_io_u_buffers(struct thread_data *td)
+{
+ struct io_u *io_u;
+ unsigned long long max_bs, min_write;
+ int i, max_units;
+ int data_xfer = 1;
+ char *p;
+
+ max_units = td->o.iodepth;
+ max_bs = td_max_bs(td);
+ min_write = td->o.min_bs[DDIR_WRITE];
+ td->orig_buffer_size = (unsigned long long) max_bs
+ * (unsigned long long) max_units;
+
+ if (td_ioengine_flagged(td, FIO_NOIO) || !(td_read(td) || td_write(td)))
+ data_xfer = 0;
+
/*
* if we may later need to do address alignment, then add any
* possible adjustment here so that we don't cause a buffer
else
p = td->orig_buffer;
- cl_align = os_cache_line_size();
-
for (i = 0; i < max_units; i++) {
- void *ptr;
-
- if (td->terminate)
- return 1;
-
- ptr = fio_memalign(cl_align, sizeof(*io_u));
- if (!ptr) {
- log_err("fio: unable to allocate aligned memory\n");
- break;
- }
-
- io_u = ptr;
- memset(io_u, 0, sizeof(*io_u));
- INIT_FLIST_HEAD(&io_u->verify_list);
+ io_u = td->io_u_all.io_us[i];
dprint(FD_MEM, "io_u alloc %p, index %u\n", io_u, i);
if (data_xfer) {
fill_verify_pattern(td, io_u->buf, max_bs, io_u, 0, 0);
}
}
-
- io_u->index = i;
- io_u->flags = IO_U_F_FREE;
- io_u_qpush(&td->io_u_freelist, io_u);
-
- /*
- * io_u never leaves this stack, used for iteration of all
- * io_u buffers.
- */
- io_u_qpush(&td->io_u_all, io_u);
-
- if (td->io_ops->io_u_init) {
- int ret = td->io_ops->io_u_init(td, io_u);
-
- if (ret) {
- log_err("fio: failed to init engine data: %d\n", ret);
- return 1;
- }
- }
-
p += max_bs;
}
- if (init_file_completion_logging(td, max_units))
- return 1;
-
return 0;
}
goto err;
}
+ td_zone_gen_index(td);
+
/*
* Do this early, we don't want the compress threads to be limited
* to the same CPUs as the IO workers. So do this before we set
close_ioengine(td);
cgroup_shutdown(td, cgroup_mnt);
verify_free_state(td);
-
- if (td->zone_state_index) {
- int i;
-
- for (i = 0; i < DDIR_RWDIR_CNT; i++)
- free(td->zone_state_index[i]);
- free(td->zone_state_index);
- td->zone_state_index = NULL;
- }
+ td_zone_free_index(td);
if (fio_option_is_set(o, cpumask)) {
ret = fio_cpuset_exit(&o->cpumask);
fi
TMPC="${TMPDIR1}/fio-conf-${RANDOM}-$$-${RANDOM}.c"
+TMPC2="${TMPDIR1}/fio-conf-${RANDOM}-$$-${RANDOM}-2.c"
TMPO="${TMPDIR1}/fio-conf-${RANDOM}-$$-${RANDOM}.o"
TMPE="${TMPDIR1}/fio-conf-${RANDOM}-$$-${RANDOM}.exe"
# NB: do not call "exit" in the trap handler; this is buggy with some shells;
# see <1285349658-3122-1-git-send-email-loic.minier@linaro.org>
-trap "rm -f $TMPC $TMPO $TMPE" EXIT INT QUIT TERM
+trap "rm -f $TMPC $TMPC2 $TMPO $TMPE" EXIT INT QUIT TERM
rm -rf config.log
;;
--disable-rbd) disable_rbd="yes"
;;
+ --disable-http) disable_http="yes"
+ ;;
--disable-gfapi) disable_gfapi="yes"
;;
--enable-libhdfs) libhdfs="yes"
fi
print_config "IPv6 helpers" "$ipv6"
+##########################################
+# check for http
+if test "$http" != "yes" ; then
+ http="no"
+fi
+# check for openssl >= 1.1.0, which uses an opaque HMAC_CTX pointer
+cat > $TMPC << EOF
+#include <curl/curl.h>
+#include <openssl/hmac.h>
+
+int main(int argc, char **argv)
+{
+ CURL *curl;
+ HMAC_CTX *ctx;
+
+ curl = curl_easy_init();
+ curl_easy_cleanup(curl);
+
+ ctx = HMAC_CTX_new();
+ HMAC_CTX_reset(ctx);
+ HMAC_CTX_free(ctx);
+ return 0;
+}
+EOF
+# openssl < 1.1.0 uses the HMAC_CTX type directly
+cat > $TMPC2 << EOF
+#include <curl/curl.h>
+#include <openssl/hmac.h>
+
+int main(int argc, char **argv)
+{
+ CURL *curl;
+ HMAC_CTX ctx;
+
+ curl = curl_easy_init();
+ curl_easy_cleanup(curl);
+
+ HMAC_CTX_init(&ctx);
+ HMAC_CTX_cleanup(&ctx);
+ return 0;
+}
+EOF
+if test "$disable_http" != "yes"; then
+ HTTP_LIBS="-lcurl -lssl -lcrypto"
+ if compile_prog "" "$HTTP_LIBS" "curl-new-ssl"; then
+ output_sym "CONFIG_HAVE_OPAQUE_HMAC_CTX"
+ http="yes"
+ LIBS="$HTTP_LIBS $LIBS"
+ elif mv $TMPC2 $TMPC && compile_prog "" "$HTTP_LIBS" "curl-old-ssl"; then
+ http="yes"
+ LIBS="$HTTP_LIBS $LIBS"
+ fi
+fi
+print_config "http engine" "$http"
+
##########################################
# check for rados
if test "$rados" != "yes" ; then
if test "$ipv6" = "yes" ; then
output_sym "CONFIG_IPV6"
fi
+if test "$http" = "yes" ; then
+ output_sym "CONFIG_HTTP"
+fi
if test "$rados" = "yes" ; then
output_sym "CONFIG_RADOS"
fi
// Advanced Hash Functions
//****************************
-void* XXH32_init (unsigned int seed);
+void* XXH32_init (uint32_t seed);
XXH_errorcode XXH32_update (void* state, const void* input, int len);
-unsigned int XXH32_digest (void* state);
+uint32_t XXH32_digest (void* state);
/*
These functions calculate the xxhash of an input provided in several small packets,
int XXH32_sizeofState(void);
-XXH_errorcode XXH32_resetState(void* state, unsigned int seed);
+XXH_errorcode XXH32_resetState(void* state, uint32_t seed);
#define XXH32_SIZEOFSTATE 48
typedef struct { long long ll[(XXH32_SIZEOFSTATE+(sizeof(long long)-1))/sizeof(long long)]; } XXH32_stateSpace_t;
*/
-unsigned int XXH32_intermediateDigest (void* state);
+uint32_t XXH32_intermediateDigest (void* state);
/*
This function does the same as XXH32_digest(), generating a 32-bit hash,
but preserve memory context.
--- /dev/null
+/*
+ * HTTP GET/PUT IO engine
+ *
+ * IO engine to perform HTTP(S) GET/PUT requests via libcurl-easy.
+ *
+ * Copyright (C) 2018 SUSE LLC
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License,
+ * version 2 as published by the Free Software Foundation..
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the Free
+ * Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+ * Boston, MA 02110-1301, USA.
+ */
+
+#include <pthread.h>
+#include <time.h>
+#include <curl/curl.h>
+#include <openssl/hmac.h>
+#include <openssl/sha.h>
+#include <openssl/md5.h>
+#include "fio.h"
+#include "../optgroup.h"
+
+
+enum {
+ FIO_HTTP_WEBDAV = 0,
+ FIO_HTTP_S3 = 1,
+ FIO_HTTP_SWIFT = 2,
+
+ FIO_HTTPS_OFF = 0,
+ FIO_HTTPS_ON = 1,
+ FIO_HTTPS_INSECURE = 2,
+};
+
+struct http_data {
+ CURL *curl;
+};
+
+struct http_options {
+ void *pad;
+ unsigned int https;
+ char *host;
+ char *user;
+ char *pass;
+ char *s3_key;
+ char *s3_keyid;
+ char *s3_region;
+ char *swift_auth_token;
+ int verbose;
+ unsigned int mode;
+};
+
+struct http_curl_stream {
+ char *buf;
+ size_t pos;
+ size_t max;
+};
+
+static struct fio_option options[] = {
+ {
+ .name = "https",
+ .lname = "https",
+ .type = FIO_OPT_STR,
+ .help = "Enable https",
+ .off1 = offsetof(struct http_options, https),
+ .def = "off",
+ .posval = {
+ { .ival = "off",
+ .oval = FIO_HTTPS_OFF,
+ .help = "No HTTPS",
+ },
+ { .ival = "on",
+ .oval = FIO_HTTPS_ON,
+ .help = "Enable HTTPS",
+ },
+ { .ival = "insecure",
+ .oval = FIO_HTTPS_INSECURE,
+ .help = "Enable HTTPS, disable peer verification",
+ },
+ },
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_host",
+ .lname = "http_host",
+ .type = FIO_OPT_STR_STORE,
+ .help = "Hostname (S3 bucket)",
+ .off1 = offsetof(struct http_options, host),
+ .def = "localhost",
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_user",
+ .lname = "http_user",
+ .type = FIO_OPT_STR_STORE,
+ .help = "HTTP user name",
+ .off1 = offsetof(struct http_options, user),
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_pass",
+ .lname = "http_pass",
+ .type = FIO_OPT_STR_STORE,
+ .help = "HTTP password",
+ .off1 = offsetof(struct http_options, pass),
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_s3_key",
+ .lname = "S3 secret key",
+ .type = FIO_OPT_STR_STORE,
+ .help = "S3 secret key",
+ .off1 = offsetof(struct http_options, s3_key),
+ .def = "",
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_s3_keyid",
+ .lname = "S3 key id",
+ .type = FIO_OPT_STR_STORE,
+ .help = "S3 key id",
+ .off1 = offsetof(struct http_options, s3_keyid),
+ .def = "",
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_swift_auth_token",
+ .lname = "Swift auth token",
+ .type = FIO_OPT_STR_STORE,
+ .help = "OpenStack Swift auth token",
+ .off1 = offsetof(struct http_options, swift_auth_token),
+ .def = "",
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_s3_region",
+ .lname = "S3 region",
+ .type = FIO_OPT_STR_STORE,
+ .help = "S3 region",
+ .off1 = offsetof(struct http_options, s3_region),
+ .def = "us-east-1",
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_mode",
+ .lname = "Request mode to use",
+ .type = FIO_OPT_STR,
+ .help = "Whether to use WebDAV, Swift, or S3",
+ .off1 = offsetof(struct http_options, mode),
+ .def = "webdav",
+ .posval = {
+ { .ival = "webdav",
+ .oval = FIO_HTTP_WEBDAV,
+ .help = "WebDAV server",
+ },
+ { .ival = "s3",
+ .oval = FIO_HTTP_S3,
+ .help = "S3 storage backend",
+ },
+ { .ival = "swift",
+ .oval = FIO_HTTP_SWIFT,
+ .help = "OpenStack Swift storage",
+ },
+ },
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = "http_verbose",
+ .lname = "HTTP verbosity level",
+ .type = FIO_OPT_INT,
+ .help = "increase http engine verbosity",
+ .off1 = offsetof(struct http_options, verbose),
+ .def = "0",
+ .category = FIO_OPT_C_ENGINE,
+ .group = FIO_OPT_G_HTTP,
+ },
+ {
+ .name = NULL,
+ },
+};
+
+static char *_aws_uriencode(const char *uri)
+{
+ size_t bufsize = 1024;
+ char *r = malloc(bufsize);
+ char c;
+ int i, n;
+ const char *hex = "0123456789ABCDEF";
+
+ if (!r) {
+ log_err("malloc failed\n");
+ return NULL;
+ }
+
+ n = 0;
+ for (i = 0; (c = uri[i]); i++) {
+ if (n > bufsize-5) {
+ log_err("encoding the URL failed\n");
+ return NULL;
+ }
+
+ if ( (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z')
+ || (c >= '0' && c <= '9') || c == '_' || c == '-'
+ || c == '~' || c == '.' || c == '/')
+ r[n++] = c;
+ else {
+ r[n++] = '%';
+ r[n++] = hex[(c >> 4 ) & 0xF];
+ r[n++] = hex[c & 0xF];
+ }
+ }
+ r[n++] = 0;
+ return r;
+}
+
+static char *_conv_hex(const unsigned char *p, size_t len)
+{
+ char *r;
+ int i,n;
+ const char *hex = "0123456789abcdef";
+ r = malloc(len * 2 + 1);
+ n = 0;
+ for (i = 0; i < len; i++) {
+ r[n++] = hex[(p[i] >> 4 ) & 0xF];
+ r[n++] = hex[p[i] & 0xF];
+ }
+ r[n] = 0;
+
+ return r;
+}
+
+static char *_gen_hex_sha256(const char *p, size_t len)
+{
+ unsigned char hash[SHA256_DIGEST_LENGTH];
+
+ SHA256((unsigned char*)p, len, hash);
+ return _conv_hex(hash, SHA256_DIGEST_LENGTH);
+}
+
+static char *_gen_hex_md5(const char *p, size_t len)
+{
+ unsigned char hash[MD5_DIGEST_LENGTH];
+
+ MD5((unsigned char*)p, len, hash);
+ return _conv_hex(hash, MD5_DIGEST_LENGTH);
+}
+
+static void _hmac(unsigned char *md, void *key, int key_len, char *data) {
+#ifndef CONFIG_HAVE_OPAQUE_HMAC_CTX
+ HMAC_CTX _ctx;
+#endif
+ HMAC_CTX *ctx;
+ unsigned int hmac_len;
+
+#ifdef CONFIG_HAVE_OPAQUE_HMAC_CTX
+ ctx = HMAC_CTX_new();
+#else
+ ctx = &_ctx;
+#endif
+ HMAC_Init_ex(ctx, key, key_len, EVP_sha256(), NULL);
+ HMAC_Update(ctx, (unsigned char*)data, strlen(data));
+ HMAC_Final(ctx, md, &hmac_len);
+#ifdef CONFIG_HAVE_OPAQUE_HMAC_CTX
+ HMAC_CTX_free(ctx);
+#else
+ HMAC_CTX_cleanup(ctx);
+#endif
+}
+
+static int _curl_trace(CURL *handle, curl_infotype type,
+ char *data, size_t size,
+ void *userp)
+{
+ const char *text;
+ (void)handle; /* prevent compiler warning */
+ (void)userp;
+
+ switch (type) {
+ case CURLINFO_TEXT:
+ fprintf(stderr, "== Info: %s", data);
+ default:
+ case CURLINFO_SSL_DATA_OUT:
+ case CURLINFO_SSL_DATA_IN:
+ return 0;
+
+ case CURLINFO_HEADER_OUT:
+ text = "=> Send header";
+ break;
+ case CURLINFO_DATA_OUT:
+ text = "=> Send data";
+ break;
+ case CURLINFO_HEADER_IN:
+ text = "<= Recv header";
+ break;
+ case CURLINFO_DATA_IN:
+ text = "<= Recv data";
+ break;
+ }
+
+ log_info("%s: %s", text, data);
+ return 0;
+}
+
+/* https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
+ * https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html#signing-request-intro
+ */
+static void _add_aws_auth_header(CURL *curl, struct curl_slist *slist, struct http_options *o,
+ int op, const char *uri, char *buf, size_t len)
+{
+ char date_short[16];
+ char date_iso[32];
+ char method[8];
+ char dkey[128];
+ char creq[512];
+ char sts[256];
+ char s[512];
+ char *uri_encoded = NULL;
+ char *dsha = NULL;
+ char *csha = NULL;
+ char *signature = NULL;
+ const char *service = "s3";
+ const char *aws = "aws4_request";
+ unsigned char md[SHA256_DIGEST_LENGTH];
+
+ time_t t = time(NULL);
+ struct tm *gtm = gmtime(&t);
+
+ strftime (date_short, sizeof(date_short), "%Y%m%d", gtm);
+ strftime (date_iso, sizeof(date_iso), "%Y%m%dT%H%M%SZ", gtm);
+ uri_encoded = _aws_uriencode(uri);
+
+ if (op == DDIR_WRITE) {
+ dsha = _gen_hex_sha256(buf, len);
+ sprintf(method, "PUT");
+ } else {
+ /* DDIR_READ && DDIR_TRIM supply an empty body */
+ if (op == DDIR_READ)
+ sprintf(method, "GET");
+ else
+ sprintf(method, "DELETE");
+ dsha = _gen_hex_sha256("", 0);
+ }
+
+ /* Create the canonical request first */
+ snprintf(creq, sizeof(creq),
+ "%s\n"
+ "%s\n"
+ "\n"
+ "host:%s\n"
+ "x-amz-content-sha256:%s\n"
+ "x-amz-date:%s\n"
+ "\n"
+ "host;x-amz-content-sha256;x-amz-date\n"
+ "%s"
+ , method
+ , uri_encoded, o->host, dsha, date_iso, dsha);
+
+ csha = _gen_hex_sha256(creq, strlen(creq));
+ snprintf(sts, sizeof(sts), "AWS4-HMAC-SHA256\n%s\n%s/%s/%s/%s\n%s",
+ date_iso, date_short, o->s3_region, service, aws, csha);
+
+ snprintf((char *)dkey, sizeof(dkey), "AWS4%s", o->s3_key);
+ _hmac(md, dkey, strlen(dkey), date_short);
+ _hmac(md, md, SHA256_DIGEST_LENGTH, o->s3_region);
+ _hmac(md, md, SHA256_DIGEST_LENGTH, (char*) service);
+ _hmac(md, md, SHA256_DIGEST_LENGTH, (char*) aws);
+ _hmac(md, md, SHA256_DIGEST_LENGTH, sts);
+
+ signature = _conv_hex(md, SHA256_DIGEST_LENGTH);
+
+ /* Surpress automatic Accept: header */
+ slist = curl_slist_append(slist, "Accept:");
+
+ snprintf(s, sizeof(s), "x-amz-content-sha256: %s", dsha);
+ slist = curl_slist_append(slist, s);
+
+ snprintf(s, sizeof(s), "x-amz-date: %s", date_iso);
+ slist = curl_slist_append(slist, s);
+
+ snprintf(s, sizeof(s), "Authorization: AWS4-HMAC-SHA256 Credential=%s/%s/%s/s3/aws4_request,"
+ "SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=%s",
+ o->s3_keyid, date_short, o->s3_region, signature);
+ slist = curl_slist_append(slist, s);
+
+ curl_easy_setopt(curl, CURLOPT_HTTPHEADER, slist);
+
+ free(uri_encoded);
+ free(csha);
+ free(dsha);
+ free(signature);
+}
+
+static void _add_swift_header(CURL *curl, struct curl_slist *slist, struct http_options *o,
+ int op, const char *uri, char *buf, size_t len)
+{
+ char *dsha = NULL;
+ char s[512];
+
+ if (op == DDIR_WRITE) {
+ dsha = _gen_hex_md5(buf, len);
+ }
+ /* Surpress automatic Accept: header */
+ slist = curl_slist_append(slist, "Accept:");
+
+ snprintf(s, sizeof(s), "etag: %s", dsha);
+ slist = curl_slist_append(slist, s);
+
+ snprintf(s, sizeof(s), "x-auth-token: %s", o->swift_auth_token);
+ slist = curl_slist_append(slist, s);
+
+ curl_easy_setopt(curl, CURLOPT_HTTPHEADER, slist);
+
+ free(dsha);
+}
+
+static void fio_http_cleanup(struct thread_data *td)
+{
+ struct http_data *http = td->io_ops_data;
+
+ if (http) {
+ curl_easy_cleanup(http->curl);
+ free(http);
+ }
+}
+
+static size_t _http_read(void *ptr, size_t size, size_t nmemb, void *stream)
+{
+ struct http_curl_stream *state = stream;
+ size_t len = size * nmemb;
+ /* We're retrieving; nothing is supposed to be read locally */
+ if (!stream)
+ return 0;
+ if (len+state->pos > state->max)
+ len = state->max - state->pos;
+ memcpy(ptr, &state->buf[state->pos], len);
+ state->pos += len;
+ return len;
+}
+
+static size_t _http_write(void *ptr, size_t size, size_t nmemb, void *stream)
+{
+ struct http_curl_stream *state = stream;
+ /* We're just discarding the returned body after a PUT */
+ if (!stream)
+ return nmemb;
+ if (size != 1)
+ return CURLE_WRITE_ERROR;
+ if (nmemb + state->pos > state->max)
+ return CURLE_WRITE_ERROR;
+ memcpy(&state->buf[state->pos], ptr, nmemb);
+ state->pos += nmemb;
+ return nmemb;
+}
+
+static int _http_seek(void *stream, curl_off_t offset, int origin)
+{
+ struct http_curl_stream *state = stream;
+ if (offset < state->max && origin == SEEK_SET) {
+ state->pos = offset;
+ return CURL_SEEKFUNC_OK;
+ } else
+ return CURL_SEEKFUNC_FAIL;
+}
+
+static enum fio_q_status fio_http_queue(struct thread_data *td,
+ struct io_u *io_u)
+{
+ struct http_data *http = td->io_ops_data;
+ struct http_options *o = td->eo;
+ struct http_curl_stream _curl_stream;
+ struct curl_slist *slist = NULL;
+ char object[512];
+ char url[1024];
+ long status;
+ CURLcode res;
+ int r = -1;
+
+ fio_ro_check(td, io_u);
+ memset(&_curl_stream, 0, sizeof(_curl_stream));
+ snprintf(object, sizeof(object), "%s_%llu_%llu", td->files[0]->file_name,
+ io_u->offset, io_u->xfer_buflen);
+ if (o->https == FIO_HTTPS_OFF)
+ snprintf(url, sizeof(url), "http://%s%s", o->host, object);
+ else
+ snprintf(url, sizeof(url), "https://%s%s", o->host, object);
+ curl_easy_setopt(http->curl, CURLOPT_URL, url);
+ _curl_stream.buf = io_u->xfer_buf;
+ _curl_stream.max = io_u->xfer_buflen;
+ curl_easy_setopt(http->curl, CURLOPT_SEEKDATA, &_curl_stream);
+ curl_easy_setopt(http->curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t)io_u->xfer_buflen);
+
+ if (o->mode == FIO_HTTP_S3)
+ _add_aws_auth_header(http->curl, slist, o, io_u->ddir, object,
+ io_u->xfer_buf, io_u->xfer_buflen);
+ else if (o->mode == FIO_HTTP_SWIFT)
+ _add_swift_header(http->curl, slist, o, io_u->ddir, object,
+ io_u->xfer_buf, io_u->xfer_buflen);
+
+ if (io_u->ddir == DDIR_WRITE) {
+ curl_easy_setopt(http->curl, CURLOPT_READDATA, &_curl_stream);
+ curl_easy_setopt(http->curl, CURLOPT_WRITEDATA, NULL);
+ curl_easy_setopt(http->curl, CURLOPT_UPLOAD, 1L);
+ res = curl_easy_perform(http->curl);
+ if (res == CURLE_OK) {
+ curl_easy_getinfo(http->curl, CURLINFO_RESPONSE_CODE, &status);
+ if (status == 100 || (status >= 200 && status <= 204))
+ goto out;
+ log_err("DDIR_WRITE failed with HTTP status code %ld\n", status);
+ goto err;
+ }
+ } else if (io_u->ddir == DDIR_READ) {
+ curl_easy_setopt(http->curl, CURLOPT_READDATA, NULL);
+ curl_easy_setopt(http->curl, CURLOPT_WRITEDATA, &_curl_stream);
+ curl_easy_setopt(http->curl, CURLOPT_HTTPGET, 1L);
+ res = curl_easy_perform(http->curl);
+ if (res == CURLE_OK) {
+ curl_easy_getinfo(http->curl, CURLINFO_RESPONSE_CODE, &status);
+ if (status == 200)
+ goto out;
+ else if (status == 404) {
+ /* Object doesn't exist. Pretend we read
+ * zeroes */
+ memset(io_u->xfer_buf, 0, io_u->xfer_buflen);
+ goto out;
+ }
+ log_err("DDIR_READ failed with HTTP status code %ld\n", status);
+ }
+ goto err;
+ } else if (io_u->ddir == DDIR_TRIM) {
+ curl_easy_setopt(http->curl, CURLOPT_HTTPGET, 1L);
+ curl_easy_setopt(http->curl, CURLOPT_CUSTOMREQUEST, "DELETE");
+ curl_easy_setopt(http->curl, CURLOPT_INFILESIZE_LARGE, 0);
+ curl_easy_setopt(http->curl, CURLOPT_READDATA, NULL);
+ curl_easy_setopt(http->curl, CURLOPT_WRITEDATA, NULL);
+ res = curl_easy_perform(http->curl);
+ if (res == CURLE_OK) {
+ curl_easy_getinfo(http->curl, CURLINFO_RESPONSE_CODE, &status);
+ if (status == 200 || status == 202 || status == 204 || status == 404)
+ goto out;
+ log_err("DDIR_TRIM failed with HTTP status code %ld\n", status);
+ }
+ goto err;
+ }
+
+ log_err("WARNING: Only DDIR_READ/DDIR_WRITE/DDIR_TRIM are supported!\n");
+
+err:
+ io_u->error = r;
+ td_verror(td, io_u->error, "transfer");
+out:
+ curl_slist_free_all(slist);
+ return FIO_Q_COMPLETED;
+}
+
+static struct io_u *fio_http_event(struct thread_data *td, int event)
+{
+ /* sync IO engine - never any outstanding events */
+ return NULL;
+}
+
+int fio_http_getevents(struct thread_data *td, unsigned int min,
+ unsigned int max, const struct timespec *t)
+{
+ /* sync IO engine - never any outstanding events */
+ return 0;
+}
+
+static int fio_http_setup(struct thread_data *td)
+{
+ struct http_data *http = NULL;
+ struct http_options *o = td->eo;
+
+ /* allocate engine specific structure to deal with libhttp. */
+ http = calloc(1, sizeof(*http));
+ if (!http) {
+ log_err("calloc failed.\n");
+ goto cleanup;
+ }
+
+ http->curl = curl_easy_init();
+ if (o->verbose)
+ curl_easy_setopt(http->curl, CURLOPT_VERBOSE, 1L);
+ if (o->verbose > 1)
+ curl_easy_setopt(http->curl, CURLOPT_DEBUGFUNCTION, &_curl_trace);
+ curl_easy_setopt(http->curl, CURLOPT_NOPROGRESS, 1L);
+ curl_easy_setopt(http->curl, CURLOPT_FOLLOWLOCATION, 1L);
+ curl_easy_setopt(http->curl, CURLOPT_PROTOCOLS, CURLPROTO_HTTP|CURLPROTO_HTTPS);
+ if (o->https == FIO_HTTPS_INSECURE) {
+ curl_easy_setopt(http->curl, CURLOPT_SSL_VERIFYPEER, 0L);
+ curl_easy_setopt(http->curl, CURLOPT_SSL_VERIFYHOST, 0L);
+ }
+ curl_easy_setopt(http->curl, CURLOPT_READFUNCTION, _http_read);
+ curl_easy_setopt(http->curl, CURLOPT_WRITEFUNCTION, _http_write);
+ curl_easy_setopt(http->curl, CURLOPT_SEEKFUNCTION, _http_seek);
+ if (o->user && o->pass) {
+ curl_easy_setopt(http->curl, CURLOPT_USERNAME, o->user);
+ curl_easy_setopt(http->curl, CURLOPT_PASSWORD, o->pass);
+ curl_easy_setopt(http->curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
+ }
+
+ td->io_ops_data = http;
+
+ /* Force single process mode. */
+ td->o.use_thread = 1;
+
+ return 0;
+cleanup:
+ fio_http_cleanup(td);
+ return 1;
+}
+
+static int fio_http_open(struct thread_data *td, struct fio_file *f)
+{
+ return 0;
+}
+static int fio_http_invalidate(struct thread_data *td, struct fio_file *f)
+{
+ return 0;
+}
+
+static struct ioengine_ops ioengine = {
+ .name = "http",
+ .version = FIO_IOOPS_VERSION,
+ .flags = FIO_DISKLESSIO,
+ .setup = fio_http_setup,
+ .queue = fio_http_queue,
+ .getevents = fio_http_getevents,
+ .event = fio_http_event,
+ .cleanup = fio_http_cleanup,
+ .open_file = fio_http_open,
+ .invalidate = fio_http_invalidate,
+ .options = options,
+ .option_struct_size = sizeof(struct http_options),
+};
+
+static void fio_init fio_http_register(void)
+{
+ register_ioengine(&ioengine);
+}
+
+static void fio_exit fio_http_unregister(void)
+{
+ unregister_ioengine(&ioengine);
+}
/*
* It fits within existing mapping, use it
*/
- dprint(FD_IO," io_u->offset %lld : fdd->libpmem_off %ld : "
- "io_u->buflen %ld : fdd->libpmem_sz %ld\n",
- io_u->offset, fdd->libpmem_off,
- io_u->buflen, fdd->libpmem_sz);
+ dprint(FD_IO," io_u->offset %llu : fdd->libpmem_off %llu : "
+ "io_u->buflen %llu : fdd->libpmem_sz %llu\n",
+ io_u->offset, (unsigned long long) fdd->libpmem_off,
+ io_u->buflen, (unsigned long long) fdd->libpmem_sz);
if (io_u->offset >= fdd->libpmem_off &&
(io_u->offset + io_u->buflen <=
return true;
}
+static int gen_eta_str(struct jobs_eta *je, char *p, size_t left,
+ char **rate_str, char **iops_str)
+{
+ bool has_r = je->rate[DDIR_READ] || je->iops[DDIR_READ];
+ bool has_w = je->rate[DDIR_WRITE] || je->iops[DDIR_WRITE];
+ bool has_t = je->rate[DDIR_TRIM] || je->iops[DDIR_TRIM];
+ int l = 0;
+
+ if (!has_r && !has_w && !has_t)
+ return 0;
+
+ if (has_r) {
+ l += snprintf(p + l, left - l, "[r=%s", rate_str[DDIR_READ]);
+ if (!has_w)
+ l += snprintf(p + l, left - l, "]");
+ }
+ if (has_w) {
+ if (has_r)
+ l += snprintf(p + l, left - l, ",");
+ else
+ l += snprintf(p + l, left - l, "[");
+ l += snprintf(p + l, left - l, "w=%s", rate_str[DDIR_WRITE]);
+ if (!has_t)
+ l += snprintf(p + l, left - l, "]");
+ }
+ if (has_t) {
+ if (has_r || has_w)
+ l += snprintf(p + l, left - l, ",");
+ else if (!has_r && !has_w)
+ l += snprintf(p + l, left - l, "[");
+ l += snprintf(p + l, left - l, "t=%s]", rate_str[DDIR_TRIM]);
+ }
+ if (has_r) {
+ l += snprintf(p + l, left - l, "[r=%s", iops_str[DDIR_READ]);
+ if (!has_w)
+ l += snprintf(p + l, left - l, " IOPS]");
+ }
+ if (has_w) {
+ if (has_r)
+ l += snprintf(p + l, left - l, ",");
+ else
+ l += snprintf(p + l, left - l, "[");
+ l += snprintf(p + l, left - l, "w=%s", iops_str[DDIR_WRITE]);
+ if (!has_t)
+ l += snprintf(p + l, left - l, " IOPS]");
+ }
+ if (has_t) {
+ if (has_r || has_w)
+ l += snprintf(p + l, left - l, ",");
+ else if (!has_r && !has_w)
+ l += snprintf(p + l, left - l, "[");
+ l += snprintf(p + l, left - l, "t=%s IOPS]", iops_str[DDIR_TRIM]);
+ }
+
+ return l;
+}
+
void display_thread_status(struct jobs_eta *je)
{
static struct timespec disp_eta_new_line;
}
left = sizeof(output) - (p - output) - 1;
+ l = snprintf(p, left, ": [%s][%s]", je->run_str, perc_str);
+ l += gen_eta_str(je, p + l, left - l, rate_str, iops_str);
+ l += snprintf(p + l, left - l, "[eta %s]", eta_str);
- if (je->rate[DDIR_TRIM] || je->iops[DDIR_TRIM])
- l = snprintf(p, left,
- ": [%s][%s][r=%s,w=%s,t=%s][r=%s,w=%s,t=%s IOPS][eta %s]",
- je->run_str, perc_str, rate_str[DDIR_READ],
- rate_str[DDIR_WRITE], rate_str[DDIR_TRIM],
- iops_str[DDIR_READ], iops_str[DDIR_WRITE],
- iops_str[DDIR_TRIM], eta_str);
- else
- l = snprintf(p, left,
- ": [%s][%s][r=%s,w=%s][r=%s,w=%s IOPS][eta %s]",
- je->run_str, perc_str,
- rate_str[DDIR_READ], rate_str[DDIR_WRITE],
- iops_str[DDIR_READ], iops_str[DDIR_WRITE],
- eta_str);
/* If truncation occurred adjust l so p is on the null */
if (l >= left)
l = left - 1;
--- /dev/null
+# Example test for the HTTP engine's S3 support against Amazon AWS.
+# Obviously, you have to adjust the S3 credentials; for this example,
+# they're passed in via the environment.
+#
+
+[global]
+ioengine=http
+name=test
+direct=1
+filename=/larsmb-fio-test/object
+http_verbose=0
+https=on
+http_mode=s3
+http_s3_key=${S3_KEY}
+http_s3_keyid=${S3_ID}
+http_host=s3.eu-central-1.amazonaws.com
+http_s3_region=eu-central-1
+group_reporting
+
+# With verify, this both writes and reads the object
+[create]
+rw=write
+bs=4k
+size=64k
+io_size=4k
+verify=sha256
+
+[trim]
+stonewall
+rw=trim
+bs=4k
+size=64k
+io_size=4k
+
--- /dev/null
+[global]
+ioengine=http
+rw=randwrite
+name=test
+direct=1
+http_verbose=0
+http_mode=swift
+https=on
+# This is the hostname and port portion of the public access link for
+# the container:
+http_host=swift.srv.openstack.local:8081
+filename_format=/swift/v1/fio-test/bucket.$jobnum
+group_reporting
+bs=64k
+size=1M
+# Currently, fio cannot yet generate the Swift Auth-Token itself.
+# You need to set this prior to running fio via
+# eval $(openstack token issue -f shell --prefix SWIFT_) ; export SWIFT_id
+http_swift_auth_token=${SWIFT_id}
+
+[create]
+numjobs=1
+rw=randwrite
+io_size=256k
+verify=sha256
+
+# This will delete all created objects again
+[trim]
+stonewall
+numjobs=1
+rw=trim
+io_size=64k
--- /dev/null
+[global]
+ioengine=http
+rw=randwrite
+name=test
+direct=1
+http_verbose=0
+http_mode=webdav
+https=off
+http_host=localhost
+filename_format=/dav/bucket.$jobnum
+group_reporting
+bs=64k
+size=1M
+
+[create]
+numjobs=16
+rw=randwrite
+io_size=10M
+verify=sha256
+
+# This will delete all created objects again
+[trim]
+stonewall
+numjobs=16
+rw=trim
+io_size=1M
if (--f->references)
return 0;
+ disk_util_dec(f->du);
+
+ if (td->o.file_lock_mode != FILE_LOCK_NONE)
+ unlock_file_all(td, f);
+
if (should_fsync(td) && td->o.fsync_on_close) {
f_ret = fsync(f->fd);
if (f_ret < 0)
ret = f_ret;
td->nr_open_files--;
+ fio_file_clear_closing(f);
fio_file_clear_open(f);
assert(f->fd == -1);
return ret;
Set this \fIcommand\fR as remote trigger.
.TP
.BI \-\-aux\-path \fR=\fPpath
-Use this \fIpath\fR for fio state generated files.
+Use the directory specified by \fIpath\fP for generated state files instead
+of the current working directory.
.SH "JOB FILE FORMAT"
Any parameters following the options will be assumed to be job files, unless
they match a job file parameter. Multiple job files can be listed and each job
assigned equally distributed to job clones created by \fBnumjobs\fR as
long as they are using generated filenames. If specific \fBfilename\fR(s) are
set fio will use the first listed directory, and thereby matching the
-\fBfilename\fR semantic which generates a file each clone if not specified, but
-let all clones use the same if set.
+\fBfilename\fR semantic (which generates a file for each clone if not
+specified, but lets all clones use the same file if set).
.RS
.P
-See the \fBfilename\fR option for information on how to escape ':' and '\'
+See the \fBfilename\fR option for information on how to escape ':' and '\\'
characters within the directory path itself.
+.P
+Note: To control the directory fio will use for internal state files
+use \fB\-\-aux\-path\fR.
.RE
.TP
.BI filename \fR=\fPstr
explicit size is specified by \fBfilesize\fR.
.RS
.P
-Each colon and backslash in the wanted path must be escaped with a '\'
+Each colon and backslash in the wanted path must be escaped with a '\\'
character. For instance, if the path is `/dev/dsk/foo@3,0:c' then you
would use `filename=/dev/dsk/foo@3,0\\:c' and if the path is
-`F:\\\\filename' then you would use `filename=F\\:\\\\filename'.
+`F:\\filename' then you would use `filename=F\\:\\\\filename'.
.P
-On Windows, disk devices are accessed as `\\\\\\\\.\\\\PhysicalDrive0' for
-the first device, `\\\\\\\\.\\\\PhysicalDrive1' for the second etc.
+On Windows, disk devices are accessed as `\\\\.\\PhysicalDrive0' for
+the first device, `\\\\.\\PhysicalDrive1' for the second etc.
Note: Windows and FreeBSD prevent write access to areas
of the disk containing in\-use data (e.g. filesystems).
.P
.BI unlink_each_loop \fR=\fPbool
Unlink job files after each iteration or loop. Default: false.
.TP
-.BI zonesize \fR=\fPint
-Divide a file into zones of the specified size. See \fBzoneskip\fR.
+Fio supports strided data access. After having read \fBzonesize\fR bytes from an area that is \fBzonerange\fR bytes big, \fBzoneskip\fR bytes are skipped.
.TP
.BI zonerange \fR=\fPint
-Give size of an I/O zone. See \fBzoneskip\fR.
+Size of a single zone in which I/O occurs.
+.TP
+.BI zonesize \fR=\fPint
+Number of bytes to transfer before skipping \fBzoneskip\fR bytes. If this
+parameter is smaller than \fBzonerange\fR then only a fraction of each zone
+with \fBzonerange\fR bytes will be accessed. If this parameter is larger than
+\fBzonerange\fR then each zone will be accessed multiple times before skipping
+to the next zone.
.TP
.BI zoneskip \fR=\fPint
-Skip the specified number of bytes when \fBzonesize\fR data has been
-read. The two zone options can be used to only do I/O on zones of a file.
+Skip the specified number of bytes after \fBzonesize\fR bytes of data have been
+transferred.
+
.SS "I/O type"
.TP
.BI direct \fR=\fPbool
(RBD) via librbd without the need to use the kernel rbd driver. This
ioengine defines engine specific options.
.TP
+.B http
+I/O engine supporting GET/PUT requests over HTTP(S) with libcurl to
+a WebDAV or S3 endpoint. This ioengine defines engine specific options.
+
+This engine only supports direct IO of iodepth=1; you need to scale this
+via numjobs. blocksize defines the size of the objects to be created.
+
+TRIM is translated to object deletion.
+.TP
.B gfapi
Using GlusterFS libgfapi sync interface to direct access to
GlusterFS volumes without having to go through FUSE. This ioengine
Poll store instead of waiting for completion. Usually this provides better
throughput at cost of higher(up to 100%) CPU utilization.
.TP
+.BI (http)http_host \fR=\fPstr
+Hostname to connect to. For S3, this could be the bucket name. Default
+is \fBlocalhost\fR
+.TP
+.BI (http)http_user \fR=\fPstr
+Username for HTTP authentication.
+.TP
+.BI (http)http_pass \fR=\fPstr
+Password for HTTP authentication.
+.TP
+.BI (http)https \fR=\fPstr
+Whether to use HTTPS instead of plain HTTP. \fRon\fP enables HTTPS;
+\fRinsecure\fP will enable HTTPS, but disable SSL peer verification (use
+with caution!). Default is \fBoff\fR.
+.TP
+.BI (http)http_mode \fR=\fPstr
+Which HTTP access mode to use: webdav, swift, or s3. Default is
+\fBwebdav\fR.
+.TP
+.BI (http)http_s3_region \fR=\fPstr
+The S3 region/zone to include in the request. Default is \fBus-east-1\fR.
+.TP
+.BI (http)http_s3_key \fR=\fPstr
+The S3 secret key.
+.TP
+.BI (http)http_s3_keyid \fR=\fPstr
+The S3 key/access id.
+.TP
+.BI (http)http_swift_auth_token \fR=\fPstr
+The Swift auth token. See the example configuration file on how to
+retrieve this.
+.TP
+.BI (http)http_verbose \fR=\fPint
+Enable verbose requests from libcurl. Useful for debugging. 1 turns on
+verbose logging from libcurl, 2 additionally enables HTTP IO tracing.
+Default is \fB0\fR
+.TP
.BI (mtd)skip_bad \fR=\fPbool
Skip operations against known bad blocks.
.TP
.TP
.BI write_iops_log \fR=\fPstr
Same as \fBwrite_bw_log\fR, but writes an IOPS file (e.g.
-`name_iops.x.log') instead. See \fBwrite_bw_log\fR for
-details about the filename format and the \fBLOG FILE FORMATS\fR section for how data
-is structured within the file.
+`name_iops.x.log`) instead. Because fio defaults to individual
+I/O logging, the value entry in the IOPS log will be 1 unless windowed
+logging (see \fBlog_avg_msec\fR) has been enabled. See
+\fBwrite_bw_log\fR for details about the filename format and \fBLOG
+FILE FORMATS\fR for how data is structured within the file.
.TP
.BI log_avg_msec \fR=\fPint
By default, fio will log an entry in the iops, latency, or bw log for every
I/O is a TRIM
.RE
.P
-The entry's `block size' is always in bytes. The `offset' is the offset, in bytes,
-from the start of the file, for that particular I/O. The logging of the offset can be
+The entry's `block size' is always in bytes. The `offset' is the position in bytes
+from the start of the file for that particular I/O. The logging of the offset can be
toggled with \fBlog_offset\fR.
.P
-Fio defaults to logging every individual I/O. When IOPS are logged for individual
-I/Os the `value' entry will always be 1. If windowed logging is enabled through
-\fBlog_avg_msec\fR, fio logs the average values over the specified period of time.
-If windowed logging is enabled and \fBlog_max_value\fR is set, then fio logs
-maximum values in that window instead of averages. Since `data direction', `block size'
-and `offset' are per\-I/O values, if windowed logging is enabled they
-aren't applicable and will be 0.
+Fio defaults to logging every individual I/O but when windowed logging is set
+through \fBlog_avg_msec\fR, either the average (by default) or the maximum
+(\fBlog_max_value\fR is set) `value' seen over the specified period of time
+is recorded. Each `data direction' seen within the window period will aggregate
+its values in a separate row. Further, when using windowed logging the `block
+size' and `offset' entries will always contain 0.
.SH CLIENT / SERVER
Normally fio is invoked as a stand\-alone application on the machine where the
I/O workload should be generated. However, the backend and frontend of fio can
GtkTreeIter iter;
struct tm *tm;
time_t sec;
- char tmp[64], timebuf[80];
+ char tmp[64], timebuf[96];
sec = p->log_sec;
tm = localtime(&sec);
int td_io_open_file(struct thread_data *td, struct fio_file *f)
{
+ if (fio_file_closing(f)) {
+ /*
+ * Open translates to undo closing.
+ */
+ fio_file_clear_closing(f);
+ get_file(f);
+ return 0;
+ }
assert(!fio_file_open(f));
assert(f->fd == -1);
assert(td->io_ops->open_file);
*/
fio_file_set_closing(f);
- disk_util_dec(f->du);
-
- if (td->o.file_lock_mode != FILE_LOCK_NONE)
- unlock_file_all(td, f);
-
return put_file(td, f);
}
char *rfname, *fname, *act;
char *str, *p;
enum fio_ddir rw;
+ bool realloc = false;
int64_t items_to_fetch = 0;
if (td->o.read_iolog_chunked) {
ipo_bytes_align(td->o.replay_align, ipo);
ipo->len = bytes;
- if (rw != DDIR_INVAL && bytes > td->o.max_bs[rw])
+ if (rw != DDIR_INVAL && bytes > td->o.max_bs[rw]) {
+ realloc = true;
td->o.max_bs[rw] = bytes;
+ }
ipo->fileno = fileno;
ipo->file_action = file_action;
td->o.size += bytes;
return false;
}
td->o.td_ddir = TD_DDIR_RW;
+ if (realloc && td->orig_buffer)
+ {
+ io_u_quiesce(td);
+ free_io_mem(td);
+ init_io_u_buffers(td);
+ }
return true;
}
extern int iolog_compress_init(struct thread_data *, struct sk_out *);
extern void iolog_compress_exit(struct thread_data *);
extern size_t log_chunk_sizes(struct io_log *);
+extern int init_io_u_buffers(struct thread_data *);
#ifdef CONFIG_ZLIB
extern int iolog_file_inflate(const char *);
return false;
}
-static uint64_t axmap_find_first_free(struct axmap *axmap, unsigned int level,
- uint64_t index)
+/*
+ * Find the first free bit that is at least as large as bit_nr. Return
+ * -1 if no free bit is found before the end of the map.
+ */
+static uint64_t axmap_find_first_free(struct axmap *axmap, uint64_t bit_nr)
{
- uint64_t ret = -1ULL;
- unsigned long j;
int i;
+ unsigned long temp;
+ unsigned int bit;
+ uint64_t offset, base_index, index;
+ struct axmap_level *al;
- /*
- * Start at the bottom, then converge towards first free bit at the top
- */
- for (i = level; i >= 0; i--) {
- struct axmap_level *al = &axmap->levels[i];
-
- if (index >= al->map_size)
- goto err;
-
- for (j = index; j < al->map_size; j++) {
- if (al->map[j] == -1UL)
- continue;
+ index = 0;
+ for (i = axmap->nr_levels - 1; i >= 0; i--) {
+ al = &axmap->levels[i];
- /*
- * First free bit here is our index into the first
- * free bit at the next higher level
- */
- ret = index = (j << UNIT_SHIFT) + ffz(al->map[j]);
- break;
+ /* Shift previously calculated index for next level */
+ index <<= UNIT_SHIFT;
+
+ /*
+ * Start from an index that's at least as large as the
+ * originally passed in bit number.
+ */
+ base_index = bit_nr >> (UNIT_SHIFT * i);
+ if (index < base_index)
+ index = base_index;
+
+ /* Get the offset and bit for this level */
+ offset = index >> UNIT_SHIFT;
+ bit = index & BLOCKS_PER_UNIT_MASK;
+
+ /*
+ * If the previous level had unused bits in its last
+ * word, the offset could be bigger than the map at
+ * this level. That means no free bits exist before the
+ * end of the map, so return -1.
+ */
+ if (offset >= al->map_size)
+ return -1ULL;
+
+ /* Check the first word starting with the specific bit */
+ temp = ~bit_masks[bit] & ~al->map[offset];
+ if (temp)
+ goto found;
+
+ /*
+ * No free bit in the first word, so iterate
+ * looking for a word with one or more free bits.
+ */
+ for (offset++; offset < al->map_size; offset++) {
+ temp = ~al->map[offset];
+ if (temp)
+ goto found;
}
- }
-
- if (ret < axmap->nr_bits)
- return ret;
-err:
- return (uint64_t) -1ULL;
-}
+ /* Did not find a free bit */
+ return -1ULL;
-static uint64_t axmap_first_free(struct axmap *axmap)
-{
- if (!firstfree_valid(axmap))
- axmap->first_free = axmap_find_first_free(axmap, axmap->nr_levels - 1, 0);
-
- return axmap->first_free;
-}
-
-struct axmap_next_free_data {
- unsigned int level;
- unsigned long offset;
- uint64_t bit;
-};
-
-static bool axmap_next_free_fn(struct axmap_level *al, unsigned long offset,
- unsigned int bit, void *__data)
-{
- struct axmap_next_free_data *data = __data;
- uint64_t mask = ~bit_masks[(data->bit + 1) & BLOCKS_PER_UNIT_MASK];
-
- if (!(mask & ~al->map[offset]))
- return false;
-
- if (al->map[offset] != -1UL) {
- data->level = al->level;
- data->offset = offset;
- return true;
+found:
+ /* Compute the index of the free bit just found */
+ index = (offset << UNIT_SHIFT) + ffz(~temp);
}
- data->bit = (data->bit + BLOCKS_PER_UNIT - 1) / BLOCKS_PER_UNIT;
- return false;
+ /* If found an unused bit in the last word of level 0, return -1 */
+ if (index >= axmap->nr_bits)
+ return -1ULL;
+
+ return index;
}
/*
* 'bit_nr' is already set. Find the next free bit after this one.
+ * Return -1 if no free bits found.
*/
uint64_t axmap_next_free(struct axmap *axmap, uint64_t bit_nr)
{
- struct axmap_next_free_data data = { .level = -1U, .bit = bit_nr, };
uint64_t ret;
+ uint64_t next_bit = bit_nr + 1;
+ unsigned long temp;
+ uint64_t offset;
+ unsigned int bit;
- if (firstfree_valid(axmap) && bit_nr < axmap->first_free)
- return axmap->first_free;
+ if (bit_nr >= axmap->nr_bits)
+ return -1ULL;
- if (!axmap_handler(axmap, bit_nr, axmap_next_free_fn, &data))
- return axmap_first_free(axmap);
+ /* If at the end of the map, wrap-around */
+ if (next_bit == axmap->nr_bits)
+ next_bit = 0;
- assert(data.level != -1U);
+ offset = next_bit >> UNIT_SHIFT;
+ bit = next_bit & BLOCKS_PER_UNIT_MASK;
/*
- * In the rare case that the map is unaligned, we might end up
- * finding an offset that's beyond the valid end. For that case,
- * find the first free one, the map is practically full.
+ * As an optimization, do a quick check for a free bit
+ * in the current word at level 0. If not found, do
+ * a topdown search.
*/
- ret = axmap_find_first_free(axmap, data.level, data.offset);
- if (ret != -1ULL)
- return ret;
+ temp = ~bit_masks[bit] & ~axmap->levels[0].map[offset];
+ if (temp) {
+ ret = (offset << UNIT_SHIFT) + ffz(~temp);
- return axmap_first_free(axmap);
+ /* Might have found an unused bit at level 0 */
+ if (ret >= axmap->nr_bits)
+ ret = -1ULL;
+ } else
+ ret = axmap_find_first_free(axmap, next_bit);
+
+ /*
+ * If there are no free bits starting at next_bit and going
+ * to the end of the map, wrap around by searching again
+ * starting at bit 0.
+ */
+ if (ret == -1ULL && next_bit != 0)
+ ret = axmap_find_first_free(axmap, 0);
+ return ret;
}
#include <stdio.h>
#include <stdarg.h>
+#include <unistd.h>
#include "lib/output_buffer.h"
__FIO_OPT_G_ACT,
__FIO_OPT_G_LATPROF,
__FIO_OPT_G_RBD,
+ __FIO_OPT_G_HTTP,
__FIO_OPT_G_GFAPI,
__FIO_OPT_G_MTD,
__FIO_OPT_G_HDFS,
FIO_OPT_G_ACT = (1ULL << __FIO_OPT_G_ACT),
FIO_OPT_G_LATPROF = (1ULL << __FIO_OPT_G_LATPROF),
FIO_OPT_G_RBD = (1ULL << __FIO_OPT_G_RBD),
+ FIO_OPT_G_HTTP = (1ULL << __FIO_OPT_G_HTTP),
FIO_OPT_G_GFAPI = (1ULL << __FIO_OPT_G_GFAPI),
FIO_OPT_G_MTD = (1ULL << __FIO_OPT_G_MTD),
FIO_OPT_G_HDFS = (1ULL << __FIO_OPT_G_HDFS),
return 0;
}
-static void __td_zone_gen_index(struct thread_data *td, enum fio_ddir ddir)
-{
- unsigned int i, j, sprev, aprev;
- uint64_t sprev_sz;
-
- td->zone_state_index[ddir] = malloc(sizeof(struct zone_split_index) * 100);
-
- sprev_sz = sprev = aprev = 0;
- for (i = 0; i < td->o.zone_split_nr[ddir]; i++) {
- struct zone_split *zsp = &td->o.zone_split[ddir][i];
-
- for (j = aprev; j < aprev + zsp->access_perc; j++) {
- struct zone_split_index *zsi = &td->zone_state_index[ddir][j];
-
- zsi->size_perc = sprev + zsp->size_perc;
- zsi->size_perc_prev = sprev;
-
- zsi->size = sprev_sz + zsp->size;
- zsi->size_prev = sprev_sz;
- }
-
- aprev += zsp->access_perc;
- sprev += zsp->size_perc;
- sprev_sz += zsp->size;
- }
-}
-
-/*
- * Generate state table for indexes, so we don't have to do it inline from
- * the hot IO path
- */
-static void td_zone_gen_index(struct thread_data *td)
-{
- int i;
-
- td->zone_state_index = malloc(DDIR_RWDIR_CNT *
- sizeof(struct zone_split_index *));
-
- for (i = 0; i < DDIR_RWDIR_CNT; i++)
- __td_zone_gen_index(td, i);
-}
-
static int parse_zoned_distribution(struct thread_data *td, const char *input,
bool absolute)
{
return ret;
}
- if (!ret)
- td_zone_gen_index(td);
- else {
+ if (ret) {
for (i = 0; i < DDIR_RWDIR_CNT; i++)
td->o.zone_split_nr[i] = 0;
}
{ .ival = "libpmem",
.help = "PMDK libpmem based IO engine",
},
+#endif
+#ifdef CONFIG_HTTP
+ { .ival = "http",
+ .help = "HTTP (WebDAV/S3) IO engine",
+ },
#endif
},
},
#ifndef CONFIG_HAVE_VASPRINTF
int vasprintf(char **strp, const char *fmt, va_list ap)
{
- va_list ap_copy;
- char *str;
- int len;
+ va_list ap_copy;
+ char *str;
+ int len;
#ifdef va_copy
- va_copy(ap_copy, ap);
+ va_copy(ap_copy, ap);
#else
- __va_copy(ap_copy, ap);
+ __va_copy(ap_copy, ap);
#endif
- len = vsnprintf(NULL, 0, fmt, ap_copy);
- va_end(ap_copy);
+ len = vsnprintf(NULL, 0, fmt, ap_copy);
+ va_end(ap_copy);
- if (len < 0)
- return len;
+ if (len < 0)
+ return len;
- len++;
- str = malloc(len);
- *strp = str;
- return str ? vsnprintf(str, len, fmt, ap) : -1;
+ len++;
+ str = malloc(len);
+ *strp = str;
+ return str ? vsnprintf(str, len, fmt, ap) : -1;
}
#endif
#ifndef CONFIG_HAVE_ASPRINTF
int asprintf(char **strp, const char *fmt, ...)
{
- va_list arg;
- int done;
+ va_list arg;
+ int done;
- va_start(arg, fmt);
- done = vasprintf(strp, fmt, arg);
- va_end(arg);
+ va_start(arg, fmt);
+ done = vasprintf(strp, fmt, arg);
+ va_end(arg);
- return done;
+ return done;
}
#endif
{
struct fio_lfsr lfsr;
struct axmap *map;
- size_t osize;
- uint64_t ff;
int err;
printf("Using %llu entries...", (unsigned long long) size);
lfsr_init(&lfsr, size, seed, seed & 0xF);
map = axmap_new(size);
- osize = size;
err = 0;
while (size--) {
if (err)
return err;
- ff = axmap_next_free(map, osize);
- if (ff != (uint64_t) -1ULL) {
- printf("axmap_next_free broken: got %llu\n", (unsigned long long) ff);
+ printf("pass!\n");
+ axmap_free(map);
+ return 0;
+}
+
+static int check_next_free(struct axmap *map, uint64_t start, uint64_t expected)
+{
+
+ uint64_t ff;
+
+ ff = axmap_next_free(map, start);
+ if (ff != expected) {
+ printf("axmap_next_free broken: Expected %llu, got %llu\n",
+ (unsigned long long)expected, (unsigned long long) ff);
return 1;
}
+ return 0;
+}
+
+static int test_next_free(size_t size, int seed)
+{
+ struct fio_lfsr lfsr;
+ struct axmap *map;
+ size_t osize;
+ uint64_t ff, lastfree;
+ int err, i;
+
+ printf("Test next_free %llu entries...", (unsigned long long) size);
+ fflush(stdout);
+
+ map = axmap_new(size);
+ err = 0;
+
+
+ /* Empty map. Next free after 0 should be 1. */
+ if (check_next_free(map, 0, 1))
+ err = 1;
+
+ /* Empty map. Next free after 63 should be 64. */
+ if (check_next_free(map, 63, 64))
+ err = 1;
+
+ /* Empty map. Next free after size - 2 should be size - 1 */
+ if (check_next_free(map, size - 2, size - 1))
+ err = 1;
+
+ /* Empty map. Next free after size - 1 should be 0 */
+ if (check_next_free(map, size - 1, 0))
+ err = 1;
+
+ /* Empty map. Next free after 63 should be 64. */
+ if (check_next_free(map, 63, 64))
+ err = 1;
+
+
+ /* Bit 63 set. Next free after 62 should be 64. */
+ axmap_set(map, 63);
+ if (check_next_free(map, 62, 64))
+ err = 1;
+
+ /* Last bit set. Next free after size - 2 should be 0. */
+ axmap_set(map, size - 1);
+ if (check_next_free(map, size - 2, 0))
+ err = 1;
+
+ /* Last bit set. Next free after size - 1 should be 0. */
+ if (check_next_free(map, size - 1, 0))
+ err = 1;
+
+ /* Last 64 bits set. Next free after size - 66 or size - 65 should be 0. */
+ for (i=size - 65; i < size; i++)
+ axmap_set(map, i);
+ if (check_next_free(map, size - 66, 0))
+ err = 1;
+ if (check_next_free(map, size - 65, 0))
+ err = 1;
+
+ /* Last 64 bits set. Next free after size - 67 should be size - 66. */
+ if (check_next_free(map, size - 67, size - 66))
+ err = 1;
+
+ axmap_free(map);
+
+ /* Start with a fresh map and mostly fill it up */
+ lfsr_init(&lfsr, size, seed, seed & 0xF);
+ map = axmap_new(size);
+ osize = size;
+
+ /* Leave 1 entry free */
+ size--;
+ while (size--) {
+ uint64_t val;
+
+ if (lfsr_next(&lfsr, &val)) {
+ printf("lfsr: short loop\n");
+ err = 1;
+ break;
+ }
+ if (axmap_isset(map, val)) {
+ printf("bit already set\n");
+ err = 1;
+ break;
+ }
+ axmap_set(map, val);
+ if (!axmap_isset(map, val)) {
+ printf("bit not set\n");
+ err = 1;
+ break;
+ }
+ }
+
+ /* Get last free bit */
+ lastfree = axmap_next_free(map, 0);
+ if (lastfree == -1ULL) {
+ printf("axmap_next_free broken: Couldn't find last free bit\n");
+ err = 1;
+ }
+
+ /* Start with last free bit and test wrap-around */
+ ff = axmap_next_free(map, lastfree);
+ if (ff != lastfree) {
+ printf("axmap_next_free broken: wrap-around test #1 failed\n");
+ err = 1;
+ }
+
+ /* Start with last bit and test wrap-around */
+ ff = axmap_next_free(map, osize - 1);
+ if (ff != lastfree) {
+ printf("axmap_next_free broken: wrap-around test #2 failed\n");
+ err = 1;
+ }
+
+ /* Set last free bit */
+ axmap_set(map, lastfree);
+ ff = axmap_next_free(map, 0);
+ if (ff != -1ULL) {
+ printf("axmap_next_free broken: Expected -1 from full map\n");
+ err = 1;
+ }
+
+ ff = axmap_next_free(map, osize);
+ if (ff != -1ULL) {
+ printf("axmap_next_free broken: Expected -1 from out of bounds request\n");
+ err = 1;
+ }
+
+ if (err)
+ return err;
printf("pass!\n");
axmap_free(map);
return 3;
if (test_overlap())
return 4;
+ if (test_next_free(size, seed))
+ return 5;
+
+ /* Test 3 levels, all full: 64*64*64 */
+ if (test_next_free(64*64*64, seed))
+ return 6;
+
+ /* Test 4 levels, with 2 inner levels not full */
+ if (test_next_free(((((64*64)-63)*64)-63)*64*12, seed))
+ return 7;
return 0;
}
[Unit]
-
-Description=flexible I/O tester server
+Description=Flexible I/O tester server
After=network.target
[Service]
-
Type=simple
-PIDFile=/run/fio.pid
ExecStart=/usr/bin/fio --server
+
+[Install]
+WantedBy=multi-user.target
--- /dev/null
+#include <stdlib.h>
+#include "fio.h"
+#include "zone-dist.h"
+
+static void __td_zone_gen_index(struct thread_data *td, enum fio_ddir ddir)
+{
+ unsigned int i, j, sprev, aprev;
+ uint64_t sprev_sz;
+
+ td->zone_state_index[ddir] = malloc(sizeof(struct zone_split_index) * 100);
+
+ sprev_sz = sprev = aprev = 0;
+ for (i = 0; i < td->o.zone_split_nr[ddir]; i++) {
+ struct zone_split *zsp = &td->o.zone_split[ddir][i];
+
+ for (j = aprev; j < aprev + zsp->access_perc; j++) {
+ struct zone_split_index *zsi = &td->zone_state_index[ddir][j];
+
+ zsi->size_perc = sprev + zsp->size_perc;
+ zsi->size_perc_prev = sprev;
+
+ zsi->size = sprev_sz + zsp->size;
+ zsi->size_prev = sprev_sz;
+ }
+
+ aprev += zsp->access_perc;
+ sprev += zsp->size_perc;
+ sprev_sz += zsp->size;
+ }
+}
+
+static bool has_zones(struct thread_data *td)
+{
+ int i, zones = 0;
+
+ for (i = 0; i < DDIR_RWDIR_CNT; i++)
+ zones += td->o.zone_split_nr[i];
+
+ return zones != 0;
+}
+
+/*
+ * Generate state table for indexes, so we don't have to do it inline from
+ * the hot IO path
+ */
+void td_zone_gen_index(struct thread_data *td)
+{
+ int i;
+
+ if (!has_zones(td))
+ return;
+
+ td->zone_state_index = malloc(DDIR_RWDIR_CNT *
+ sizeof(struct zone_split_index *));
+
+ for (i = 0; i < DDIR_RWDIR_CNT; i++)
+ __td_zone_gen_index(td, i);
+}
+
+void td_zone_free_index(struct thread_data *td)
+{
+ int i;
+
+ if (!td->zone_state_index)
+ return;
+
+ for (i = 0; i < DDIR_RWDIR_CNT; i++) {
+ free(td->zone_state_index[i]);
+ td->zone_state_index[i] = NULL;
+ }
+
+ free(td->zone_state_index);
+ td->zone_state_index = NULL;
+}
--- /dev/null
+#ifndef FIO_ZONE_DIST_H
+#define FIO_ZONE_DIST_H
+
+void td_zone_gen_index(struct thread_data *td);
+void td_zone_free_index(struct thread_data *td);
+
+#endif
projects / fio.git / commitdiff