summaryrefslogtreecommitdiff
path: root/Makefile
diff options
context:
space:
mode:
authorErwan Velu <e.velu@criteo.com>2021-05-12 09:54:11 +0200
committerErwan Velu <e.velu@criteo.com>2021-06-29 16:18:32 +0200
commitb50590bc0ac49efb6550c2eeb893c5e6e3224841 (patch)
tree7cf2d05a352914d7f57baaad952b7d7910f95e0c /Makefile
parentea51055cbb2fcbca3935e25c78e8b6d358ca2b3f (diff)
downloadfio-b50590bc0ac49efb6550c2eeb893c5e6e3224841.tar.gz
fio-b50590bc0ac49efb6550c2eeb893c5e6e3224841.tar.bz2
engines: Adding exec engine
When performing benchmarks with fio, some need to execute tasks in parallel to the job execution. A typical use-case would be observing performance/power metrics. Several implementations were possible : - Adding an exec_run in addition of the existing exec_{pre|post}run - Implementating performance/power metrics in fio - Adding an exec engine 1°) Adding an exec_run This was my first intention but quickly noticed that exec_{pre-post}run are executed for each 'numjob'. In the case of performance/power monitoring, this doesn't make sense to spawn an instance for each thread. 2°) Implementing performance/power metrics This is possible but would require lot of work to maintain this part of fio while 3rd party tools already take care of that perfectly. 3°) Adding an engine Adding an engine let users defining when and how many instances of the program they want. In the provided example, a single monitoring job is spawning at the same time as the worker thread which could be composed of several worker threads. A stonewall barrier is used to define which jobs must run together (monitoring / benchmark). The engine has two parameters : - program: name of the program to run - arguments: arguments to pass to the program - grace_time: duration between SIGTERM and SIGKILL - std_redirect: redirect std{err|out} to dedicated files Arguments can have special variables to be expanded before the execution: - %r will be replaced by the job duration in seconds - %n will be replaced by the job name During the program execution, the std{out|err} are redirected to files if std_redirect option is set (default). - stdout: <job_name>.stdout - stderr: <job_name>.stderr If the executed program has a nice stdout output, after the fio execution, the stdout file can be parsed by other tools like CI jobs or graphing tools. A sample job is provided here to show how this can be used. It runs twice the CPU engine with two different CPU modes (noop vs qsort). For each benchmark, the output of turbostat is saved for later analysis. After the fio run, it is possible to compare the impact of the two modes on the CPU frequency and power consumption. This can be easily extended to any other usage that needs to analysis the behavior of the host during some jobs. About the implementation, the exec engine forks : - the child doing an execvp() of the program. - the parent, fio, will monitor the time passed into the job Once the time is over, the program is SIGTERM followed by a SIGKILL to ensure it will not run _after_ the job is completed. This mechanism is required as : - not all programs can be controlled properly - that's last resort protection if the program gets crazy The delay is controlled by grace_time option, default is 1 sec. If the program can be limited in its duration, using the %r variable in the arguments can be used to request the program to stop _before_ the job finished like : program=/usr/bin/mytool.sh arguments=--duration %r Signed-off-by: Erwan Velu <e.velu@criteo.com>
Diffstat (limited to 'Makefile')
-rw-r--r--Makefile1
1 files changed, 1 insertions, 0 deletions
diff --git a/Makefile b/Makefile
index f57569d5..a5c5e67c 100644
--- a/Makefile
+++ b/Makefile
@@ -57,6 +57,7 @@ SOURCE := $(sort $(patsubst $(SRCDIR)/%,%,$(wildcard $(SRCDIR)/crc/*.c)) \
smalloc.c filehash.c profile.c debug.c engines/cpu.c \
engines/mmap.c engines/sync.c engines/null.c engines/net.c \
engines/ftruncate.c engines/filecreate.c engines/filestat.c engines/filedelete.c \
+ engines/exec.c \
server.c client.c iolog.c backend.c libfio.c flow.c cconv.c \
gettime-thread.c helpers.c json.c idletime.c td_error.c \
profiles/tiobench.c profiles/act.c io_u_queue.c filelock.c \