cpus_allowed: use __NRPROCESSORS_CONF instead of __SC_NPROCESSORS_ONLN for non-sequential CPU ids When disabling SMT on some systems, the ID of some available CPU may be larger than the value of sysconf(_SC_NPROCESSORS_ONLN). Without this patch, fio complains that the expected CPU ID is invalid. Here's an example from my server: $ ./fio --cpus_allowed=14 --ioengine=libaio --direct=1 --name=test --numjobs=1 --blocksize=128k --iodepth=1 --rw=read --filename=/dev/nvme0n1 fio: CPU 14 too large (max=11) fio: failed parsing cpus_allowed=14 System information: $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0,2,4,6,8,10,12,14,16-19 Off-line CPU(s) list: 1,3,5,7,9,11,13,15 ... Model name: 12th Gen Intel(R) Core(TM) i7-12700 BIOS Model name: 12th Gen Intel(R) Core(TM) i7-12700 ... $ uname -a Linux localhost.localdomain 4.18.0-348.el8.x86_64 #1 SMP Tue Oct 19 15:14:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux $ cat /etc/redhat-release CentOS Linux release 8.5.2111 $ cat /proc/cmdline BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-348.el8.x86_64 root=/dev/mapper/cl-root ro nosmt isolcpus=0,2,4,6,8,10,12,14 crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet $ cat test.c #include <stdio.h> #include <unistd.h> int main(int argc, char *argv[]) { printf("_SC_NPROCESSORS_ONLN=%ld _SC_NPROCESSORS_CONF=%ld\n", sysconf(_SC_NPROCESSORS_ONLN), sysconf(_SC_NPROCESSORS_CONF)); } $ gcc test.c $ ./a.out _SC_NPROCESSORS_ONLN=12 _SC_NPROCESSORS_CONF=20 Signed-off-by: mayuanpeng <mayuanpeng@dapustor.com>
Add Windows support for --server. This required working around two calls to fork() The first fork() was in accept_loop() where it creates a new process to work on handle_connection(). This change set uses Window's CreateProcess() in place of fork(). In order to support this, it duplicates the socket connection via WSADuplicateSocket() and passes the resulting WSAPROTOCOL_INFO to the process via a named pipe. The pipe name is then passed to the process via a new command line argument(--server-internal). The second fork() was in handle_run_cmd() where it creates a new process to work on fio_backend_thread(). This change set runs fio_backend_thread() as a thread via pthread_create() instead of a forked process. There is also some supporting work in the monitoring of spawned processes/threads in fio_server_check_conns() and fio_server_check_jobs(). Signed-off-by: james rizzo <james.rizzo@broadcom.com>
engines: Adding exec engine When performing benchmarks with fio, some need to execute tasks in parallel to the job execution. A typical use-case would be observing performance/power metrics. Several implementations were possible : - Adding an exec_run in addition of the existing exec_{pre|post}run - Implementating performance/power metrics in fio - Adding an exec engine 1°) Adding an exec_run This was my first intention but quickly noticed that exec_{pre-post}run are executed for each 'numjob'. In the case of performance/power monitoring, this doesn't make sense to spawn an instance for each thread. 2°) Implementing performance/power metrics This is possible but would require lot of work to maintain this part of fio while 3rd party tools already take care of that perfectly. 3°) Adding an engine Adding an engine let users defining when and how many instances of the program they want. In the provided example, a single monitoring job is spawning at the same time as the worker thread which could be composed of several worker threads. A stonewall barrier is used to define which jobs must run together (monitoring / benchmark). The engine has two parameters : - program: name of the program to run - arguments: arguments to pass to the program - grace_time: duration between SIGTERM and SIGKILL - std_redirect: redirect std{err|out} to dedicated files Arguments can have special variables to be expanded before the execution: - %r will be replaced by the job duration in seconds - %n will be replaced by the job name During the program execution, the std{out|err} are redirected to files if std_redirect option is set (default). - stdout: <job_name>.stdout - stderr: <job_name>.stderr If the executed program has a nice stdout output, after the fio execution, the stdout file can be parsed by other tools like CI jobs or graphing tools. A sample job is provided here to show how this can be used. It runs twice the CPU engine with two different CPU modes (noop vs qsort). For each benchmark, the output of turbostat is saved for later analysis. After the fio run, it is possible to compare the impact of the two modes on the CPU frequency and power consumption. This can be easily extended to any other usage that needs to analysis the behavior of the host during some jobs. About the implementation, the exec engine forks : - the child doing an execvp() of the program. - the parent, fio, will monitor the time passed into the job Once the time is over, the program is SIGTERM followed by a SIGKILL to ensure it will not run _after_ the job is completed. This mechanism is required as : - not all programs can be controlled properly - that's last resort protection if the program gets crazy The delay is controlled by grace_time option, default is 1 sec. If the program can be limited in its duration, using the %r variable in the arguments can be used to request the program to stop _before_ the job finished like : program=/usr/bin/mytool.sh arguments=--duration %r Signed-off-by: Erwan Velu <e.velu@criteo.com>
windows: drop XP support Windows 2003 has been EOL since 2015 and Windows XP has been EOL since 2014. Technically fio hasn't been properly targetting XP anyway - (see 16d40a15783f9181d64082fc00a970dff485798f 'configure: be explicit about "XP" Windows API version'). - Strip out support for XP and change the documentation to no longer mention it - Stop targetting XP in the CI 32 bit build Next stop, Windows 7... Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
filesetup: fix win raw disk access and improve dir creation failure msg The commit df18600fd06258b96ae6f6b530ecdff541c2a82d ("filesetup: fix directory creation issues") broke Windows raw/physical disk access because Windows doesn't consider a path that only consists of a namespace (such as the device namespace "\\.\" - see https://docs.microsoft.com/en-gb/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#namespaces for information on Windows' namespaces) to exist as a directory(!). Workaround the issue for raw devices by explicitly considering the device namespace subcomponent to always be valid. Further, intermediate path components in UNC paths or paths starting with a namespace might also not "exist" so introduce backwards scanning for the longest pre-existing directory to sidestep this. The function doing this is made available for non-windows platforms so a similar code path is used everywhere. Tests done: Windows: > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename 'fio.tmp' > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\\?\C\:\Windows\Temp\fio\fio.tmp' > Clear-Disk 1 -RemoveData -Confirm:$false # Destroys partition data! > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\\.\PhysicalDrive1' > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\fio.tmp' > ./fio.exe --name=dtest --thread --size=16k --rw=write ` --filename '\\LOCALHOST\Users\User\fio\fio.tmp' macOS: $ rm -rf /tmp/fio $ ./fio --name=dtest --size=16k --filename /tmp/fio/fio.tmp \ --rw=write Finally, change the directory creation error message to give a human error message rather than just an errno. Fixes: https://github.com/axboe/fio/issues/916 Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
filesetup: fix directory creation issues - Fix double directory prefix concatenation that was happening after 95af8dd57a7a9a564f99492bb1e04bb6d13b95b1 ("filesetup: don't call create_work_dirs() for ioengine with FIO_DISKLESSIO") refactored some code - Stop always using '/' and switch to using FIO_OS_PATH_SEPARATOR when concatenating the directory value to the filename in set_name_idx() - Change the permissions directories are created with to have the execute bit so they can be entered(!) - Fix bug where filenames that were absolute unix paths would prevent the creation of intermediate directories - Remove the configure mkdir test and introduce fio_mkdir to hide mkdir platform differences - Make Windows mkdir emulation more robust against multiple slashes after the drive letter and when dealing dealing with with paths containing drive letters in general This commit makes the following work on Windows: ./fio --thread --name=doubleslashtest \ --filename='C\:\\Windows\Temp\fio.tmp' --size=4k --bs=4k --rw=read The next example now works when D:\1\ exists but D:\1\2\ doesn't: ./fio --thread --name=mkdirtest --directory='D\:\1' \ --filename='2\fio.tmp' --size=4k --bs=4k --rw=read The following now works on macOS/Linux when /tmp/1/ exists but /tmp/1/2/ doesn't: ./fio --name=mkdirtest --directory='/tmp/1' \ --filename='2/fio.tmp' --size=4k --bs=4k --rw=read Fixes https://github.com/axboe/fio/issues/784 , https://github.com/axboe/fio/issues/864 and https://github.com/axboe/fio/issues/870 Tested-by: Astolfo Rueda <astolfor@microsoft.com> Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
windows: target Windows 7 and add support for more than 64 CPUs Introduce support for targeting the build to either Windows XP or Windows 7 (the default). When targeting Windows 7 this allows us to use the process group APIs which are required are required to support more than 64 CPUs. These changes mean if you want a Windows binary that supports versions of Windows below Windows 7/Windows Server 2008 R2 you will need to explicitly use the --target-win-ver=xp configure option at build time. Such builds will lack features (such as the ability to access CPUs beyond those in fio's default process group) that are dependent on recent Windows APIs. Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
os-windows: fix cpumask operations Fix cpumask manipulation on (64-bit) Windows systems. cpus_allowed=nn values greater than 32 does not work, due to * the compiler not promoting expressions like "1 << cpu" to 64 bits * the clear function clearing the wrong bit (using "cpu - 1"), * the clear function using XOR to clear (which only works if the bit was previously set) * the check function returning a 64-bit value through a 32-bit return value Example problems (from extra debug prints): * setting CPU 32 really sets bit 0 (CPU 0) Set mask of 0000000000000000 to add 0000000000000001 (32) * setting CPU 63 really sets bit 31 (CPU 31) Set mask of 0000000000000000 to add 0000000080000000 (63) * clearing CPU 0 really clears bit 63 (CPU 63) Clear mask of 0000000055555555 to remove 8000000000000000 (0) * clearing CPU 2 really clears bit 2 (CPU 2) Clear mask of 0000000055555555 to remove 0000000000000002 (2) * checking claims CPU 32 is not in a mask of CPUs {32,34,36,38} for 0000005500000000 check if bit 32 is set ==> FALSE Tested with x86_64-w64-mingw32-gcc 6.4.0 from cygwin on a system with 64 CPU cores (all fitting in one Windows processor group). Signed-off-by: Jens Axboe <axboe@kernel.dk>
Drop struct thread_data dependency from os headers Since os/os.h is included by fio.h before other functions/structures are even defined, it's better for os/os*.h not to have dependency on those (in this case it's struct thread_data and duplicated td_fill_ rand_seeds() prototypes) unless really needed. os/os*.h are basically collections of small wrappers over syscalls/etc whose idea is similar but with difference interface among supported platforms, thus they normally don't need to have dependency on fio functions/structures, and in fact they're normally designed that way. This commit gets rid of struct thread_data argument from an inlined function init_random_state(), which was needed only to call another function td_fill_rand_seeds(td). This can simply be called from setup_random_seeds() after init_random_state() without making any functional difference. Also rename init_random_state() to init_random_seeds() as it only initializes random seeds now. With this commit, struct thread_data is completely gone from os/. struct fio_file is still there, but this is needed by Windows where the idea of fd differs from unix like in general. Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@fb.com>
Use ENOTSUP if OS doesn't support blkdev page cache invalidation The return value of blockdev_invalidate_cache() has been platform dependent (some pretend success, others return EINVAL) while Linux is the only one actually supports this via Linux specific ioctl. This commit makes all the rest return a consistent value ENOTSUP, so __file_invalidate_cache() prints a blkdev invalidation failure message that is consistent and makes sense. Signed-off-by: Tomohiro Kusumi <tkusumi@tuxera.com> Signed-off-by: Jens Axboe <axboe@fb.com>