test: improve evaluation of t0020.fio and t0021.fio

We previously used an arbirtrary test based on a count of consecutive
offsets to decide whether the offsets from t0020.fio and t0021.fio were
random or not. Replace this with a Wald-Wolfowitz runs test. This new
test calculates a statistic based on whether each offset is in the first
or second half of the LBA space. So it is not particularly powerful
although it has a more sound statistical justification than what we
originally had.

Recent changes to fio's default random number seeds produced false
positives with t0020.fio and prompted this change.

Consider calculating the test statistic directly instead of relying on
the statsmodel import in order to reduce the dependencies required.

References:
https://www.statology.org/runs-test-python/
https://www.geeksforgeeks.org/runs-test-of-randomness-in-python/
https://www.investopedia.com/terms/r/runs_test.asp
https://www.statsmodels.org/stable/generated/statsmodels.sandbox.stats.runs.runstest_1samp.html

Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

diff --git a/ci/actions-install.sh b/ci/actions-install.sh
index 5057fca39cc602b915b67e38d4d8312918bfe497..fb3bd141c8ae8e4540833f6ca6078709a6d3fb8c 100755 (executable)
--- a/ci/actions-install.sh
+++ b/ci/actions-install.sh
@@ -62,6 +62,7 @@ DPKGCFG
     pkgs+=(
         python3-scipy
        python3-sphinx
+       python3-statsmodels
     )
 
     echo "Updating APT..."
@@ -85,7 +86,7 @@ install_macos() {
     echo "Installing packages..."
     HOMEBREW_NO_AUTO_UPDATE=1 brew install cunit libnfs sphinx-doc
     brew link sphinx-doc --force
-    pip3 install scipy six 
+    pip3 install scipy six statsmodels
 }
 
 main() {

diff --git a/ci/appveyor-install.sh b/ci/appveyor-install.sh
index 3137f39ebe91424dc5cfbfc23d002796dc31cf1a..1e28c4540326b2b48dacac167bec7d2d9ba3dbb4 100755 (executable)
--- a/ci/appveyor-install.sh
+++ b/ci/appveyor-install.sh
@@ -37,7 +37,7 @@ case "${DISTRO}" in
         ;;
 esac
 
-python.exe -m pip install scipy six
+python.exe -m pip install scipy six statsmodels
 
 echo "Python3 path: $(type -p python3 2>&1)"
 echo "Python3 version: $(python3 -V 2>&1)"

diff --git a/t/run-fio-tests.py b/t/run-fio-tests.py
index 7e0df7ed2e9d53f954d5675d3223f363a9d705d1..4fe6fe46f6ec2429a1fd1cab169a7731e6131bd6 100755 (executable)
--- a/t/run-fio-tests.py
+++ b/t/run-fio-tests.py
@@ -53,6 +53,7 @@ import traceback
 import subprocess
 import multiprocessing
 from pathlib import Path
+from statsmodels.sandbox.stats.runs import runstest_1samp
 
 
 class FioTest():
@@ -598,24 +599,16 @@ class FioJobTest_t0020(FioJobTest):
 
         log_lines = file_data.split('\n')
 
-        seq_count = 0
-        offsets = set()
+        offsets = []
 
         prev = int(log_lines[0].split(',')[4])
         for line in log_lines[1:]:
-            offsets.add(prev/4096)
+            offsets.append(prev/4096)
             if len(line.strip()) == 0:
                 continue
             cur = int(line.split(',')[4])
-            if cur - prev == 4096:
-                seq_count += 1
             prev = cur
 
-        # 10 is an arbitrary threshold
-        if seq_count > 10:
-            self.passed = False
-            self.failure_reason = "too many ({0}) consecutive offsets".format(seq_count)
-
         if len(offsets) != 256:
             self.passed = False
             self.failure_reason += " number of offsets is {0} instead of 256".format(len(offsets))
@@ -625,6 +618,11 @@ class FioJobTest_t0020(FioJobTest):
                 self.passed = False
                 self.failure_reason += " missing offset {0}".format(i*4096)
 
+        (z, p) = runstest_1samp(list(offsets))
+        if p < 0.05:
+            self.passed = False
+            self.failure_reason += f" runs test failed with p = {p}"
+
 
 class FioJobTest_t0022(FioJobTest):
     """Test consists of fio test job t0022"""