mmap: catch invalid msync() errors early