Remove ->mutex after last use, not on job exit