Don't clear 'refill_buffers' unconditionally for pattern fill