backend: don't allocate memory buffers for non-data workloads