net/mlx5: HWS, rework rehash loop

net/mlx5: HWS, rework rehash loop

Reworking the rehash loop - simplifying the code and making it less
error prone:
 - Instead of doing round-robin on all the queues with batch of rules in
   each cycle, just go over all the queues and move all the rules that
   belong to this queue.
 - If at some stage of moving the rule we get a failure (which should
   not happen), this can't be rolled back. So instead of aborting
   rehash and leaving the matcher in a broken state, allow the loop
   to continue: attempt to move the rest of the rules and delete the
   old matcher. A rule that failed to move to a new matcher will loose
   its match STE once the rehash is completed and the old matcher is
   deleted, so the rule won't match any traffic any more. This rule's
   packets will fall back to the steering pipeline w/o HW offload.
   Rehash procedure will return an error, which will cause the rule
   insertion to fail for the rule that started this whole rehash.

Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com>
Reviewed-by: Vlad Dogaru <vdogaru@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Link: https://patch.msgid.link/1746992290-568936-10-git-send-email-tariqt@nvidia.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>