clk: wm831x: get rid of the implementation of remove function