drm/dp/mst: Remove port after removing connector.