When provider security groups are removed, the corresponding
bindings could have already been removed by
_update_port_preprocess_security.
This change ensures binding deletion is done only when needed,
and avoids failures in case the bindings have already been
removed.
Change-Id: Iaccf4f3ddb9fef6d8dcb254bc978883b99c947f3