User error during an update to the networking configuration in the Kubernetes cluster prevented new nodes to be added to the cluster. For services that required more than the existing cluster resources were unable to start.
The networking configuration are normally controlled with configuration-as-code (Terraform). A manual update was made to the cluster to facilitate the debugging of a user issue. The manual update should not have been applied. The configuration-as-code process should have been used as it would have avoided the manual error and provided a more rigorous review of the change.
Going forward, our process will require all network configuration changes to be made via configuration-as-code (Terraform).