Fix #2291: Add safe-to-evict annotation to audit pod

[Mallikarjunadevops]

Fixes #2291 by adding the cluster-autoscaler.kubernetes.io/safe-to-evict: "true" annotation to the audit deployment. This unblocks cluster autoscaler downscaling on nodes running the audit pod.

[linux-foundation-easycla]

• ❌ - login: @Mallikarjunadevops / name: Mallikarjunadevops. The commit (2d50043) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds the Kubernetes cluster-autoscaler safe-to-evict pod annotation to Gatekeeper-related workloads so the autoscaler is allowed to evict them during scale-down/optimization.

Changes:

• Add cluster-autoscaler.kubernetes.io/safe-to-evict: "true" to Gatekeeper audit-controller pod templates in rendered manifests.
• Add the same annotation to the Helm chart template for the audit deployment.
• Add the same annotation to the manager deployment manifest.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
manifest_staging/deploy/gatekeeper.yaml	Adds safe-to-evict annotation to the audit-controller PodTemplate metadata in staging manifest.
manifest_staging/charts/gatekeeper/templates/gatekeeper-audit-deployment.yaml	Hard-codes safe-to-evict annotation into the Helm template for the audit deployment.
deploy/gatekeeper.yaml	Adds safe-to-evict annotation to the audit-controller PodTemplate metadata in deploy manifest.
config/manager/manager.yaml	Adds safe-to-evict annotation to the manager PodTemplate metadata.

[JaydipGabani]

@Mallikarjunadevops why cant podAnnotations helm variable be used to solve the problem in the original issue?

[Mallikarjunadevops]

@JaydipGabani Good point! You are absolutely right that users can manually configure this via podAnnotations. However, since Gatekeeper's audit and manager deployments use emptyDir by default for /tmp, they silently block cluster downscaling out-of-the-box for users who might be unaware of this interaction.

To provide a better out-of-the-box experience while remaining configurable, I've updated this PR to remove the hardcoded annotations from the templates, and instead set them as defaults in the Helm values.yaml under both podAnnotations and auditPodAnnotations. This fully resolves the issue for Helm users by default while allowing them to override it if they prefer. Let me know if this approach looks better!