PGO Fails To Resolve Cluster Domain?

[jsievenpiper]

Please ensure you do the following when reporting a bug:

• Provide a concise description of what the bug is.
• Provide information about your environment.
• Provide clear steps to reproduce the bug.
• Attach applicable logs. Please do not attach screenshots showing logs unless you are unable to copy and paste the log data.
• Ensure any code / output examples are properly formatted for legibility.

Note that some logs needed to troubleshoot may be found in the /pgdata/<CLUSTERNAME>/pg_log directory on your Postgres instance.

An incomplete bug report can lead to delays in resolving the issue or the closing of a ticket, so please be as detailed as possible.

If you are looking for general support, please view the support page for where you can ask questions.

Thanks for reporting the issue, we're looking forward to helping you!

Overview

Sometime around an upgrade to PG 18 & PGO 6, backups started failing with DNS resolution errors due to a trailing dot:

 [HostConnectError] unable to get address for 'test-pgha-ln9q-0.test-pods.postgres.svc.':

It seems like the Kubernetes domain lookup is failing silently and returning an empty string, but I don't have a lot of insight into why 😅 In my cluster, the domain should resolve to cluster.local

Environment

Please provide the following details:

• Platform: (Kubernetes, OpenShift, Rancher, GKE, EKS, AKS etc.)
  • kubernetes
• Platform Version: (e.g. 1.20.3, 4.7.0)
  • k3s 1.31.14

• PGO Image Tag: (e.g. ubi8-5.x.y-0)

• Postgres Version (e.g. 15)
  • 18.4
• Storage: (e.g. hostpath, nfs, or the name of your storage class)
  • local ebs zfs

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

1. Any manual or scheduled backup

EXPECTED

1. backup proceeds as expected

ACTUAL

1. all hosts fail to resolve

Logs

time="2026-07-29T03:02:23Z" level=info msg="crunchy-pgbackrest starts"
time="2026-07-29T03:02:23Z" level=info msg="debug flag set to false"
time="2026-07-29T03:02:23Z" level=info msg="backrest backup command requested"
time="2026-07-29T03:02:23Z" level=info msg="command to execute is [pgbackrest backup --stanza=db --repo=1 --type=full]"
time="2026-07-29T03:02:24Z" level=info msg="output=[2026-07-29 03:02:24.112 P00   WARN: unable to check pg1: [HostConnectError] unable to get address for 'test-pgha-ln9q-0.test-pods.postgres.svc.': [-2] Name or service not known\n2026-07-29 03:02:24.115 P00   WARN: unable to check pg2: [HostConnectError] unable to get address for 'test-pgha-rjb5-0.test-pods.postgres.svc.': [-2] Name or service not known\n2026-07-29 03:02:24.117 P00   WARN: unable to check pg3: [HostConnectError] unable to get address for 'test-pgha-wlbr-0.test-pods.postgres.svc.': [-2] Name or service not known\n2026-07-29 03:02:24.117 P00  ERROR: [056]: unable to find primary cluster - cannot proceed\n                                    HINT: are all available clusters in recovery?\n]"
time="2026-07-29T03:02:24Z" level=info msg="stderr=[]"
time="2026-07-29T03:02:24Z" level=fatal msg="command terminated with exit code 56"

Additional Information

Patroni, pgbouncer, and other systems seem to be just fine! Thank you in advance for your time!!