DOCS-2997: Write the L2 network troubleshooting guide - #2949
Conversation
✅ Deploy Preview for calico-docs-preview-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview succeeded!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Pull request overview
Adds a symptom-oriented troubleshooting guide for Calico Enterprise L2 bridge networking, focusing on where to find actionable signals (Felix vs workload/CNI/Multus) and what diagnostics to collect when bridge state isn’t captured automatically.
Changes:
- Expanded the page from a brief stub into a full troubleshooting flow organized by symptom.
- Added guidance on log locations, diagnostic bundle collection, and bridge-state commands to gather manually.
- Added multiple specific failure-pattern sections plus a consolidated “Additional resources” list.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| 1. Run the standard diagnostics bundle: | ||
|
|
||
| ```bash | ||
| kubectl exec -n calico-system <calico-node-pod> -- calicoctl cluster diags |
| 2. **The pool is not marked for L2 use.** A pool without `allowedUses: L2Workload` is not a | ||
| candidate for an L2 interface. |
There was a problem hiding this comment.
L2Workload is valid — this troubleshooting step is correct as written.
On the EP2 build (2026-08-20-v3-24-2-kettle) the IPPool CRD enum accepts it:
"HostSecondaryInterface" "L2Workload" "LoadBalancer" "Tunnel" "Workload"
and a live pool on the validation cluster uses it: l2-vlan10 -> allowedUses: ["L2Workload"].
The useful half of this comment is the other one though: reference/resources/ippool.mdx:52 really does still list only Workload/Tunnel/HostSecondaryInterface/LoadBalancer, and the allowed-combinations list at lines 70-71 doesn't cover it either. So the reference page is the stale one, not this page.
Worth fixing alongside this docs set, since #2947 tells readers to create a pool with allowedUses: ["L2Workload"] and the reference page currently implies that is not a legal value.
1db20db to
f3ca5a4
Compare
Fill in the troubleshooting guide for L2 bridge networking, organized by symptom rather than by tool. The guide opens with where the signals are, because L2 problems report themselves somewhere unusual. Bridge problems appear only in the Felix log inside calico-node, with no Kubernetes event, no resource condition, and no metric. Attachment problems appear in the workload status and the CNI and Multus logs. The diagnostics section says that bridge state is not part of the standard bundle and has to be collected by hand, so a support case does not arrive without it. Entries cover the failures that are silent, late, or self-concealing: a bridge that does not meet its prerequisites, a workload with two interfaces called eth0 because the multi-interface mode was missed, a node taken off the network while its bridge was being configured, and a packet capture that masks the hardware filtering problem it was opened to investigate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f3ca5a4 to
9e74793
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
calico-enterprise/networking/l2-bridge/troubleshoot.mdx:45
- The diagnostics bundle command here assumes running
calicoctlinside acalico-nodepod. Elsewhere in the Calico Enterprise docs, the standard guidance is to installcalicoctlas a kubectl plugin and runkubectl calico cluster diags(see calico-enterprise/operations/troubleshoot/troubleshooting.mdx:9-21). Aligning this page with that standard avoids requiring readers to identify a specific calico-node pod and avoids implyingcalicoctlmust be present inside the pod.
1. Run the standard diagnostics bundle:
```bash
kubectl exec -n calico-system <calico-node-pod> -- calicoctl cluster diags
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
calico-enterprise/networking/l2-bridge/troubleshoot.mdx:46
- The diagnostics command is executed inside a calico-node pod, which means the generated
.tar.gzbundle will be written into the container filesystem and is not straightforward to retrieve. The calicoctl reference also documentscluster diagsas runnable from any location with kubeconfig access. Consider switching this step to a directcalicoctl cluster diagsinvocation (or add explicitkubectl cpretrieval instructions).
1. Run the standard diagnostics bundle:
```bash
kubectl exec -n calico-system <calico-node-pod> -- calicoctl cluster diags
**calico-enterprise/networking/l2-bridge/troubleshoot.mdx:65**
* This section instructs users to increase Felix logging to Debug, but only says to set it back to `Info` without providing the command to do so. Adding the explicit rollback command helps prevent leaving clusters in a noisy Debug logging state.
Set it back to Info afterwards.
</details>
Add a comment at the top of the page naming the user story it exists to serve. This page serves one story only. Diagnosis is framed by the reader's goal, which is why it is a how-to rather than a section appended to the setup guides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
calico-enterprise/networking/l2-bridge/troubleshoot.mdx:40
kubectl describe pod <pod>omits the pod namespace, which can be misleading because the workload pod usually isn’t in the current/default namespace. Include an explicit namespace placeholder so the command is copy/pasteable.
kubectl describe pod <pod>
calico-enterprise/networking/l2-bridge/troubleshoot.mdx:53
- The diagnostics bundle step currently runs
calicoctl cluster diagsviakubectl execinto acalico-nodepod. That’s inconsistent with the calicoctl reference, which describes running this command anywhere with kubeconfig, and it also leaves readers without a clear local path to the generated.tar.gzbundle. Prefer documenting the directcalicoctl cluster diagsinvocation from a machine with cluster access.
1. Run the standard diagnostics bundle:
```bash
kubectl exec -n calico-system <calico-node-pod> -- calicoctl cluster diags
</details>
| kubectl exec -n calico-system <calico-node-pod> -- calicoctl cluster diags | ||
| ``` | ||
|
|
||
| 2. Collect the bridge state by hand on each affected node. |
There was a problem hiding this comment.
I have a PR up to add this to calicoctl, not sure if it'll land in time for EP2
There was a problem hiding this comment.
Ok, will keep an eye out.
| bridge fdb show | ||
| ``` | ||
|
|
||
| 3. If the problem is not obvious, raise Felix's log level and reproduce it: |
There was a problem hiding this comment.
Not recommended as written, debug logging is perf impacting unless you combine with a filename regex
There was a problem hiding this comment.
Ok, we'll scrap this.
lwr20
left a comment
lwr20
left a comment
There was a problem hiding this comment.
Checked against 2026-08-20-v3-24-2-kettle on OpenShift. This page covers the bridge gotchas well — MAC pinning, VLAN-filtering ordering, self-VID membership and the promiscuous-mode masking trap all match what I hit in practice.
|
|
||
| If the value is not `Multus`, set it as described in [Set the multi-interface mode](connect-vlan.mdx#set-the-multi-interface-mode), then delete and recreate the affected workloads. | ||
|
|
||
| ## The workload has no IP address, or will not start |
There was a problem hiding this comment.
Missing the failure I actually hit here, which is a plausible first-week user error.
A NAD referencing a Network that no longer exists leaves the VM stuck in Starting indefinitely, with virt-launcher at Init:0/3, no restarts and no pod IP. The real cause appears only in the launcher pod's events:
failed to create pod network sandbox ... error adding container to network "vm-migrated-vlan10":
failed to get Network "vlan-trunk" ... resource does not exist: Network(vlan-trunk) not found
The VMI condition says only GuestNotRunning, and kubectl get vm shows Starting forever. Three VMs on the test cluster had been wedged this way for over two days. Nothing on this page points a KubeVirt reader from the VM to the launcher pod, which is where the error lives.
| tcpdump -p -i <interface> | ||
| ``` | ||
|
|
||
| ## Workloads lost connectivity after a Network was deleted |
There was a problem hiding this comment.
This entry covers workloads that were already running when the Network was deleted. The other half — new workloads created against a NAD whose Network is already gone — behaves differently and is not covered; see the note on line 121.
| $[prodname] rejects the interface rather than choosing. | ||
| Make them agree, or remove one. | ||
| 2. **The address is not the one you asked for.** | ||
| Check the `network-status` annotation to see what $[prodname] actually applied, and confirm the annotation you set is scoped to the right interface name. |
There was a problem hiding this comment.
Worth qualifying: on this build the network-status annotation is not protected by anything.
#2948 states an admission policy restricts it to the CNI plugin, but no such policy exists (CORE-13288), so a disagreement between request and status can also mean somebody wrote to it. That makes this check less conclusive than it reads.
|
|
||
| If the value is not `Multus`, set it as described in [Set the multi-interface mode](connect-vlan.mdx#set-the-multi-interface-mode), then delete and recreate the affected workloads. | ||
|
|
||
| ## The workload has no IP address, or will not start |
There was a problem hiding this comment.
This section has no entry for an unsupported cniVersion in the NAD, which presents exactly as the symptom in the heading.
A NAD declaring cniVersion: 1.1.0 fails CNI ADD — Calico's plugin supports up to 1.0.0. The pod stays in ContainerCreating and the only signal is in pod events:
plugin type="multus" ... failed (add): error adding container to network "<nad>":
incompatible CNI versions; config is "1.1.0", plugin supports [... "1.0.0"]
Nothing on the NAD, the Network, or (for a VM) the VMI says anything — a reader has to know to run kubectl describe pod on the sandbox. Suggest a row pointing there and naming the supported list.
Tested on RKE2 v1.33.5+rke2r1, EE 2026-08-20-v3-24-2-kettle, Multus v4.3.0. Claude-assisted, verified on a live cluster.
Fills in the troubleshooting guide for L2 bridge networking. Seventh in the DOCS-2997 sequence, and near the end because it references the setup guides.
The guide is organized by symptom rather than by tool, so a reader arrives with a problem and finds it.
It opens with where the signals are, because that is genuinely unusual here and looking in the obvious places wastes time. Bridge prerequisite failures appear only in the Felix log inside calico-node: no Kubernetes event, no condition on the Network, no status on the workload, no metric. Attachment failures appear in the workload status and the CNI and Multus logs instead.
The diagnostics section states that bridge state is not collected by the standard bundle and has to be gathered by hand. A support case from a cluster in this state otherwise arrives with the Felix warning and nothing else about the bridge.
The entries deliberately favour failures that are silent, late, or self-concealing:
Two notes:
Changed page: https://deploy-preview-2949--calico-docs-preview-next.netlify.app/calico-enterprise/next/networking/l2-bridge/troubleshoot