Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
256 changes: 249 additions & 7 deletions calico-enterprise/reference/resources/network.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@
description: API for this Calico Enterprise resource.
---

{/*
User story, primary
S9: As a network engineer, I want the field-level definition of the Network resource, so that I can author and review manifests precisely.

User stories, subsidiary
S5: the bridge configuration contract that the prepared-bridge guide points at is defined here.

Story ladder and page plan: DOCS-2997.
*/}

# Network

:::note
Expand All @@ -11,8 +21,12 @@
:::

A `Network` resource represents a logical network within a $[prodname] cluster. Each
`Network` has a type (currently `vrf`) that determines how pods on that network are
isolated and how their traffic is routed.
`Network` has a type — either `l2Bridge` or `vrf` — that determines how workloads on that network are isolated and how their traffic is routed.
Exactly one type must be set, and a `Network` cannot be changed from one type to the other.

A `Network` of type `l2Bridge` puts workloads directly onto 802.1Q VLANs on your existing network.
$[prodname] creates or uses a VLAN-aware Linux bridge on each selected node, attaches the trunk interface that carries those VLANs to your fabric, and attaches each workload as an access port on a single VLAN.
Workloads keep their own IP and MAC addresses and share a broadcast domain with machines outside the cluster, while still being subject to $[prodname] network policy.

A `Network` of type `vrf` configures a Linux Virtual Routing and Forwarding (VRF)
domain. $[prodname] creates a Linux VRF device on each selected node, moves the
Expand All @@ -21,7 +35,7 @@
isolated from pods on other networks (including the default flat pod network)
unless they are explicitly bridged outside the cluster.

For an end-to-end how-to, see [Configure multi-VRF networking](../../networking/configuring/multi-vrf.mdx).
For end-to-end how-tos, see [Connect workloads to an existing VLAN](../../networking/l2-bridge/connect-vlan.mdx) and [Prepare an existing bridge](../../networking/l2-bridge/byo-bridge.mdx) for `l2Bridge` networks, and [Configure multi-VRF networking](../../networking/configuring/multi-vrf.mdx) for `vrf` networks.

For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/),
the following case-insensitive aliases may be used to specify the resource type
Expand All @@ -30,6 +44,74 @@

## Sample YAML

### L2 bridge network, $[prodname]-managed bridge

A single VLAN, with $[prodname] creating and owning the bridge on every node.

```yaml
apiVersion: projectcalico.org/v3
kind: Network
metadata:
name: vlan-trunk
spec:
l2Bridge:
hostConfig:
- nodeSelector: ''
bridge:
managedBridge: {}
hostConnections:
- trunkPort:
interface:
name: eno2
vlans:
- vlan: { id: 10 }
subnets:
- cidr: 10.10.0.0/24
```

### L2 bridge network, bridge prepared by the user

Several VLANs on a bridge the user built, with different trunk interface names on different nodes.
The node's own address lives on this bridge, which is why $[prodname] does not own it.

```yaml
apiVersion: projectcalico.org/v3
kind: Network
metadata:
name: vlan-trunk
spec:
l2Bridge:
hostConfig:
- nodeSelector: kubernetes.io/hostname == "node-1"
bridge:
existingBridge:
name: br-l2
hostConnections:
- trunkPort:
interface:
name: enp8s0f1
- nodeSelector: ''
bridge:
existingBridge:
name: br-l2
hostConnections:
- trunkPort:
interface:
name: enp2s0f1np1
vlans:
- vlan: { id: 10 }
subnets:
- cidr: 10.10.0.0/24
- vlan: { id: 20 }
subnets:
- cidr: 10.20.0.0/24
- vlan: { id: 100 }
subnets:
- cidr: 10.10.100.0/24
```

### VRF network

```yaml
apiVersion: projectcalico.org/v3
kind: Network
Expand Down Expand Up @@ -60,11 +142,163 @@

### Spec

Exactly one of the network-type fields must be set. Currently only `vrf` is supported.
Exactly one of the network-type fields must be set.
The choice is fixed for the life of the resource.

| Field | Description | Schema |
| -------- | ------------------------------- | --------------------------------------- |
| l2Bridge | L2 bridge network configuration. | [L2BridgeSpec](#l2bridgespec) |
| vrf | VRF network configuration. | [VRFNetworkSpec](#vrfnetworkspec) |

Check failure on line 151 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'vrf'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'vrf'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":151,"column":3},"end":{"line":151,"column":6}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}

### L2BridgeSpec

| Field | Description | Schema |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| vlans | The VLANs this network carries, and the subnets on each. At least one entry is required. Overlapping or duplicated VLAN entries are accepted and combined rather than rejected. | List of [L2VLANSpec](#l2vlanspec) |

Check warning on line 157 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Terms] Use 'VLANs' instead of 'vlans'. Raw Output: {"message":"[Vale.Terms] Use 'VLANs' instead of 'vlans'.","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":157,"column":3},"end":{"line":157,"column":8}}},"severity":"WARNING","code":{"value":"Vale.Terms"}}
| hostConfig | Per-node bridge and trunk configuration. At least one entry is required. When several entries are present, each node is matched against them in order and the **first matching entry wins** — all others are ignored for that node. Entries are not merged. | List of [L2HostConfig](#l2hostconfig) |

Check failure on line 158 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'hostConfig'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'hostConfig'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":158,"column":3},"end":{"line":158,"column":13}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}

### L2VLANSpec

| Field | Description | Schema |
| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| vlan | The VLAN or range of VLANs this entry describes. Must be specified. | [L2VLANMatch](#l2vlanmatch) |

Check warning on line 164 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Terms] Use 'VLAN' instead of 'vlan'. Raw Output: {"message":"[Vale.Terms] Use 'VLAN' instead of 'vlan'.","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":164,"column":3},"end":{"line":164,"column":7}}},"severity":"WARNING","code":{"value":"Vale.Terms"}}
| subnets | The subnets present on this VLAN. $[prodname] uses these to choose an IP pool for a workload, and to decide which addresses are reserved. | List of [L2Subnet](#l2subnet) |

### L2VLANMatch

Identifies a VLAN, or a contiguous range of them.
Exactly one field must be set.

| Field | Description | Accepted Values | Schema |
| ----- | ------------------------------------ | --------------- | ------------------------------- |
| id | A single 802.1Q VLAN ID. | 1 – 4094 | int |
| range | A contiguous range of VLAN IDs. | | [L2VLANRange](#l2vlanrange) |

### L2VLANRange

A contiguous, inclusive range of VLAN IDs.

| Field | Description | Accepted Values | Schema |
| ----- | -------------------------------------------------- | --------------- | ------ |
| start | First VLAN ID in the range. | 1 – 4094 | int |
| end | Last VLAN ID in the range. Must not be less than `start`. | 1 – 4094 | int |

### L2Subnet

| Field | Description | Accepted Values | Schema |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- | ------------------------------- |
| cidr | The subnet present on this VLAN. An IP pool is a candidate for a workload on this VLAN only if the pool's CIDR falls inside this one. | A valid CIDR | string |

Check warning on line 190 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Terms] Use 'CIDR' instead of 'cidr'. Raw Output: {"message":"[Vale.Terms] Use 'CIDR' instead of 'cidr'.","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":190,"column":3},"end":{"line":190,"column":7}}},"severity":"WARNING","code":{"value":"Vale.Terms"}}
| routes | Additional routes to program into the workload's routing table for this interface, beyond the connected route for the subnet. | | List of [L2Route](#l2route) |

$[prodname] does not assign the network address, the broadcast address, or the gateway address of a subnet.
On a `/31` or `/32` no addresses are reserved, because those prefixes have no network or broadcast address to withhold.

| Field | Description | Schema |
| ----- | ---------------------------- | ----------------------------------- |
| vrf | VRF network configuration. | [VRFNetworkSpec](#vrfnetworkspec) |
### L2Route

| Field | Description | Accepted Values | Schema |
| ----------- | ------------------------------------------------------------------------------ | --------------- | ----------------------------------------- |
| destination | CIDR prefix for this route. Use `0.0.0.0/0` for a default route. | A valid CIDR | string |
| action | What to do with traffic matching this route. Exactly one action must be set. | | [L2RouteAction](#l2routeaction) |

### L2RouteAction

Exactly one field must be set.
Only `nextHop` is available in this release.

| Field | Description | Schema |
| ------- | ---------------------------------------------------------------------------------------------------- | ------ |
| nextHop | Forward matching traffic to this gateway address. The address must be reachable on the workload's VLAN. | string |

Check warning on line 210 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Terms] Use '[nN]exthop[s]?' instead of 'nextHop'. Raw Output: {"message":"[Vale.Terms] Use '[nN]exthop[s]?' instead of 'nextHop'.","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":210,"column":3},"end":{"line":210,"column":10}}},"severity":"WARNING","code":{"value":"Vale.Terms"}}

### L2HostConfig

Describes the bridge and the uplink on a set of nodes.

| Field | Description | Schema |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- |
| nodeSelector | $[prodname] selector expression choosing the nodes this entry applies to. An empty selector applies to all nodes. **The first entry whose selector matches a node is applied and all others are ignored for that node**, so place specific selectors before general ones. | [selector](bgppeer.mdx#selector) |

Check failure on line 218 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'nodeSelector'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'nodeSelector'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":218,"column":3},"end":{"line":218,"column":15}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}
| bridge | The bridge to use on the selected nodes. Must be specified. | [L2BridgeDevice](#l2bridgedevice) |
| hostConnections | How the bridge reaches the rest of the network on the selected nodes. Must be specified. | List of [L2HostConnection](#l2hostconnection) |

Check failure on line 220 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'hostConnections'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'hostConnections'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":220,"column":3},"end":{"line":220,"column":18}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}

### L2BridgeDevice

Exactly one field must be set.

| Field | Description | Schema |
| -------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| managedBridge | $[prodname] creates the bridge and owns it. Use this when $[prodname] can take over an interface of its own. | [L2ManagedBridge](#l2managedbridge) |

Check failure on line 228 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'managedBridge'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'managedBridge'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":228,"column":3},"end":{"line":228,"column":16}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}
| existingBridge | Use a bridge that already exists on the node. Required when the node's own IP address depends on the interface that must carry the VLANs. | [L2ExistingBridge](#l2existingbridge) |

Check failure on line 229 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'existingBridge'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'existingBridge'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":229,"column":3},"end":{"line":229,"column":17}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}

### L2ManagedBridge

| Field | Description | Accepted Values | Schema | Default |
| ----- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------ | ------- |
| stp | Enable Spanning Tree Protocol on the bridge. Every access port waits out a forwarding delay before passing traffic, which delays workload startup. | true, false | bool | `false` |

Check failure on line 235 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'bool'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'bool'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":235,"column":178},"end":{"line":235,"column":182}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}

Check failure on line 235 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'stp'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'stp'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":235,"column":3},"end":{"line":235,"column":6}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}

$[prodname] derives the bridge name per node, so it is not something you choose.
It also assigns the bridge its own MAC address, so the address does not change as workloads attach and detach.

### L2ExistingBridge

| Field | Description | Accepted Values | Schema |
| ----- | ---------------------------------------------------------------------- | ----------------- | ------ |
| name | Name of the existing bridge device on the node. | 1 – 15 characters | string |

#### Bridge configuration requirements

$[prodname] checks the following on a bridge it did not create, and **refuses to use the bridge** if any of them is not met.
It does not set them for you, because your host network configuration owns these properties and would undo the change.

| Property | Required value | Why |
| ---------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| VLAN filtering | Enabled | Without it the bridge ignores VLAN tags, so segments are not isolated from one another. |
| MAC address | Explicitly set, and not the trunk's | Otherwise the kernel borrows a port's address and re-chooses on every port change. Reusing the trunk's address can leave the bridge non-functional on some network card drivers. |
| VLAN tag protocol | 802.1Q | $[prodname] programs 802.1Q membership and creates 802.1Q sub-devices. Nothing matches on an 802.1ad bridge. |

$[prodname] reads two of these from sysfs.

Check failure on line 257 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'sysfs'? Raw Output: {"message":"[Vale.Spelling] Did you really mean 'sysfs'?","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":257,"column":35},"end":{"line":257,"column":40}}},"severity":"ERROR","code":{"value":"Vale.Spelling"}}
A value it cannot read is treated as "cannot determine" and allowed, rather than stranding the network on that node.

The following are **not** required, and $[prodname] handles them itself:

- Enslaving the trunk interface.
It only has to exist and be named in `trunkPort`.
- Any particular default PVID on the bridge.
$[prodname] neither reads nor changes it.
- Removing an existing VLAN sub-device on the bridge.
$[prodname] reuses one that is already present.
- Bringing up a bridge that is administratively down.
- The bridge device's own VLAN memberships, and their PVID and untagged flags.
$[prodname] never adds or removes these.

:::caution

Do not give an existing bridge a name beginning with `calb-`.
$[prodname] treats that prefix as indicating a bridge it owns, and deletes such bridges when they are no longer needed.
This is not validated when the `Network` is created.

:::

When a requirement is not met, $[prodname] programs nothing for that `Network` on that node: no trunk enslavement, no VLAN membership, and no workload interfaces.
Workloads stay administratively down rather than attaching to a bridge that cannot isolate them.
The condition is reported in the Felix log in `calico-node`, naming the bridge, the `Network`, and the command that resolves it, and is retried indefinitely.

### L2HostConnection

Exactly one field must be set.
Only `trunkPort` is available in this release.

| Field | Description | Schema |
| --------- | ------------------------------------------------------------------ | ----------------------------------------- |
| trunkPort | An interface carrying tagged VLANs between the bridge and your fabric. | [L2HostTrunkPort](#l2hosttrunkport) |

### L2HostTrunkPort

| Field | Description | Accepted Values | Schema |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- | ----------------------------------------- |
| interface | The interface to use as the trunk. Must be specified. Must be the trunk interface itself; a VLAN sub-device is not supported here. | | [InterfaceMatch](#interfacematch) |
| vlans | Restrict the trunk to a subset of the network's VLANs. If omitted, the trunk carries every VLAN the network defines. | | List of [L2VLANMatch](#l2vlanmatch) |

Check warning on line 298 in calico-enterprise/reference/resources/network.mdx

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Terms] Use 'VLANs' instead of 'vlans'. Raw Output: {"message":"[Vale.Terms] Use 'VLANs' instead of 'vlans'.","location":{"path":"calico-enterprise/reference/resources/network.mdx","range":{"start":{"line":298,"column":3},"end":{"line":298,"column":8}}},"severity":"WARNING","code":{"value":"Vale.Terms"}}
| nativeVLAN | The VLAN to assign to untagged traffic arriving on this interface. Must name a VLAN this network defines, which is enforced when the resource is created. | 1 – 4094 | int |

A node supports one trunk in this release.

### VRFNetworkSpec

Expand Down Expand Up @@ -150,6 +384,10 @@

## Limitations (tech preview)

The limitations below apply to `vrf` networks.
For `l2Bridge` networks, see [L2 bridge support and limitations](../l2-bridge-support.mdx).


- **Dataplane**: only the [nftables dataplane](../../operations/nftables.mdx) is supported. iptables and eBPF are not supported.
- **NodePort services** are not supported on VRF networks; advertise services as `LoadBalancer` cluster IPs instead.
- **Egress gateways** cannot be placed on a VRF network.
Expand All @@ -164,6 +402,10 @@

## Requirements

The requirements below apply to `vrf` networks.
`l2Bridge` networks have different requirements, including the eBPF data plane and a newer kernel — see [L2 bridge support and limitations](../l2-bridge-support.mdx).


- Linux kernel **5.6 or later** (for the `meta sdifname` nftables match used by VRF policy dispatch).
- The `vrf` kernel module must be loaded on every node. On Ubuntu this is part of `linux-modules-extra-$(uname -r)`. Confirm with `sudo modprobe vrf && lsmod | grep '^vrf '`.
- $[prodname] must be installed with `linuxDataplane: Nftables` and kube-proxy must also be in nftables mode.
Loading