Private networks for VMs on Kubernetes

Why this conversation

Many teams moving off VMware want to run their virtual machines on Kubernetes, next to their containers. The difficulty is the network. Kubernetes networking was built for pods that come and go, while VMs live for a long time and expect to keep their addresses, their subnets and their place in the data centre’s network. The host met Raymond de Jong, Field CTO EMEA at Isovalent, at TalosCon in Amsterdam on 15 October, where Isovalent was showing its answer: Isovalent Private Networks. Everyone the show has met from the company is on Isovalent: the people on Udienza.

A VPC for the Kubernetes cluster

Raymond describes private networks as basically a VPC for the Kubernetes cluster. The aim is to solve VM connectivity without the complexity of Kubernetes networking. The two worlds differ: pod IPs are ephemeral, but a VM’s IP is sticky to the VM, and VMs are long-lived. Private networks let you configure persistent networking for virtual machines and define subnets and routes for each private network, which gives proper multi-tenant networking for VMs on top of Kubernetes.

The host asked whether this is the primary network or a secondary interface, as with Multus. It is the primary network. Under the hood it relies on KubeVirt, which connects the pod network to the VM’s interface; private networks abstract that away. A simple annotation gives a VM a static IP, and another says which private network it belongs to. The VM is then connected to that network in its own routing domain, and can reach the network fabric with direct VLAN access or through EVPN.

For teams coming from VMware with traditional VLAN-based port groups, the path is direct VLAN access. You configure a VLAN for a private network and define the interface the host should use, and a VM on that network gets layer 3 access, for example to a default gateway on that VLAN. The alternative is overlay networking with EVPN and VXLAN, with BGP as the control plane advertising private networks to the fabric, so that a network administrator can route them across the network and between data centres.

Tenants, clusters and IPv6

For multi-tenancy, each tenant can have one or more private networks, each with a default and per-subnet configuration. On top of that, Cilium network policies isolate tenants for security. The result is two layers of separation: routing, and network policy.

Spanning clusters is on the roadmap. Today a private network lives within one cluster; the work under way will let it span several, in an experience similar to Cluster Mesh. IPv6 is already supported: you can define IPv6 networks and subnets and do BGP peering over IPv6. The integration still to come is with the egress gateway and load balancing capabilities. Once that is done, Raymond says, the multi-tenant capabilities will cover the whole ingress and egress stack.

Recovery across clusters

The host asked about disaster recovery and disconnected environments. What Raymond sees most is VMware-based customers who used NSX for availability or recoverability: several clusters, load balancing traffic across them, or recovering workloads in a different cluster. Private networks that span clusters are meant to deliver the same outcome.

Where to follow it

New capabilities are announced on the Isovalent website, with blog posts and YouTube videos from the team. Announcements also come through the Cisco website, and for integration with Cisco technology there are Cisco integration and validated design guides.