📌 30 de Junho, 2024

Replace Proxmox with Incus / LXD

Informática · ISP / Redes · Linux

📌 30 de Junho, 2024

Replace Proxmox with Incus / LXD

Informática · ISP / Redes · Linux

This article goes through all the reasons why you might want to replace Proxmox with Incus / LXD and how it could save you from a lot of headaches down the line. While free Proxmox, much like VMWare ESXi was, is a potential disaster waiting to happen.

Container Technology Evolution

For a long time, Proxmox and many other solutions used OpenVZ as their container engine. However, due to its strong ties to Virtuozzo (company), it was a risky situation, eventually leading the industry to create the Linux Containers project.

With significant initial investments, primarily from IBM and Canonical, LXC became much more advanced than OpenVZ and eventually even Proxmox decided to transition to LXC. It’s important to note that LXC is solely a container technology. Later on, Canonical developed LXD, which added a higher-level management layer.

Proxmox vs Incus / LXD

Proxmox is a “hypervisor”, a full operating system you can download and use to deploy and manage virtual machines and containers across multiple hosts thanks to its clustering features. While being free and open-source software, Proxmox requires a payed license for the stable version and updates. Furthermore the Proxmox guys have been found to withhold important security updates from non-stable (not paying) users for weeks.

Incus / LXD is essentially an alternative that offers most of the Proxmox’s functionality while being fully open-source – 100% free. It can be installed on most Linux systems and provides a management and automation layer that makes things work smoothly – essentially what Proxmox does but properly done. You can create clusters, download, manage and create OS images, run backups and restores, bootstrap things with cloud-init, move containers and VMs between servers (even live sometimes).

Both Proxmox and Incus/LXD aren’t about replacing existing virtualization techniques such as QEMU, KVM and libvirt, they’re about augmenting them so they become easier to manage at scale and overall more efficient.

Depending in your real needs you might be able to replace your Proxmox instances with Incus and avoid a few headaches in the future. Read more about Incus features here.

Incus or LXD? What’s the difference?

For a very long time LXD was an independent project under the Linux Containers project umbrella however heavily funded by Canonical. Recently they announced that LXD would not be independent any more, and removed it from the Linux Containers project and moved it to in-house development.

One potential reason for this move was that the former lead engineer of LXD, Stéphane Graber, left the company. He later on stated he planned to continue contributing to the project and will be working under the handle of his own company, Zabbly. After this the SUSE developer, Aleksa Sarai, launched the Incus project as a fork and community driven alternative to Canonical’s LXD. Today, it’s led and maintained by Stéphane Graber and much of the same people that once created LXD under Canonical.

My Experience with Proxmox

Proxmox is considerably cheaper than ESXi so people use it for businesses like we did, but it is far from perfect. Homelabbers also like Proxmox because it’s free, easy to install and provides a reasonably well built management WebUI.

I run Promox since 2009 and until very recently, professionally, in datacenters, multiple clusters around 10-15 nodes each which means that I’ve been around for all wins and fails of Proxmox. I saw the raise and fall of OpenVZ, the subsequent and painful move to LXC and the SLES/RHEL compatibility issues.

While Proxmox works most of the time and their payed support is decent I would never recommend it to anyone since Incus became a thing. The Promox PVE kernel has a lot of quirks, for starters it is build upon Ubuntu’s kernel – that is already a dumpster fire of hacks waiting for someone upstream to implement things properly so they can backport them and ditch their own implementations – and then it is a typically older version so mangled and twisted by the extra features garbage added on top.

I got burned countless times by Proxmox’s kernel. Broken drivers, waiting months for fixes already available upstream or so they would fix their own bugs – it all happens in Proxmox.

As practice examples, at some point OpenVPN was broken under Proxmox’s kernel, the Realtek networking has probably been broken for more time than working. ZFS support was introduced only to bring kernel panics. 🙂 Upgrading Proxmox is always a shot in the dark and half of the time you get a half broken system that is able to boot and pass a few tests but that will randomly fail a few days later.

Proxmox’s startup is slow, slower than any other solution – it even includes management daemons that are there just there to ensure that other daemons are running. Most of the built-in daemons are so poorly written and tied together that they don’t even start with the system properly on the first try.

Incus

Incus also provides a unified experience to deal with both LXC containers and VMs, no need to learn two different tools / APIs as the same commands and options will be used to manage both. Even profiles defining storage, network resources and other policies can be shared and applied across both containers and VMs. The same thing can’t be said about Proxmox, while it tries to make things smoother there are a few inconsistencies and incompatibilities there.

I now draw your attention to containers because for most people virtualization isn’t even required. In a Home Lab with LXC (typically called system containers) you can run a full operating systems (minus the kernel) with persistency.

Either way Incus allows for both and you can easily mix and match and use what you require whenever you require. What about Docker? Well you can even run Docker inside a LXC container for your applications.

For eg. I run the official Home Assistant image as a VM (because we all know how hard is to get that thing running), however my NAS / Samba shares are just a Debian 12 container with Syncthing, Samba 4 and FileBrowser. I also run a few services exposed to the Internet on VMs in order to increase security.

Incus also comes with a WebUI:

Incus is free can be installed on any clean Debian system with little to no overhead and on the release of Debian 13 it will be included on the repositories.

Another interesting advantage of Incus is that you can move containers and VMs between hosts with different base kernels and Linux distros. If you’ve bought into the immutable distro movement you can also have your hosts run an immutable with Incus on top.

Incus Under Debian 12

If you’re on stable Debian 12 then you’ve a couple of options:

  1. Run the LXD version provided on their repositories: this will give you LXD 5.0.2 LTS that is guaranteed to be compatible with Debian 13’s Incus. Note that this was added before Canonical decided to move LXD in-house;
  2. Use the backported version as described here: https://linuxcontainers.org/incus/docs/main/installing/;
  3. Get the latest Incus pre-compiled from https://github.com/zabbly/incus and install as described above.

In the first option you’ll get a Debian 12 stable system with a stable LXD 5.0.2 LTS, it works really well however it doesn’t provide a WebUI. The second and third options will give you the latest Incus but they might not be as stable.

Personally I was running LXD from Snap since Debian 10, and moved to LXD 5.0.2 LTS repository under Debian 12 because I don’t care about the WebUI. I can see how some people, particularly those coming from Proxmox, would like the WebUI so getting the latest Incus might be a good option.

Closing Remarks

I believe most people running Proxmox today will, eventually, move to Incus and never look back, I just hope they do before Proxmox GmbH changes their licensing schemes or something fails. If you don’t require all features of Proxmox then Incus works way better with less overhead, is true open-source, requires no subscriptions, and doesn’t delay important security updates.

If you’re running a modern version of Proxmox then you’re already running LXC containers so why not move to Incus that is made by the same people? Why keep dragging all of the Proxmox overhead and potencial issues?

Proxmox is yet another disaster waiting to happen.
Something akin to what happened with VMWare ESXi or with CentOS licensing.