Pros:

• you run a home lab

Cons:

• you run a home lab

If you know your way around Linux you most likely don’t need Proxmox and its pseudo-open-source… you can try Incus / LXD instead.

Avoid Proxmox and safe yourself a LOT of headaches down the line. Go with Debian 12 + Incus/LXC, it runs VMs and containers very well. Proxmox ships with an old kernel that is so mangled and twisted that they shouldn’t even be calling it a Linux kernel. Also their management daemons and other internal shenanigans will delay your boot and crash your systems under certain circumstances.

LXD/Incus provides a management and automation layer that really makes things work smoothly - essentially what Proxmox does but properly done. With Incus you can create clusters, download, manage and create OS images, run backups and restores, bootstrap things with cloud-init, move containers and VMs between servers (even live sometimes).

Another big advantage is the fact that it provides a unified experience to deal with both containers and VMs, no need to learn two different tools / APIs as the same commands and options will be used to manage both. Even profiles defining storage, network resources and other policies can be shared and applied across both containers and VMs.

I draw your attention to containers (not docker), LXC containers because for most people full virtualization isn’t even required. In a small homelab if you can have containers that behave like full operating systems (minus the kernel) including persistence, VMs might not be required. Either way LXD/Incus will allow for both and you can easily mix and match and use what you require for each use case. Hell, you can even run Docker inside an LXC container.

For eg. I virtualize the official HomeAssistant image with Incus because we all know how hard is to get that thing running, however my NAS / Samba shares are just a LXD Debian 12 container with Samba4, Nginx and FileBrowser. Same goes for torrent client that has its own container. Some other service I’ve exposed to the internet also runs a full VM for isolation.

Like Proxmox, LXD/Incus isn’t about replacing existing virtualization techniques such as QEMU, KVM and libvirt, it is about augmenting them so they become easier to manage at scale and overall more efficient. I can guarantee you that most people running Proxmox today it today will eventually move to Incus and never look back. It woks way better, true open-source, no bugs, no delayed security updates, no BS licenses and way less overhead.

Also, let’s consider something, why use Proxmox when half of it’s technology (the container part) was made by the same people who made LXD/Incus? I mean Incus is free, well funded and can be installed on a clean Debian system with way less overhead and also delivers both containers and VMs.

Yes, there’s an optional WebUI for it as well!

• Context on Incus vs LXD: https://www.theregister.com/2023/08/04/incus_lxd_fork/
• A sum of my experience with Proxmox over the years: https://lemmy.world/comment/7476411

Some documentation for you:

• https://linuxcontainers.org/incus/
• https://github.com/lxc/incus
• https://wiki.debian.org/Incus
• https://linuxcontainers.org/incus/docs/main/installing/
• https://github.com/canonical/lxd
• https://www.techtarget.com/searchitoperations/definition/LXD-Linux-container-hypervisor

For me, pros are:

• Fun to learn something new
• Easy to test different systems. For example, I can play with different router or NAS software without having a separate computer around.
• I’ve been able to create different “computers” that serve different needs and require different levels of security.
• Currently, a cluster is probably overkill, it was a fun experiment.

Cons

• Updating all the different systems can be a pain. I could probably automate it, but I haven’t made the time to learn it yet.
• As a beginner, I’m throwing a bunch of parts together and hoping it will work. I should probably be more strategic in my implementation, but I don’t know what to prioritize. I’m sure I’ll have to start over in the future.
• With the previous point, the storage setup doesn’t seem very intuitive. I probably need to set up that better.
• I haven’t quite figured out backups yet. My VM backups all seem too big. I need to figure that out and automate it.

Hope this is helpful.

Proxmox is based on kvm/qemu, and is very resource conservative. There is virtually no impact on performance due to the hypervisor, even on older processors. Scheduling on the cpu and hypervisor makes running multiple VMs at the same time trivial as well. RAM and I/O bandwidth are the two things that can affect performance. Running out of RAM due to too many VMs will grind you to a halt, but so would running too many applications or containers on bare metal. Running everything off of one spinning sata disk will make it impossible, but again, same downfall on bare metal.

Those minimal impacts to performance are a minor nuisance compared to the ability to run experiments and learn on sandboxed VMs. Now that TrueNAS has better virtualization support, it has caught my eye as a better homelab solution, but I will always have a proxmox server running somewhere in my stack just due to the versatility it gives me.

VMs under KVM are pretty much bare metal and Proxmox doesn’t use much for resources itself, it’s basically a headless Debian with a webserver interface to do all the KVM stuff.

Proxmox, especially if you use ZFS for the VM datastore, makes a home lab so much easier to revert, backup and deploy/clone VMs and LXCs. I highly recommend it if you’re just starting out. Once you wrap your head around it, it gets out of the way and lets you just tinker with your projects, and not have to manually do everything in VirtManager or at the command line.

Combined with Proxmox Backup Server, it’s a production ready hypervisor for anything you decide to keep. Also, the HA features work well enough that I had my main routing OPNsense VM jump between nodes when the primary node lost a drive, and I didn’t notice for a week, it was that seamless.

I used Proxmox for a couple years and it’s good if you run a lot of VMs or LXCs, but I found that I’m not really the target audience. I ended up only running one Debian VM for my Docker containers. It was fine, but I eventually felt that Proxmox added no value for me, and the end result was sacrificing some memory and performance from using virtio emulations for CPU/GPU/RAM/filesystems. If your machines only have 8-16GB of RAM I don’t think it would be a good idea, as I’ve seen the rule of thumb is to dedicate 2GB for Proxmox’s usage, which is in addition to any guest OS’s requirements. Meanwhile I have a Debian install on a VPS that takes about 450MB of RAM.

For me, pros:

• Native ZFS support - invaluable, ZFS is terrific. MergerFS+SnapRAID is a decent replacement but the dodgy tooling and laundry list of footguns makes me nervous to use it on important data. ZFS is idiot-proof, as long as you know what you’re doing during the initial setup. RAIDZ expansion is coming this year and you can still use mixed-size disks in a RAIDZ as long as you accept that all disks are equivalent to the smallest one, so I personally feel ZFS is acceptable for grab-bag disk usage now
• Separation of bare metal and server environment, which means you can spin up another server VM from scratch without impacting the previous one, then switch with zero downtime. In the end, I replaced Proxmox with Debian on ZFS root (ZFSBootMenu) and wrote a few hundred lines of bash to automate the installation, so when I switched it only took about 30 minutes of downtime start to finish.
• Isolation of different environments. If my VM gets hacked, it will have a harder time reaching my Proxmox host etc. I run all services in isolated Docker environments anyway so this isn’t that big of a perk for my threat profile.

Cons:

• Partitioning RAM for ZFS ARC, Proxmox, and VM leads to inherent inefficiencies at the margins.
• I usually give my VM n-1 CPU cores, which is still less power than if I had just used the CPU natively.
• GPU passthroughs to VM can be less efficient, depending on the GPU and how it handles it. My iGPU is less performant when using its ~SR-IOV feature
• Learning requirement - not a huge learning curve but it’s a lot of knowledge that I will not use now that I’ve stopped using Proxmox
• Hosting your data pool on the Proxmox host or a dedicated data VM means that your server VM needs to use NFS to access its data, which lacks a handful of features (e.g. inotify) and is a pain
• Need to maintain two systems for updates, downtimes, etc
• More points of failure
• Extra startup time
• Run by a company that thinks it’s okay to use winrar-style nag popups every time you load the console, and requires you to manually dig through the source to disable that. I understand it’s their business model, it doesn’t change how it affects me the end user who lacks $120/year to spend on disabling a popup

It all depends on how you want to homelab.

I was into low power homelabbing for a while - half a dozen Raspberry Pis - and it was great. But I’m an incessant tinkerer. I like to experiment with new tech all the time, and am always cloning various repos to try out new stuff. I was reaching a limit with how much I could achieve with just Docker alone, and I really wanted to virtualise my firewall/router. There were other drivers too. I wanted to cut the streaming cord, and saving that monthly spend helped justify what came next.

I bought a pair of ex enterprise servers (HP DL360s) and jumped into Proxmox. I now have an OPNsense VM for my firewall/router, and host over 40 Proxmox CTs, running (at a guess) around 60-70 different services across them.

I love it, because Proxmox gives me full separation of each service. Each one has its own CT. Think of that as me running dozens of Raspberry Pis, without the headache of managing all that hardware. On top of that, Docker gives me complete portability and recoverability. I can move services around quite easily, and can update/rollback with ease.

Finally, the combination of the two gives me a huge advantage over bare metal for rapid prototyping.

Let’s say there’s a new contender that competes with Immich. They offer the promise of a really cool feature no one else has thought of in a self-hosted personal photo library. I have Immich hosted on a CT, using Docker, and hiding behind Nginx Proxy Manager (also on a CT), accessible via photos.domain on my home network.

I can spin up a Proxmox CT from my custom Debian template, use my Ansible playbook to provision Docker and all the other bits, access it in Portainer and spin up the latest and greatest Immich competitor, all within mere minutes. Like, literally 10 minutes max.

I have a play with the competitor for a bit. If I don’t like it, I just delete the CT and move on. If I do, I can point my photos.domain hostname (via Nginx Proxy Manager) to the new service and start using it full-time. Importantly, I can still keep my original Immich CT in place - maybe shutdown, maybe not - just in case I discover something I don’t like about the new kid on the block.

That’s a simplified example, but hopefully illustrates at least what I get out of using Proxmox the way I do.

The cons for me is the cost. Initial cost of hardware, and the cost of powering beefier kit like this. I’m about to invest in some decent centralised storage (been surviving with a couple li’l ARM-based NASes) to I can get true HA with my OPNsense firewall (and a few other services), so that’s more cost again.

Incus is way easier to work with than Proxmox, and it sits on your OS of choice instead of being the OS you must use. For home use it’s way easier to use with the web ui, it even has clustering if you want to go hard.

So you can install Incus when you want a VM/LXC container and not have to commit to a VM/LXC container OS from the start.

Also Proxmox free just had a bad update that björked some stuff if you updated when it was live. Proxmox free is rolling and apparently lacks basic sanity checks for updates.

I don’t prefer proxmox, but I will say that when you have even a machine with 8 or 16gb RAM, virtualizing a workload on it just makes sense. At that point the cost is 12% resources, and the benefits IMHO farrr outweight that.

Using ProxMox has been extremely useful for me. It has allowed me to experiment with a lot more things than I ever did before—it is very easy to spin up a new VM to test things out.

I would recommend it to anyone running a home server.

I have tried a couple of Proxmox clusters, one with overkill specs and one with little Mini PCs. Proxmox does eat up a fair amount of memory, but I have used it with Ceph for live migrations. Its really useful to me to be able to power off a machine, work on it, then bring it back up, and have no interruptions in my services. That said, my Mini PCs always seemed to be hurting for RAM. So that’s my pros and cons.

You need to understand what Proxmox gives you, which primarily is ability to run/manage/backup/etc VMs easily. If you don’t care about that, don’t use it. I have a fairly well spec’d desktop I use for homelab and I use proxmox because I often do experiments in VMs where snapshots and ability to jump to snapshots is essential. So is being able to spin up a new VM with new OS (like Windows) for example to do some testing. You can still do VMs without proxmox, but proxmox does make it a lot easier for living with daily.

I run it on a 4GB Fujitsu Futro S920! 😆 All the RAM seems to be used by 3 VMs. Some SWAP is been used, ok, but the Proxmox overhead doesn’t seem that much.

If you want to get things working then never “tinker” with things, maybe it’s not worth it. But if you want to learn and be able to try new things it is really helpful. Having a new VM not breaking existing VMs reduces risk when trying something new.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

[Thread #839 for this sub, first seen 29th Jun 2024, 15:25] [FAQ] [Full list] [Contact] [Source code]