This was the exact problem I had, so I switched to NixOS. It takes a bit to learn about the nix language and understanding how things work (I had a few years of linux experience already, so do keep that in mind), but it has been so great once I understood things. I can now keep my config in a git repo.

Being able to just nix shell nixpkgs#[pkg] (or the old version nix-shell -p [pkg]) and getting a short lived shell where a package is available is grand.
I want to download music from extrememusic.com, but yt-dlp doesn’t have it. Not to worry, there is a pull request to add it, so I just patch yt-dlp. And I don’t have to think about updating it since I kept the version pinned to the nixpkgs version (rev = "${oldAttrs.version}";) so when it updates nix will complain that the hash is wrong and I just replace the old hash with the new one.