Are they so different that it’s justified to have so many different distributions? So far I guess that different package manager are the reason that divides the linux community. One may be on KDE and one on GNOME but they can use each other’s packages but usually you are bound to one manager
As another user mentioned, package managers are specific to distributions rather than DEs. The main difference between them is that they’re developed by the respective distribution teams, but there are some practical differences too. For example, apt supports versioned dependencies while pacman doesn’t because of the different distribution models between Debian and Arch (monolithic vs. rolling release). This affects their dependency resolution strategy with each being better suited for it’s respective distribution.
To address your point about package managers being the main difference between distros, this isn’t quite true. As mentioned, different distros have different distribution models, priorities, and overall biases/opinions that affect the user experience in a variety of ways and make them better suited to different use cases. I would never dream of putting Arch on one of my servers in the same way that you’d probably never catch me installing Debian on my gaming machine.
All my devices, including servers and pi, on Arch testing (Only nvidia has fucked me twice):