Linux

I wouldn't worry too much about the package manager, just worry about whether the distro has a good package repository. If it has all the software you want to use, then use it. In my opinion, most package managers (dnf, apt, pacman, xbmp) are basically the same, and you would only notice a big difference if you ever tried to make your own package for your own software.

That said, a few package managers are very different from all the rest:

• Crux OS "prt-get": simple and stupid: just downloads and installs tar archives.
• Gentoo "emerge": builds all software from source code when you install it. This provides some guarantees that the source code was not tampered with by the distro maintainers, this is great if you need to review all of the source code that is running on your system, but terrible for most people who don't want to spend so much computing power on compiling stuff every time you do a software update.
• Nix and Guix: creates its own blockchain-like database of isolated package dependency chains on your system, allowing you to instantly roll-back to the previous set of installed packages if you ever install something that breaks your system. It also guarantees that the software can be checked bit-for-bit (using SHA hash) traced back to the exact version and dependencies of the source code that built it. Nix and Guix packages also live peacefully side-by-side with any other package manager since all Nix/Guix apps are completely self-contained within its own database. In a way, it is sort of like one big AppImage or Docker container, but you can just keep adding or removing stuff to it as often as you want.
• Silverblue, SteamOS, VanillaOS, BlendOS, CarbonOS: distributes "immutable images," so it is impossible modify the operating system at all. Updates will ship an entirely new operating system with all packages built-in. However you are allowed to install software into your home directory, and you can install FlatPacks and AppImages. This provides a great deal of security in exchange for a tiny bit of inconvenience.

My personal preference: I use ordinary Debian or Ubuntu to install the critical software that needs to be stable and reliable, and I use Guix OS on the side to install the bleeding-edge things that might break a lot.

As another user mentioned, package managers are specific to distributions rather than DEs. The main difference between them is that they're developed by the respective distribution teams, but there are some practical differences too. For example, apt supports versioned dependencies while pacman doesn't because of the different distribution models between Debian and Arch (monolithic vs. rolling release). This affects their dependency resolution strategy with each being better suited for it's respective distribution.

To address your point about package managers being the main difference between distros, this isn't quite true. As mentioned, different distros have different distribution models, priorities, and overall biases/opinions that affect the user experience in a variety of ways and make them better suited to different use cases. I would never dream of putting Arch on one of my servers in the same way that you'd probably never catch me installing Debian on my gaming machine.

Oh no, there are 5 package managers out there and they're all wildly different

I know! I'll make a standard, universal package manager that'll be better than all the others that everyone will use!

There are now 6 different package managers