It was patch day for the test environment, the group of servers dedicated to testing locally developed code before it is pushed into QA, and then into production. Patches for RedHat Enterprise Linux are applied using "yum update" in our setup, keeping it simple. Things were going well, no conflicts, no problems with our applications after the patches were applied. Then, one of the server's yum processes died with an error--a very familiar error.
Yum was trying to install a patch that depended on a certain library being at a certain level. Our library in question was higher than the level required. That library had been updated prior to patch day, so rolling it back would break whatever package installed it. I was immediately flashed back ten years...dependency hell.
If you remember the "old" days, back before package management was awesome on Linux, it was fairly easy to get into this predicament. Then it was normally done by source; you'd download the source for an application you'd like to run, say a music player, then chase down all of it's dependencies, compile them all, and happily listen to your music. Then you'd try to launch your email client, or word processor, or some other random application, and it would crash because you'd overwritten a library that it depended on in the process of installing the music player. This was a very real problem.
Package managers and repositories were a lifesaver for Linux. As the package management tools grew in sophistication and popularity, their associated repositories grew with software that was mostly guaranteed to work, and not muck up your system. I called it "staying in the box". As long as you stick with what was provided by the default package managers of your distribution, you more than likely won't run into the kind of dependency conflicts that were common in the earlier days of Linux.
Staying in the box is even more important with commercial Linux distributions like RedHat Enterprise Linux or SuSE Linux Enterprise Server. While the software itself is open source, what you really pay for is the support. When we ran into the server with the dependency issues, a little digging revealed that one of the application administrators with pseudo privileges had been adding third party repositories left and right. By doing so, he overwrote some of the RedHat supported libraries, breaking our patching functionality and breaking our support contract.
Luckily enough, it was the test environment, and this is what testing is for. Problems were easily fixed by spinning up a new virtual machine. It turns out the problem was just as much our fault as it was his. What we should have done from the start is install the yum-protectbase plugin, which would have protected our core packages from being overwritten, and allowed us to safely use third party repositories.
If you've got similar stories of package management run amok, I'd love to hear about them in the comments!