Package Management and Perl
Anyone who lived through the bad old days of compiling software from source on Linux remembers well the frustration of upgrading one package only to find that it breaks another. I like to think that those days are behind us; and, for the most part, they are. Unfortunately, I found myself in an eerily similar situation after patching a CentOS 6 server, and then trying to run a scheduled Perl job.
Package managers are the life blood of a Linux distribution. Without them, the entire house of cards starts to fall down. The open nature of Linux lends to many different people working on many different projects, each bit fitting into another bit in just the right way. I always tell the teams I work with, "stay in the box", as much as possible, you should stay within what the distribution you are working with gives you in the package manager. Yes, it might be a bit out of date, but the frustration you save by having a reliable, and upgradeable system is well worth the cost. There are, obviously, times when stepping outside the box is warranted, but do so knowing full well what you are getting in to, how it works, and what else you might be affecting.
When just using a Linux system, this advice works well, but developing on one can be a slightly different story, especially when working with a system that comes with it's own package manager like CPAN, the Comprehensive Perl Archive Network. A developer contacted me the other day complaining that my patches broke his Perl script, and demanded that I restore a library from backup. My response was that course of action was probably not a good idea, since the entire system was upgraded at once, and that he should fix the script. It was the type of classic back and forth, sysadmin to developer, banter that is generally, and rightfully, looked down upon in this devops world. When he told me that simply fixing the script wasn't an option, I went in to dig a bit deeper and sort out the issue.
The script was fairly short and easy enough to sort out what part was failing. The script compiled some data, created a file, and then scp'd the file to another server. Commenting out one line that called the scp let the script finish without error, but why was that method causing the error? Turns out, the script was using parts of Perl that were upgraded during patching, and calling other parts that were not. After talking to the admin who setup the machine, it became clear that this system had been Frankensteined.
The initial Perl install and the first modules were all installed from the official CentOS repositories with yum, but that wasn't enough. The repository was either too out of date, or didn't contain the modules that the developers wanted to use, so the admin used CPAN to fill in the blanks. Now there are two versions of Perl modules installed managed by two different package managers, but that wasn't quite enough yet. To top it off the admin had installed a few more modules from source, so now there were the two package managers plus modules that had to be updated completely separately. It became readily apparent that the wonder was not that the system was broken, but that it had ever worked at all.
So, after digging into yum, cpan, and downloading and compiling one module from source, the developers Perl script ran again, and all was right with the world. Until the next time something is updated. So, the moral of the story is not just to stay in the box, it is more to pick a box and then stay in the new box. Perl is a powerful language, and the modules are easy to work with, but both the language and the modules should be used with care.