Do not clean your file system off installation packages

Seriously. Don’t. (if you  really do not have to – eg. large iso images, clean up of space etc.). Double don’t if your documentation isn’t bulletproof.

Yesterday this saved my time/work life/put_something_meaningful_here. The ability to backtrack what has been done on the system from installation perspective can be invaluable. Especially on very service-dense systems.

I was performing an update-upgrade-distupgrade of one debian system running lots of different services (postfix, mailscanner,spamassassin, samba, apache, openssh etc. – basically think about a service – it was there, it was crucial and it wasn’t documented).

Everything went smoothly (not that much, but in tolerable widened scope of word smooth – some configs changes etc.), services went up after minor reconfigs and emails didn’t work… Well they did work, but only till the first reboot (which was initiated on purpose). Then they did not. No log errors, just prior to the reboot, but in that time everything worked, so no problems there, right? (no, there was a problem). Misleading call to previous admin (can’t blame him, he is off the job for 2 years and for the hand out of the services we had 3,5 days – great managerial decision) cost me another 1 hour of troubleshoot, but then going from service startup – startup scripts – files (no files, strange) – configs (configs there, good) got me to a point where upgrade really deleted some files of the mailscanner binary, which is running postfix in case you want to use it. Since we had postfix configured to work with it, we were holding emails in queue to process them by mailscanner (which wasn’t working after reboot – couldn’t load into memory – didn’t have what). There I was able to backtrack the thoughts of the previous admin based on the fact he had in root folder all the configs, scripts and packages he was installing (apart from apt-get). This helped us a ton – first we could doublecheck we are not going against something else (this is one of the few linux systems we have and by far the most complex one) and second we could use it to reinstall the files from deb archive.

Learned a ton on that. Especially from tshoot point of view – backtracking of someone’s thoughts => procedures is possible if he leaves a trace. And it can really help and save a day in the end.

 

Leave a Reply

Your email address will not be published. Required fields are marked *