ServersOnFireRsync -rf /4 August 2006 I've seen things you people wouldn't believe. Chrooted daemons running on kernels present only in RAM. I watched TCP/IP packets flood across timezones at 10Mb/s. All our data lost to rsync like flip-flops in an e-m field. Time to blog. That should give a decent impression of how crazy this week has been. I awoke on Monday morning to a phone call saying that our lead developer had just toasted the newly-installed not-backed-up new server by copying the data from the old server over its hard drive. This left the new server running a shiny up-to-date kernel which was no longer on disk over a bunch of really ancient applications from the old server. Unsurprisingly, very little was working. With the old server scheduled to be taken away by the old hosting company at the end of the day, Monday's task was to rebuild the new server. Without rebooting it One not found. Much chrooting and manual RPM dependency resolving later, we had a chrooted system which could run yum. Since then it's been a slow, but somewhat saner migration of things like mail, Apache, jabber and our own applications to the chroot. I finally got DNS reconfigured this evening. We're still running everything in the chroot under the kernel which was installed before the hard drive got overwritten. And, of course, at some point we have to reboot into a kernel which is actually on the drive. :) And finally, a link to a some previews of a cool new animated series Amazing Screw-On Head. [1] This additional constraint was imposed by the lengthly time (about 3-4 hours we didn't have) taken by the new hosting company to reboot completely messed up machines into fresh installs. A useful piece of information we had discovered on Friday. |