portupgrade system destruction?

[Andrew Sinclair]

Moved to freebsd-questions by Andrew Sinclair. Eric Anderson wrote:

> I have a few dedicated servers at a hosting company (about 3 hours 
> drive time away). On one of the systems I ran a 'portupgrade -arR' 
> this morning, and then disconnected (I ran it in a screen session). 
> About an hour later, I realized I could not log in anymore via ssh. 
> Seems that I can connect, but my passwords fail (permission denied). I 
> can't FTP in, or check mail with any username/password combos. Even my 
> preshared SSH keys do not work. When connecting via POP, I get this 
> message:
>
> Connected to hostname.
> Escape character is '^]'.
> /usr/libexec/ld-elf.so.1: Shared object "libc-client4.so.8" not found
> Connection closed by foreign host.
>
> Can anyone help me figure out what may have gone wrong? And even how I 
> might be able to fix it remotely, or walk someone through a fix?
>
Portupgrade makes a mess at the best of times. A recursive portupgrade 
is not so clever about dependencies, particually on a live system. On 
occasion, it even seems to tamper with core libraries which is what 
would have occured in your case.

With all due respect, you better have a disaster recovery plan.

You said one of the systems. That's a good sign. Core library 
dependancies like libc are a bitch to deal with. My approach would be to 
reinstall a release on the existing system image, then restore the 
overwritten /etc files from a recent backup or an identical server. 
First, install the same release version on your PC. Build a custom 
kernel for the server (it's better to monitor the build locally). 
Tarball the files to be installed, send it to work and get one of the 
admins to do the following:

1. Burn your chosen release CD
2. Insert into affected servers slot-load and reboot, reinstalling 
everything (including sources)
2. Extract the kernel and LKM's tarball you uploaded to /
3. Restore /etc from backup.
4. Reboot and watch for errors on the console

This should get you running again but you might have to fix some ports 
manually. The reason I didn't suggest restoring your complete OS from 
backup is because an older version may not like your ports. I think it 
saves time but it's your call.