The Glade 4.0

Media Temple is going to be taking down the server's physical host tonight to upgrade the RAM in it. They estimate the downtime should be about an hour. Hopefully everything comes up hunky dory once they are done.

_________________

/MMO Forum Style Nerd Rage On

$%*($* you and your #$!)) % %#**@#$!!

/MMO Forum Style Nerd Range Off

/Friendly Glade Denizen On

W00T MORE RAM !!

...

I apologize for the downtime. When the server came up, it did not have an IP address, and it took until 4am CST for them to fix it. There's still an issue with the services not starting, and I'm going to work with Stathol to resolve this.

Feel free to vent or cheer here.

_________________

We're back! I choose to vheer!

Wow, Media Temple is incompetent.

_________________

can I chent?

Thank goodness. I can feel the tremors and cold sweats dying down already.

_________________
"Hysterical children shrieking about right-wing anything need to go sit in the corner and be quiet while the adults are talking."

I'm pretty sure this isn't Media Temple's fault. I think we've run into a bug in upstart (Ubuntu's init daemon). The server is still in a really wonky state right now. I'm in class at the moment, but I'll see if I can get it sorted out when I'm done.

_________________
Sail forth! steer for the deep waters only!
Reckless, O soul, exploring, I with thee, and thou with me;
For we are bound where mariner has not yet dared to go,
And we will risk the ship, ourselves and all.

Very much so. A few months back they decided to physically move our server to a different data center without telling us. 50+ client sites down for the majority of a day and they basically said "tough ****".

Stathol is quite competent.

_________________

An IP address that does not persist across shutdown is certainly Media Temple's fault. And while I haven't examined the device, I find the claim of a bug in upstart somewhat dubious. I suspect if you simply examine /etc/init.d and /etc/rc2.d, you'll find missing (or bad) scripts.

_________________

From what stathol told me, the problem is that init did not work properly, preventing the startup scripts from running properly. So if up didn't run, ergo no IP address. IMO, they shouldn't have taken all day to figure out there was no IP address, though.

_________________

I'm almost positive we're dealing with bug #543506. The problem isn't that the IP address wasn't persistent across a reboot. Rather, the "networking" and "networking-interface" init scripts (along with a shitton of others) are stuck in wait state. Basically init failed spectacularly due to what seems to be a race condition between /dev/console and the rc-sysinit upstart script which normally starts all the traditional system V (i.e. non-upstartified) init scripts. All of the system's symptoms match up exactly.

Unfortunately this is something of a mandelbug. It's not clear precisely what conditions trigger it, and affected systems may only see this happen every Nth boot. Further, since 12.04 was released (the newer LTS version of Ubuntu), this bug has pretty much languished without any complete resolution. There's a potential workaround which I've implemented, but it won't take effect until the server is rebooted. We're holding on that for now because Mookhow only has access to a tablet right now, and if things go tits up again on reboot, it would be nice if he had access to a full PC.

Basically, this would have inevitably happened the next time the server was rebooted, regardless of who did it or why. Upstart was quite frankly shoved out the door by the Ubuntu devs before it was really ready. I could rant about other dumb **** I've seen upstart do, but it suffices to say that I'm not terribly surprised to see it blow up entirely.


Yeah, I will fault them for that. ifupdown is acting all kinds of weird because of the init problems, but running "ifconfig" or "ip link show" still would have made it obvious that the interfaces weren't configured at all (not even loopback).

_________________
Sail forth! steer for the deep waters only!
Reckless, O soul, exploring, I with thee, and thou with me;
For we are bound where mariner has not yet dared to go,
And we will risk the ship, ourselves and all.

Do you think we should switch to Debian or fedora?

Sent from my Galaxy Nexus using Tapatalk 2

_________________

I would probably just recommend upgrading to 12.04. Upstart had a rough launch, but I think they've gotten it ironed out by and large. That's why this bug isn't likely to ever receive an official fix -- so far as I know, it disappeared in the process of developing 12.04+

If you want to jump on a totally different distro, Debian is the only thing I have particularly much experience with. It would also be the most direct path from Ubuntu since the latter is bascially just a downstream fork of the former (maybe a little less so these days, but still). I've never used Redhat, so I can't really comment on Fedora.

_________________
Sail forth! steer for the deep waters only!
Reckless, O soul, exploring, I with thee, and thou with me;
For we are bound where mariner has not yet dared to go,
And we will risk the ship, ourselves and all.

My company has RHEL, and my old vps was running debian. My preferred Linux is Slackware, but that isn't an option.

looking at the options, the newest Ubuntu they offer is 11.04.


Sent from my Galaxy Nexus using Tapatalk 2

_________________

I'm going to be rebooting shortly. Hopefully everything will come up correctly now. If not, hang tight.

_________________
Sail forth! steer for the deep waters only!
Reckless, O soul, exploring, I with thee, and thou with me;
For we are bound where mariner has not yet dared to go,
And we will risk the ship, ourselves and all.

This last restart broke my dns for this site.

I had to flush my dns cache to connect again.

_________________
Memento Vivere

I have local knowledge.
That sandbar was not there yesterday!

Stathol's workaround didn't work, so we'll have to do something else. Probably not for a little while though.

_________________