| The Glade 4.0 https://gladerebooted.net/ |
|
| Server downtime tonight https://gladerebooted.net/viewtopic.php?f=2&t=9627 |
Page 1 of 1 |
| Author: | Mookhow [ Mon Jan 28, 2013 5:54 pm ] |
| Post subject: | Server downtime tonight |
Media Temple is going to be taking down the server's physical host tonight to upgrade the RAM in it. They estimate the downtime should be about an hour. Hopefully everything comes up hunky dory once they are done. |
|
| Author: | Midgen [ Mon Jan 28, 2013 7:56 pm ] |
| Post subject: | |
/MMO Forum Style Nerd Rage On $%*($* you and your #$!)) % %#**@#$!! /MMO Forum Style Nerd Range Off /Friendly Glade Denizen On W00T MORE RAM !! ... |
|
| Author: | Mookhow [ Wed Jan 30, 2013 8:26 am ] |
| Post subject: | Re: Server downtime tonight |
I apologize for the downtime. When the server came up, it did not have an IP address, and it took until 4am CST for them to fix it. There's still an issue with the services not starting, and I'm going to work with Stathol to resolve this. Feel free to vent or cheer here. |
|
| Author: | FarSky [ Wed Jan 30, 2013 9:29 am ] |
| Post subject: | |
We're back! I choose to vheer! |
|
| Author: | shuyung [ Wed Jan 30, 2013 9:34 am ] |
| Post subject: | |
Wow, Media Temple is incompetent. |
|
| Author: | TheRiov [ Wed Jan 30, 2013 10:14 am ] |
| Post subject: | Re: |
FarSky wrote: We're back! I choose to vheer! can I chent? |
|
| Author: | Diamondeye [ Wed Jan 30, 2013 10:58 am ] |
| Post subject: | |
Thank goodness. I can feel the tremors and cold sweats dying down already. |
|
| Author: | Stathol [ Wed Jan 30, 2013 11:40 am ] |
| Post subject: | Re: |
shuyung wrote: Wow, Media Temple is incompetent. I'm pretty sure this isn't Media Temple's fault. I think we've run into a bug in upstart (Ubuntu's init daemon). The server is still in a really wonky state right now. I'm in class at the moment, but I'll see if I can get it sorted out when I'm done. |
|
| Author: | Lenas [ Wed Jan 30, 2013 11:40 am ] |
| Post subject: | Re: |
shuyung wrote: Wow, Media Temple is incompetent. Very much so. A few months back they decided to physically move our server to a different data center without telling us. 50+ client sites down for the majority of a day and they basically said "tough ****". |
|
| Author: | Mookhow [ Wed Jan 30, 2013 12:18 pm ] |
| Post subject: | Re: Server downtime tonight |
Stathol is quite competent. |
|
| Author: | shuyung [ Wed Jan 30, 2013 12:19 pm ] |
| Post subject: | Re: Re: |
Stathol wrote: I'm pretty sure this isn't Media Temple's fault. I think we've run into a bug in upstart (Ubuntu's init daemon). The server is still in a really wonky state right now. I'm in class at the moment, but I'll see if I can get it sorted out when I'm done. An IP address that does not persist across shutdown is certainly Media Temple's fault. And while I haven't examined the device, I find the claim of a bug in upstart somewhat dubious. I suspect if you simply examine /etc/init.d and /etc/rc2.d, you'll find missing (or bad) scripts. |
|
| Author: | Mookhow [ Wed Jan 30, 2013 12:47 pm ] |
| Post subject: | Re: Server downtime tonight |
From what stathol told me, the problem is that init did not work properly, preventing the startup scripts from running properly. So if up didn't run, ergo no IP address. IMO, they shouldn't have taken all day to figure out there was no IP address, though. |
|
| Author: | Stathol [ Wed Jan 30, 2013 1:14 pm ] |
| Post subject: | Re: Re: |
shuyung wrote: An IP address that does not persist across shutdown is certainly Media Temple's fault. And while I haven't examined the device, I find the claim of a bug in upstart somewhat dubious. I suspect if you simply examine /etc/init.d and /etc/rc2.d, you'll find missing (or bad) scripts. I'm almost positive we're dealing with bug #543506. The problem isn't that the IP address wasn't persistent across a reboot. Rather, the "networking" and "networking-interface" init scripts (along with a shitton of others) are stuck in wait state. Basically init failed spectacularly due to what seems to be a race condition between /dev/console and the rc-sysinit upstart script which normally starts all the traditional system V (i.e. non-upstartified) init scripts. All of the system's symptoms match up exactly. Unfortunately this is something of a mandelbug. It's not clear precisely what conditions trigger it, and affected systems may only see this happen every Nth boot. Further, since 12.04 was released (the newer LTS version of Ubuntu), this bug has pretty much languished without any complete resolution. There's a potential workaround which I've implemented, but it won't take effect until the server is rebooted. We're holding on that for now because Mookhow only has access to a tablet right now, and if things go tits up again on reboot, it would be nice if he had access to a full PC. Basically, this would have inevitably happened the next time the server was rebooted, regardless of who did it or why. Upstart was quite frankly shoved out the door by the Ubuntu devs before it was really ready. I could rant about other dumb **** I've seen upstart do, but it suffices to say that I'm not terribly surprised to see it blow up entirely. Mookhow wrote: IMO, they shouldn't have taken all day to figure out there was no IP address, though. Yeah, I will fault them for that. ifupdown is acting all kinds of weird because of the init problems, but running "ifconfig" or "ip link show" still would have made it obvious that the interfaces weren't configured at all (not even loopback). |
|
| Author: | Mookhow [ Wed Jan 30, 2013 1:33 pm ] |
| Post subject: | Re: Server downtime tonight |
Do you think we should switch to Debian or fedora? Sent from my Galaxy Nexus using Tapatalk 2 |
|
| Author: | Stathol [ Wed Jan 30, 2013 1:54 pm ] |
| Post subject: | |
I would probably just recommend upgrading to 12.04. Upstart had a rough launch, but I think they've gotten it ironed out by and large. That's why this bug isn't likely to ever receive an official fix -- so far as I know, it disappeared in the process of developing 12.04+ If you want to jump on a totally different distro, Debian is the only thing I have particularly much experience with. It would also be the most direct path from Ubuntu since the latter is bascially just a downstream fork of the former (maybe a little less so these days, but still). I've never used Redhat, so I can't really comment on Fedora. |
|
| Author: | Mookhow [ Wed Jan 30, 2013 2:02 pm ] |
| Post subject: | Re: Server downtime tonight |
My company has RHEL, and my old vps was running debian. My preferred Linux is Slackware, but that isn't an option. looking at the options, the newest Ubuntu they offer is 11.04. Sent from my Galaxy Nexus using Tapatalk 2 |
|
| Author: | Stathol [ Wed Jan 30, 2013 3:52 pm ] |
| Post subject: | |
I'm going to be rebooting shortly. Hopefully everything will come up correctly now. If not, hang tight. |
|
| Author: | Squirrel Girl [ Wed Jan 30, 2013 4:43 pm ] |
| Post subject: | Re: Server downtime tonight |
This last restart broke my dns for this site. I had to flush my dns cache to connect again. |
|
| Author: | Mookhow [ Wed Jan 30, 2013 5:24 pm ] |
| Post subject: | Re: Server downtime tonight |
Stathol's workaround didn't work, so we'll have to do something else. Probably not for a little while though. |
|
| Page 1 of 1 | All times are UTC - 6 hours [ DST ] |
| Powered by phpBB® Forum Software © phpBB Group https://www.phpbb.com/ |
|