?

Log in

No account? Create an account

Previous Entry | Next Entry

Servers and such

It's pretty bad when your laptop has better uptime than your servers.

I've been planning a new storage cluster server for a while. I finally bought the hardware and had done the design testing and was about to load test it when the storage array on my main server begins to fail.

This server's been out of storage for a while, so this was something I was going to be putting in anyway, and planning to do it just the way I am.

So, I bump up plans to put the server in, having made sure the design of the system was sound, I figured the stress-test was just me seeing how much faster it could go than my current machine.

I come up to Ouray in the middle of the night and put the new server in the rack, move the data over to it and go home. Total downtime? 5 hours. So far, so good.

The only problem? The new NFS server to handle the storage locks up under heavy IO on occasion.

Back to Ouray.

Update some things. Try to find the source of the lockup. It's an obscure, possibly SMP-only related bug in the Linux kernel — or my hardware. I get "NMI Watchdog detected lockup on CPU", and one of the CPUs in the system is frozen, and tasks running on it killed. Then the other one goes some time later.

Apply a few trial fixes. Seems stable for hours. Go home.

Server's frozen.

Wash. Rinse. Repeat.

I gotta learn to drive.

This morning, same thing. Hitch to Ouray. I'm changing kernels now, and with luck, that'll take care of it. I think I'll just spend my day up here. Ouray's nice in the off-season.

Tags:

Comments

( 8 comments — Leave a comment )
lightwalker
Apr. 14th, 2007 04:06 pm (UTC)
Geez.

Maybe next time you're waiting for it to break you could forage around for someplace that has a good brunch menu?
aredridel
Apr. 14th, 2007 04:07 pm (UTC)
Hehe. During the off season, bagels is about it. There's a couple places that serve breakfast starting in May, but now, I'd have to go to Ridgway for breakfast. It's sad.

Oh, well. The bagel shop has wireless.
lightwalker
Apr. 14th, 2007 05:40 pm (UTC)
That's remarkably pathetic. Astral projection of a ginger miso tempeh scramble coming your way.
branwen
Apr. 14th, 2007 08:37 pm (UTC)
Maybe if one restaurant opened for breakfast it would get some good business, since it would be the only one? Or maybe there really isn't anyone open for a good reason.
aredridel
Apr. 14th, 2007 08:37 pm (UTC)
Nah, town's /dead/
branwen
Apr. 14th, 2007 08:36 pm (UTC)
Friday the 13th week has been like that.

I noticed the other day that the server was down because an image I have on myspace that's hosted on your server wasn't showing up. Freddy freaked. I just thought the server's just down, don't have a cow, man.

dykotez
Apr. 14th, 2007 09:48 pm (UTC)
Heh, where is Ouray anyway? I've never heard of it before.
aredridel
Apr. 14th, 2007 09:49 pm (UTC)
Rural Colorado. About here.

I live just south of that reservoir, ten miles north.
( 8 comments — Leave a comment )