Aria Stewart ([info]aredridel) wrote,
@ 2007-11-05 01:14:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
Entry tags:polis

Well, that sucks.
Polis is down for a somewhat extended count, as it's doing an intensive check on a huge disk.

It's going to take about ten hours.


(Post a new comment)


[info]whetherwoman
2007-11-05 02:41 pm UTC (link)
Thanks for letting us know.

(Reply to this)

[info]rcoder
2007-11-05 06:54 pm UTC (link)
For a long time, we had a primary/secondary IMAP server failover model. One fateful day, power and network issues conspired to cause a split-brain problem where both nodes had the shared mail spool SAN volume mounted at the same time.

Hello, 18-hour fsck on a single 400GB Reiser3 logical volume. That was a fun day. (Did I mention that it was my second or third week on the job, too?)

Amazingly, it pretty much worked, modulo a few hundred "orphan" files which couldn't be automatically re-associated with a parent directory inode. Thankfully, they were all basically RFC822 text, so we were able to grep through the recovery directory for any user that noticed something amiss in their inbox.

Now, we keep filesystems in the <250GB range, and just spread mail out onto a large number of logical volumes. Also, each mail server has its own local RAID, and boxes are divided between them and copied as a read-only replica to the other node.

Regardless, good luck. Hope you have a good version of tetris to keep yourself busy watching those inodes churn.

(Reply to this)

[info]raininglight7
2007-11-06 04:16 am UTC (link)
Yuck.

(Reply to this)(Thread)

[info]aredridel
2007-11-06 04:16 am UTC (link)
Yes, but it's better!

(Reply to this)(Parent)
Idetrorce
(Anonymous)
2007-12-15 02:59 pm UTC (link)
very interesting, but I don't agree with you
Idetrorce

(Reply to this)

Create an Account
Forgot your login?
Login w/ OpenID
English • Español • Deutsch • Русский…