Recently in UFies.org Category

In case anyone has thought that here is a good place to get info on why UFies.org is maybe having some problems have hit a jackpot! Unfortunately, I don't know all that much. There was a network "incident" this afternoon where we were down completely due to a switch being broken at the hosting company. Currently the nameservers that UFies uses seem to be down, so I'm re-routing things so that mail can start going properly again. Oh, and their support number voicemailbox is full, so I'm guessing I'm not the only one to be reporting this :)

Will update when there are updates.

Update #1: Looks like there are still routing issues, which is causing DNS resolving issues (even after putting in the openDNS servers to use). So my guess is that the hosting company's network is a bit FUBAR right now, and there's not much I can do about it on my end here. Hopefully this will resolve itself soon.

Update #2: And like magic, everything is fixed in the morning. Looks like the hosting company's router is back up and going.

UFies.org is currently down due to co-lo provider issues. Nothing I can do :( When it's up, it's up.

Probably another rogue backhoe.

Update 12:00 - Back up now...

Slashdot brought up a story called Gentoo On Server Considered Harmful, a note on a blog entry by Cuzimatter on his experiences, both good and bad, with Gentoo Linux. One of his assertions was that Gentoo was too risky for servers:

The best way to keep a system stable is to get it working and then not changing anything. This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge.

A day after a rebuttal came out :)

Jumping Ship
Anyway, this is about me, so here's my story. I've been running Gentoo on my Linux Desktop since 2002 when it first crept onto the scene. The attraction to me at the time (over Debian) was that it was far easier to manage as far as different software versions, USE flags, and being able to go back to previous versions if something went wrong. Also, at the time anyway, the Debian version scheme of stable (2 years out of date software), testing (not as often updated with fixes) and unstable (bleeding edge, sometimes b0rks things for no reason) had bit me a couple of times too often.

So I jumped.

I updated my desktop machine first though, partially because at the time it was the first distro with the new GNOME (or maybe KDE) desktop available on the day of release, and you could install it all by simply typing "emerge kde" as root. Anyway, that was more attractive at the time than dealing with finding sources lists, incompatible versions, one guys archive of .debs conflicting with another, etc. Nowhere near the mess of RPM based distros (IMHO), but for folks who wanted to get the latest and greatest.

Using Gentoo on the desktop made me like it even more, so when I redid my main server in 2006, I decided to go with what I knew, and installed Gentoo on it.

Getting Bit
As the original article states, Gentoo's philosophy is to update everything, so it's very possible that an update to package "D" will have dependancies on "C", "B" and "A", and you may not want to update "A". When you do a full system update it's possible that the packages you don't want updated will be mixed up in the list of files that are going to be updated and you may end up sitting there not long after with a brand new version of MySQL or "PostgreSQL" that you weren't expecting. This happened to me not that long ago when I updated without paying attention.

The opposite of the "update everything" philosophy is that you don't have to, you can just update specific packages. The issue with this of course is because Gentoo is all compiled packages, updating, say, "libdb" to a brand new version could render many programs unrunnable. Sometimes this can go unnoticed for a while because when linux loads programs it loads the libraries into memory (or something like that) so that a running program (or it's dependant libraries) can be replaced unnoticed while it continues to happily run.

Until of course you restart the service and find that it fails to start up with a "missing library" error. This has happened to me too.

I have always been very careful with updating updated configuration files "etc-update", so while I see the issue with the potential of overwriting your carefully crafted configuration files, I haven't been hit with it yet.

So Why Update?
Why is this an issue of course? Well, Gentoo is supposed to be bleeding edge so there are updates constantly. This is great for people who have their desktop environments compiled out of CVS and who have to have the absolute latest, but this isn't good for servers. Servers really should only be updated for major security issues or hardware updates, not at a constant rate. Again, fantastic for those jonesing for the latest Tomboy release, not good for the datacenter with lots of users.

To Gentoo or Not to Gentoo?
Would I do it again? Good question.

On the desktop I think it's great. Issues can be fixed easily enough if you know what you're doing (a requirement it seems of using the distro), and if you're down for a day recompiling X and all of GNOME to get the new stuff that day or cause someone decided to update a library that forced everything else to be updated isn't the end of the world. It sucks, but if you can live with that to be on the bleeding edge, more power to you. Heck, that's why I'm there.

On the server, probably not. I've matured in the last 4 years of so as a sysadmin, and understand the value in not screwing with what works. I'm also working as a sysadmin right now in a prominently Debian shop, so I'm learning some of the tricks that I didn't know when I was using Debian ages ago. My original complaint still holds though, there's lots of way outdated packages in Debian, and RPM still sucks :)

Debian stable is probably how I'd go these days and just live without some of the new wizbang stuff, most of which is for the desktop anyway. Stability is key I'm learning, and now that I'm getting more and more people on the server, I'm appreciating that all the more. Question of course is how the heck to move back now :)

New hardware is coming along. Other than sounding like a 747 taking off, it's a sweet box. Sadly the default Gentoo install CD didn't include all the right drivers needed to boot up properly, so I burned an Ubuntu install CD and plan to do a standard Gentoo install via chroot from a minimal install.

Well, last night I headed out and met up with Fred for dinner and to collect the new UFies.org server. Good to see Fred again as well, caught up with each other (damn you for moving so far away!). The server is gorgeous.... a 2U, dual hyperthreaded 64 bit system with 6 gigabytes of RAM and tons of 15k SCSI drives. The system also has a management console thingy that is basically a built in KVM over IP, which means I don't need to be scared about the system booting slowly and wonder if it's just doing a disk check or is hung, which'll take a load off my mind. Also of course everything is redundant and hot-swappable. So the next couple of weeks will be testing, configuring, playing with different filesystems and EVMS and various other bits. We're looking at a February install sometime, which will be interesting.

Below is a list of the things currently not working on UFies.org (which came up just fine it seems, many thanks to cat5 and Fred).

• /home copied over
• /home permissions Pretty sure this is ok now
• user accounts accessable
• permissions tweaked for blog users etc Far as I can tell anyway
• apache
• suexec support
• tdiclub site
• ufies.org site
• peer2peer personals
• xfmail.org
• postfix (mail)
• webmail
• webmail sending without errors
• https
• jabber
• rdiff-backup
• local scripts
• pop-before-smtp
• dns
• mailman
• postgres
• iambe comments
• mysql databases
• vnstat
• ipaudit
• bigbrother
• bogofilter (You may have to run bogoupgrade to get things to work properly though)
• crontab All previous crontabs restored (some minor edits on stuff that is old, check your own if you're unsure) and all previous crontab users added into the "cron" group. Looking for a way to allow all users though.
• Server side includes
• Supermicro stuff - Nope, won't work until they support 2.6.x kernels :(

I'll check 'em off as things get done. Or think they are done.

Tomorrow is the big day. The system that's been sitting (hulking is more like it) on my desk beside me, towering to the roof, the system that has had it's drives bashed and broken and plundered, whose newly installed OS has been tested, retested, and then tested some more, will find out if it makes the cut. Tomorrow it will be swapped in as ufies.org and hopefully all will go well.

If errors start popping up in a day or week, and if they're hardware related, I will simply unplug it, get a @hotmail address, find a job where I can die a ripe old age without ever seeing a computer again, and be happy about it.

So a note to all those who read this who are hosted on the box, do your backups, just in case, m'kay? The data won't be copied, moved or manipulated, but better safe than sorry.

The only thing I'm worried about is the order that Linux brings up it's raid arrays. The current system has three RAID arrays, md0, md1, and md2. The new UFies.org set up has three raid arrays, all named the same. Hopefully the kernel will bring them up in the right order, that is, the order I want them to come up in. There's really no good information that I could find out there on this :(

The kernel is told to load /dev/md0 as the root drive, and if it can find the right "md0", the rest doesn't matter. The system can run without /var or /home until I can come in and rename thing. However, if it loads up the old md0, I'll have to get Fred and Cat5 to play with GRUB boot commands until they can find the right one. Again, not a huge deal, but a potential PITA. Of course, after they find it and boot up on it I have to make sure that it stays there properly. I'll make sure I pass along a LiveCD for them just in case they need to fix things on the fly.

So yea, tomorrow, late afternoon, between 3:30-5:30 or so is when it'll go down, when it comes back up no on knows, but if you can't log in don't worry about it too much, just try again in a bit.

Over lunch today I finally resolved the last blocker issue on the newly set up UFies.org box. Everything went in fine but I found while doing some testing on a site with large database tables that performance sucked! The problem could be narrowed down to a query which takes 0.1s on a heavily loaded debian unstable system running on an XP1800 with 1G of ram was taking 10-20s on an unloaded dual P3-1.13Ghz gentoo system with 2G of ram. I recompiled MySQL with different options until the cows came home, even tried the binary distribution from the webpage, with no changes in the performance problems...

Update: Looks like the indexing issue is working properly again in 4.0.17. So 4.0.14 good, 4.0.16 bad, 4.0.17 good. Odd.

Things are really close now. I've basically spent the last few evenings and mornings testing, installing, configuring, bashing my head against the desk, and repeating. However, things are done as much as are needed at this point I think. I have mail, apache and the databases going pretty much as much as I want to without the system being up "for real."

So it looks like all my problems were caused by a buggy kernel or buggy drives in the boot CD. I feel like such a dumbass for not trying a different CD when I was there. I booted up on a Knoppix CD and had full speed on IDE on the motherboard and no problems doing things on the SCSI. I started building the new OS last night and it's busy doing a stage 2 install as I type this. Doing it nice and fast as well I might add, SMP kicks ass!

There's still lots of work to be done once the base system is installed, installing all the apps, finding all the perl modules, reconfiguring apache, mailman, mysql, etc etc etc to get things back to a usable state. Probably another couple of nights getting things going, and then, the hardware, software, OS and every other deity and minor deity out there willing, a working system. If I'm lucky I can get a system that works for the majority of the time back in before next week (*knocking crossed fingers on wood*), in at least enough of a state that only little tweaks and corrections are needed.

In theory....