Download the tag of the latest stable euca2ools release from github. Install virtualenv with 'sudo easy_install virtualenv' and homebrew following their instructions if you haven't already.

Using virtualenv and homebrew isn't mandatory. Homebrew makes it much easier to install many tools and libraries like swig, and virtualenv lets you keep isolated python environments.

brew install swig

mkdir -p ~/.virtualenvs/euca2ools
virtualenv ~/.virtualenvs/euca2ools
source ~/.virtualenvs/euca2ools/bin/activate

pip install boto
pip install m2crypto

# the filename will vary depending on what version is the latest
tar xzf eucalyptus-euca2ools-2.1.0-0-gb5c5889.tar.gz
cd eucalyptus-euca2ools-c5c7caa
python setup.py install

After this is done, you'll have your euca2ools in '~/.virtualenvs/euca2ools/bin'. In every new shell you start, you'll have to run 'source ~/.virtualenvs/euca2ools/bin/activate' before you can use them.

There are a lot of underappreciated monitoring utilities for linux. From time to time, I plan to dive into one and write up what it's useful for and how it works. I'll start with nethogs. Unlike most network monitoring tools, nethogs breaks down traffic per-proccess rather than per-host or per-protocol. Its focus is answering the question “What program is using my bandwidth?”. It provides a top-style interface for real-time monitoring. So how does it work?

On linux the file /proc/net/tcp lists all established TCP connections. It includes the local and remote socket addresses and the inode number for the local socket. Nethogs uses libpcap to sniff traffic and associate it with its entry in /proc/net/tcp. It takes the inode from there and scans through /proc/*/fd/ looking for the file descripter that has that inode to determine which process has the socket open. Once it finds the process it adds it to a table of inode to process id mappings so it doesn't have to scan through /proc again the second time a packet for that connection comes through.

Stale environment variables, particularly those related to SSH agent and X11 forwarding, are a well-known problem for GNU Screen users. Search google and you'll find many solutions to this problem (including one from my fellow Athenian Sam Rowe), but I wasn't happy with having to manually run a command to fix each screen window. Thus, I whipped up a script I call rescreen.

Rescreen depends on two tricks to automate fixing the environment variables to match your new SSH connection. First, it uses screen's setenv command to set the environment for any new screen windows you create. Second, it uses a signal handler to have bash processes running in your existing screen windows alter their environments.

To start using rescreen, you need to download it into your $PATH and then place the following in your bashrc.

trap "if [ -f "$HOME/.rescreen_environment" ]; then source $HOME/.rescreen_environment; fi" SIGUSR1

Now you can attach a detached screen session by running rescreen with a single argument, the session name. SSH_CLIENT, SSH_TTY, SSH_AUTH_SOCK, SSH_CONNECTION, and DISPLAY will be reloaded if necessary. If you need to change more variables, just add to the list starting in line 11. If you left a screen session attached, you'll have to detach it with screen -d sessionname before using rescreen. You could alter rescreen to do this automatically, but I prefer to think about whether I really want to “call the Dr”.

I've added a script, export_google_site_news, to my catchall repository on github. It's for downloading the section of a Google Sites hosted webpage generated by the News Gadget. It takes two arguments, the address of your site and the name of the news section. It will download all the news stories it finds as html files into your working directory.

For example, to download the news stories at https://sites.google.com/a/medinacommunityband.org/www/announcements, you would run

export_google_site_news \
https://sites.google.com/a/medinacommunityband.org/www \
announcements

When Free It Athens moved our website from Google Sites to Drupal, we started from scratch rather than importing our old content. I realized on Wednesday that the news posts on the site were interesting historical information, yet I'd never archived them.

First, I tried a recursive wget.

wget -r --no-parent --no-clobber \
"https://sites.google.com/a/freeitathens.org/foo/news"

This failed to work because Google pointlessly used javascript rather than anchor tags to link between the news listing pages.

Next I found and tried to use the google-sites-export tool from Google's Data Liberation Team, but I was never able to authenticate succesfully from it.

At this point I was worried I'd need to use a tool like Selenium to run the javascript, but then I realized the news listing pages took a single paramater to determine how far along in the pagination they were. It wouldn't take more than a C-style for loop to download them all.

for i in $(seq 0 10 120); do
    wget "https://sites.google.com/a/freeitathens.org/foo/news?offset=$i" \ 
"-Onews.$i"
done

After doing that, I looked at the first one and determined a pattern that would match the relative URLs of individual news stories. I then extracted all the URLs.

grep -E -h -o '/a/freeitathens.org/foo/news/[a-z0-9\-]+' news.* | 
sort -u > news_links

Once I had the list of URLs, it was simple to have wget download them all.

wget -i news_links -B https://sites.google.com

Since I didn't find any other guides to doing this, I decided to flesh out what I'd done into a simple tool and write about it here.

Dell has several methods for updating their BIOS.

Out of all the different approaches, this is what I cam up with that successfully let me install the latest BIOS without taking a trip to the data center.

• Install smbios-utils from EPEL
• Download the BIOS package from the Dell website, here is an example
• Rather than running what you downloaded, just extract the files, e.g.

  ./PER610_BIOS_LX_2.1.15.BIN --extract bios_files

  would extract the files to a directory named bios_files

• The actual BIOS image is in the subdirectory named payload. Install it using the dellBiosUpdate program, e.g.

  dellBiosUpdate -u R610-020115C.hdr

• Reboot

A GUI that's a part of the Lifecycle Controller. I think you get to it by pressing F10 for System Services during boot.

• Able to update all firmware on the system
• Fetches the latest version
• No dependencies on installed OS

• Must go to data center since console redirection does not work with it
• Extensive downtime as updates are downloaded

Dell's Drivers and Downloads page has BIN files you can download and run from within linux to update the BIOS. These are a combined shell script and archive. In theory, you just run them and the new BIOS is installed.

• Can always get latest version

• Must manually browse site and identify proper download
• Only updates the BIOS, not other firmware
• In my experience, this does not work. A report of the kind of error I ran into and some troubleshooting steps are here.

Dell officially maintains and supports a yum repository with rpms for their firmware and some tools for installing it.

• Able to update all firmware on the system

• Has software that is also packaged in EPEL, meaning you must assign a higher priority to this repository in your yum config
• The repository is not kept up to date with the latest fimrware. It seems that it's updated in batches every few months.
• In my experience, this does not work. See these error reports over a period of months where the R610 BIOS update fails to install.

Matt Domsch from Dell maintains an unsupported yum repository just with BIOS updates. You need to install some other tools from Dell's community repository (or maybe smbios-utils and firmware-addon-dell from EPEL would be enough, haven't tested this yet) to use it.

• Should always be latest version

• The script that generates the repository is fragile and frequently breaks
• Doesn't seem to have all the BIOS I need, e.g. it didn't find any updates for my R610s

Update: vwaelchli on reddit points out that you can also update the firmware by running wsman commands from the lifecycle controller.

When new software packages are released, you don't want to blindly apply them to all your systems. You should have a test environment, or at least a set of less-critical systems, where you test things first. If you normally stick to a consistent update schedule, such as once a week, you can use the date a package was created to determine whether it should be updated in your production environment. I couldn't find an way to do this directly through yum, so I created a shell script named update_production to do this. As written, it applies all updates, except for packages released to CentOS or EPEL in the last week. It's designed to be easy to tweak the repositories you delay updates from or how long the delay is. Ideally this functionality should be available as a plugin to yum, but that looked much more complicated to develop.

If you accidentally set up a server with the wrong software raid type, as long as you used LVM you can use mirroring to temporarily move the filesystem off the RAID device, recreate the RAID properly, then move the filesystem back onto it. Needless to say if things go wrong during this process you'll be reinstalling. Below is an example where md1 on sda2 and sdb2 is being converted from RAID0 to RAID1, and an iSCSI volume serves as the temporary storage. In it VG is the volume group name. LV is the only logical volume being moved, and DEVICE is the device linux creates for the iSCSI target. If doing this yourself, you'll want to frequently check the sanity of what you're doing with commands like 'pvs' and 'lvs -a -o +devices'. If feasible, you can save a lot of time by turning off swap, removing its logical volume during the conversion, then recreating it afterwards.

pvcreate $DEVICE
vgextend $VG $DEVICE
lvconvert -m 1 --mirrorlog core ${VG}/$LV $DEVICE
 
# repeat these two steps for every LV on the VG
lvconvert -m 0 ${VG}/$LV /dev/md1
vgreduce $VG /dev/md1
 
mdadm --stop /dev/md1
mdadm --remove /dev/md1
mdadm --create /dev/md1 --level=raid1 --raid-devices=2 /dev/sda2 /dev/sdb2
mdadm --detail --scan >> /etc/mdadm.conf
pvcreate /dev/md1
vgextend $VG /dev/md1
lvconvert -m 1 --mirrorlog core ${VG}/$LV /dev/md1
 
# repeat these two steps for every LV on the VG
lvconvert -m 0 ${VG}/$LV $DEVICE
vgreduce $VG $DEVICE
 
mkinitrd -v --with=raid1 /boot/newraid $(uname -r)
# make the first entry use your new initrd
vim /boot/grub/grub.conf

I recently had a problem on CentOS 5 where the best solution I could think of required running a command when a network interface is brought up. I'm used to having this functionality as a part of the networking scripts on Debian-derived distributions, but I discovered that Red Hat-derived distributions lack it. I've written a small daemon called nethook to provide it. You can find the code and documentation for nethook on github.

Update: Moved project from google code to github.

If you are using virt-install to create KVM virtual machines on RHEL or CentOS 5, be sure to specify the '–accelerate' option. If you don't use accelerate, virt-install starts '/usr/bin/qemu-system-x86_64' rather than '/usr/libexec/qemu-kvm'. This isn't what you want, and it will fail with the error message “internal error Domain $YOUR_VM didn't show up”. What happened in the background is that libvirt set the machine type to rhel5.4.0, which lets qemu-kvm know it can use virtio, but this machine type is not understood by qemu-system-x86_64. If you check '/var/log/libvirt/qemu/$YOUR_VM.log', you'll see a detailed error like

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/ /usr/bin/qemu-system-x86_64 -S -M rhel5.4.0 -no-kqemu -m 512 -smp 1 -name $YOUR_VM -uuid a3e38d9d-d958-0da1-4b3a-57179ee04f29 -monitor pty -pidfile /var/run/libvirt/qemu/$YOUR_VM.pid -no-reboot -boot d -drive file=$YOUR_INSTALLER.iso,if=ide,media=cdrom,index=2 -net nic,macaddr=54:52:00:76:5f:f4,vlan=0 -net tap,fd=36,script=,vlan=0,ifname=vnet10 -serial pty -parallel none -usb -vnc 127.0.0.1:10 -k en-us

Supported machines are:
pc Standard PC (default)
isapc ISA-only PC

If you receive the following messages when starting collectd

network plugin: Option `SecurityLevel' is not allowed here.
network plugin: Option `AuthFile' is not allowed here.


It is because you configured collectd to sign or encrypt its communication, but collectd was not compiled with libgcrypt support. This is true of the version in EPEL.

$ yum info collectd | grep Version
Version : 4.10.2
$ ldd /usr/lib64/collectd/network.so
libpthread.so.0 ⇒ /lib64/libpthread.so.0 (0x00002b620d29d000)
libdl.so.2 ⇒ /lib64/libdl.so.2 (0x00002b620d4b8000)
libc.so.6 ⇒ /lib64/libc.so.6 (0x00002b620d6bc000)
/lib64/ld-linux-x86-64.so.2 (0×0000003223400000)


To solve the problem, either disable signing and encryption or rebuild collectd.