Kernel panics on Leopard

Recently, since upgrading to OS X 10.5.3 it seems I’ve been experiencing some kernel panics, as much as once per day. After the last one I enabled kernel debug with the editing /Library/Preferences/SystemConfiguration/com.apple.Boot.plist and adding the following:

<key>Kernel Flags</key>
<string>-v debug=0x144</string> <—- ADD THIS

The -v flag is for verbose output during startup instead of the gray Apple screen. So once it kernel panicked again, it handily printed out the mac and IP address and stated it’s awaiting a remote debugger connection. On another Mac, download the 10.5.3 KernelDebugKit from ADC and fire up the debugger as follows:

gdb -arch i386 /Volumes/KernelDebugKit/mach_kernel

Then use gdb as follows:

(gdb) target remote-kdp
(gdb) attach 192.168.100.3
Connected.
(gdb) source /Volumes/KernelDebugKit/kgmacros
Loading Kernel GDB Macros package.  Type “help kgm” for more info.
(gdb) paniclog
panic(cpu 1 caller 0x002420F2): “ip_output: mix of packet in packetlist is wrong=0x577ab700″@/SourceCache/xnu/xnu-1228.5.18/bsd/netinet/ip_output.c:1494
Backtrace, Format – Frame : Return Address (4 potential args on stack)
0x5b7c7778 : 0x12b0fa (0x459294 0x5b7c77ac 0x133243 0x0)
0x5b7c77c8 : 0x2420f2 (0x475128 0x577ab700 0x0 0x0)
0x5b7c7918 : 0x24883c (0x577ab700 0x2 0x0 0x5fc9390)
0x5b7c7968 : 0x24a02b (0x2 0x0 0x0 0x1)
0x5b7c7a98 : 0x24732a (0x5fc9400 0x5b7c7c38 0x8e8544ca 0x364a8c0)
0x5b7c7c78 : 0x23bebd (0x57645f00 0x14 0xd7f47686 0x5358a4)
0x5b7c7cb8 : 0x23dc3f (0x57645f00 0x14 0x6 0x0)
0x5b7c7dd8 : 0x23dc7b (0x57645f00 0x0 0x6 0x21c5)
0x5b7c7df8 : 0x229bd3 (0x2 0x57645f00 0x5b7c7e48 0x141de3)
0x5b7c7e38 : 0x218be2 (0x2 0x57645f00 0x5b7c7e68 0x63a5fb58)
0x5b7c7ec8 : 0x215321 (0x671ae04 0x2 0x57645f00 0x3e95bc)
0x5b7c7ef8 : 0x215a7a (0x5278e4 0x5d71988 0x2 0x5b7c7f74)
0x5b7c7f68 : 0x215cd8 (0x0 0x57645f00 0x0 0x64f762c)
0x5b7c7fc8 : 0x19ebdc (0x64f7604 0x0 0x1a20b5 0x608e4f0)
Backtrace terminated-invalid frame pointer 0

BSD process name corresponding to current thread: kernel_task

Mac OS version:
9D34

Kernel version:
Darwin Kernel Version 9.3.0: Fri May 23 00:49:16 PDT 2008; root:xnu-1228.5.18~1/RELEASE_I386

I googled the output from the ip_output.c module and found several references, all indicating the Cisco VPN client’s kernel module and usually running a torrent client at the same time.

So for the moment I avoid using both at the same time and also removing the Cisco kernel modulel while not using the VPN:

[user@nefilim ~]$ sudo /System/Library/StartupItems/CiscoVPN/CiscoVPN stop
Stopping Cisco Systems VPN Driver
kextunload: unload kext /System/Library/Extensions/CiscoVPN.kext succeeded

So far so good. Sigh Cisco… won’t be the first time and certainly won’t be the last.

Resizing off screen windows in OS X (Leopard)

I run into this problem just frequent enough that I don’t remember what I did last time. Usually with Eclipse when moving back and forth between home and office (connecting to external monitor).

Here’s a whole bunch of tips. What usually does it for me is changing the doc position to the left (external monitor is on the right), zoom window and then resetting the doc back to the bottom.

H264 quality differences in players

I am surprised at the significant difference in quality by various players, well to be more precise, VLC & MPlayer. The difference between MPlayer & Quicktime (other than colourspace conversion?) is negligible. For comparing I tried to make sure I use the same frame, which is probably impossible but it seems pretty close and the trends I’ve observed holds true after many tries, QuickTime ~= MPlayer > VLC

Here’s a SD comparison, all 3 players in one screenshot for comparison:

Here’s a scene from 720p content, make sure you view this at 100% (which usually means clicking on the full size image in firefox), first MPlayer:

and then VLC:

and finally, both together for easier comparison. Look especially at the detail on the landing pad and the boat’s surface:

UPDATE: here’s another shot with XBMC (OSX port), looks to be on par with MPlayer (ffmpeg + OpenGL vs Core Video/OpenGL?):

ImageMagick on Leopard

Latest binary build for ImageMagick on their site is pretty broken, the actual binaries isn’t included, just the wrapper scripts!

Make sure you install both X11User and X11SDK, both from the Leopard DVD. Then install the latest X package from http://trac.macosforge.org/projects/xquartz/wiki/Releases

You might also need a symlink from /usr/X11 to X11R6. “fink install imagemagick” kept on asking me about system-xfree86 packages until I get all these things right.

Automount on Leopard

at last! Easily added with Directory Utility if you’re not a vim junkie:

Monitoring HDD temperature

While the Antec P180 case is a favourite amongst enthusiasts it really is a poor option for a RAID array, the bottom cage especially. It houses 4 drives and has a single fan (in my case the stock fan was replaced with a Scythe SFLEX fluid bearing fan for lower noise – the server & desktop under my desk are virtually noiseless as I like them – but that’s another post) in between the cage and the power supply, with limited space for a lot of cabling. The fan also have no grills on either side. I have already lost one 500GB Seagate drive (full of data) when a cable got stuck in the fan and caused the drives to overheat. The bottom cage has no ventilation at all when the fan stops working for one or another reason. It really should give you the ability to add another fan in front of the cage, like the the center cage. Perhaps I can jury rig something.

This prompted me to create a 3x500GB linux software RAID5, using all WD RE2 drives this time around. So far so good, until last night on a whim I checked hdd temperatures (as I do from time to time – call me paranoid) and found the 3 drives in the bottom cage sitting at 60C!! I dove under my desk and found that the bottom fan was not working, looks like the power connector got bumped out when I cleaned up a bit there with velcro ties recently after upgrading some hardware. No idea how long they’ve been running like that, could be up to a few days 😦 Short of replacing the case with some better suited, for the short term I thought some pro active monitoring was in order. I installed hddtemp with yum (no need to parse the response otherwise could have just used smartmontools instead) and whipped up a quick script which I run through cron every 15 minutes:

[root@gatekeeper bin]# crontab -l
15,30,45,59 * * * * /usr/local/bin/monitorhddtemp.sh    

[root@gatekeeper bin]# cat monitorhddtemp.sh
#!/bin/bash
HDDS="/dev/sda /dev/sdb /dev/sdc /dev/sdd"
HDT=/usr/sbin/hddtemp
LOG=/usr/bin/logger
DOWN=/sbin/shutdown
ALERT_LEVEL=40
SHUTDOWN_LEVEL=55
for disk in $HDDS
do
  if [ -b $disk ]; then
        HDTEMP=$($HDT -n $disk)
        if [ $HDTEMP -ge $ALERT_LEVEL ]; then
           $LOG "hard disk : $disk temperature $HDTEMP°C crossed its alert limit"
           echo "hard disk : $disk temperature $HDTEMP°C crossed its alert limit" | mail -s "HDD TEMPERATURE WARNING" your@email.here
        fi
        if [ $HDTEMP -ge $SHUTDOWN_LEVEL ]; then
           $LOG "System going down as hard disk : $disk temperature $HDTEMP°C crossed its critical limit"
           sync;sync
           $DOWN -h 0
        fi
  fi
done

This script will email me when any of the listed hdd’s temperature exceeds the alert level (40C) and shutdown the machine when they reach over (55C – manufacturer’s operating limit)

Eclipse Europa and Leopard problems

Often dies with a SIG_BUS especially when using the Open Type (cmd+shift+t) functionality. Bug report here

Easy fix, download latest nightly build, I used M20071115-1247 from http://download.eclipse.org/eclipse/downloads/

Untar and copy the two swt jars to your current Europa installation (eclipse/plugins/).

Macbook Pro wireless woes

I’ve had some intermittent wireless problems with both Macbook Pros, especially with WEP networks. Colleagues with Macbook Pros have had the same problems from time to time. I can see networks in my list but when I try to connect to them I just see the generic “An error has occured” message. Really infuriating when everyone else arounds you connects just fine. There’s usually something along the lines of:

Mar 23 19:30:57 xxxx /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport: Error: WirelessAssociate2() = 88001006 for network xxxxx. The operation timed out.

in /var/log/system.log

The Apple discussion boards are full of people having very similar problems, it’s really hard to understand why Apple is not stepping up to the plate and fixing these issues once and for all. After numerous Airport updates over the past 9 months we’re still not there. Perhaps the problem lies with Atheros, the manufacturer of the wireless chipset?

Either way, I think I’ve come up with a reasonably consistent way to overcome this problem. Unfortunately you’ll need Windows (yea, really) installed. Boot to windows and make sure you have the latest Atheros drivers installed (6.02.75 at the time of writing, I had 6.01.75) otherwise you will have the exact same problems in Windows as in OSX. With the latest drivers installed Windows should be able to connect to said wireless network without problem. Reboot back to Mac OSX and reconnect with Airport, which should work now. I suspect the wireless firmware or something is left in an unstable state and the Windows drivers knows how to correctly reset this. It’s worked for me on two occasions now where OSX did not want to connect.

UPDATE (2 May 2007): Apple releases Airport updates again: “This update is recommended for all Intel-based Macintosh computers and includes compatibility updates for certain third-party access points configured to use WPA or WPA2 security.”. Hopefully this fixes some of the above problems. Interestingly enough a battery update, that will update each battery when inserted, is included as well. Time to charge my spare battery!

Budget reconciliation … UNIX style

Alright this is as geeky as it gets. Unexpected rain kept us from DNA Lounge tonight so I decided it’s a good time to look at some budget reconciliation after a prior discussion.

Bank of America online banking has a nice download feature that allows you to download a specific range of transactions in various formats. Fortunately one (Microsoft Excel format) is CSV. The description, amount and balance fields are ” delimited but not the date field. A quick one liner got me the results I wanted, most of the time:

grep -i "whole foods" march07.csv | sed -e "s/\"//g" | awk -F','  '{total+=$3;}END{print total}'

This worked great to get the total as along as there is no commas in the description field, which inevitably there were for some lines. Next stop Perl, some very, very rusty Perl at that:

#!/usr/bin/perl

$file = $ARGV[0];
$keyword = $ARGV[1];
print "searching $file for $keyword:\n";
open(FILE, $file);
@lines = ;
$linecount = 0;
$sum = 0;
foreach $line (@lines)
{
        $linecount++;
        $line =~ /(\d\d\/\d\d\/\d\d\d\d),\"(.+?)\",\"(.+?)\",\"(.+?)\"/;
        $date = $1;
        $description = $2;
        $amount = $3;
        $total = $4;

        if ($description =~ m/$keyword/i)
        {
                print "$date -> $description -> $amount\n";
                $sum += $amount;
        }
}

print "sum = $sum\n";

So a typical invocation to see what I’ve spent on food and groceries in a particular month:

./keyword.pl march07.csv “whole foods|chipotle|satsuma|jitlada|tofu|albertson|trader|safeway|pluto|asqew|longs”

(Almost not a day passes without visiting Whole Foods… got to have that organic food, not great on the budget:)

or gas:

./keyword.pl march07.csv “chevron|shell”

you get the picture…

Load testing AJAX with JMeter

Testing normal AJAX (using DWR) connections with JMeter is reasonably straight forward. Note that if you are using Spring’s CharacterEncodingFilter and force UTF-8 (since JMeter does not set the character encoding), POST parameters will not be read by the DWR servlet, in this case either disable your filter or use a GET for your AJAX requests and keep the parameter size in check.

Setup a POST to login to your application (or whatever is necessary to establish a session).

GET dwr/engine.js, parse the response and bind the HTTP Session ID and scriptSessionId to JMeter variables:

And now to make an actual AJAX request:

To inspect your existing AJAX calls I recommend using a proxy such as Charles, Firebug doesn’t show you (or at least me:) the request parameters. Parameters to note are:

page:

where the click happened

batchId:

DWR seems fairly unbothered by this but I click through a sequence of events, setup my test plan to match that sequence and then use those batchIds (Charles’s sequence view is very useful for this)

c0-paramX:

Be sure to pass the required parameters.

Coming in Pt II … how to load test reverse AJAX connections. Watch this space.