Using grep in vim

One of the things I like about Vim (and vi) is the ability to invoke Unix utilities to manipulate text in ways that might be hard or impossible with just the regular editor commands.  It's definitely written with the Unix Philosophy in mind.
One thing I do frequently while debugging problems is to add log messages with a distinct pattern so I can find them easily in the log file.  For purposes of this example, let's assume the pattern I use is XYZ.  If I open the log file in Vim, I can issue the following command to isolate just the lines in the log file which contain XYZ.
^[ggVG!grep XYZ^M
That looks like a pretty complicated command, doesn't it?  But if we examine it piece by piece, it's not really that bad.
At the beginning of the command we've got ^[ which is the escape key.  Look at this ASCII chart if you're not familiar with the caret followed by a letter shorthand for control characters.  I issue the escape key to make sure Vim is in command mode.  While we're talking about control characters, the ^M at the end of the command is shorthand for the carriage return (AKA the Enter key).  That causes the command to be executed.
The ggVG in the command serves to do a visual selection of all the text in the file.  The gg causes the cursor to be placed at the first line of the file.  The V invokes visual marking of text.  And finally, the G causes the cursor to be placed on the final line of the file.  That causes the marked area to contain all the lines of the file.
The real meat of the command is the next part - !grep XYZ.  The exclamation mark pipes the marked text to the external Unix utility which follows which is the grep command.  This particular command line searches for lines which match a pattern of XYZ.
Issuing this command will cause Vim's current data buffer (the full contents of the log file) to be replaced with the output of the external Unix utility which will be just the lines within the log file which contain the pattern XYZ.
That makes it really simple to isolate just the log commands I've added.  Once I'm done, I can either exit Vim without saving or just issue the u command (undo last text manipulation) to leave the log file untouched.

Unix tool: xargs

One of the reasons I like Unix style operating systems so much is the Unix Philosophy.  One of the principles is it's better to include a bunch of small, fast tools which can be combined together to accomplish a variety of tasks than it is to build large special purpose tools which are complicated to use.  One of my favorite of these Unix tools is xargs.  What xargs is good at is taking a bunch of separate lines of input and changing those into arguments for another command.
Perhaps an example will serve to illustrate better than a dry explanation.
Let's imagine we want to find all the source files from the current directory (recursively) which contain the string "stdio.h" and to edit each of those files using vi.  The following line will accomplish that.  Of course in Unix, there are numerous ways to accomplish the same task.  This happens to be my favorite way to perform the task and serves to illustrate the use of xargs nicely.
find . -name "*.c" -print | xargs grep -l "stdio.h" | xargs vi
The first part of the command (that before the first pipe character) is a simple find command.  The only thing worthy of explanation is the quotes around the file pattern.  Unix shells will substitute matches for a wildcard such as this before the find command gets invoked.  If we invoke this long command in a directory containing files which match the pattern *.c, the matches would be substituted on the command line instead of the actual *.c pattern.
The second part of the command is a simple grep command but combined with the xargs command.  This passes the file names output by the find command to be used as arguments to the grep command.  The grep command is looking for files which contain the string "stdio.h", a common c library header file.
The third part of the command simply takes the matching files found by grep and passes them to vi.
This example should work on either Linux or Mac OS X.  It will also work on Windows provided you've installed a Unix environment such as Cygwin.  Cygwin is one of the first things I install on any Windows machine I have to use on more than a casual basis.
Play around with the xargs command to get a feel for what it can do.  It's a handy part of any Unix tech's grab bag of tricks.

Portable tools

Having written software on a variety of development systems including Sun workstations, Windows PCs, Linux PCs, and Macs, I've never regretted the decision to choose easily portable tools for use in my daily work.  By doing so, I learn a tool once and am able to use it regardless of what type of development system I am given at a new company.  I also have a preference for free software.  In the past it's been difficult if not impossible to get companies to pay for software tools.  Having a ready arsenal of free tools for any system I use is a huge bonus.
My editor of choice these days is one of the GUI versions of Vim (MacVim at home and GVim on the Windows PCs at work).  Besides the obvious enhancements which either version of Vim adds to the standard vi commands, the developer who started the project, Bram Moolenaar, runs the project as charity-ware with donations going to help children in Uganda.  I've got no problems donating to such a worthy cause.
I continue to use vi clones because vi was available on most of the systems I've used over the past 20+ years.  Using PCs in the early days, I chose vi over emacs because early ports of emacs were extremely resource intensive.  While vi would fit easily in a 40 KB .COM file which could be executed from a floppy disk on base PCs, the emacs .EXE file (necessary because it required more than 64 KB of code space) typically required a PC with an expensive EMS board.
Were I to be starting out these days, I believe I'd probably choose to learn UltraEdit or SlickEdit since both have Linux, Windows, and Mac versions.  Both editors also have a much less daunting learning curve.  At best, vi commands are perplexing to new users.  You really need to understand the history of vi to grasp why the commands are designed the way they are.  In a nutshell, software in the early days of Unix had to support a variety of dumb terminals.  Commands had to be formed from the common subset of keys available on all terminals.
If you combine Vim with Exuberant CTAGS, you have a very powerful programming editor which makes it easy to learn and to navigate unfamiliar source code.
I've also chosen the Bash shell as my daily command line interface.  It's available on Linux and Mac.  It can also easily be added to Windows systems as part of Cygwin.  Bash is much more powerful than the standard Windows command shell.  I've heard good things about PowerShell but since it's only available on Windows, it's not really an attractive option for me.
Aside from the tools above, I'm fond of AWK and SED.  Used either separately or combined with their Unix based brothers like xargs, find, and grep, they give an nearless endless variety of ways to manipulate text files.
I've got one last recommendation.  There's a great grep replacement for software developers called ack.  It makes it easy to restrict the files searched to just source files.  You can do the same thing with grep but the command line gets very complicated.

Long hours

For years I've been subscribing to Jack Ganssle's excellent newsletter, The Embedded Muse. It's a must read for anyone working with embedded systems. It's also very good as a general resource for engineers who work with any type of hardware or software. If you're a software or hardware engineer, I urge you to take a look at a few back issues and then sign up for the free newsletter.

 The issue which arrived today contains an interesting discussion about the number of hours engineers are asked to work. I fall heavily into the camp which questions the wisdom of working more than 40 hours. Lest you think I'm a slacker, consider that I worked for 8 straight years at startup companies, and I probably averaged 55-60 hours per week during most of that period. I can speak from personal experience that the weeks with extremely long hours were not nearly as productive as those where I worked 40-45 hours.

Of particular significance are the stories where upper management gets upset when they find no one in the lab after hours. Whenever you find upper management in the lab after hours, that's also a result of poor management. Good managers will be able to communicate with line managers without checking up on them. Good upper managers will also be sufficiently removed from day to day operations that they realize they may not have the entire picture. Employees connected to lab equipment from home are awfully hard for anyone to detect. If you think that you can only achieve results by having a negative impact on the engineer's personal life, I can safely say I don't want to work for you and I suspect most experienced engineers would say the same.

 I would suggest that any project where management asks for a long term commitment of long hours is a direct result of inexperienced or poor management. Any manager with a reasonable amount of time in the industry knows that while you may get a delivery out with a short burst of concentrated effort but if you ask for it over a period of months (or more), you're going to get a product which requires more and quicker maintenance releases thanks to the mistakes even the best engineers make when they're overtired. You're also asking for a large turnaround in your engineering team unless you're able to offer some form of extreme compensation (large bonuses and/or stock options).

For young engineers, I would advise that you consider what you're getting out of any company which is asking for extreme time commitments. If you're getting some form of financial compensation or some form of experience which will prove extremely valuable in the future, I'd say go for it although I'd get it in writing if at all possible. Otherwise, I would suggest that you update your resume and start looking around for a company whose management might not be so out of touch with reality.

Golden age of nerds

It suddenly struck me today that nerds today have it much easier than nerds of my generation. It's easy to feed unconventional tastes in books, comics, music, TV, movies, food, beer, etc than it's ever been before.

It struck me as I was having a very enjoyable but quiet lunch by myself (thanks to a last minute cancellation by a friend who wasn't feeling well) at my favorite craft brew establishment. In my youth, I would have never considered dining solo but luckily my iPhone affords me ample entertainment for such occasions so there's no need to skip an tasty lunch. I have more media than I could consume in several weeks at my beck and call.

In my misspent youth, nerds were forced to read a wide range of magazines and visit geeky bookshops run by knowledgeable staff to keep up with upcoming media they might find of interest. Proximity to a great bookshop could spell the difference between keeping in touch with new happenings in your slice of the nerd universe and playing a perpetually unsatisfying game of nerd culture catchup. Fortunately I was close enough to a couple great bookshops that on weekends I could convince my Mother (who instilled the love of reading in me) to drive me to one or the other to spend my hard earned paper route money on another stack of books.

Now I can accomplish the same thing with a few subscriptions to e-magazines, email newsletters, RSS feeds, and podcasts. Ordering said items of nerdy desire are usually just a matter of pointing at the correct website and furnishing a valid credit card.

I chuckle when I hear geezers (not that I'm not one myself) bemoaning the death of paper books, magazines, newspapers as well as vinyl as a storage medium for music). They're missing the point that the artist's message is becoming easier than ever for the public to consume and frequently without the unnecessary middlemen who have all too often served as unwanted arbiters of what the public was ready to accept.

I for one embrace this golden age of nerds. Let the mass consumption begin!

Advice for the family tech support person (or mechanic)

An acquaintance of mine has been roped into providing computer tech support for a family friend and is entirely too polite to complain about the situation to the correct people to prevent a reoccurrence. I've got friends who are mechanics or at least very mechanically inclined who face similar problems in their field.

Hearing about the headaches associated with this has brought back memories of being in similar situations. Once people realize you have expertise in computers, or any other field for that matter, you may find yourself getting pressured into providing computer or car maintenance assistance. It has made me wish there were a tactful way of establishing guidelines for providing help to friends and family members. What better way than an anonymous blog post you can point to as a gentle suggestion of how to behave when asking for help?

1) Use Google to see if you can't solve your own problem or at least narrow down the possible causes. You'd be surprised at how many problems can be solved without intervention from experts.

2) Just because your friend or family member is willing (or at least too polite to refuse) to offer you assistance, don't assume that he or she wants to extend the same courtesy to your friends or coworkers. Don't volunteer them to do so without asking first and take "no" for a answer gracefully. Just because they're good at their job, don't expect them to be thrilled about volunteering extra hours above and beyond their normal work week on related problems.

3) Should your indentured tech support servant be kind enough to agree to help, don't make the imposition any worse by being picky about where or when the service should be offered. No one enjoys being stuck in a cramped back room working on a computer or on a cold, hard driveway working on a car. If they ask you to bring the PC or vehicle to their place, do so cheerfully. If they do agree to come to your house, clean up a bit to make service less painful. I can't tell you how many times I've had to ask for vacuum cleaners to clean off fan vents sufficiently to make disassembly possible without a big mess. No one enjoys working with a jumble of cables or surrounded by so many knick-knacks that it's difficult to get to the computer. Be flexible about time too. Remember, your potential tech support person has a life too. Don't interfere with work or their other commitments.

4) Find a way to reciprocate and make sure it's something of comparable value. If your tech support person has spent 2 hours fixing your computer, don't let a plate of cookies be your only thanks for their efforts. Think about some skill you have that they may need and offer that, be it mowing their lawn or taking them to dinner. Remember how much you would have spent at the local big box electronics store.

5) If the support you're asking for is advice or if advice is offered as a method of preventing future problems, don't blithely ignore it. You've asked this person for help because of their expertise. If you ignore it or instead follow advice from some less experienced friend or coworker or even worse, fall prey to some slick advertising or advice from one of the poorly trained hourly grunts at your local big box electronics store, how likely do you think it is that your tech support person will ever give you meaningful advice again?

6) Do whatever preventive maintenance is recommended to try to prevent future problems. No family tech support person wants to hear your plaintive cries about how you can't afford to lose some valuable file. So do your backups (or oil changes in the case of cars) and don't expect your family tech support to perform miracles on neglected PCs or cars.

Following these steps will help ensure less resentment on the part of the person helping you.

iPhone/iPod Touch databases

One of the many things I use my iPod Touch for is to store databases of information I want access to on the go such as books I have, books I'm on the lookout for, car maintenance histories, etc. Bento fits the bill nicely for my purposes and there's a Mac version which allows you to synchronize your databases between a Mac desktop or laptop and your mobile device.

It's a full featured database with optional field types such as check boxes, choice boxes, text, dates, numbers, and other options. It makes data entry very easy which is important on a mobile device. One of the things I like about Bento is it's one of the few inexpensive lightweight single user databases available these days. In the old days there were lots of database choices for users on a budget.

I read today that there's a version of Filemaker Go for the iPhone/iPod Touch/iPad. It synchronizes with Filemaker Pro which is available for Mac or Windows. Filemaker Pro finally makes it easy to sync between a mobile device and Windows. The upside is it's a more powerful database than Bento is if your database requirements are a bit more advanced than Bento's feature set. The downside is it's more expensive than Bento.

So you've got another database option if you're a iPhone/iPod Touch/iPad user. It's good to have choices, eh?

iPod Touch update

I'm relieved to report my enthusiasm for the iPod Touch as a replacement for the long line of Palm PDAs continues unabated. This little device continues to amaze me.

I managed to find this very useful article which helped me get secondary calendars in Google Calendar syncing with the iPod Touch. That was really the last piece of the PDA puzzle I needed to solve.

Tonight I downloaded the free Apple Remote application which allows me to control our Apple TV from the iPod Touch. It's not earth shattering but rather a bit of fun with a device which is already ranks among the best technical purchases I've made.

I typically find several reasons to be dissatisfied with any new gadget after a month of using it. Not so with the iPod Touch. It makes using a PDA type device fun again. I recall feeling this sense of enjoyment when I first started playing with most of the new Palm devices I've owned (aside from the Centro which was horribly disappointing).

By the way, the iPod Touch serves as a very nice music and video player in addition to its stellar performance as a PDA. It also works nicely to do some light web browsing at night. The mobile version of Safari is by far the most useful web browsing experience I've seen on a small device.

If you're in the market for a small device which can serve multiple purposes, investigate the iPod Touch. You won't be sorry.

Life after Palm

I've finally settled upon using my iPod Touch as the best PDA replacement device for my Palm. Having used Palm PDAs since I bought my first one at the employee price working for U.S. Robotics in 1996, I've come to rely heavily on a number of key applications to keep my life organized. Fortunately the iPhone application market has matured to the point that I could easily find replacements for all my "must have" applications.

First and foremost, I needed a way to sync my contact information from my Mac to the iPod Touch. Luckily, Apple anticipated that by adding that capability into iTunes.

It was also imperative that I be able to sync memos between my desktop computer and the iPod Touch. Mark/Space has a product called The Missing Sync which takes care of that problem for me.

It was also important that I be able to sync calendars between my desktop system and the PDA. Fortunately a combination of iCal on my Mac, iCal on the iPod, and Google Calendar was able to handle that thorny issue. This article on using CalDAV with Google Calendar and iCal helps explain the rather tricky configuration.

For a number of years I've been using a Palm application called SplashID to keep my passwords secure and yet easily accessible. The good folks at SplashData have created a version which runs on the iPod Touch. Using a new version of the same application made conversion very simple.

I've also grown reliant on having access to a simple database application on my PDA to keep track of things such as books I'm interested in reading as well as a number of other topics. I've used a great little Palm application called JFile for many years to fill this need. One of the best known names in Mac databases, FileMaker, has recently released a personal database application called Bento. They also have an iPhone/iPod Touch version available which can sync with the Mac version of Bento.

The last important thing I required on my PDA was a way to read books. I'd been using the mobile version of Adobe's Acrobat Reader on the Palm. I discovered that there's a free product called Stanza Desktop which is available for both Mac and Windows machines. They also have an iPhone/iPod Touch version which can sync books from the desktop version. Couple this with the fact that Google Books has announced that they have over a million public domain books available in the EPUB format (which Stanza supports) and my needs for a PDA book reader is more than met.

It's been a bit of a challenge but I've finally got all these applications installed and my data transferred. The only problem which remains is I'm still not able to see subordinate calendars from Google Calendars yet. Aside from that, I'm very happy with the new device. It's thinner, easier to read, and has better battery life than my Palm did. Plus it functions as a very good music player to boot.

e-audiobooks rock!

For a long time I've been listening to audiobooks to make commuting or exercise more bearable. Our local library used to have a great selection of audiobooks on cassette. The last time I checked, they were slowly transitioning to CD based audiobooks. I never borrowed a book in that form as they only seemed to stock recent bestsellers and I don't find most recent books as engrossing as some of the old classics by authors like Nevil Shute, Rudyard Kipling, and Robert Louis Stevenson among others.

Fast forward to a couple years ago. I discovered that our local library had started carrying e-audiobooks which could be downloaded to an MP3 player with support for Windows Media DRM aka protected WMA files). With mixed feelings, I purchased a refurbished MP3 player capable of playing protected WMA files. It irked me that I couldn't play this protected media on the iPod I used for most of my music. The MP3 player gave me a years worth of service before developing some interface quirks which became so bothersome as to be unbearable. It's hard to believe that with an iPod as a model of how to design a decent user interface, other manufacturers do such an abysmal job.

The refurbished replacement I purchased from another manufacturer was slightly better but still paled in comparison to the iPod's interface. It lasted me another year before refusing to connect using its USB interface without jumping through a strange series of steps I found on a user forum for a different model player from the same manufacturer. The prospect of spending more money on yet another unsatisfactory MP3 player didn't thrill me.

Fortunately, my library wasn't sitting idle during this time. They've recently made available a new service from Overdrive, the company which offers e-audiobooks for download. They offer an MP3 version of some of the e-audiobooks. Sadly this is a fledgling service at the moment so the selections aren't nearly as extensive as the WMA selections are yet but I'm hoping this will improve with time.

So far I've listened to Kipling's Captains Courageous and Stevenson's Kidnapped. Both were wonderful and made me choose longer paths when walking the dog so I could listen longer. If you've got activities which might benefit from distractions such as a good book being read by a skilled narrator, I recommend you check with your local library to see whether the offer e-audiobooks.

one blind mouse

My old bluetooth mouse, a Logitech which I really loved, recently developed a problem which caused it to skip erratically. Needless to say this served to annoy me greatly. I finally got fed up with the problem and being a bit reluctant to get another mouse which might fail prematurely, purchased a Microsoft Notebook Mouse 5000 (see image above) instead.

The new mouse is smaller than the Logitech which means it doesn't fit my hand quite as well as the Logitech did. On the plus side, at 1000 dpi the tracking is incredibly precise. That's a nice change from the way the old mouse had behaved recently. I'm pretty sure that's much better tracking resolution than the old mouse had when it was new.

One thing I don't care for is the scroll wheel which moves in a jerky fashion. I guess it's designed to keep you from scrolling too quickly and going past whatever you're trying to find. I prefer the smoother scroll wheels which Logitech mice typically have.

The manual estimates the battery life at 3 months on two AAA batteries. If it really lasts that long, that will be slightly better than the old mouse. I've got to get more rechargeable AAA batteries now.

All in all, I think I like this mouse. The fact that it was cheaper than replacing the old one with another Logitech would have cost me more is a nice bonus too.

Less than perfect

In case anyone thinks I'm one of the Apple fanboys oblivious to their shortcomings, I'll disclose my top pet peeve about Macs. Their default HFS file system is case insensitive. This can cause problems for software engineers like me as some engineers use the file extension .C for C++ source code (I prefer .cpp myself) and .c for C source code. It's not out of the question to want to have source files with identical base names and with file extensions which would be identical aside from their case within the same directory. In a development environment with both Linux and Mac systems available, this can cause Macs to fail to be able to check out source directories with these potentially duplicate filenames.

With the last few versions of OS X version 10.5, there has been an option to format disks with a case sensitive version of HFS. This should not be undertaken lightly. I've read accounts that this can cause problems for older programs (backup programs seem particularly vulnerable for some reason).

I've taken the approach that the devil you know is preferable to the devil you don't. I know how to avoid difficulties on a case sensitive file system. I'm not sure I can find workarounds for some of my older software were I to make the switch to case sensitive HFS.

Happy 40th birthday to my favorite OS

Unix celebrates its 40th anniversary this summer. This article from ComputerWorld recounts the history of the world's most productive OS.

Not a week goes by that I'm not somehow reminded of just how powerful Unix is. Treating all devices as files was brilliant. Being able to combine command line utilities via I/O redirection gives an amazing amount of power and flexibility to the Unix/Linux user interface.

It's no wonder the first thing I do whenever presented with a stock Windows PC is to download the Cygwin utilities. I still can't see how anyone can get any work done using only the built-in Windows commands.

XnView proves handy

One complaint my wife had was after the data restoration I spent much of the weekend on, all of her photos ended up with yesterday's date. This causes her problems locating photos because while she can typically remember the approximate date when she took a picture, she always can't recall for certain what she named the directory. Recently she's taken to my simple approach of prefacing the directory name with a date. I use a date of the form YYYYMMDD (year, month, day) which causes the directories to sort nicely.

I remembered JPG files contained metadata fields which stored useful information like landscape or portrait orientation (for cameras which have appropriate sensors), GPS data (for cameras with a GPS installed), photo settings, and the date the photo was taken. There's even a standard for how these metadata fields are recorded called EXIF.

A bit of searching turned up some software called XnView which allows one to perform a number of date related functions including the following:

- changing the create and/or modified file time stamps based on the date from the EXIF fields (very useful in this case)
- changing the EXIF time stamps based on the current file time stamp (useful when the camera's date was not set when the photo was taken)

As you might imagine, I was very happy to discover I didn't have to cobble together a solution on my own. While I enjoy writing the odd bash or Python script, doing so unnecessarily didn't strike me as much fun.

Windows data recovery

Last night while poking about the web in my typical pseudorandom fashion, I found a new tool which may help prepare for the next time Windows decides not to boot. TestDisk is a tool designed to run under multiple OSes which can perform lots of handy data recovery functions including repairing and restoring an NTFS boot sector.

It turns out it's already available on the System Rescue CD I've been using. I just needed to poke around a bit more.

I'll be making backups of the MBR and boot record since I don't expect this to be the last time that Windows goes belly up. *sigh*

Windows strikes again

Despite the fact I made the switch to a Mac about 6 years ago, Windows still manages to be a thorn in my side from time to time. Yesterday morning my wife discovered that her XP laptop which had been working Friday night had decided not to boot. What was even more entertaining was it decided to display a completely blank screen so there was no clue what was causing the boot to fail. What's even more disturbing is this is the second time I've seen these same symptoms.

I poked around a bit after booting the system under Linux using a System Rescue CD. That allowed me to see which files had been added or modified since the last full backup a month ago. To do so, I had to issue a few Linux commands.

Here's the command to mount the Windows NTFS file system on the /mnt/windows mount point.

ntfs-3g /dev/sda1 /mnt/windows

Here's the commands to mount an external USB disk containing a Windows FAT32 file system on the /mnt/usbdisk mount point. Note that you have to create the mount point first. Also note that the device name of the USB drive may be different than /dev/sdb1. You can figure this out by issuing the command "ls -l /dev/sd*" before and after connecting the USB disk. The device name which appears after connecting the disk will be the correct one.

mkdir /mnt/usbdisk
mount -t vfat /dev/sdb1 /mnt/usbdisk

Here's the command to show you the files which have a later modification date than the last backup. This will give us an idea of what files need to be backed up before we perform any potentially destructive operations to the old disk.

find /mnt/windows -newer /mnt/usbdisk/backup -print

Once I did this, I had a quick look at the partition table/MBR to see if that might be the cause of the failure to boot. That required dumping sector 0 via a program which can display binary data in hexadecimal form. The xxd command will display hexadecimal equivalents for binary data and we can use the dd command to copy the first sector to xxd. Here's the command I used.

dd if=/dev/sda1 count=1 | xxd

A cursory glance indicated the partition table looked okay. It ended with the required 0xaa55 (since PCs are little endian this appears as 0x55aa when viewed as bytes).

I verified the rest of the files involved in the early portion of the XP boot process appeared to be present and had a reasonable size and time stamp. All appeared fine.

Now I was almost ready to try to fix the problem. Since this involved issuing some potentially destructive commands, I booted from the Acronis True Image CD to create a new "since save" backup of the data which had changed since the last full backup.

First I tried booting into the XP Recovery Console to issue the fixmbr command. Sadly, that didn't fix the problem. It did issue some stern looking warnings about the drive having a nonstandard partition table. Since I had used Windows to set up the disk, I found this very irritating. Why would Windows use a partitioning scheme which it would later declare to be nonstandard, especially when the default method is chosen? By this point, there aren't many of Windows' quirks which surprise me. They do make me all the happier that I deal mostly with MacOS and Linux these days.

I also tried an XP Repair Installation. Strangely enough that didn't change the boot failure symptoms at all.

What finally fixed the problem? Doing a full restore with Acronis True Image followed by a restore of the Acronis "since save" data. I may be completely disillusioned with Windows but Acronis makes a very nice product. I don't think I'd consider running Windows without it.

Apple makes switching phones incredibly easy

One of the things I've always dreaded about switching phones the laborious process of entering the phone numbers of friends and family members. I've got all those in my Mac Address Book so why can't I sync it straight to my phone? I've gotten used to doing so with my Palm Centro but wasn't sure it was easily accomplished using my old Razr.

Apple comes to the rescue again. It turns out they've included enough intelligence in their Bluetooth software that I can choose groups of contacts from my Address Book to sync.

But wait, there's more. I was also able to add my favorite wallpaper image to the old phone by using Apple's Bluetooth browsing feature. In the future, I'll be able to transfer photos taken with my cell phone via Bluetooth instead of being charged a fee for using the phone's data connection.

The procedure for accomplishing this was pretty easy to figure out... after all, it is an Apple. But here's the procedure spelled out if you don't feel like noodling it out yourself.

I did a little work with Google but I didn't see a similar procedure available for Windows. That doesn't mean it doesn't exist but if it does, it must not be quite so obvious as Apple's.

A farewell to Palms

The phone portion of my Palm Centro has been slowly dying for the past few months. The signal strength varies wildly over time. Occasionally I'll look at the Centro's status display and see the full 4 bars indicating a strong signal. Minutes later at the same location it may show a single bar or worse yet, the dread "SOS Service Only" message. A visit to Google indicates that this is a very common problem with Centros.

So much for my attempt to combine PDA and phone functionality into a single device. I had such high hopes for the Centro. Having been a Palm PDA user since I got the original Pilot as part of a special employee purchase back when I worked for U.S. Robotics, I've grown to depend on my PDA heavily. Palm PDAs provided the ideal combination of application support and built-in features. If only their phone capability worked as well as the rest of the device.

I've been doing a bit of research and so far I don't see a suitable replacement device, at least not among the subset of devices available from my cellular provider. The iPhone comes closest but I'm too cheap to want to be saddled with a $30 per month data plan when a little planning can make it completely unnecessary. The iPhone doesn't make any sense without the data plan. Many of the other smart phones suffer from poor battery life, poor signal quality, crashes, or insufficient application support.

For the time being I've switched back to my old Motorola Razr phone which despite its age has much better signal strength and battery life than the Centro did when it was new. I'm using the Centro without a SIM card as a standalone PDA device.

I'm considering the Motorola Q Windows Mobile device. I can find Windows Mobile equivalents for all the applications I've come to depend upon (contact management, calendar, to-do list, memos/notes, database, and secure password storage). I've got a few months before I'm eligible for a reduced cost equipment upgrade. I'm hoping a better option presents itself between now and then as the prospect of switching to a Windows Mobile device doesn't thrill me.

FAT32 sucks!

I'm doing my monthly offsite backups and am seriously frustrated by the limitations of the FAT32 file system again. FAT32 is something of a de facto standard since most external hard drives and USB flash storage devices come pre-formatted with FAT32. However FAT32 carries with it the limitation that the size of an individual file may not grow beyond 4 GB. Many backup programs don't handle this limitation gracefully since this requires the backup program create multiple output files.

So what are my options for getting around this limitation? I'm currently using external drives with 2 partitions equally split between FAT32 so the Windows machines can write to it and HFS so our Macs won't be limited by the arbitrarily small maximum file size. Another option is formatting the entire drive with NTFS and using MacFUSE/NTFS-3g to allow the Macs to read and write to the NTFS partition. I like this approach better as it doesn't force me to correctly predict how much space I'll need for each type of machine.

Ultimately I think I'd prefer to use drivers to allow me to mount, read, and write to Linux EXT file systems but this requires more investigation. This is mainly because I really hate the thought of trusting my Mac data to NTFS.

Vim continues to amaze me

Every time I think I've learned all of Vim's tricks, it manages to surprise me with some new feature. There are two things I've discovered recently which have made it more enjoyable to use.

The first is Vim's ability to edit files on remote systems which can be reached via scp or ftp. Since I frequently need to edit files on embedded Linux systems which don't have X11 installed, this feature is a huge timesaver for me. The following commands illustrate the basic commands for either editing a file over scp or ftp.

gvim scp://username@system//home/username/filename
gvim ftp://username@system//home/username/filename

This feature is referred to as NetRW.

The other thing I discovered was completely by accident. I was using MacVIM (my favorite Mac port of the GUI version of Vim made available from the fine folks at Google) when I found I could drag the file icon off the titlebar and into a bash shell. Once dragged into the window containing the bash shell, the full pathname of the file being edited was placed on the command line. Granted this isn't something I need to do often but when I do, it saves me from having to manually type a long pathname.

Python is slowly winning me over

I've been learning Python at work and I have to admit that it's slowly winning me over.

I was skeptical at first. I've been a low level programmer (firmware, diagnostics, operating systems, device drivers, etc) for most of my lengthy career. A mixture of C and assembly language has served me well for most of that time with a little shell scripting thrown in for good measure.

When I started my current job, I discovered they use a mixture of C and Python. The parts of the product which aren't performance sensitive such as the GUI are implemented in Python. This code is much smaller than C code to accomplish the same purpose would be. Thanks to the interactive nature of Python it's also much quicker to develop this code and to get instant feedback.

What finally won me over was examining a short script designed to generate some proprietary TCP/IP packets. This script had a small problem when talking to one of our products. It produced the integer and float portions of the structure with the wrong endianness. A little searching with Google showed me that adding a single "<" character to the format string used by the "struct" module (which prepares individual fields within a structure) would correct the problem.

It took a minute for the full impact of that to sink in to my deeply ingrained C mindset. A single character would correct my problem. Couple that with the fact that the script to generate these packets was probably less than a third of the length of a comparable C program and I was forced to admit that Python is much more useful than I originally thought.

I was tempted to end with a comment about old dogs learning new tricks but I suspect that phrase is far too tired to carry the proper impact. I am finding that old engineers can learn new tools with sufficient incentive. Seeing how much more productive I could be with solid Python experience under my belt is more than enough incentive.

The Neuros OSD is so cool!

I just watched part of the first recording from the Neuros OSD (a Linux based DVR) my wife got me for Christmas. What a cool little device! I'm serious about the little part... it's only about 5.5 inches square and just 1.25 inches tall. Just check out the image. The device itself isn't much bigger than the small remote included. It records programs onto SD or compact flash cards. There's also a USB port so you can record to a USB thumb drive or hard drive.

The first order of business was to upgrade the firmware. There are two ways of doing that. If you've connected the Neuros OSD to an ethernet port, it can upgrade the firmware directly. If you don't have a network connection, you can download the new firmware image to a flash card and upgrade from that. During the ~10 minutes which the upgrade requires, the Neuros shows you a little pong game on the TV screen. The on-screen score appears to be the percentage of the upgrade currently completed.

Is it perfect? Not exactly but it's pretty darned close. It's especially attractive given the low price. It appears to have been primarily created for recording a single recording at a time. That fits my requirements nicely since this was just intended to supplement our existing full-featured DVR. Occasionally my wife will have a couple simultaneous recordings set up which takes up both tuners. It's nice to have the option of not worrying about how much space is used on the main DVR or whether both tuners are spoken for at the time my obscure SciFi program is on. There's a couple other restrictions. Your connection from the set top box must provide composite video (one of those 3 connector cables with 1 video and 2 audio connections).

The other restriction is not really a shortcoming of the Neuros but rather of the FAT-32 format used on many of the flash devices. FAT-32 has a maximum file size of 4 GB. That's probably somewhere between 3 and 3.5 hours of recording. So you might need to choose a flash device capable of recording the maximum length program you'd be interested in watching. If you're keen on catching some type of program marathon, you're probably looking at a small portable hard drive rather than a flash device.

Well, enough of my blathering on about it. Check it out for yourself at the links above. It's a very nice gadget and a useful too.

The sound of many Zunes freezing...

My neck is sore from so much head shaking over the Zune freeze problem. I would have hoped that the development teams at Microsoft would have held regular code reviews which should have caught a glaring error like this one. I understand why they may not have done so. It's quite hard to get people to review your code if company procedures don't demand it. This is especially true in consumer electronics where the schedules seem to be compressed to ridiculous levels.

So what do we learn from this public relations fiasco?
1) Code reviews should be mandatory, not optional.
2) Developers must do much better unit testing. In the past I've had coworkers criticize me because I insist on running functions under a debugger to force execution of tough to test code paths.

I'm always amazed that software engineers can take such a cavalier approach to verifying their code performs as expected. Then again I'm old and cranky. Hey you kids... get off my lawn! :-)

The new printer is working... finally!

I finally got our new Epson Workforce 600 all-in-one printer working properly today. It wasn't an easy installation by any stretch of the imagination. Their setup software isn't what I'd call user friendly. I was close to having it work on several occasions but something always went awry. The most annoying problem was when the printer found the WiFi network during setup and allowed me to enter the WPA key even though it refused me to allow me to paste the key into the form (try entering a 64 character hexadecimal key manually twice with no errors). The printer sat there trying to connect via WiFi in excess of 10 minutes. When it finally failed, it gave no indication of why the connection attempt didn't work.

I won't bore you with a step-by-step recounting of all the problems I encountered. Instead I'll present a list of lessons learned. Most of these only apply if using the WiFi connection.

1. Make sure your WiFi router has some of its IP address range reserved for statically assigned IP addresses (as opposed to DHCP assigned addresses). Using a static IP address for your printer will make the installation go easier.
2. If you're using WPA-PSK, specify the key as a 63 character ASCII string. The Epson doesn't seem to do well with a hexadecimal key. I used Steve Gibson's Perfect Password page to pick a strong random WPA key.
3. Use the printer's front panel to enable and configure the WiFi adapter. Sadly this procedure isn't covered in the manual which insists you use the software to configure everything.
4. Configure the WPA key for the WiFi adapter. This is done with the up and down arrow keys. You can toggle among the uppercase, lowercase, and numeric/special character sets by using the menu key located at the bottom left of the menu navigation keypad (the keys surrounding the "OK" button).
5. Assign an IP address which was excluded from the router's DHCP range in the first step.

Now that the printer is configured, it works like a dream. Don't be put off by this confusing WiFi setup. I've read worse stories about configuring the WiFi adapters for all-in-one printers from other manufacturers. There are far too many combinations of WiFi routers and broadband providers for any manufacturer to ensure no problems will arise during installation. I'm sure at least part of the problems are a result of the router supplied by our broadband provider. I've encountered problems attempting to connect other devices to it in the past. If it weren't for the requirement of MoCA connectivity I would have replaced it with a better router because of the issues encountered.

My new gadget fix

Last week I finally got fed up with the numerous quirks exhibited by the cheap flash only MP3 player which will remain nameless (suffice it to say I should have never strayed away from the Apple fold for price alone) and ordered a refurbished iPod Touch. I figured I could use it as a light MP3 player when I was exercising or walking the dog. Plus it gives me the ability to play with some of the more compelling iPhone features without having to pay the monthly fee for the data service.

It arrived a few days ago and I've been having a great time playing with it ever since. I think it's as very nearly as revolutionary as the original iPod was. The first thing which struck me was how thin it was. I'm amazed at how such a tiny device can perform such an amazing number of tasks.

The mobile version of the Safari browser works well for light web browsing. I get a big kick out of using the gestures to zoom in and out on portions of web pages. Being able to rotate the device and having the display orientation automatically adjust itself is also very cool. Having a built-in IMAP email client is pretty darned handy too. I've also discovered a few free applications which complement the standard set nicely.

I like the mobile version of the NetNewsWire RSS reader. I've been using the Mac version for quite a while now. I discovered that I had to sign up for a second NewsGator account since my old account contains a lot more RSS feeds than the mobile version of NetNewsWire can comfortably handle.

I also downloaded Labyrinth LE, a game which takes advantage of the orientation sensors in the iPod Touch to allow you to control a ball's movement through a maze by tilting the Touch. It's more fun than I would have thought.

I'm sure I'll be discovering more cool applications for the Touch in the coming weeks. In the meantime, I'm having a blast playing with this clever little gadget.

NASA's steadily shrinking budget

According to this article at least some politicians are aware of the problems posed by NASA's steadily dwindling budget. With the economy as strained as it is now, there's very little chance they'll get the increase they so desperately need. The problem isn't widely understood either. According to this article, most Americans grossly overestimate how large a part of the nation's budget NASA gets. It's hard to believe they only account for 0.6% of the budget. Where else can you get that type of return for such a small investment?

We need to revisit our priorities because correcting NASA's funding deficits isn't merely something to do to make ourselves feel good, it's an absolute imperative. Have a look at this chart of the number of near Earth asteroids (NEAs) observed within the last 60 days. Do we really want to take the chance that none of these will come close enough to do massive damage, especially when funding NASA is so affordable compared to a lot of the other budget items?

Most scientists credit an asteroid strike with causing the extinction of dinosaurs. That may be the way we're headed unless we find a way to fund more space research.

Gripping television

We watched the finale of The Shield last night, a fact which fills me bittersweet happiness. It was a thoroughly enjoyable series. It's hard to believe that it's been seven years since it started. I'm going to miss marveling at the havoc Vic Mackey and the rest of the Strike Team managed to wreak each week.

The writing on the series was amazing, especially when contrasted with the drivel that passes for entertainment on broadcast TV channels. It was surprising that they didn't have a slow season unlike so many other series. I wish I could say as much for The Sopranos which came close to boring me to tears for much of its last three seasons. Unlike The Sopranos, The Shield's finale was every bit as good as the series had ever been.

I feel the same bittersweet happiness remembering The Wire. It's refreshing to see what a great team of writers including George Pelecanos, who also has an impressive career as a novelist, can create when given the kind of freedom that an innovative network such as HBO can offer.

So where do we turn for TV entertainment now that those two great series have run their course? I'm hopeful that True Blood, Dexter, Sons of Anarchy, and Brotherhood will all continue providing us with thoughtful entertainment for years to come. I'm fairly certain that the broadcast networks won't abandon their current fascination with crappy reality TV programs long enough to produce something capable of entertaining people who who have an attention span of longer than 30 seconds any time soon.

You can get a virus just by visiting a web site

Just in case anyone doubted that the web is a dangerous place, this article discusses the very real possibility of getting a virus just by visiting a web site with scripting enabled. See, it happens to the ubergeeks among us too. ;-)

The safest way by far to browse the web is using Firefox with the NoScript plug-in installed. I keep scripting disabled by default and only enable it on sites I trust such as my bank, Amazon, etc. For those web sites which don't display properly without scripting enabled, I resort to browsing to them in a Virtual Machine under VMware Fusion. It's far easier to restore the single file containing my virtual machine than it is to have to either restore my entire machine or reinstall software.

Using OpenDNS is also highly recommended. It helps cut down on phishing by warning you when a link has directed you to a dangerous web site.

Those of us using Macs shouldn't feel too smug. With the Mac's popularity increasing so quickly, it's only a matter of time before the people who write malicious software start targeting Macs in earnest.

Firefox smart keywords

A lot of people I've talked to don't seem to explore features added by new releases of their software. A case in point is the smart keyword feature of Firefox. It allows you to enter a quick search on the web site of your choosing just by entering a few works in the Firefox's address bar.

An example will probably make this a bit clearer. I frequently visit the Internet Movie Database to look up movies I'm interested in seeing. So I've created a smart keyword called "imdb" which saves me a lot of time. When I want to look up the movie Time Bandits, I click in the Firefox address bar and enter the command "imdb time bandits". That takes me directly to IMDB's information page on Time Bandits.

To enter a new smart keyword, perform the following procedure.

1) Visit www.imdb.com.
2) Right click in the search box and choose the "Add a Keyword for this search..." selection.
3) Enter a name and the actual keyword for your saved search. For this example, I've used "imdb" for both the name and keyword.
4) I like to group all my keywords in a special keyword bookmarks folder. Then again, I'm an organization freak.

More about strokes

After a few visits with a friend who has recently suffered from a major stroke, I think it's a good idea to pass along some information because strokes are not well understood by the public at large.

It's important for people to learn to recognize the symptoms of a stroke. If you or your family members don't recognize the fact that you're having a stroke, you may not get treatment quickly enough to prevent unnecessary long term brain damage. It's especially important to remember when symptoms first appeared. The most promising treatment, TPA, must be given within 3 hours of the first appearance of symptoms. It can actually cause harm if given after 3 hours have passed.

The aftereffects of strokes are also somewhat misunderstood. Most people recognize the temporary or permanent paralysis of of one side of the body as classic stroke symptoms. The brain swelling which takes place frequently causes problems with speech or control of the limbs on one side of the body which can require months of therapy to correct.

You may also want to ask your doctor about taking a daily low dose aspirin as a preventative for strokes and heart attacks. For some reason the body seems more prone to developing blood clots as we get older.

A podcast I listen to regularly, Skepticality, features a host who had a major stroke a few years ago. He returned to podcasting after about 4 months. At first his delivery wasn't as fluid as it had been before the stroke but with continued speech therapy, it's now hard to tell that he ever had a stroke. I'm hoping therapy proves just as effective for my friend.

Leopard's Stacks feature

One of the first Leopard features I started using was Stacks. I used it to replace the program launcher I'd been using. The Dock is a great way to organize your most frequently used apps but since it also gets populated with an icon for every program currently running on your Mac, it can get a bit cluttered. Fortunately Stacks offers a great way to tame that unruly list of icons.

Leopard comes with two Stacks items by default, Documents and Downloads. I've gotten rid of Documents since I tend to keep my documents stored in a hierarchical directory structure rather than shoving them all in one big documents folder. I've also made a minor change to the Downloads item. Stacks has the annoying habit of changing the icon for the Downloads or any other Stacks item in the dock to icon of the first item in that folder. I find that disconcerting to have my Downloads folder sometimes look like a folder and sometimes look like the icon of whatever program I've happened to download. So I've added an alias for a folder called _0_dummy_folder to the Downloads folder (in your home directory). The name causes the folder to appear first in normal sort order so a folder icon is used for the Downloads Stacks item.

I've also added a few new Stacks items of my own to the Dock. I did this by following the procedure below:

1) Create a folder to contain the entries you'd like to appear in your new Stacks items. For example, I created a folder called ~/Dock/Utilities to hold aliases for all my frequently used utility programs. Note that the tilde is Unix shorthand for your home directory.
2) Create and drag aliases for each utility program you want appearing in this new Stacks item to the folder you created in the previous step.
3) Create and drag an aliases for a dummy folder to the ~/Dock/Utilities folder. I use an alias to a folder called _0_dummy_folder for reasons outlined above.
4) Drag the folder ~/Docks/Utilities to your actual Dock. I dragged mine to the left of the Downloads Stacks item.

You can repeat the above procedure to create more categories of frequently used programs or frequently accessed documents. The best part is I don't have to keep paying to update my program launcher. This also takes up less space than my old program launcher.

Recognizing the symptoms of a stroke

I found out tonight that a close friend of mine suffered a stroke last week. He recognized the symptoms and was able to describe them to the doctors before his speech was too affected. Knowing how bright he is and how much he reads, it doesn't surprise me that he'd taken the time to learn this valuable information.

The Stroke Association has a list of symptoms which everyone should commit to memory. It can make all the difference!

Useful Mac software

A friend of mine just took the Mac plunge so I've been sending him links about useful Mac software I use most frequently. I thought it might be nice to collect all those recommendations here.

ClamXav is a high performance virus scanner for Macs. With the Mac's steadily increasing market share, it's becoming a more attractive target for malware authors.

TNEF's Enough is a handy little utility which unpacks the strange attachments which some Windows mail programs use (usually called winmail.dat).

MacJanitor allows you to manually run the 3 cron jobs (daily, weekly, and monthly) used to do periodic maintenance on Macs. This can be important on laptops which typically don't get left on all the time or desktop systems which may be asleep when the scheduled time passes.

Flip4Mac is a free QuickTime component that allows you to play WMV files.

JView is the fastest image viewer I've found.

MonoLingual is a utility to remove unnecessary language files on Mac OSX. This can save a fair amount of disk space.

NetNewsWire is the best RSS reader I've found for the Mac.

Transmission is a good open source BitTorrent client.

The VLC media player can play most types of video files.

NeoOffice is the Mac port of OpenOffice, an excellent and free alternative to Microsoft Office.

Here's an article on how to run Windows or Linux on your Mac in a virtual machine using some free software called VirtualBox. VirtualBox isn't the only virtualization software available for the Mac. At home I use Parallels which was the first virtualization software available for the Intel based Macs. At work I use VmWare Fusion.

MacVim

I'm a long time vi/vim user. I don't know that I'd recommend anyone who doesn't already know vi go through the heavy learning curve necessary to become proficient at using vi. There are a number of gui editors which are more easier for new users to learn. However vim (vi improved) is available on very nearly any computing platform you might use. I also find that vim allows me to accomplish some pretty complex editing operations faster than most of my former coworkers using other editors. Having used vi for over 20 years now, I have yet to find an editor which would make me more productive and believe me I've looked. I'm constantly searching for new programming tools in my spare time.

I've been looking for a decent port of vim for the Mac for a while now. The version pointed to by vim.org always seems to lag behind a version or two. It also has a few deficiencies. It used to have screen draw problems and would leave pixel residue behind after scrolling. It also never handled the "-" command line argument properly. This argument causes vim to read its data from stdin which is very handy for piping output from other commands into vim for easier manipulation.

My search is over. The team at Google Code has created a great port of vim for the Mac they call MacVim. It's fast and features none of the problems I'd experienced with other ports. Thanks guys for a great porting job!

Data recovery, part two

I've managed to recover a fair number of files from my stepdaughter's failing hard disk. You might recall I used a program called dd_rescue to do a raw copy of the sectors of the failing hard drive to an image file on a larger USB hard disk. That was important because the old hard disk seemed to be getting progressively worse, as hard disks which have experienced a partial crash are wont to do.

There was good news and bad news regarding the copy. The good news was dd_rescue managed to copy about 33 GB from the 60 GB drive before encountering constant errors. The bad news is that left about 27 GB of data which hasn't yet been recovered.

Next, I used a handy (and free) program called PhotoRec to recover photos and a number of other data types from the partial image of her drive. All told, it found about 4000 jpg files large enough to be her pictures. Some of them probably come from a browser cache but a good deal of them are vacation photos which she'd be pretty upset to lose.

So far it's been at least a partial success. I'll post more if there are significant updates in the future.

By the way, let this be a reminder to you to go back up your data. Hard disks sometimes fail with no warning and not all drives fail gracefully enough to allow some data to be recovered.

No-name router problems

I spent part of this morning doing some remote troubleshooting of a problem my in-laws were having with their broadband connection. Their broadband provider supplied a no-name router. Somehow it had decided that my wife's laptop had made too many outbound connections and therefore must have a virus. Once having decided this (and quite erroneously so), this poorly designed router continued making this assertion even when her laptop was no longer plugged into the router. In fact it seemed completely unable to determine which were active computer connections and which had timed out. Its status page listed two computers which didn't match any computers currently connected to the network nor had there been any such computers connected that my in-laws could remember.

You might wonder how I could determine that the complaint about too many outbound connections was erroneous beyond the shadow of a doubt. I simply enlisted the use of the "netstat" command. The netstat command exists in all major OSes (Windows, MacOS, Linux, and BSD). It allows you to determine the state of network connections for the computer on which you execute the command. Using the "-b" option allowed us to see which programs had open connections. As I suspected, only iTunes, Thunderbird, and Firefox had network connections and none of the three applications had an unusual numbers.

And yet this silly router continued complaining about the number of outbound connections from this one computer even when the computer was disconnected to the network and through several power cycles of the router. So I walked my stepdaughter through the procedure to disable this poorly implemented portion of the firewall (the detection of number of outbound connections) because it obviously wasn't working properly.

My advice is to stick with a name brand router (Linksys or Netgear) whenever you're presented with the option. Sadly since this router was supplied by their ISP, they don't have a choice in the matter. Using a no-name router may cost you more than any initial cost savings realized by purchasing a cheap device in the amount of troubleshooting time you spend on poorly implemented features such as this one.

Recovering data from a failing hard drive

I've been trying to recover the data off a failing hard drive for a family member. I've found a few programs which claim to be able to do just that but they always get hung up by the numerous retries the drive keeps doing in the failing areas. Then I came up with the idea of using the dd command to make a copy of drive image which I could then manipulate having gotten the retries out of the way during the initial copy process. I'd used dd pretty heavily during the development of an SD card driver I'd done at my last company. Once the drive image has been copied to a file, the resulting image file can be mounted using the mount command... well it can on Linux and Mac OS X at least. You poor folks running Windows are out of luck.

After looking around on the web, I discovered a great little program called dd_rescue which does intelligent retries if errors are encountered, slowly lowering the block size being requested to find the boundaries of the affected area. I think the standard dd command would try to do retries until the read worked or until the copy was aborted. dd_rescue also allows an offset to be specified when the command is invoked so the copy may be done in several stages. Since it's taken about 4 hours, off and on, to copy the first 33 GB from the failing 60 GB drive, I'm anticipating having to make heavy use of this feature to complete the copy process over the next day or two.

I made a few minor changes to the source to allow me to curtail the retries to speed up the copy. So far it's copied about 32 GB from the failing 60 GB drive. Once the data has been copied then I'll start trying to recover files from it. Wish me luck, I think I'm going to need it!

quick and dirty shell command

Today I was working on some old code at work. I discovered at least one duplicate include file which is a personal pet peeve. It's far too easy to allow multiple include files get out of sync so you have different versions for different source files.

What I needed was a quick way of finding all the duplicated include files within this project directory (and subdirectories). It turns out stringing together a few Unix/Linux/Mac OS commands with some I/O redirection makes this task pretty easy.

The first thing we need is to be able to locate all the include files. In the C programming language, these files typically end with the ".h" file extension. We can use the find command to give us a list of the files which end with .h.

The next problem to be solved is that the matching files will have not only their filenames but also the directory in which they're located printed out. So we need a way of extracting just the "base" filename. Fortunately bash has any easy method of accomplishing this with the basename command.

The next logical step in figuring out whether there are duplicate filenames is to sort the matching filenames to make it easier to see matches with the sort command.

Finally we can use the uniq command to show just the filenames which appear more than once. The uniq command has other options. You can choose to show just items which are unique as well.

If we put all the portions of this command together, we come up with the following command. It's doing a lot of work to save us the trouble of manually sifting through all the filenames ourselves. That's what computers are supposed to do for us, eh?

find . -name "*.h" -print | xargs basename | sort | uniq -d

Palm Centro

I've used a Palm PDA without interruption since the first one was introduced in 1996. I was working for U.S. Robotics which owned Palm at the time and the employee pricing helped me decide to take the plunge into PDA life. After all these years, I've come to rely heavily upon a few key PDA applications (in addition to the standard PDA applications).

I use SplashID to securely store the multitude of passwords I need to remember both at work and at home. Without it, I'd have to resort to using weak passwords in order to stand a chance of remembering them all which compromises security.

I use SplashMoney to record credit card transactions while I'm away from my computer. This ensures I stay within budget and helps guarantee that I recognize any erroneous charges which might pop up.

JFile is invaluable for storing databases I design myself. I use this to keep track of all manner of data such as books I've got and those I'm interested in reading. Before I did this, I occasionally bought multiple copies of a book.

SlovoEd is a portable dictionary which allows me to look up words I don't recognize when reading without a print dictionary handy. My Centro takes up a lot less space on my nightstand than a conventional dictionary.

Adobe Reader for Palm allows me to read PDF documents on my PDA. This is handy to ready books in non-traditional settings. It's nice always having a book handy to read for those occasions when unexectedly left with extra time to kill.

A couple months ago, the time seemed ideal to upgrade my phone and PDA. My wife's phone was acting up and my stepdaughter wanted a cheap PDA. So it made sense to get a device which fulfilled both those functions for me, freeing up my phone and PDA for them. This also had the added benefit of allowing me to pare down the devices I carried from two to just a single gadget.

The Palm Centro is smaller than I expected but keyboard surprisingly useable. The software upgrades work to make the smaller sized device more intuitive to use than older Palm devices. Is it perfect? No, but it does seem a better compromise device than the other affordable multi-use devices I've seen.

If you're interested in an affordable combination mobile phone and PDA device, check out this review from Engadget.

I/O Redirection

One of the most useful features of the Linux, Unix, MacOS, and to a lesser extent Windows (more on that later) is the concept of I/O redirection. In this discussion, we'll restrict ourselves to the pipe form of redirection which is invoked by the vertical bar character "|". This tells the command interpreter to take all output from the first part of the command and send (or pipe) it to the second part. The other characters which invoke I/O redirection are the less than "<" and the greater than ">" characters. Those are primarily used to send output to a file or cause the program to take its input from a file.

If you're a long time computer user, you may want to skip ahead to the examples below. You may have heard the term "I/O redirection" before but what does it really mean? I/O redirection gives you the ability to chance where a program's input and/or output is bound for. Normal command line programs have both their input, aka stdin (standard input), and output, aka stdout (standard output), directed to the console (which is just shorthand for saying the input comes from your computer's keyboard and the output goes to the portion of the screen where you're running the program). Note that if one of the portions of the command line produces errors, you may be surprised to find the error messages may not get redirected with the pipe command. This is because many Unix/Linux style programs also make use of a third I/O stream called stderr. WIthout taking special action, stderr output is almost always directed to the console to bring the error condition to the user's attention.

The simple example

For this example, let's suppose that you've got a huge tar file, aka tarball) which is really an archive file containing many other files. Now suppose you want to look to see whether it contains a text file but you really don't recall the name of the text file. Perhaps you recall something else about the text file such as it was located in the /projects directory. You could always get a listing of all the files within the tar file and manually search through them but computers were created to relieve users of the need to do such labor intensive tasks. How about we put I/O redirection to work?

To start with, we need to obtain a listing of all the files in our tar file which for purposes of illustration we'll call sample.tar. That can be accomplished with the command below.

tar -tvf sample.tar

That gives you a complete listing of all the files but chances are if it's a big tar file, the names and details of the files scrolled off the screen and perhaps overwhelmed even the scroll back buffer of your terminal window (aka command interpreter or shell). In any case, being lazy computer types we don't feel like searching through this huge amount of data.

The first thing we want to do is to weed out all the non-text files. Hopefully we've been disciplined about our file naming conventions and have added a ".txt" file extension to all our text files. So let's show only the files which end with that file extension. We'll use our old friend "grep" to match just the output lines which contain the string ".txt". Note that we're using the "-i" parameter to specify that we want to match the string ignoring the case of the letters in the string. This may be important if not everyone adding files to the tar file was careful about adding a .txt and not a .TXT file extension. Toy OSes like Windows don't make this distinction but you'll find they don't handle I/O redirection properly either. The Windows shell is very simplistic so even if you've added Linux style utilities like "tar" to its repertoire, you may be disappointed to find that it doesn't do multitasking. A proper OS will handle I/O redirection real time so when you type the command below, you'll see output data appear quickly. Windows creates a temporary file containing all output from the first part of the command which it then sends to the second part of the command once the first is completed. It makes the Windows command line feel much slower than it is and believe me it doesn't need much help. If you've ever manipulated large tar files on both Windows and Linux systems, you'll quickly discover that the Windows command line isn't performance oriented by any stretch of the imagination.

tar -tvf sample.tar | grep -i ".txt"

That gives us a listing of just the files which contain the string ".txt" which hopefully only appears as a file extension.

Something we notice about the output which makes life a bit tougher is that the tar shows you the file names and other details in the order they were added to the tar file. It would be nice if we could see the files sorted by the directory names in which they appear. Fortunately there's a simple solution to that desire.

tar -tvf sample.tar | grep -i ".txt" | sort

This command does the trick but it also illustrates some odd behavior. The output doesn't appear piecemeal the way it had been doing previously. If we think about it, the reason becomes obvious. The sort command can't really sort correctly unless all input to be sorted is present. So it must wait until the commands up to that point in the command line are complete before starting to sort the output.

Since we might be fans of the graphical version of the vim (vi improved) editor, we can add another labor saving twist to our command line. We can send the output of our command to gvim. This has the advantage of being able to search the output using editor commands. Doing this in gvim will also cause the search terms to be highlighted within the text making it much easier to pick out from the surrounding text.

tar -tvf sample.tar | grep -i ".txt" | sort | gvim -

Obviously the commands above were simple examples to make explanation easier. I'll add a few slightly more advanced examples below with a brief explanation of what they do. Once you get the hang of it, you'll find you quickly come to rely upon this powerful feature. Most of the Unix/Linux style command line utilities are written so they can be easily combined to create more powerful command lines similar or more sophisticated than the ones we've been exploring.

A few more advanced examples

The command below uses the find command to search for all files which end with the ".txt" file extension. It then searches them to see which of them contain the string "project". Note the "-l" parameter causes grep to only output the filenames which match the search criteria. If you omit the "-l" you'll see a list of matching lines from within the files. Also note the use of the xargs command which may seem unfamiliar. It's a method of appending multi-line output from previous commands to form arguments for the command specified after xargs.

find . -name "*.txt" | xargs grep -l project

This command does essentially the same thing but sends the list of matching file names to the vim editor. It issues the command to search for the string "project" so that term will be highlighted in the file and the cursor will be placed on the first occurrence within the first file.

find . -name "*.txt" | xargs grep -l project | xargs vim -c /project

Try coming up with ways to use I/O redirection which make your time at the computer easier. You'll be glad you did.

I knew it!

Here's an interesting article about Ernest Hemingway. The part I find most interesting occurs on page 4 where Hemingway that the symbolism which English teachers so often attribute to stories is not premeditated. He states "No good book has ever been written that has in it symbols arrived at beforehand and stuck in".

This quote supports my long held belief that the symbolism English teachers claim to find in books was usually not intended by the author and as such is entirely subjective. In school I always hated being criticized by an English teacher for not seeing the symbolism they claim is the "only" valid interpretation. Frequently these teachers would speak as if they had some sort of notebook from the author containing their secret thoughts about hidden subtext they had woven into their novel. What a crock!

Backing up data

I've been struggling trying to find a decent solution for backing up my wife's laptop computer. The program I'd been using ended up not backing up some key files. We almost ended up losing all her photos when the hard disk in her laptop started dying recently. It's never a good sign when you hear clicking noises when you try to list a directory. Fortunately the use of SpinRite and a little luck allowed the drive to continue functioning long enough to manually copy the files to an external drive.

After trying a number of different backup programs and not being fully satisfied with any of them, I decided to use something simple. The tar program has been around since the early days of Unix. Tar stands for Tape ARchiver and it was originally used to archive a group of files to a tape device. The beauty of tar is there are versions available for almost any operating system you can think of. That makes it easy to examine the tar files on any system to verify I've backed up all the files which needed backing up. I'd long ago installed cygwin (a version of the most common Unix utilities for Windows) on her laptop so I was good to go. Cygwin can't fix all of Windows' shortcomings but it is able to make Windows much more useful. The Windows command line tools are so woefully underpowered that I no longer consider using them for anything.

Having decided to use tar, there were still a couple other problems which needed to be solved. The resulting tar file for even a partial backup is likely to be quite big. By default tar concatenates all the files with some file information (file name, size, permissions, etc) about each file added. So using gzip to compress the tar files is highly recommended to avoid using too much space on the backup device.

The gzipped tar file for the backup of just the data files on her laptop ends up having a size of just over 11 GB. The external USB hard drive and NAS (network attached storage) drive we have are both formatted as FAT32 to make it easy to use on Windows, Mac, and Linux systems. That presents a problem since FAT32 drives have a maximum file size of 4 GB. So I was forced to use the Unix split program to split the huge tar file into smaller files which can be copied to a FAT32 drive.

After building the tar file, I was able to dump a list of all the files contained in the tar file. I was also to use the find command (the cygwin/GNU version not the lame Dos/Windows version) to build a list of all the files on her hard disk. Then it was a simple matter to use grep to get a list of all the JPG, GIF, DOC, etc files in both the listing of all files and the listing of all files in the tar file. That made it easy to verify that I've managed to back up all the data files.

Whew! It was a lot of work but now I can finally rest easier knowing all her data files are safely backed up.

Bad advice from an IT guy

A relative called recently asking about her friend's laptop computer which was running slowly. Her friend had asked someone from the IT department at work how to make the laptop run quicker. The IT guy's response was to suggest they replace the hard disk with a faster model.

This is wrong on so many levels that it makes my head hurt. First, the IT tech didn't ask any questions to determine what the underlying cause of the slowdown might be. Laptop computers typically come with relatively slow hard drives since lower RPM drives create less heat. Laptops always have trouble dissipating heat because of the small cases. Chances are a faster drive may not be available or at least may not be affordable for the average user. A slow hard drive typically only causes delays in one of two circumstances: loading programs and reading or writing data files. Those two cases comprise a fairly small percentage of the overall usage time and will most likely not produce a noticeable delay.

A better approach to speeding up an older computer is to add more memory. Application software always seems to get larger over time. Data files also have a tendency to grow with use. Users also tend to use more applications simultaneously as they get more sophisticated. All of these conditions probably require more memory than originally came with the laptop. When the laptop doesn't have enough physical memory, Windows will be forced to swap unused applications and portions of the data files out to the swap file on the disk. Hard disk accesses are always much slower than memory accesses.

Sadly, bad advice like this is not at all uncommon. Hang around in the computer section at any big box electronic store any you'll undoubtedly hear something similar. Amazingly enough, $8.50 an hour and a few months of experience doesn't always produce quality technical advice. Imagine that...

PDA replacement

I finally found an excuse to take the plunge and replace my aging Palm Tungsten T3. This is not a transition I make lightly. I've been using Palm PDAs since I bought the first Palm device introduced back in 1996, the Palm Pilot 1000. I got it for special employee pricing since I was working for U.S. Robotics. You may or may not remember that they owned Palm when the first Palm PDA was introduced.

As a longtime and very satisfied Palm user, I would have loved to choose another Palm PDA but Palm's future doesn't look quite so rosy these days. The company has been making some strange moves such as spinning off their software division. They've also apparently lost focus and haven't introduced any new innovative PDAs models in a while. Even the Treo line they've been depending on so heavily has lost ground when compared with other smartphones.

The device I chose as a replacement was the Nokia N810. It includes pretty much all the features the Tungsten has but also features a slide out keyboard, a Wifi adapter, a built-in webcam, a GPS receiver, and can even use a bluetooth cellphone for an Internet connection if a wireless access point isn't available.

Now comes the struggle to figure out how to duplicate all the features I depend on my Palm for on the new device. Fortunately, Access has introduced a Palm emulator called the Garnet VM which should help ease the transition somewhat.