Using grep in vim

One of the things I like about Vim (and vi) is the ability to invoke Unix utilities to manipulate text in ways that might be hard or impossible with just the regular editor commands.  It's definitely written with the Unix Philosophy in mind.
One thing I do frequently while debugging problems is to add log messages with a distinct pattern so I can find them easily in the log file.  For purposes of this example, let's assume the pattern I use is XYZ.  If I open the log file in Vim, I can issue the following command to isolate just the lines in the log file which contain XYZ.
^[ggVG!grep XYZ^M
That looks like a pretty complicated command, doesn't it?  But if we examine it piece by piece, it's not really that bad.
At the beginning of the command we've got ^[ which is the escape key.  Look at this ASCII chart if you're not familiar with the caret followed by a letter shorthand for control characters.  I issue the escape key to make sure Vim is in command mode.  While we're talking about control characters, the ^M at the end of the command is shorthand for the carriage return (AKA the Enter key).  That causes the command to be executed.
The ggVG in the command serves to do a visual selection of all the text in the file.  The gg causes the cursor to be placed at the first line of the file.  The V invokes visual marking of text.  And finally, the G causes the cursor to be placed on the final line of the file.  That causes the marked area to contain all the lines of the file.
The real meat of the command is the next part - !grep XYZ.  The exclamation mark pipes the marked text to the external Unix utility which follows which is the grep command.  This particular command line searches for lines which match a pattern of XYZ.
Issuing this command will cause Vim's current data buffer (the full contents of the log file) to be replaced with the output of the external Unix utility which will be just the lines within the log file which contain the pattern XYZ.
That makes it really simple to isolate just the log commands I've added.  Once I'm done, I can either exit Vim without saving or just issue the u command (undo last text manipulation) to leave the log file untouched.

Unix tool: xargs

One of the reasons I like Unix style operating systems so much is the Unix Philosophy.  One of the principles is it's better to include a bunch of small, fast tools which can be combined together to accomplish a variety of tasks than it is to build large special purpose tools which are complicated to use.  One of my favorite of these Unix tools is xargs.  What xargs is good at is taking a bunch of separate lines of input and changing those into arguments for another command.
Perhaps an example will serve to illustrate better than a dry explanation.
Let's imagine we want to find all the source files from the current directory (recursively) which contain the string "stdio.h" and to edit each of those files using vi.  The following line will accomplish that.  Of course in Unix, there are numerous ways to accomplish the same task.  This happens to be my favorite way to perform the task and serves to illustrate the use of xargs nicely.
find . -name "*.c" -print | xargs grep -l "stdio.h" | xargs vi
The first part of the command (that before the first pipe character) is a simple find command.  The only thing worthy of explanation is the quotes around the file pattern.  Unix shells will substitute matches for a wildcard such as this before the find command gets invoked.  If we invoke this long command in a directory containing files which match the pattern *.c, the matches would be substituted on the command line instead of the actual *.c pattern.
The second part of the command is a simple grep command but combined with the xargs command.  This passes the file names output by the find command to be used as arguments to the grep command.  The grep command is looking for files which contain the string "stdio.h", a common c library header file.
The third part of the command simply takes the matching files found by grep and passes them to vi.
This example should work on either Linux or Mac OS X.  It will also work on Windows provided you've installed a Unix environment such as Cygwin.  Cygwin is one of the first things I install on any Windows machine I have to use on more than a casual basis.
Play around with the xargs command to get a feel for what it can do.  It's a handy part of any Unix tech's grab bag of tricks.

Portable tools

Having written software on a variety of development systems including Sun workstations, Windows PCs, Linux PCs, and Macs, I've never regretted the decision to choose easily portable tools for use in my daily work.  By doing so, I learn a tool once and am able to use it regardless of what type of development system I am given at a new company.  I also have a preference for free software.  In the past it's been difficult if not impossible to get companies to pay for software tools.  Having a ready arsenal of free tools for any system I use is a huge bonus.
My editor of choice these days is one of the GUI versions of Vim (MacVim at home and GVim on the Windows PCs at work).  Besides the obvious enhancements which either version of Vim adds to the standard vi commands, the developer who started the project, Bram Moolenaar, runs the project as charity-ware with donations going to help children in Uganda.  I've got no problems donating to such a worthy cause.
I continue to use vi clones because vi was available on most of the systems I've used over the past 20+ years.  Using PCs in the early days, I chose vi over emacs because early ports of emacs were extremely resource intensive.  While vi would fit easily in a 40 KB .COM file which could be executed from a floppy disk on base PCs, the emacs .EXE file (necessary because it required more than 64 KB of code space) typically required a PC with an expensive EMS board.
Were I to be starting out these days, I believe I'd probably choose to learn UltraEdit or SlickEdit since both have Linux, Windows, and Mac versions.  Both editors also have a much less daunting learning curve.  At best, vi commands are perplexing to new users.  You really need to understand the history of vi to grasp why the commands are designed the way they are.  In a nutshell, software in the early days of Unix had to support a variety of dumb terminals.  Commands had to be formed from the common subset of keys available on all terminals.
If you combine Vim with Exuberant CTAGS, you have a very powerful programming editor which makes it easy to learn and to navigate unfamiliar source code.
I've also chosen the Bash shell as my daily command line interface.  It's available on Linux and Mac.  It can also easily be added to Windows systems as part of Cygwin.  Bash is much more powerful than the standard Windows command shell.  I've heard good things about PowerShell but since it's only available on Windows, it's not really an attractive option for me.
Aside from the tools above, I'm fond of AWK and SED.  Used either separately or combined with their Unix based brothers like xargs, find, and grep, they give an nearless endless variety of ways to manipulate text files.
I've got one last recommendation.  There's a great grep replacement for software developers called ack.  It makes it easy to restrict the files searched to just source files.  You can do the same thing with grep but the command line gets very complicated.

Long hours

For years I've been subscribing to Jack Ganssle's excellent newsletter, The Embedded Muse. It's a must read for anyone working with embedded systems. It's also very good as a general resource for engineers who work with any type of hardware or software. If you're a software or hardware engineer, I urge you to take a look at a few back issues and then sign up for the free newsletter.

 The issue which arrived today contains an interesting discussion about the number of hours engineers are asked to work. I fall heavily into the camp which questions the wisdom of working more than 40 hours. Lest you think I'm a slacker, consider that I worked for 8 straight years at startup companies, and I probably averaged 55-60 hours per week during most of that period. I can speak from personal experience that the weeks with extremely long hours were not nearly as productive as those where I worked 40-45 hours.

Of particular significance are the stories where upper management gets upset when they find no one in the lab after hours. Whenever you find upper management in the lab after hours, that's also a result of poor management. Good managers will be able to communicate with line managers without checking up on them. Good upper managers will also be sufficiently removed from day to day operations that they realize they may not have the entire picture. Employees connected to lab equipment from home are awfully hard for anyone to detect. If you think that you can only achieve results by having a negative impact on the engineer's personal life, I can safely say I don't want to work for you and I suspect most experienced engineers would say the same.

 I would suggest that any project where management asks for a long term commitment of long hours is a direct result of inexperienced or poor management. Any manager with a reasonable amount of time in the industry knows that while you may get a delivery out with a short burst of concentrated effort but if you ask for it over a period of months (or more), you're going to get a product which requires more and quicker maintenance releases thanks to the mistakes even the best engineers make when they're overtired. You're also asking for a large turnaround in your engineering team unless you're able to offer some form of extreme compensation (large bonuses and/or stock options).

For young engineers, I would advise that you consider what you're getting out of any company which is asking for extreme time commitments. If you're getting some form of financial compensation or some form of experience which will prove extremely valuable in the future, I'd say go for it although I'd get it in writing if at all possible. Otherwise, I would suggest that you update your resume and start looking around for a company whose management might not be so out of touch with reality.

Golden age of nerds

It suddenly struck me today that nerds today have it much easier than nerds of my generation. It's easy to feed unconventional tastes in books, comics, music, TV, movies, food, beer, etc than it's ever been before.

It struck me as I was having a very enjoyable but quiet lunch by myself (thanks to a last minute cancellation by a friend who wasn't feeling well) at my favorite craft brew establishment. In my youth, I would have never considered dining solo but luckily my iPhone affords me ample entertainment for such occasions so there's no need to skip an tasty lunch. I have more media than I could consume in several weeks at my beck and call.

In my misspent youth, nerds were forced to read a wide range of magazines and visit geeky bookshops run by knowledgeable staff to keep up with upcoming media they might find of interest. Proximity to a great bookshop could spell the difference between keeping in touch with new happenings in your slice of the nerd universe and playing a perpetually unsatisfying game of nerd culture catchup. Fortunately I was close enough to a couple great bookshops that on weekends I could convince my Mother (who instilled the love of reading in me) to drive me to one or the other to spend my hard earned paper route money on another stack of books.

Now I can accomplish the same thing with a few subscriptions to e-magazines, email newsletters, RSS feeds, and podcasts. Ordering said items of nerdy desire are usually just a matter of pointing at the correct website and furnishing a valid credit card.

I chuckle when I hear geezers (not that I'm not one myself) bemoaning the death of paper books, magazines, newspapers as well as vinyl as a storage medium for music). They're missing the point that the artist's message is becoming easier than ever for the public to consume and frequently without the unnecessary middlemen who have all too often served as unwanted arbiters of what the public was ready to accept.

I for one embrace this golden age of nerds. Let the mass consumption begin!

Advice for the family tech support person (or mechanic)

An acquaintance of mine has been roped into providing computer tech support for a family friend and is entirely too polite to complain about the situation to the correct people to prevent a reoccurrence. I've got friends who are mechanics or at least very mechanically inclined who face similar problems in their field.

Hearing about the headaches associated with this has brought back memories of being in similar situations. Once people realize you have expertise in computers, or any other field for that matter, you may find yourself getting pressured into providing computer or car maintenance assistance. It has made me wish there were a tactful way of establishing guidelines for providing help to friends and family members. What better way than an anonymous blog post you can point to as a gentle suggestion of how to behave when asking for help?

1) Use Google to see if you can't solve your own problem or at least narrow down the possible causes. You'd be surprised at how many problems can be solved without intervention from experts.

2) Just because your friend or family member is willing (or at least too polite to refuse) to offer you assistance, don't assume that he or she wants to extend the same courtesy to your friends or coworkers. Don't volunteer them to do so without asking first and take "no" for a answer gracefully. Just because they're good at their job, don't expect them to be thrilled about volunteering extra hours above and beyond their normal work week on related problems.

3) Should your indentured tech support servant be kind enough to agree to help, don't make the imposition any worse by being picky about where or when the service should be offered. No one enjoys being stuck in a cramped back room working on a computer or on a cold, hard driveway working on a car. If they ask you to bring the PC or vehicle to their place, do so cheerfully. If they do agree to come to your house, clean up a bit to make service less painful. I can't tell you how many times I've had to ask for vacuum cleaners to clean off fan vents sufficiently to make disassembly possible without a big mess. No one enjoys working with a jumble of cables or surrounded by so many knick-knacks that it's difficult to get to the computer. Be flexible about time too. Remember, your potential tech support person has a life too. Don't interfere with work or their other commitments.

4) Find a way to reciprocate and make sure it's something of comparable value. If your tech support person has spent 2 hours fixing your computer, don't let a plate of cookies be your only thanks for their efforts. Think about some skill you have that they may need and offer that, be it mowing their lawn or taking them to dinner. Remember how much you would have spent at the local big box electronics store.

5) If the support you're asking for is advice or if advice is offered as a method of preventing future problems, don't blithely ignore it. You've asked this person for help because of their expertise. If you ignore it or instead follow advice from some less experienced friend or coworker or even worse, fall prey to some slick advertising or advice from one of the poorly trained hourly grunts at your local big box electronics store, how likely do you think it is that your tech support person will ever give you meaningful advice again?

6) Do whatever preventive maintenance is recommended to try to prevent future problems. No family tech support person wants to hear your plaintive cries about how you can't afford to lose some valuable file. So do your backups (or oil changes in the case of cars) and don't expect your family tech support to perform miracles on neglected PCs or cars.

Following these steps will help ensure less resentment on the part of the person helping you.

iPhone/iPod Touch databases

One of the many things I use my iPod Touch for is to store databases of information I want access to on the go such as books I have, books I'm on the lookout for, car maintenance histories, etc. Bento fits the bill nicely for my purposes and there's a Mac version which allows you to synchronize your databases between a Mac desktop or laptop and your mobile device.

It's a full featured database with optional field types such as check boxes, choice boxes, text, dates, numbers, and other options. It makes data entry very easy which is important on a mobile device. One of the things I like about Bento is it's one of the few inexpensive lightweight single user databases available these days. In the old days there were lots of database choices for users on a budget.

I read today that there's a version of Filemaker Go for the iPhone/iPod Touch/iPad. It synchronizes with Filemaker Pro which is available for Mac or Windows. Filemaker Pro finally makes it easy to sync between a mobile device and Windows. The upside is it's a more powerful database than Bento is if your database requirements are a bit more advanced than Bento's feature set. The downside is it's more expensive than Bento.

So you've got another database option if you're a iPhone/iPod Touch/iPad user. It's good to have choices, eh?