lots of options...
The dedicated recording devices are really good these days...
I have a Zoom H4N
http://www.amazon.com/Zoom-Handy-Portab ... B001QWBM62They have decent internal stereo microphones, and have XLR Mic inputs if you want to upgrade to something more like what you'd use when performing.
I record in raw wave format, to the local SD card, and then copy them to my computer and process them locally with Audacity or something similar.
The USB Microphones like the Blue YETI, and such are pretty good too. They are USB sound devices, and primarily designed for podcasting and such. You would just record directly into something like Audacity, and save yourself a step.
There are advantages to both. The portable digital recorder is a bit pricier, but I think the quality is probably better, especially if you use your own mic's. They are also, well, portable... which can be handy if you want to record on site, or take it out caroling with you....