defaults write com.apple.screencapture type png
Then you must issue:
killall SystemUIServer
to make the change take effect.
Note: You can replace png in the above with any of these: png, pdf, tiff, pict, bmp, gif, psd, sci, or tga.
source