I have just about everything of his, and I always recommend "Ease Down the Road" as my favorite of his. Most other people say that "I See A Darkness" is his best. Any of the stuff he's put out under the Bonnie Prince Billy name is worth hearing, IMO.
The Palace stuff is also worth checking out, but can be more hit or miss.
* "Summer In The Southeast" - nice live album
* "Sings Greatest Palace Music" Nashville-ized re-recordings of older stuff, including Palace stuff
By the way, here's a complete (?) discography: http://users.bart.nl/~ljmeijer/oldham/discog.htm