I had an enormous amount of trouble setting that stuff up, partly because I also wanted to use Common Lisp. I also was unable to uninstall any package.

Everything about package.el and ELPA has always given me the feeling that it's a bit rough. My hope is that Emacs 24 will straighten it all out.