ummm... are you rendering this in maya? or pulling it in from actual footage? if you rendered it in maya, you can map to the alpha channel, that should remove at least some of the pixelation problems - meaning you're not removing anything, because there's nothing there. also, probably upping the quality will help, as well. If you're using actual footage... you're probably going to clean it up by hand... even with the best computers, I think most studious still go back and clean up the edges of things (although the have to do less cleaning with green than with blue).
Book Wise
https://X4nd5r.deviantart.com