Okay, this is weird. I tried creating a render layer override on the color gain, which works fine in the render view (current frame), but when I do a batch render, it includes the IBL.
I added more layer overrides for "visible in final gather" "visible as environment" etc. and that worked, but then when I removed those overrides, it still worked and respected the color gain override. Weird.