Hey there nickynacky!
Thanks for the response!

My primary vis was turned off, that wasn't the problem.
The problem was, that although not selected for specific render layer, the IBL still contributed to it (reflection and indirect lighting).
Still asking myself what the hell went wrong.

I guess that I would have to use an override for the "visible as environment" checkbox, but still, very strange..