greenscreen transparency issues

# 1 18-09-2012 , 09:30 PM

Registered User
Join Date: Jun 2008

Location: Los Angeles, CA

Posts: 229

This is a rough key test. My character needs to be inside a ball at all times, keyed from greenscreen out of After Effects. I'm using Straight Alpha, not Premultiplied.

This is a render of a surface shader plane in the bg, the ball model and texture I've settled on, and a surface shader w/ keyer alpha image sequence plane for the character, inside the ball.

This is a photoshop comp - top layer is just the png coming into maya. the other is the tga for the bg plane. Just to show that the key doesn't suck as much as it looks like it does in maya. (the key DOES suck, but that's just cause this is a test)

Any ideas why non-extreme transparency values are rendering strangely? Things I've done:

Lambert (incandescent + transparency) instead of surface shader
Removed BG plane
Removed Ball object
Removed ALL objects except character
Used TGAs instead of PNGs
Tried Maya Software AND Mental Ray renderers

The other option is to perform the composite in After Effects, but though importing camera motion from maya to ae is pretty straight forward (once you know how to dodge the common issues), it takes quite a bit of time that I won't have once I start rendering. Plus, I don't know how to comp her INSIDE a semi-transparent maya render.

--
Peter Srinivasan
Producer

Quote

# 2 19-09-2012 , 05:52 AM

Registered User
Join Date: Jun 2008

Location: Los Angeles, CA

Posts: 229

Update: Switching to a mr_material_x_passes shader, dropping file.color into mr_material_x_passes.additionalColor and the alpha into mr_material_x_passes.cutoutOpacity.

Here's the render. Black lines instead of white... improvement?

Attached Thumbnails

Click image for larger version

Name: better ball.jpeg
Views: 517
Size: 330.0 KB
ID: 52447

--
Peter Srinivasan
Producer

Quote

# 3 19-09-2012 , 06:17 AM

Registered User
Join Date: Jun 2008

Location: Los Angeles, CA

Posts: 229

Next update: it only gives that black outline when the sphere AND BG are visible when using cutout and additional color (I had to turn backface culling on on the sphere shader, otherwise the black outlines remained)! So I'm pretty close to the solution it seems. But not there yet. With this method, removing either the sphere or the BG gives me perfect alpha. So there has to be some kind of transparency sorting issue with multiple layers of transparency.

--
Peter Srinivasan
Producer

Last edited by petersrin; 19-09-2012 at 06:49 AM. Reason: incomplete

Quote

# 4 19-09-2012 , 06:58 AM

Registered User
Join Date: Jun 2008

Location: Los Angeles, CA

Posts: 229

Final update so anyone looking for the a solution will have it.

In Render Settings>Mental Ray>Quality>Raytracing (must be active), set Refractions and Max Trace Depth to as many as necessary. In my case, 2 was the lowest value that would work.

That was the key. Once that was set I could keep backface culling off. Apparently culling was simply reducing the number of required refraction traces by enough that the default raytrace settings could handle all the necessary transparency effects. Once I keyed in the right Raytrace quality settings, turning culling off was fine user added image

--
Peter Srinivasan
Producer

Quote

# 5 19-09-2012 , 07:41 AM

The Nurb Herd
Join Date: Oct 2007

Location: London

Posts: 2,381

This is a rough key test. My character needs to be inside a ball at all times, keyed from greenscreen out of After Effects. I'm using Straight Alpha, not Premultiplied.

Originally Posted by petersrin

If you are asking what are the white lines. You partly answered your own question. Your image is unpremultiplied.

Quote

# 6 19-09-2012 , 04:19 PM

Registered User
Join Date: Jun 2008

Location: Los Angeles, CA

Posts: 229

I mean, technically with unpremult there will be white lines (or whatever color bg you choose) on the edges in the RGB channels, but that's what the alpha is for. The real issue turned out to be the number of times a ray was allowed to be refracted. Since I was exceeding it, MR just resorted to 0 and 1 for transparency values, thereby making the unpremult lines visible. The final settings removed the unpremult as expected.

--
Peter Srinivasan
Producer

Quote

# 7 19-09-2012 , 04:40 PM

The Nurb Herd
Join Date: Oct 2007

Location: London

Posts: 2,381

haha, sure thing. whatever works for you.

Quote

# 8 20-09-2012 , 03:44 AM

Registered User
Join Date: Jun 2008

Location: Los Angeles, CA

Posts: 229

Interesting. I switched back to surface shader just to see why it was giving me the outline and found my answer here: https://forums.cgsociety.org/showthre...r+transparency

Apparently, unlike ALL OTHER SHADERS, the surface shader applies transparency BEFORE outColor, and anything that is non-transparent is black. Then it ADDS the outColor to the entire image, including transparent portions. This satisfactorily explains and solves a number of my issues better than the solution I gave above. Unfortunately, it also seems completely preposterous. Is it some strange attempt to save processing power? Do surface shaders in other programs, renderers, and/or game engines work like that? Or was the link mistaken, but the solution still fine?

--
Peter Srinivasan
Producer

Quote

greenscreen transparency issues

Posting Rules Forum Rules

Similar Threads

Topics

Free Courses

Full Courses

VFX News