Typically, the Fresnel effect causes images to soften around the edge of the reflective surface. So save a copy of the scene and then delete all the textures. Create a new texture with no channels activated, except for the color channel. Make the color channel 100% white and apply it to all objects in the scene. Now remove all light sources and create a new light point that is placed in the same location as your camera.
If you were animating this scene, you would want to bundle the light to the camera if the camera was moving. This light source should have no drop or shadows and be 100% full intensity. Render this pass and call it “Fresnel.tif”. Now, to control the fresnel effect of the reflection, you must have four separate files of four render passes: Mask Pass, 100% Reflection Pass, 0% Reflection Pass, and Fresnel Pass.
The composition of these passes can be done in Photoshop or any image editing program that allows multiple layers. However, it’s very easy in AfterEffects, and if you’re doing animations, AfterEffects is the only way to go. Therefore, we will look at the compositing process in After Effects. Within AfterEffects, import all four passes into a project (File>Import>Media Files).
Once they are all imported, create a new composition (Composition>New Composition). Be sure to size the composition to match the size of your renderings and your desired final size. In this case, the renditions are 800 x 600 pixels in size, and since it is a still image, the length does not need to be more than one frame. In this first composition, we’ll want to start affecting how the fresnel effect plays out.
Drag the 100Reflection.tif file from the Project window to the Composition window or Time Layout window. Then do the same for Fresnel.tif. At the bottom of the Time Layout window is a button called “Switches/Modes”. Click this button so you can see the Mode and Tracking Matte features of this composition. Change the mode of 100Reflection.tif to “Add” and change the TrkMat setting to “Luma Inverted Matte”.
What this does is make Fresnel.tif (or our Fresnel pass) a matte for the 100% reflection pass. A luma inverted matte defines pixels to be opaque when the luma value is 0%. Notice in the composition window that the center of the ball is now dark and “fresnelled”. Now we need to mask the effects of the fresnel pass from the rest of the image. To do this, create a new composition (Composition>New Composition) and place “Comp 1” from the Project window into the Composition or Time Layout window.
Now place MatteMask.tif in the same way and click on the Switches/Modes button. This time change Comp 1 Mode to “Add” and change the TrkMat to “Luma Matte”. What this does is use the first comp (the one where the fresnel pass was applied to the 100% reflection pass) and mask everything except where the ball is visible as defined in the MatteMask.tif render pass. Create a third composition (Composition>New Composition), and in this composition first place 0Reflection.tif in the Composition or Time Layout window and then place Comp 2.
This places the raw file with textures and lighting for all the objects at the bottom, and then places a bit of the sphere that has been altered via the layers we did in Comp 1 and Comp 2 on top. Click the Switches/Modes button and set the Mode for Comp 2 to “Add”. You can further refine the effect by applying a blur effect to the reflection. You’ll want to do this inside Comp 2 and apply the effect to Comp 1.
To do this within the Time Layout window, click on the Comp 1 line (not the tab) and then go to Effects>Blur & Sharpen>Gaussian Blur. A new window should appear in your work area that allows you to adjust the degree of this effect. Adjust to your liking, making sure to click the Comp 3 tab within the Time Layout window to occasionally check the results. When you like what you see in the final Comp 3, go to Composition>Save Frame As>File…. This will allow you to save the file as a Photoshop document which you can then save as jpg, tiff, or whatever format you need.
If it was an animation, I’d go to Composition>Make Movie. It would take a while for After-Effects to process the adjustments you’ve made to all the footage you’ve imported. You may be thinking, “Well, if I have to render in AfterEffects anyway, why not rework it in the 3D application?” In some cases this may be a smarter way to do it; however, in most cases, AfterEffects will render many frames in a minute instead of one frame in many minutes, as most 3D applications do with large, complex files.
Also, in some situations like the one discussed here, we’re creating effects that can’t be easily achieved within the 3D application, so AfterEffects (or Photoshop) is the only way to do it. It’s yet another argument for never getting too attached to any one program. Use the app or combination of apps that do the job to create the visual effect you want.