REAL D Blog

A Step Towards the Classification of 3-D Displays

2008-06-27T21:42:00Z

A recent article in a prestigious journal has inspired this blog. In it was included a foldout chart classifying stereoscopic moving image systems. The chart was obscure and confusing. I prefer to have people understand stereoscopic imaging, and the chart is of no help. It isn’t as if the classification of stereoscopic imaging systems is at the same level of complexity as that of the Periodic Table. But classifying a technology family, a system created by the human mind, can also be challenging.

I divide the field into two parts: projection systems and other displays such as flat panels. There are only three ways that a plano-stereoscopic image (a two-view system) can be selected. You can select the image based on color (or wavelength), time, and polarization. One interesting thing about this is that the time selection technique can work with the other two. In fact, for the present generation of stereoscopic projection systems using the Texas Instruments DMD engine, time is combined with the other two methods to provide a shuttering system; so you can’t classify any of the present projection systems under one category, except for shuttering eyewear that use time for selection.

You can have a train of left-right images, variously described as field-sequential or time-multiplexed, in which shuttering eyewear like CrystalEyes are used. XpanD eyewear are used in some theatrical installations and their eyewear shutters open and close in synchrony with the video field rate and, by a well-known principle, when the left image is on the screen the left eye is seeing an image (and the right is blocked), and vice versa.. Provided that the repetition rate is high enough, you see left images with the left eye, right images with the right eye, and a good-quality stereoscopic image (if everything else is done correctly) results. This is time-division multiplexing or temporal multiplexing, and it uses a shutter. There are two other systems that I alluded to. One uses color and the other uses polarization. As noted, both are combined with the temporal multiplexing or shuttering approach for projection.

Color selection has been called the “anaglyph,” and I am not going to depart from that terminology. Generally anaglyphs use broad filtering for two halves of the visual spectrum – one towards the reddish end and one towards the blue end. Such a technique can use two projectors, or the images can be combined on a single file or print and projected using a single projector.

The Dolby system is the modern version of the anaglyph, and they license the technology from INFITEC. It is a wavelength selection system but it uses very narrow spikes of filtration at three parts of the visual spectrum which are at different locations for the left and right eye. In this way you can get good color images, unlike the traditional anaglyph (which I find to be an abomination for 3-D projection). Dolby uses a spinning filter incorporated into the projector so that the output is a sequence of field-sequential color-encoded images. When combined with proper selection device eyewear, the left eye sees only the left train of images and the right eye sees the right train of images. One advantage is that you don’t need shuttering glasses, which are electronically driven devices that presumably would be more costly than passive devices that employ filtration. Unfortunately, this isn’t the case with regard to the Dolby system in which the selection devices’ lenses, using retarder stacks, cost about as much as the shuttering eyewear.

The next system I’ll describe uses polarization. There are two kinds of polarization: linear and circular. Circular polarization is outputted by my invention, the ZScreen. The ZScreen, when combined with the single DMD projector, produces a selection technique that can be classified as both polarization-selection and temporal-selection. When you’re designing a system like this you have to pay attention to both the polarization and shuttering aspects of the design. That means you need a polarization-conserving screen, which both the Dolby and the XpanD shuttering system don’t require. The ZScreen alternates the characteristic of polarized light at the frame rate to produce alternate trains of left and right images with polarization encoding. When you put on polarizing glasses using this system you’re actually looking through a shutter and the parts of the shutter are distributed among the eyewear, the screen, and the ZScreen. You can say the same thing about the Dolby system. The Dolby system is a shuttering system as well as a color-selection system, with the parts of the shutter distributed among the eyewear, the screen, and the spinning color wheel incorporated in the projector.

For polarization, what has been done since the late ‘30s (and perhaps even earlier) is to put polarizing filters over the left and right projectors. You need a polarization-conserving screen and everybody wears 3-D glasses with polarization filters which could be circular or linear.

For projection systems the wavelength and polarization techniques are combined with the temporal technique for a single-display-device projector (in other words, a single projector coming out of a single optical path). For flat panels or electronic displays of any type we can do the same thing, and we can also use a dot- or line-sequential (spatial) approach. If you have a single display, somehow or other on the surface of the screen, either in time or in space, you need to share the image. This sharing is then combined with the other selection techniques by a means that is analogous to what has been described with regard to single projector systems.

Let’s take the case of a liquid crystal display, because that is the dominant display and will be the dominant display for years to come. If you could make liquid crystal displays run fast enough, you could view the display stereoscopically using shuttering eyewear, for example. The only viable means art this time is line-sequential selection combined with microscopic polarizer. The dominant player in this field is Arisawa Corporation, and they manufacture a micro-retarding device that is applied to the surface of a liquid crystal display. Because the display already uses a linear polarizer for image formation the combination produces areas of left and right handed circularly polarized light. This produces a line-sequential display that alternates, in the case of their embodiment, left- and right-handed circular polarization states in horizontal lines.

The CRT display is history for the most part but for years it dominated desktop stereo displays using the field-sequential technique. Flat panel displays can, without equivocation, locate each pixel, but you can’t do that with a CRT. However, CRTs are fast enough to use either shuttering eyewear or a polarization modulator placed in front of the screen. CRTs have been supplanted almost entirely by liquid crystal displays. That’s too bad, because they work very well for stereoscopic desktop applications for scientific imaging and visualization.

The other type of display that has characteristics similar to a CRT display is the rear-projection television (RPTV) sets made by a number of companies who license the technology from Texas Instruments. To get a high resolution of 1920 pixels, and because of the physics of the DMD engine, they use the diagonal interlace technique, also called the checkerboard technique. Because of the rapid refresh capability of the DMD they are able to use a time-sequential technique, so shuttering eyewear can be used with these kinds of monitors and produce an excellent image (albeit half resolution in each eye).
After I completed this blog article I hit upon the classification system I will present in the next blog (how I hate the sound of that word) article.

WRONG, WRONG, WRONG:MYTHS OF STEREOSCOPIC FILMMAKING

2008-06-11T20:59:00Z

Every field has its mythology. Myths can build a collective spirit, and help people reach for a goal. But sometimes myths are destructive because they are based on fiction and can prevent people from making properly informed decisions. This is especially true in a nascent field like stereoscopic filmmaking. Although stereoscopic filmmaking has been around for a long time, people haven’t had a chance to practice their skills, so from a craft point of view, it’s still in a relatively early stage. Until lately there has been a lack of proper technology so that theatrical filmmakers could learn from experience. You learn from experience, which often means your mistakes. Although the maxim is learning from mistakes, you also learn from what you do right and what works. Who’s to know which is more important – doing it right or doing it wrong? Obviously, nobody wants to do it wrong, so people tend to get conservative and cautious, especially in an undeveloped field like stereoscopic filmmaking.

One of the vexing issues has to do with the perception of projected 3-D images. We’re not used to looking at stereoscopic images on a big screen. We see stereoscopically in the real world every day, but looking at 3-D movies is not the same as looking at the real world, just as looking at motion in the real world is not the same as looking at movies – that is, photographed images that move. With that in mind, I am going to rail against some of the things I have read and heard recently that annoyed me. I don’t know the answers to all 3-D issues, but many people have made assumptions that they could test if they took the trouble to do so.

The first wrong, wrong, wrong myth is that you can’t do fast cuts in stereo. A recent test done by DreamWorks Animation proved the wrong-headedness of this notion. The test was supervised by our own “Captain 3-D,” Phil McNally. He took a section of Kung Fu Panda, about five minutes long, that has rapid cuts. It’s an action sequence of a snow leopard being released from imprisonment, and it’s spectacular. It leaves the audience gasping and cheering. Why people think you can’t do rapid cutting in 3-D is probably related to the fact that prior 3-D systems of photography and projection were so terrible that fast cuts got blamed for the pain. But what Phil has supervised proves that it’s not the rate of cutting that’s responsible for discomfort.

The next myth is that the breakdown of convergence and accommodation applies to the stereoscopic cinema. Wrong, wrong, wrong. The breakdown of convergence and accommodation is the habituated response we have between the neurological pathway for eye muscles that control vergence (fixation on a point in space) and a separate set of eye muscles and their neurological pathway controlling focusing. These are coordinated in the visual field. When you look at something in the visual world what you are focused on is also converged on. But for projected stereoscopic images that’s not the case for much of what’s going on on the screen. Only action that’s at the plane of the screen with at or near zero parallax involves no breakdown of convergence and accommodation. The very word “breakdown” is a scary term. It sounds like something is falling apart, and the whole ball of wax is going to crumble. That’s the accepted nomenclature and it refers to the fact that this habituated response doesn’t happen for plano-stereoscopic stereoscopic displays. It is a habituated response but the two neurological systems are independent.

When you’re viewing stereoscopic movies, typically you’re sitting tens of feet from a big screen. That’s good, because at such distances there is no breakdown of convergence and accommodation. It’s possible that people sitting in the first row or so of a small theater will have some problem. But the way it works is this: Past a certain distance A/C breakdown doesn’t happen. I’m tired of reading ill-informed articles saying that the breakdown of convergence and accommodation is a big deal for projected 3-D movies. It of concern for small screens like TVs and desktop monitors, because people sit close to them.

Here’s another wrong, wrong, wrong: Films that are composed for IMAX won’t work in RealD. But two shows that I consulted on from National Geographic – Sea Monsters: A Prehistoric Adventure and Lions 3D: Roar of the Kalahari – work perfectly fine when translated from IMAX to RealD. The major concern was aspect ratio – cropping from 1.4 to 1.85. Another concern was to make sure that the zero-parallax plane was properly adjusted. IMAX doesn’t care about making the zero-parallax condition correspond with the plane of the screen, because the screen surround is almost beyond the periphery of the visual field. Real D cinemas have the equivalent effect by using floating windows. I’ve been looking at a clip of The Polar Express for three years on a RealD screen. It looks great and it was prepared for IMAX. You can translate from one to the other. You can translate from RealD to IMAX and vice versa, and the results look good.

Another wrong, wrong, wrong is that everything in a 3-D movie needs to be sharp. For photographers that means lots of depth of field. Perversely, the nomenclature that has grown up for people in CG animation is to “turn off the depth of field.” If photographers say they want a lot of depth of field, the people in CG say, “Let’s turn off the depth of field.” Maddeningly enough, they mean the same thing. For a long time people have thought you need to have everything in a 3-D movie in focus.

The out-of-focus or depth-of-field cue that occurs in cinematography is a unique depth cue because it was created by photography. In the real world you have so much depth of field that nothing is out of focus so there is no resultant depth cue. But we have learned to associate out-of-focus images with images that are in the background. Depth of field is a depth cue but an invented imaging depth cue. It is a depth cue that you won’t see in paintings prior to the invention of photography, but it’s one that we’ve learned to appreciate because of photography. The history of stereoscopy and the history of photography are linked. Wheatstone enunciated stereoscopic imaging in 1838, and a year later he made the first stereoscopic photographs. His initial work was with drawings.

It turns out that, based on tests I’ve done and seen, there’s nothing wrong with out-of-focus backgrounds for 3-D. The question that comes up is: do you need this cue when you’re doing stereoscopic imaging? You probably don’t need the background focus cue that much anymore, because you’ve got the stereoscopic cue. But the background cue doesn’t hurt, and you can use it. However, the foreground out-of-focus thing – like the tip of a finger coming off-screen – might make sense in a 2-D show, but in a 3-D show you’re better off having off-screen objects sharp.

Another wrong, wrong, wrong myth of stereoscopic filmmaking that people have been rapidly learning is wrong in the past year or so, is that stereoscopic photography should be done with the interaxial (distance between camera lens axes) and the interocular or interpupillary (distance between the eyes) with the same value. It turns out that a lot of good stereoscopic cinematography can be done – and, especially on a set, must be done – with the interaxial less than the interpupillary separation. If you don’t do this you can blow out background points. That is, you will have strongly divergent background points if you insist upon having the interaxial equal to the interpupillary. You can also wind up with exaggerated stereoscopic depth effects in which objects look elongated. The point of stereoscopic filmmaking is to make images that are enjoyable to look at, and that leads me to my next point.

Wrong, wrong, wrong: Stereoscopic movies must be orthostereoscopic. There’s a concept in photography called orthoscopy in which certain geometric constraints have to be taken into account in order for an image to have the same appearance that it would have in the visual world. Given a certain focal length of a lens you have to be a certain distance from a projected image, given its magnification, for the image to be orthoscopic. In other words, it needs to subtend the same height on your retina as it would have in the visual field from a given distance. Human beings can’t change the focal lengths of their eyes (unless you’re looking through a telescope or binoculars). But filmmakers can. Typically, for orthoscopy to work, images shot with wide angle lenses have to be viewed close, and images shot with telephoto lenses have be viewed from far. To pull it off, everybody in a theater would have to keep moving around to different seats or you’d have to change the size of the projected image for on a shot-by-shot basis. But this is not done because we have learned how to look at projected images shot with different lenses.

The planar orthoscopy conditions apply to orthostereoscopic conditions. The orthostereoscopic condition also includes the condition that the interaxial equals the interpupillary. This is a condition that can only be fulfilled for some of the people in the audience. That’s because the mean interaxial separation for human beings is 65 millimeters. I don’t know the value of the standard deviation, but from kids to adults (with big heads) it’s anyplace from 45 to 74 millimeters. So that means that if you shot for one set of eyes the majority of people would not be seeing an orthostereoscopic image. For an orthostereoscopic image to be seen, in addition, you need to be sitting in the dead nuts middle of a row, at exactly the right distance given the focal length and magnification. And there’s no good reason to do any of that, because the point of stereoscopic filmmaking is to create an enjoyable image, not a scientifically correct one.

Another wrong, wrong, wrong, myth, has to do with divergence – that you’re going to tear people’s eyeballs out of their heads if you have more than 2-1/2 inches (the nominal interpupillary separation) of background parallax. It’s a good thing to avoid divergence, but think about it this way: For people who are sitting an average viewing distance from the screen, the difference between 2-1/2 inches and 3 or 4 inches of background parallax is mice nuts. That’s because the best measure of parallax is angular measure, not that which you can measure by laying a ruler on a screen. Large values of parallax, either off-screen or divergent parallax, have the biggest effect on people who are sitting up close. But for the vast majority of people in a theater, a little divergence is probably not going to hurt them very much. Discomfort also depends on how long the shot is on the screen, how sharp it is, and a whole bunch of other things. Stereoscopic filmmaking is an art. But for CG animation there’s no excuse, if you’re shooting for a certain size screen, to have divergence. Cinematography (with cameras) is a more difficult art.

But there’s a lot of flexibility, and you’ve got to use common sense. Human beings have a flexible visual system.

“Silver screens are crap.” Wrong, wrong, wrong. There’s a prejudice about silver screens, despite the fact the movie industry itself is described by the term “the silver screen.” There are some powerful producers and technical people in this town who hate silver screens. It’s not altogether an unreasoned prejudice, because there have been problems with silver screens in the past. There has been blotching, visible seams, and hot spotting. Modern silver screens made in the past couple of years can be good. Well-made screens no longer show seams, no longer have blotching, and no longer have hot spotting. Silver screens are different from matte screens. They have higher gain, and they conserve polarization. But they can be used for showing both 2-D and 3-D. The major disadvantage of silver screens is the same disadvantage you have when you sit in the worst seats in the house and you’re looking at a matte screen. Not only is the image “keystone” distorted because you’re way off on the side, but you get shading. By “shading,” I mean that one side of the picture is going to be brighter than the other. The worst seats in the house for matte screens are even worse for silver screens; there’s no denying it. But the vast majority of seats in a normal theater are good. Silver screens, when you project 2-D images on them, have more contrast. 2-D can look better on a silver screen.

Silver screens sometimes have different colorimetric characteristics. Some silver screens may slightly tint the image so that it’s colder. If they tinted the image so that the image was warmer, or more towards the red, I don’t think people would object. But it’s the coldness that people don’t like, and that can be corrected easily when projecting digitally, which is the major use of silver screens.

THE SCULPTURAL CINEMA

2008-02-29T16:30:00Z

A change has taken place in the way some stereographers look at the most basic aspect of stereoscopic composition. This change made its appearance only a few years ago. It provides what could well be described by that tired cliche as a paradigm shift. It's a new way of looking at the world of stereoscopic composition.

So that we're all on the same page, I'll just say a little bit about how people have viewed controlling of the depth effect in stereoscopic images. It is well known and understood, and first enunciated by MacAdam in the SMPTE Journal in the early '50s, that the stereoscopic depth effect is weighted or scaled by extra-stereoscopic or monocular depth cues. That is, the depth cues you can see with only one eye tend to strengthen the stereoscopic depth effect. That doesn't mean that you can't have a stereoscopic depth effect without monocular cues, as Wheatstone first demonstrated in 1838, or was shown by Julesz in his Foundations of Cyclopean Perception.

The strongest extra-stereoscopic depth cues are perspective and motion parallax as noted by MacAdam (who was an important contributor to Kodak's color photography efforts). Perspective is easily grasped. I can think of two examples, one of which is the use of a wide angle lens in which the background appears to be farther away than it would have if a long focal length lens was used. A good example of geometric perspective is a shot of railway ties receding to infinity. Another depth cue, as mentioned, is motion parallax, which involves the temporal rejuxtaposition of foreground and background. In this case there are perspective changes with time, and often achieved in cinematography with a traveling shot in which the direction of the lens axis is more or less perpendicular to the direction of camera travel. It doesn't matter what direction you're moving in. You can be moving vertically, and you will increase the apparent stereoscopic depth effect of the image using a moving camera.

Te basic parameter setting, without regard to monocular cues, and the most direct way to increase the stereoscopic depth effect, is to change the distance between the cameras, or more properly the camera lenses, so that the interaxial separation is greater. There is an easy and enlightening experiment to do involving a series of images of the same scene gradually increasing the distance between the cameras. In the limiting case when there is no distance between the cameras (i.e. they coincide), the image has no stereoscopic depth effect. It is the job of the stereographer to understand how far apart the cameras (or camera heads) should be placed, based on the distance to the principle object and other objects in the scene, the focal length used, and, as mentioned, the extra-stereoscopic depth cues.
This is the heart of my little article: A relatively recent change in the way stereographers can control the depth in a scene is based on the insight that it is possible to use more than one interaxial spacing for objects in a shot or for foreground and background. I can tell you that it's a technique that occurred to me years ago in conjunction with the work StereoGraphics was doing with computer generated images, but there was no motive to employ it. We were happy to have application developers simply add stereo drivers to their software and asking them to do more was, I thought, looking for trouble. The first place I read about the idea was in a paper by Nick Holliman given at the SPIE Stereoscopic Displays and Applications Conference, in which Nick suggested changing the distance between the cameras or lenses for different parts of the scene. (It's also described in U.S. Patent application number 10/596,536 in terms of a specific algorithm for the selection of parameters.)

The most convenient way to do this is with a computer generated image. And indeed this technique has been used, and is being used, by stereoscopic supervisors such as Phil McNally, and Rob Engle. Phil used it on Meet the Robinsons, and Rob used it extensively on the Beowulf. I watched him use it for one shot in particular - the shot where Angelina Jolie appears out of the pool in the cave. I saw several versions of the shot in which she originally appeared covered with scales, in the next with mud, and in the final version with liquid gold dripping from her body (or somebody's body). In this case, Rob judiciously used different interaxial separations for Angelina and for the cave wall that was behind her.

For CG animation the technique it's a good idea, because nobody cares about reality - whatever that is. Stereoscopic supervisors who are working in CG have no impediment to having interaxial separations that are different for different parts of the shot. They can use two, three, four, or even more to control the depth of characters and the stereo strength of the backgrounds.

This makes me think of those old stereo cards I've seen, where somebody has taken a picture of Niagara Falls, for example, and the left and right images are essentially identical - so when you look through the stereoscope you see a planar image surrounded by a stereo window. If that's what people think stereo is, fine, but I think that's not asking for much. That shot of Niagara Falls should have been taken with the cameras placed feet or maybe yards apart, to get a decent 3-D effect. What's the point of having a 3-D picture if it doesn't have a stereoscopic effect, unless the lack of depth is the effect sought?

Live action can use the technique too with blue or green screen. In fact, without trying it's automatically going to happen. Blue screen or green screen is almost always going to be CG originated these days, and the interaxial the director of special effects chose - hopefully in conjunction with the stereographer on the shoot - is not going to be the same as what was used in the camera setup. It won't be for reasons of scale. I've seen recent fun examples of this combination of live action and CG in Journey to the Center of the Earth 3D. In addition, science fiction movies, in order to be satisfying, are going to need camera separations that might be miles or hundreds or thousands of miles apart to get decent depth effects for planets or spaceships.

In addition to these are examples of interaxial separations that can be controlled based on the ability to have multiple stereo-cameras in the shot there is another approach that also tosses out the (stereo) window the notion of a fixed interaxial. The technique that has arisen I'm going to call the "virtual interaxial separation." For converting (stereoizing or dimensionalizing) planar to 3-D, the technique that's usually used - and there are really just a couple of them - is to outline the principle object, typically a person, against a background. Roundness is added to the foreground figure by treating it with an operator selected depth map. The hidden or missing background material is filled in by cloning or an artistic endeavor. Philips, in their patents, calls it "plugging," which is a pretty good term - you're plugging a hole. Depth maps can be created for the background as well.

What you're creating is a bas-relief or a false front; and this false front can be manipulated so that the objects are sculpted to be elongated or manipulated in the Z axis. The concept of interaxial may not make a lot of sense because you'e actually creating stereo-pairs independent of cameras and their limitations. Sometimes you might be simply accepting the left view and then creating a new right view, and how that right view is sculpted will determine the depth effect in the shot. Another way to handle conversion is to take a 2-D skin and lay it on a 3-D model, in which case you can also manipulate the Z axis locally and place "cameras" in the space any which way. The stereo strength of objects can be sculpted. This technique has been used in CG animated movies.

There is a concept called orthostereoscopy, and one condition it must fulfill is that the interaxial separation must be equal to the observer's interpupillary separation (obviously impossible in most cases since peoples' eyes have different Interocular separations). Other ortho conditions have to do with the geometry of the shot so that the perspective considerations are maintained, with regard to the projected image and the conditions of photography. The concept of orthostereoscopy doesn't have much importance in entertainment. The creative people in the industry have grasped this and entirely new techniques are being developed that heretofore hadn't been imagined. There is now, in this the early days of the stereoscopic cinema renaissance, a breakthrough in the concept of stereoscopic composition, making the creation of images more nearly related to sculpture than photography. I'm sure there are more innovations to follow.

CONVERGENCE VS. CONVERGENCE (or struggling to find meaning in the world of stereoscopic filmmaking)

2008-02-18T20:02:00Z

One of the most confusing and misused terms in the field of stereoscopic cinematography is the word convergence. In common usage the word means a coming together. Marketing people in consumer electronics like to use it to refer to the coming together of PC and TV technology. It's also a technical term that is used in the jargon of displays. For example, for projection of displays that use three image engines – red, blue and green – the term is used to describe how well the three optical paths are combined into one image. If they don't combine nicely you'll get color fringing.

We have to draw the line when using the term convergence or we will wind up with everything coming together in a reverse of the big bang, and then where would we be? Everything in the world would be the same thing and there would be no difference between Haagen Dazs and De Beers.

In stereoscopic filmmaking, the term convergence can mean different things leading to even more confusion in a world already rife with mind numbing farragoes. Human beings have a visual system that uses two eyes to see the world and our eyes' optical axes cross or converge on the object on which we are concentrating. The eyes need to verge either inward or outward. If they verge inward the term is called convergence, and if they verge outward the term is called divergence. Depending upon where you are looking in space – if you're looking at something that is farther or closer to your point of convergence – your eyes will have to either converge of diverge.

The purpose of vergence for your eyes and your eye-brain in the visual world is to place the image of greatest interest – what you're concentrating on – on your fovea centralis. This is the part of the fovea that has most of the cones. It is where the eye-brain provides high definition and where fusion is brought about; that's the ability of the eye-brain to turn these two views of the world with a single point of convergence into the stereoscopic depth sense.

The term convergence, or vergence, means something to visual scientists and it means something else to stereographers. For stereoscopic displays the meaning is different enough to be worth talking about. If portions of left and right projected images superimpose perfectly they can be said to be converged, and have zero parallax. They appear to be (more or less) in the plane of the screen. The term convergence is also used to refer to the rotation of camera heads to achieve that end, because if the optical axes of the left and right heads' lenses cross on a point in space, that point will have zero parallax upon projection.

Some workers talk about converging the cameras (or camera heads), but others talk about toe-in to mean the same thing. Toe-in refers to the inward rotation of the cameras to achieve a zero parallax setting. (Sometimes one would have to toe-out to achieve the zero parallax condition.)

People who are doing CGI animation don't have to use toe-in or its geometric equivalent rotation. Toe-in is not required because these workers can use cameras that have parallel lens axes, and these parallel lens axes look straight ahead at the world and the "convergence" can be achieved by horizontally shifting the left and right images with respect to the lens axes to achieve zero parallax. Without any toe-in one can achieve zero parallax without distortion.

I'm talking about trapezoidal distortion, which is sometimes referred to as keystoning. Rotation to achieve the zero condition will produce trapezoidal distortion because the two images have, in effect, placed one portion of the object closer to the camera sensor than another resulting in differential magnification across the image field. One way to beat this with real-world photography is to horizontally shift the lenses or the sensors to achieve so-called convergence. I call that HIT for horizontal image translation. The term convergence is confusing in these contexts because it can relate to both the zero parallax condition and the means to achieve it during photography by toe-in or horizontal lens axis shifting. Because of this confusion I call this the zero parallax setting (ZPS). I don't like the term convergence when I am talking about this idea.

From the get-go stereographers have been arguing about convergence. They say: It's okay to toe-in the cameras because the eyes toe in. But eyes and the brain are not the same as cameras and projectors.

Toe-in, not convergence, should be the terminology used for the inward or outward rotation of cameras to achieve a zero parallax condition. Although it's undesirable to use toe-in because of the design of some camera rigs, it's expedient and its resultant trapezoid distortion, which leads to vertical parallax in projection. However, in this the age of digital post trapezoidal distortion can be successfully countered in post using image distortion tools. Although I can't kick about toe-in for features because or the rectification possible in post, for live camera feeds to theaters, for example, toe-in would probably create some real problems.

In a nut shell: Don't use the term convergence when you are talking about setting the zero parallax point (or plane) especially if you are doing CG because it could mean toe-in. Note: Toe-in cannot produce a zero parallax plane. A plane of zero parallax can only be achieved by means of horizontal translation of the lenses' axes with respect to the image sensors. Toe-in can only produce a warped surface with a zero parallax locus.

Make a distinction when using the term convergence between projected superimposed image points and the act of toeing-in the cameras heads.

As the poet said: A rose by any other name smells as sweet but please don't smell my feet.

Stereoscopic Filmmaking Chronology

2008-02-15T18:06:00Z

Introduction

Those of us who love stereoscopic filmmaking are also interested in its history. Our most distinguished stereoscopic cinematographer, Peter Anderson ASC, has put together the chronology that is reproduced here. It’s meant to be a work in progress and Peter and I are looking forward to additional entries (or corrections) from readers. Neither Peter nor I are historians, but we are co-chairs of the ASC Technology Committee Sub-committee on Stereoscopic Cinematography, and we’d like to advance the cause of scholarship in the field by offering an accurate chronology.


A brief overview of Three-Dimension film milestones
by Peter Anderson, ASC

1838 Sir Charles Wheatstone (England) developed the stereoscope.

1844 David Brewster (Scotland) developed the (still) photo stereoscope.

1862 Oliver Wendell Holmes (U.S.) perfected and popularized the photo stereopticon.

1886 Louis le Prince (France and Scotland) patented and built a basic motion picture camera system capable of producing 3-D images. He and his equipment mysteriously disappeared while on a train ride in France.

1890 Joseph d'Alemedid (France) projected 3-D anaglyphic still images.

1916 The Society of Motion Picture Engineers (now the SMPTE) (U.S.) is incorporated. The Society’s first president (and co-inventor of the motion picture projector) C. Francis Jenkins predicted that 3-D films will become popular if one could get away from using special glasses.

1922 Harry Fairall (U.S.) produced what is probably the first true 3-D movie "The Power of Love" in the anaglyphic stereo process.

1923 William Cassidy (U.S.) presented "Teleview" using individual handheld motorized interlocked viewers and dual synchronized projectors.

1925 Able Glance (France) filmed parts of his epic "Napoleon" in anaglyphic stereo, but deleted it from the final film.

1936 Dr. Edward Land and George Wheelwright (Polaroid Corp.) (U.S.) developed and presented the first polarized 3-D movie.

1939 John Norling and Polaroid Corp. (U.S.) produced Chrysler’s "In Tune With Tomorrow", a 3-D film for the New York World's Fair in black & white

1940 They replaced, the above, "In Tune With Tomorrow"
with a color 3-D version. 1.5 million see this film.

1940 Dr. Land (Polaroid Corp. U.S.) developed full color Vectographs.

1952 Arch Oboler and United Artists (U.S.) released the color theatrical 3-D feature "Bwana Devil". It is critically panned but financially successful. Hollywood gears up for $$$ 3-D.

1953 Warner Bros. (U.S.) produced "House of Wax" which was successfully directed by Andre de Toth, who was blind in one eye.

1952 to 1954 Hollywood produced more than 40 feature 3-D films. While several good 3-D films made, many are of poor quality. 3-D’s popularity with the audiences plummeted and thus the studios lose interest in the process.

1954 Paramount (U.S.) released the 3-D Technicolor printed feature "Money From Home" with Martin and Lewis. Other 3-D films were released on a variety of print stocks including Ansco, Dupont and Eastman.

1954 Technicolor and Polaroid Corp. (U.S.) announced a joint venture to create color Vectographic motion picture prints. However, with the demise of 3-D feature production, development is stopped.

1960s and 1970s 3-D production survived mainly in three different domains: exploitation films, experimental projects and the Russian cinema.

1975 and onward Multiplex theaters with their single projector automated projection systems replaced the 3-D compatible dual "changeover" projectors of the conventional theaters ending the era of viable neighborhood 3-D theaters.

1978 Marineland (U.S.) produced Murray Lerner's "Sea Dream". This is the first of the contemporary Special Venue 3-D productions.

1978 Dr. Vetter (Todd A.O.) (U.S.) developed his twin 65mm 3-D systems for large format 3-D filming.

1981 Anthony, Guintano and Lupo (U.S.) released "Comin at Ya!" on single strip 3-D and the studios start looking at 3-D again. Single strip over/under systems temporarily become popular.

1982 Disney (U.S.) produced "Magic Journey" for its EPCOT theme park with their state of the art 3-D twin 65mm camera systems and custom projection system. This was the first 3-D film to combine live action and computer elements in the same image.

1983 Several studios produce 3-D features. Stereovision, Arri and Panavision develop new 3-D lens systems. But, the 3-D films produced could not find an adequate audience.

1984 Disney (U.S.) produced "Captain Eo" for its theme parks. Jackson, Lucas and Coppola teamed up for a big budget 3-D project.

1984 to date Many major theme parks and world's fairs develop high quality large format dedicated 3-D films and theaters.

1990 Imax (Canada) produced the Omnimax film "Echoes of the Sun" for the Japanese World's Fair using their new liquid crystal display (LCD) Solido Glasses.

1990 Disney (U.S.) produced "MuppetVision 3-D" with Jim Henson for its theme park. This was the first 3-D film to continuously adjust the camera's convergence and interocular during filming in order to enhance the 3-D experience and minimize eyestrain and other 3-D problems. The “MuppetVision” theater was designed to make the in theater experience, the animatronic figures, live actors, additional projection systems and physical gags support and interact with the 3-D filmed presentation.

1992 Bayley Silleck (U.S.) filmed "Concerto for the Earth" in Showscan (twin 65mm at 60 fps) for the Spanish World's Fair.

1994   Sony and Imax (Japan and Canada) opened the New York Imax Theater for specially produced 3-D Imax films. This theater uses individual P.S.E. headsets which combine wireless LCD (Solido) glasses with their own enhanced stereo audio sound system. This concept soon expands large format into other Multiplex entertainment centers.

1995 Warner Bros. Recreation produces their “Marvin the Martin” twin 70mm 3-D film. This is the first Large Format Dual Strip 3-D all CGI animated production.

1996 Universal Studios opened its "T2 3-D" venue. This venue is a three-screen interactive 3-D film based experience.

1999 Universal Studios (Florida) opened its "Spiderman 3-D" interactive multi screen ride attraction. This is the first CG film show where the 3-D is actually designed to spatially track (follow) with the guests, in proper 3-D perspective, as they ride through the experience in moving vehicles.

2004 Warner Bros. theatrically released "Polar Express", a Motion Capture based CG animated film worldwide in 2-D. The CG graphics were converted to 3-D and “Polar Express” was also simultaneously released as a successful 3-D Imax film.

2005 Disney released "Chicken Little", a CG animated film in both 2-D and 3-D. The 3-D version was the first 3-D general release in the Real D digital 3-D   theaters.

2007 National Geographic Films released "Lions 3-D" in Imax 3-D. This 3-D live    action film was a 2-D to 3-D conversion of the 2-D live action “Roar: Lions of the Kalahari” 2003 Imax film.

2007 "Beowulf" first film released in all theatrical selection systems: Imax 3-D, Real D, Dolby 3D, and shuttering eyewear.

2008 First 3D concert films released, "Hannah Montana", and "U2 3D".

©Peter Anderson, ASC

Compositional differences: Real D VS. IMAX

2008-02-15T18:04:00Z

The history of western art, of painting and then photography, has been informed by the rectangle. Virtually all painting and photography exists within the confines of a rectangular frame, and it is the edges of the rectangle that create the compositional boundaries and structure. You don’t look through a rectangle when you’re seeing the visual world but the rectangle abruptly limits the visual field in a painting, or a photograph, or a projected motion picture image. The placement of objects in space within the limits of the rectangle determines the composition. Up until now visual artists have had to deal with the rectangle.

With the introduction of head-mounted displays, the rectangle could take a rest – or at least try to disappear. The idea of virtual reality or augmented reality as exemplified by HMDs is a relatively new idea. You can find references to the concept in science fiction in which images are piped directly into the brain.

Granted, most HMDs have a limited field of view and the rectangle remains apparent. But the idea, in its full embodiment, would be for the image to subtend most or all of the visual field, and over the years I’ve seen devices that were designed to do just that – to immerse the viewer in a visual experience. The reason that I take the trouble to describe this is that the IMAX experience is one that attempts to immerse the viewer in the totality of the image – to remove the rectangular boundaries. The Real D experience, and most other motion picture projection, involves the rectangle. The audience member will be aware of the rectangle; and the traditional concepts of composition apply. The kind of balance, juxtaposition and placement of objects within the image field is critical in classical composition but is different for IMAX screens.

In IMAX, in which people are sitting close to a giant screen, the periphery of the screen is more difficult to discern and the rectangle becomes relatively unimportant. The idea behind IMAX is to immerse you in the experience. So people who shoot IMAX movies have to think about a different kind of composition. I’m concerned with the stereoscopic cinema so my remarks are predicated on that interest. In the stereoscopic cinema, the rectangular boundaries are important because of the well-known effect of the conflict of stereoscopic cue of parallax and the extra-stereoscopic cue called interposition. If off-screen (negative) parallax values are occluded by the screen edges – and this is especially true for the vertical edges of the screen – there will be a conflict of cues, which some people (possibly most people) interpret as a region of confusion. Some people may say that the image looks like it’s pulled back into the plane of the screen; some will report that the image looks odd. In any event, it’s something that has to be dealt with in the conventional stereoscopic cinema and doesn’t need to be dealt with in IMAX because the screen is so large that it’s hard to see the edges of the surround. (Another thought to put into the mix that might further confusion rather than understanding is that we are in a time of transition in which people are learning how to look at stereo movies and maybe with the passage of time the screen edge conflict will come to be accepted.)

The big screen changes the way IMAX composes a stereoscopic image. In fact, when looking at IMAX movies in stereo, everything appears to be playing into theater space. I define theater space as that which is in front of the plane of the screen, and screen space is that which plays behind the plane of the screen, and the boundary layer between the two (at the zero parallax condition) is the plane of the screen. In IMAX, the plane of the screen more or less disappears. It becomes not necessarily unimportant, but it certainly has a different meaning from that in the conventional stereoscopic cinema, or the rectangle-bounded stereoscopic cinema. A lot of IMAX films are set up so that the background points will have about 2-1/2 inches of positive parallax – which is the average interpupillary separation for the adult male population – so in most cases it will avoid producing divergence to produce a comfortable result when viewing background points.

The net effect of looking at an IMAX screen for many people is that everything is playing out into the theater. Some people report that they feel immersed in the screen or in the image and that they feel within the image – and that’s certainly what IMAX is trying to accomplish. Since they don’t have to worry about the conflict of cues at the screen surround, and they’ve got such a big image, their compositional theory is a different one from the conventional stereoscopic cinema.

The upshot of this for IMAX is that it has a tremendous parallax budget. Since they don’t have to worry about a conflict of cues at the screen surround, they’re free to have large values of parallax. In fact, the last IMAX film I saw in a preview (and because it was in preview I don’t think it’s fair to critique it or mention what the film was) had parallax that was measured not in inches, not in feet, but in yards!

To some people this is fun. But to me, it produced an obtrusive image that I felt like I couldn’t get away from. The truth is that, for me anyway, the IMAX stereo experience is not a lot of fun. Most people may not share that experience and the last time I went to see an IMAX 3-D feature with my family my wife and kids had a good time even if I didn’t. I also find the big parallax values tiring on the eyes, and the increased ghosting with just a little bit of head-tipping not to be a plus.

For the Real D cinema and for other stereoscopic processes projecting on conventional cinema screens, which are typically between 20 and 50 feet in scope, the experience is different. Directors of stereography like Phil McNally and Rob Engle, at DreamWorks and at Sony Imageworks respectively, have been using a technique that allows the conventional cinema to also have a very large parallax budget.

It was almost two years ago that I sat with Phil McNally in the Egyptian Theatre on Hollywood Boulevard and watch films at the 3D Expo. We saw shorts that had been shot by Raymond and Nigel Spottiswoode in the ‘50s. These were in black and white in the 1.3:1 aspect ratio. Before we went into the theater, Phil had talked about the idea of the floating window that he said he’d read about it in my book, Foundations of the Stereoscopic Cinema. Phil and I sat together and watched the Spottiswoode films with the floating window, and Phil said (with a British accent): “This is gonna work”; and he used it while he was the stereo supervisor at Disney for Meet the Robinsons. He used vertical edges that were added to the left and right edges of the picture frame to build a virtual surround – in other words, a floating window that hovers in space between the screen and the audience so that images with large parallax values at the edges of the screen will not have a conflict of cues.

One paradox of the stereoscopic cinema is that from the plane of the screen to stereo-optical infinity there is a parallax budget of only 2-1/2 inches (after which divergence sets in). But you can have many inches of off-screen parallax without bothering anybody’s eyes. While it may not be an order of magnitude – maybe it’s half an order of magnitude – that gives you a big parallax budget, certainly good enough to represent any shot. The problem that is solved is that there is no longer a parallax-occlusion conflict occurring at the edges of the screen. Movies like Meet the Robinsons and Beowulf have used the floating window to good effect to increase the parallax budget.

Since the allowable theater space parallax is much greater than the screen space parallax, this gives the filmmaker a chance to exploit the medium and to increase the depth effect. In fact, it is now on a par with that which is in IMAX theaters. However, the image is still contained by the rectangle. People talk about this stereoscopic method as being immersive, but who knows exactly what that means? They may very well feel more immersed in the picture because it is a stereoscopic picture with a bigger range of parallax. One thing is certain: A trade-off has been made between parallax budget and off-screen effects. That’s because the amount an object appears in front of the screen is compared to the plane of the screen that is usually defined by the screen surround (the physical black border around the screen edges). With a printed on surround the frame of reference is now literally the virtual window floating in space. But the parallax values of the windows can be varied from shot to shot, and even within a shot and the values for the left edge don’t have to match the right edge so shots requiring off-screen effects can be had. The virtual window can be thought of as an animated window that can be placed at the Z location at will so off-screen effects can be accomplished.

A Glossary for Stereoscopic Filmmaking

2008-02-08T19:15:00Z

Introduction

In any discipline nomenclature turns out to be of obvious importance. It’s crucial for all the people who are doing the thing to agree on the same set of definitions. Without that, it’s impossible to communicate – or it’s impossible to communicate without ambiguity. Part of my job at Real D is to work with filmmakers and studios to help them (and me) figure out how to make stereoscopic movies. This is an art that is being invented. Digital projection has opened the door for advances in stereoscopic content creation. Whereas in the past stereoscopic filmmaking was an erratic or a sporadic activity, or at best one that was practiced only by a handful of people typically for large format or theme park attractions, the door has now been opened for the conventional theatrical cinema, which has a greater activity level.

In my trips to studios and filmmakers, I have learned that people are using different terminology. Sometimes they are trying to say the same thing with different words, and sometimes they are trying to say different things with the same words. It’s confusing, and it doesn’t help to develop a nascent art form to have such ambiguity or downright confusion. So here is my stab at a glossary that is based on the one that appeared in the CrystalEyes handbook that I put together when I was at StereoGraphics. I am hoping that in the weeks and months that follow the readers and I will have a chance to add to it. So if you have entries or ideas for concepts that need to be defined, send them to me.

I’ll provide a relatively benign example of what can go wrong. In this example of a lack of precision everybody can more or less figure out what’s meant. It’s the matter of interaxial versus interpupillary or interocular. When doing stereoscopic cinematography, whether you are talking about virtual cameras in a computer-generated world or real cameras in the visual field, there needs to be some accepted term for the distance between the cameras; or more specifically the distance between the camera lenses; and to be even more specific, the distance between the camera lens axes. The interaxial is the distance between the optical axes of the camera heads’ lenses. I call them camera heads since a stereo rig (or camera) has two heads and if we call the twin lensed camera a camera we have failed to distinguish between the camera and the camera heads. It’s the distance between the lens axes that, to a large extent, determines the strength stereoscopic effect.

The distance between people’s eyes, depending upon what field of medicine or science you’re in, is called the interpupillary or interocular distance. The distance between your eyes’ lens axes when your eyes are converged at infinity is a number that doesn’t change once you reach adulthood, and the spread is between 55 and 75 millimeters for the adult population. Typically, the interpupillary or interocular distance is given as an average of 65 millimeters for men. This has led some misguided workers to believe that stereoscopic movies should be shot with camera lenses whose axes are always the interpupillary separation.

People who are involved with the capture of stereoscopic images will use the term “interpupillary” or “interocular” when they mean interaxial. A camera lens, whether it’s a virtual or a real camera, has a lens axis that passes through the optical center of the lens and, if the lens is set up correctly, is orthogonal to the plane of the image sensor. The distance between the left and right axes is called the interaxial separation. So if you’re talking about the distance between people’s eyes, please call it interpupillary or interocular and everybody will know what you’re talking about. If you’re talking about the distance between camera lenses, call it interaxial so we’re all speaking the same language.

The Glossary (a work in progress)

Accommodation. The focusing of the eyes -- or more properly the ability of the eyes’ lenses to change shape in order to focus.

Accommodation/Vergence Relationship. The learned relationship established through early experience between the focusing of the eyes and verging of the eyes when looking at a particular object point in the visual world. Usually called the accommodation/convergence relationship (or the convergence accommodation relationship.)

Anaglyph. Wavelength selection using complimentary colored images and color filters to filter or pass the appropriate perspective views to the appropriate eyes.

Autostereoscopic. Sometimes called auto-stereo, which can be confused with a car sound system.

Beamsplitter. Technically this is a couple of prisms cemented together with a semi-silvered layer to split a light beam into two halves. For the rig used for stereo-cinematography a thin sheet of glass is used in the optical path that is semi-silvered and such a device is more properly called a pellicule (or pellicle).

Binocular. Two eyes. The term binocular stereopsis (two-eyed solid seeing) is used in some psychology books for the depth sense more simply described as stereopsis.

Circular polarization. A form of polarized light in which the tip of the electric vector of the light ray moves through a corkscrew in space.

Conjugate Points. See Corresponding Points.

Convergence. The inward rotation of the eyes, in the horizontal direction, producing fusion. The more general term is vergence which includes inward and outward rotation. The term has also been used, confusingly, to describe the movement of left and right image fields or the rotation (toe-in) of camera heads.

Corresponding Points. The image points of the left and right fields referring to the same point on the object. The distance between the corresponding points on the projection screen is defined as parallax. Also known as conjugate or homologous points.

Crosstalk. Incomplete isolation of the left and right image channels so that one leaks (leakage) or bleeds into the other. Looks like a double exposure. Crosstalk is a physical entity and can be objectively measured, whereas ghosting is a subjective term.

Depth Range. A term that applies to stereoscopic images created with cameras. The limits are defined as the range of distances in camera space from the background point producing maximum acceptable positive parallax to the foreground point producing maximum acceptable negative parallax.

Disparity. The distance between conjugate points on overlaid retinae, sometimes called retinal disparity. The corresponding term for the display screen is parallax.

Extrastereoscopic Cues. Those depth cues appreciated by a person using only one eye. Also called monocular cues. They include interposition, geometric perspective, motion parallax, aerial perspective, relative size, shading, and textural gradient.

Field-Sequential. In the context of cinema-stereoscopy, the rapid alternation of left and right perspective views projected on the screen.

Floating windows. Invented by Raymond and Nigel Spottiswoode, this is the use of printed vertical bands to create a surround to supplant the physical screen surround. The result is a so-called virtual window that is floating in space to eliminate the screen edge cue conflicts and to extend the parallax budget of the projected image.

Fusion. The combination, by the mind, of the left and right images -- seen by the left and right eyes -- into a single image.

Ghosting. The perception of crosstalk is called ghosting.

HIT. Horizontal image translation. The horizontal shifting of the two image fields to change the value of the parallax of corresponding points. The term convergence has been confusingly used to denote this concept.

Homologous Points. See Corresponding Points.

Interaxial distance. Also interaxial separation. The distance between camera lenses' axes. See t.

Interocular distance. See t.

Interpupillary Distance. Also interpupillary or interocular separation. The distance between the eyes' axes. See t.

Linear polarization. A form of polarized light in which the tip of the electric vector of the light ray remains confined to a plane.

Monocular Cues. See Extrastereoscopic Cues.

Multiplexing. The technique for placing the two images required for an stereoscopic display within an existing bandwidth.

Parallax. The distance between conjugate points. It may be measured with a ruler or, given the distance of an observer from the screen, in terms of angular measure. In the latter case the parallax angle directly provides information about disparity.

Parallax budget. The range of parallax values, from maximum negative to maximum positive, that is within an acceptable range for comfortable viewing.

Planar. Flat. Two-dimensional. A planar image is one contained in a two-dimensional space, but not necessarily one which appears flat. It may have all the depth cues except stereopsis.

Plano-Stereoscopic. A stereoscopic projected image that is made up of two planar images.

Ramsdell rig. See beamsplitter.

Retinal Disparity. See Disparity.

Rig. Dual camera heads in a properly engineered mounting used to shoot stereo movies.

Screen Space. The region appearing to be within a screen or behind the surface of the screen. Images with positive parallax will appear to be in screen space. The boundary between screen and theater space is the plane of the screen and has zero parallax. See theater space.

Selection Device. The hardware used to present the appropriate image to the appropriate eye and to block the unwanted image. For 3D movie the selection device is usually eyewear used in conjunction with a device at the projector, like a polarizing device.

Stereo. Short for stereoscopic. If you are trying to learn about multi-channel sound you are in the wrong place.

Stereoplexing. Stereoscopic multiplexing. A means to incorporate information for the left and right perspective views into a single information channel without expansion of the bandwidth.

Stereopsis. The binocular depth sense, literally, "solid seeing."

Stereoscope. A device for viewing plano-stereoscopic images. It is usually an optical device with twin viewing systems.

Stereoscopy. The art and science of creating images with the depth sense stereopsis.

Surround. The vertical and horizontal edges immediately adjacent to the screen.

t. In stereoscopy, t is used to denote the distance between the eyes, called the interpupillary or interocular distance. tc is used to denote the distance between stereoscopic camera heads' lens axes and is called the interaxial.

Theater Space. The region appearing to be in front of the screen or out into the audience. Can also be called audience space. Images with negative parallax will appear to be in theater space. The boundary between screen and theater space is the plane of the screen and has zero parallax. See screen space.

Window. The stereo window corresponds to the screen surround unless floating windows are used.

ZPS. Zero parallax setting or the means used to control screen parallax to place an object in the plane of the screen. ZPS may be controlled by HIT, or toe-in. We can refer to the plane of zero parallax, or the point of zero parallax (PZP) so achieved. Prior terminology says that left and right images are converged when in the plane of the screen. That term should be avoided because it may be confused with the convergence of the eyes, and because the word implies rotation of camera heads Such rotation produces geometric distortion and may be expedient in camera rigs but is unforgiveable in a CG virtual camera rig.

The Three Dimensional Cinema and Digital Technology: A Match Made in Heaven

2008-01-28T20:23:00Z

The revival of the stereoscopic theatrical cinema is intimately linked to the rise of digital technology for the production and projection of motion pictures. The term “digital” when applied to cinema means many things. Most people would assume a strong linkage with computers; and indeed computers play an important part in the digital cinema, from image capture or generation to projection. Whether the computers are servers or in projectors, the digital cinema depends not only on this technology but on modern display technology, including the Texas Instruments DLP light engines. My purpose is to acquaint the reader with some understanding of how the stereoscopic medium and the digital medium work together nicely for the capture or the creation of stereoscopic images.

It would be more pleasing to me, for one, to call this the electronic cinema, rather than the digital cinema, but this isn’t an article about technology definitions and everybody knows what I am talking about. Oddly, it is electronic movies or television that has begun to replace chemical-based photography, because the current digital cinema is clearly an outgrowth of television. It is this isomorphism that gives the studios such fits because the distinction between the 1920 TV standard and the 2K theatrical standard is of interest to and possibly only noticeable to experts.

It is my purpose to illuminate why the combination of stereoscopy and digital technology is such a neat one, providing so many benefits. First let’s take a look at the content creation aspects of the medium, which fall into several categories: Live-action photography, animation by means of computer generated images, animation by means of performance capture, and conversion from planar to stereo.

The differentiation between computer generated animation and performance capture is one that does not have a sharp dividing line. There are movies that are touted as having performance capture, such as Beowulf, and there are movies such as Monster House that also use performance capture but make no mention of it in their promotion or advertising. In a naïve time the use of rotoscoping, the progenitor of motion capture, was a hush-hush affair and reports of its use in Snow White were denied by Disney. But they obviously used it.

For computer generated images and for performance or motion capture, digital technology plays a powerful role – and indeed it is utterly impossible to conceive of this means of content creation without digital technology. In the 3D boomlet of the fifties, except for a few cell animation shorts, all the features were live action. But in this, the first two years of the renaissance, until recently, all the features were CG animation. Which is history looping back on itself, because stereoscopy was invented using drawings, before photography was invented.

All of the major animation studios that produce computer generated images have physicists and imaging specialists who are attempting to produce a computer world that can be rendered with remarkable real-world fidelity or with controlled departures from the real world, to produce a beautiful visual effect. The people who create this content – the animators, background artists and other specialists – for the most part deal with content creation on an intuitive level. They aren’t doing calculations, but they are using computers. They need to be able to do what they do as any creative artist does, using on intuition to work the medium.

Whether their endeavors are based on animator’s skills or the artist’s ability to create backgrounds, generally speaking they are dealing with three-dimensional databases that exist as algorithms and numbers in a computer. These three-dimensional databases have to be fully rendered and captured by a virtual camera, and for a stereoscopic version what is required are two perspective views; so there must be two virtual cameras. These two cameras must be set up and coordinated according to the geometry of stereoscopic image capture.

The same kind of remarks can be made for performance capture, in which motion vectors of the actors’ bodies and faces are turned into a database. That database is then manipulated into characters that are inserted into a computer generated world, or for that matter the characters could be placed into photography of the real world.

For camera-captured images digital (or electronic) technology leaves film-based photography of stereoscopic images in the dust. Cameras that depend on modern CMOS and CCD technology aren’t digital, but produce analog signals that are captured digitally and are then recorded digitally either on hard drives or tape drives. One benefit of these video cameras is that they can be lighter and more compact than film cameras. This is important because two cameras make a rig, and two big heavy cameras become a big, heavy, clunky rig. Also, it is very good to be able to get the lenses as close together as possible especially for close-ups but also for medium shots.

During capture and immediately after capture it is desirable to look at the images. It is possible to look at the images on various kinds of stereoscopic monitors during photography, and without the need to process film and look at dailies (typically the next day), the cinematographer, the director and other creative and technical people can look at the images right away. In fact, they can often look at them on large screens – sometimes on a theater-size screen. It is very important to be able to do this, because it is so hard to visualize how stereoscopic images will look. It turns out to be a real bear to be able to predict the stereoscopic effect. If you have to resort to calculators and rules to try to figure out whether the image is going to look good, stereoscopic photography becomes difficult to do. But if you can actually see what you’ve done real-time (or shortly thereafter), you can improve and correct and tweak what you’re shooting. The same remarks that are made here with regard to the ability to view stereoscopic camera-captured images apply to computer generated images, because the content creators are able to look at stereoscopic images real-time on their desktops or in their sweatboxes. (A sweatbox is a little theater.)

The same considerations apply to conversion technology. There are a number of firms that now specialize in converting planar to stereoscopic movies. They are all doing more or less the same thing depending on artists and computers to help them get a decent result with reasonable throughput. The basic idea involved is outlining of foreground objects, laying the skins of those objects on a wire frame mesh or a depth map, and treating the background by filling in missing data and modeling the background where required. All of which would be impossible without digital technology.

We have looked at the major ways in which content can be created. We will now look at a vital portion of the filmmaking process, which is post-production. Post-production involves an array of procedures that create the film after photography. These include the manipulation of picture elements and sound elements into a finished product that can then be released to the theaters. When a stereoscopic film is cut it’s a good thing to be able to see it in 3D so that the editor and the director can understand how shots interact with each other. There are many prejudices, opinions and myths about stereoscopic cutting – about what works and what doesn’t work in 3-D movies: for example, whether a lot of depth-of-field is required, whether fast cuts are allowable or slow cuts are better to allow the stereoscopic effect to build. Theories matter only to a small extent. The eyes of the beholder rule. So if editors and directors can see what they are doing stereoscopically, that’s a tangible benefit. And the well-known advantages of cutting a film digitally apply here in spades. It is beneficial because of the difficulties in visualizing stereoscopic images and how shots interact.

An important process for camera generated material in particular is called rectification, which is a term that comes to us from aerial photography. If the left and right images have any distortions or magnification errors, they can, to a large extent, be fixed in post-production by tweaking the geometry of the two images so they correspond. This becomes important for zoom lenses, because zoom lenses have great big problems in terms of optics centration, which causes spurious generation of parallax values. Problems can be fixed in post-production and there are both proprietary and off-the-shelf tools for doing so. For the most part, these errors can be eliminated; and they can also include color and density shifts in the left and right images that can occur in cinematography.

The most important and well-developed element in this current phase of the stereoscopic cinema is stereoscopic projection. By emerging from a single DLP light engine projector, a stereoscopic image can be created using the time-multiplex or field-sequential mode. I am the first person to create flicker-free images for the time-sequential process, and the primary inventory of the major selection techniques used with field-sequential stereoscopic presentations: CrystalEyes active shuttering eyewear and the ZScreen, the electro-optical modulator used by Real D. It fits in front of the projection lens and switches the characteristics of polarized light in synchrony with the projection fields. Other systems are extant, such as shuttering eyewear systems, or the Dolby system which is an advanced form of anaglyph.

Only the projectors made by manufactures licensing DLP technology from Texas Instruments, Christie, Barco, and NEC, meet the required specification for field-sequential 3D. In order to make the Real D, NuVision shuttering eyewear, or Dolby systems work, you have to have a rapid sequence of frames projected on the screen. And only the DLP can refresh fast enough. In the case of material captured at the film standard rate of 24 frames per second, these systems work best when projecting at 144 frames per second. There are two 24-fps images for 48 fps, and each image is repeated three times for a total of 144 fps. The images are concatenated, and a train of images (left, right, left, right, left, right, and so on) reach the eyes. Half the time when you look at the image your right eye is seeing only the right images and is seeing nothing in the left eye, and vice versa. If everything is done right, the result is a good because the left and right images are treated identically by the projector in terms of geometry and illumination. The repetition rate of 144 frames per second lets us approach left and right frame projection simultaneity, another important factor.

Dual-projection systems require a lot of tweaking; and even after they have been tweaked they can drift out of spec. It’s not that it is impossible to make dual-projection systems work. It is simply that they are not a real product that you can count on given current technology in digital cinemas, which requires not only the projection of a beautiful image but a dependable process and an image that does not require constant monitoring.

Digital technology –content creation, post-production, and projection – has enabled the stereoscopic medium to become a part of the filmmaking armamentarium; not only to provide beautiful projection but to provide a dependable product, free from the mistakes of the past, that I don’t want to dwell on because they’re such a bummer. But today’s modern 3D digital projection is free from fatigue and eyestrain, and can now allow content creators to do their best to discover the art of this new medium. We’re going to see several years of experimentation and discovery, and at the end of that time the stereoscopic medium will be on a firm foundation. Creative people will never stop creating, but we will reach a plateau where many of the creative and production technical processes become routinized. Oddly enough, the reintroduction of the stereoscopic cinema comes down to turning that which had been more or less a laboratory experiment into a routine.

And none of this would have been possible without DLP projection which is the invention of Larry Hornbeck, who I just had the pleasure of meeting at the SPIE Stereoscopic Displays and Applications Conference in San Jose. Larry asked me for my autograph, so I asked him for his. As you can see, his autograph is on the back of a pair of paper 3D eyewear, which is entirely appropriate.

The Last Great Innovation: The Stereoscopic Cinema

2008-01-24T21:37:00Z

Attached to this post is an article I wrote for the November/December 2007 issue of SMPTE Motion Imaging Journal. Enjoy.

Download the article here.

3D Zooms

2008-01-24T21:23:00Z

In 1972 or 73 (and I admit it’s so long ago that I can’t remember which year it was) I began to devote myself to stereoscopic filmmaking and technology. That led eventually to the publication of my book Foundations of the Stereoscopic Cinema, and it also led to my career as an inventor in the field.

The first experiments I did involved using super-8 cameras and projectors – specifically Beaulieu and Nizo cameras, and Eumig projectors. The Eumig projectors could be interlocked using a timing belt with a declutching mechanism, and I could really achieve good precision with regard to shutter phase, which allowed me to reproduce some of the experiments that had been described in the literature, specifically by Jones and Shurcliff in the SMPTE Journal of the early fifties. The cameras were interlocked using hardware that I got from Super8 Sound with devices originally made for synchronizing a super-8 camera with a magnetic recorder.

I did a lot of work trying to figure out the basic parameters for doing good stereoscopic photography, and I actually had a fairly flexible system. I projected on a Kodak Ektalite screen that did a wonderful job of conserving polarization and had very high gain. For quite a few years I did experiments in my laboratory in Point Richmond, California.

I wanted to do stereoscopic zooms and in 1973, or maybe 74, I tried it and I learned a lesson that other people have learned: they’re difficult to do. At the time I thought I may have been one of the first people to try successfully to pull it off. I was informed on the subject by an article that was written by a Kodak researcher, MacAdam, which one can also find in the SMPTE Journal. Stereoscopic zooms, it was alleged by some, were assumed to be a no-no – something you can’t do, something that won’t work. The idea was that the effect would be perceptually disturbing. As I look back on it now, I wonder why people say no to something, not having seen it.

As a matter of fact, stereoscopic zooms work well perceptually but there are all kinds of technical “gotchas”. One of the most important gotchas has to do with something that I defined as the centration vector, and I filed a patent for the cure for the stereoscopic zoom problem in U.S. Patent No. 4,418,993, filed on May 7, 1981. This was my first patent filed years after I did the work. At the time I had no idea I’d be starting a business and when I did I filed the patent.

There are a couple of problems with stereoscopic zooms, and these are mechanical/optical problems, and have to do with the fact that as you zoom the lenses the optical centers wander and the left and right images shift side to side or up and down depending upon on the location of the new optical center. When you have two zoom lenses there’s no guarantee – in fact, it’s improbable – that the lenses’ centers are going to shift in the same direction and in the same way. This means that you are going to create spurious or unwanted parallax in the vertical and/or the horizontal direction. One leads to difficult fusion and the other to more or less parallax than you bargained for changing the location of the zero parallax plane – or that which is meant to appear in the plane of the screen.

Another problem has to do with linking the two zooms so the focal lengths are exactly the same frame-for-frame, or field-for-field. And there are other problems such as those related to focus. You’ve got to focus the two lenses in a coordinated fashion, and focusing can change magnification too. I mentioned my early efforts to Peter Anderson, and he is possibly the most experienced stereoscopic cinematographer in the world having shot many theme park and IMAX shows. Peter and I are co-chairs of the ASC Technology Committee Stereoscopic Cinema Subcommittee. I’m going to let Peter talk for himself and here’s his take on stereoscopic zooming in the early days:

In the sixty’s, while I was attending Art Center and USC, I saw a couple of amateur 8mm dual camera 3-D rigs where the zooms were tied together via timing type belts and idlers and even one dual camera rig with push / pull lever zooms with the levers mechanically tied together by a cross brace. One person also had a single zoom lens Bolex 16mm with the Bolex 3-D mirror box mounted on it. These were sometimes shown at the various 3-D clubs in LA and info on them might be available in the UC Riverside’s 3-D archives.

The first time that I saw zoom lenses on a “professional” 3-D system would have been in the late 1970’s, at Paramount Studios, where Zoran Perisic [Zoeoptic] was demonstrating his zoom lens 3-D film system. Zoran is best known for his matched zoom front projection work on the Superman movies and less so for his patent around the same time for electronically tying zoom lenses to dolly moves in order to hold a constant subject size while dollying. In the 80s, I worked with Zoran in England on Disney’s “Return to Oz” and then later at Universal Studios.

My first use of a zoom lens on 3-D would have been at Heartland in 79 or 80. We used a single six to one Cook zoom as a varifocal long body lens on single camera for dual pass mo-co miniatures on 3-D tests for Buck Rogers and Battlestar Galatcia.

Chris Condon, probably in the early to mid 80s, showed me a dual zoom 3-D system he had.

We tried matching zooms for a HD shoot on the Hines rig at Sony LA in late 80 / early 90s, but could not find two lenses that visually matched. This was attempted after Sony Japan had done some side by side 3-D shoots that, while they had to wide of an IO, worked okay otherwise.

Slightly later, with Paradise, we shot a short 3-D film at Ed DiGiulo’s (of Cinema Products) house which even included a 3-D reverse zoom dolly shot.

Other things need to be taken into consideration, namely that the zero parallax plane that is use at the start of the zoom may not be the one that is best employed at the end of the zoom, and the same remarks can be made for interaxial separation. In other words, that which is desired to be at the plane of the screen at the start of the shot may now be required to be at some other Z location, and similarly, the strength of the stereo effect may need to be altered by smoothly and continuously varying the distance between the lenses. When zooming in on a subject, because of perspective considerations, the subject may tend to flatten out requiring a greater interaxial lens separation at the completion of the shot. All this would argue for interactive linkage of the ZPS (zero parallax setting) and t_c (interaxial setting) with focal length.

If you have information about this obscure but interesting subject – if you know anything about early attempts to do stereoscopic zooms – I’d love to hear about it. Right now, stereoscopic zoom defects or artifacts are usually cured in post. Vince Pace, in conjunction with Jim Cameron, has worked on stereoscopic zooms using high-def Sony cameras and Steve Schklair has camera and post-systems system at 3ality. Quantel has devised a post system that is now being used at FotoKem. All three outfits (Pace, 3ality, and Fotochem) are in Burbank. I’ve seen all demonstrated, and they can work great. The Quantel system is the first one off-the-shelf and any post-production studio can buy one, but Steve’s is proprietary. Points are picked at the beginning and the end of a shot, for example, and then the equipment more or less automatically rectifies the stereo pairs. What could have been a tedious frame-by-frame operation can now be automated. Let the machine suffer. In addition, creative, rather than corrective, decisions can be made about the placement of objects in Z space.

Obviously, zoom lenses that have centration or magnification artifacts aren’t suitable for real-time broadcast – for example, special events in Real D theaters. So work has to be done to solve the problem at the shooting end, and that might involve electronic correction with lookup tables and so forth and so on – which I believe I suggested in my patent.

Click here to download a copy of my 1983 Patent for Stereo Zooms

How Things Get Invented

2007-07-27T16:59:00Z

The history of motion pictures is an interesting one, and I am learning more about it in the context of my present work inventing stereoscopic motion picture systems, and in connection with the work I am doing with studios and filmmakers. I am taking working with filmmakers seriously because the quality of the Real D system is judged by the content projected on our screens. I was recently appointed as the co-chair (Peter Andersen is the other co-chair) of the sub-committee of the ASC Technology Committee tasked to help figure out workflow production pipeline and stereoscopic cinematographic issues. These subjects are tentative and need to be developed and we’re all learning together.

The stereoscopic cinema, in its present incarnation, as manufactured by Real D, is entirely dependent upon digital and computer technology. Digital projection allows for a single projector, while other stereoscopic systems use two projectors. Two projectors work well in IMAX theaters, based on my observations. I cannot say the same for theme parks, whether they use film or digital technology, because there are occasions when the projected image is out of adjustment.

Replacing multiple machines with a single machine – i.e. a projector – is the way to go, especially in today’s projection booths; because typically there is no projectionist in the booth at the time the film is being projected. There is a technician who will assemble the film reels and make sure everything is going to project well, but then somebody else – maybe the kid at the candy counter – who actually works the projector and makes adjustments. (Interestingly the kid at the candy counter may be well qualified to work the servers and projectors because of his or her PC experience.)

The product that I invented, the projection ZScreen®, has been used for years for the projection of CAD and similar images for industrial applications. Real D turned the ZScreen into a product that had to work even better for theatrical motion picture applications. It turns out that the film industry has very high standards when it comes to image quality. This is easy to understand, because the industry lives or die by image quality.

The stereoscopic cinema has had a long gestation. To date, this is the longest gestation of any technology advance in the history of the cinema. For example, within about three decades of the invention of the cinema, sound was added. There were numerous efforts to make sound a part of the cinema and make it a bona fide product. In the three-year period from about 1927 to 1930, rapid advances were made both in sound technology and in aesthetics. If you take a look at movies that were made in 1927, and then you see movies that were made in 1930 or 1931, there’s a gigantic difference. Movies made in the early 1930s look a lot like, and sound like, modern movies. There was a tremendous advance in the technology and in filmmaker know-how in a short period of time.

It is the creative professionals who will perfect the stereoscopic medium. That’s exactly what they did every time a new technology came along, whether it was sound, color, widescreen, or computer-generated images. In fact, those are the major additions to the cinema, and they all took decades to become an ongoing part of the cinema. Ads for movies never say, “This is a sound movie,” or “This is a color movie,” or “This movie is in the widescreen (or ‘scope) aspect ratio.” It’s assumed. It’s a rare movie that is in black-and-white. It’s an even rarer movie that is silent. And nobody is going back to shooting 4:3 Edison aspect ratio movies. (Curiously, that’s more or less the aspect ratio used by IMAX for their cinema of immersion.)

An attempt was made in the early 1980s to use a single projector with the above-and-below format – essentially two Techniscope frames that could be projected through mirrors or prisms or split lenses, optically superimposed on the screen, and polarized. The audience used polarizing glasses to view the images in 3-D. I was the chairman of the SMPTE working group that established the standards for the above-and-below format. But as soon as the standards were established, the above-and-below format was more or less abandoned. A few films like Comin’ At Ya! or Jaws 3-D, and one I worked on, Rottweiler: Dogs of Hell were projected above-and-below, an approach that was technically inadequate. For one thing it was hard to adjust properly and set up the projector to achieve even illumination. I know; I set up a few, and it was tough to do a good job because of the design of the lamp housings and the projectors.

Curiously it was the above-and-below format that led me to the first flicker-free stereoscopic field-sequential computer and television systems. I noticed that the above-and-below format was applicable to video, because that which is juxtaposed spatially can, with the injection of a synchronization pulse between the two frames, become juxtaposed temporally when played back on a CRT monitor; so the first StereoGraphics systems used the above-and-below format.

The above-and-below video format, which is applicable to video or computer graphics, results in a field-sequential image that can be viewed using shuttering or related polarizing selection techniques. I design the first flicker-free field sequential system in 1980. It used early electro-optics that were clunky, but the flickerfree principal was established. Using 60 Hz video, for example, with the above and below format, one achieved a 120 Hz result, that is to say, 60 fields per second per eye. The field sequential system is what is used for the Real D projection system. The electro-optics are different. There’s the ZScreen modulator used in the optical path in front of the projection lens, and audience members wear polarizing eyewear. (The combination of ZScreen and polarizing eyewear actually form a shutter. You can classify the system as either shuttering for selection or polarization, but in fact a proper classification is that it uses both polarization and shuttering.) But the principal is the same as that used for the early stereo systems I developed. The right eye sees the right image while the left sees nothing and vice versa, ad infinitum, or as long as the machine is turned on.

The issue I had to solve in 1980 was this: How to make an innately 60 Hz device work twice as fast. And the above-and-below format did just that. We had to modify the monitors to run fast, but for a CRT monitored it wasn’t that hard. There are two parts to stereoscopic systems’ issues: The selection device design and content creation. Today we are faced with the same design issue I was faced with in 1980. In addition, content creation has always been a major issue and that’s why I am working with the film industry to work out compositional and workflow issues.

Engineer Jim Stewart (left) and I are working on the first electronic stereoscopic field-sequential system that produced flickerfree images (Circa 1980). We used two black and white NTSC TV cameras as shown, and combined the signals to play on a Conrac monitor, which, without modification, could run at 120 Hz. The images were half height, but we proved the principal. Stewart is wearing a pair of welder’s goggles in which we mounted PLZT (lead lanthanum zirconate titanate) electro-optical shutters we got from Motorola. The shutters had been designed for flash blindness goggles for pilots who dropped atomic bombs. I kid you not.