Dawn of the new everythi.., p.18
Dawn of the New Everything, page 18
Here was Fred Brooks, an authentic Southern gentleman. Fred had run the team that created the first commercial operating system, for IBM, and defined ASCII, the way letters are represented by bits. He was one of the people who launched the Digital Age. He also wrote one of the few great books about computers, The Mythical Man-Month, which sensitively explained for the first time what it was like for humans to program computers.
On top of all this, Fred was a pioneering VR researcher. At the time of my first visit, Fred was particularly interested in haptics, which was also becoming Margaret’s passion, so we spent a lot of time with robot arms feeling the boundaries of virtual objects.
The other pillar of the lab at UNC is Henry Fuchs, who is also one of my favorite collaborators. He’s a darting genius, hardly able to talk fast enough to get his flow of amazing ideas out. His students have comprised the absolute top tier of the field decade after decade. Modern VR would not exist without Henry.
UNC probably had the fastest graphics computers of the time, and they enchanted me. Henry’s group built their own computers for the visual side of VR, at extraordinary expense. Pixel Planes was one of the first computers optimized for the kinds of graphics VR would need, though on my first visit it still took a few seconds to render each frame. But improvement would surely come, according to the Law, so we all imagined living in the future.
Soon it would be time to head back to California. I didn’t have any formal arrangement to return to, but it felt important to go. Something was brewing.
Ants on a Mission
“Oh wow, didn’t realize you were back!”
“Oh hi! Missed you. Just now got in. It’s incredible how much light there is here. And the air. You can breathe. The air around MIT in the summer is a hot dirty syrup.”
Ann had what was then known as the “Seattle look.” Long straight dark hair, big doe eyes. “Enjoy it while you can. This is all going away. The other dirt road in Palo Alto just got paved.”
“Oh no! That’s awful. But listen to them! Wow, I missed that sound.” There was a cat rescue place down the road and you could hear a hundred cats meowing, a turbulent string section.
“Yeah, you’re kidding yourself. It will be driving you insane in twenty minutes and I’ll have to listen to you complain. Oh, and you should know you have ants.”
“Oh, who cares.”
“No, I mean tons of ants, rivers of ants.”
Our little group had colonized the decrepit stretch of disabled orchard around my hut. Ann and Young and their kids had moved into a mirror-image hut across from mine on the same dirt road. Other people in the group lived in nearby huts from time to time.
I wish Silicon Valley still had corners of weirdness of the same caliber, but I fear those days are long gone. Next door was a woman who was one of the few top-tier female computer business consultants of the day, but it turned out she suffered from multiple personality disorder and you never knew who she’d be. There might be an acid-tongued punk rocker harassing me, or a silver-tongued power broker enlisting me to help out with a thrust into one of the big tech companies.
It turned out the refrigerator had failed and was not just colonized by ants, but had been completely filled by them, as if Archimedes had been performing an experiment with ants instead of water. I had to walk the rusty Space Age relic out back by the creek to empty it on its side. It looked like a flattened rocket that might have flown on the cover of one of Hugo Gernsback’s 1950s pulp science fiction mags, the ones in which Ellery wrote his popular science pieces. The spaceship disgorged a massive invading force that had not survived the trip. Chrome caught the sun and made me squint.
Another voice behind me. The multiple personality neighbor, today sounding completely normal, which was a little disorienting: “You look festive! What’s the occasion?”
Some of my Cambridge friends and I had the not-really-worth-it idea that we’d cut new holes in big colorful windbreakers from Bali so that they could be worn sideways. Two limp sleeves hung off to the left.
“I guess it’s clear-out-the-ants day. No really, this is how I always look. Everyone around here’s gotten more straight while I was in Cambridge; what’s going on?”
“I guess that’s right. I hadn’t noticed until you pointed it out.”
While I was gone, the hacker dress code shifted from exotic hippie to what would today be called “normcore.” It brought out the snark in me. “Everyone looks ordinary except not as good as ordinary. Schlumpy. What’s up with that?”
“I guess it’s our way of showing we don’t care.”
Fridge hosed, swiveling it a step at a time back to the hut: “Maybe it’s a sign that the world really needs VR now!”
“Oh, you don’t know. We’re calling it Shallow Alto.”
“It changed that much?”
“It just seems like everything interesting isn’t here anymore. The Suicide Club is in the city, the Whole Earth Catalog moved to Marin—sigh—and Survival Research Lab doesn’t even come around anymore. No one interesting can afford the rents anymore.”
You might not know about these early Silicon Valley institutions. The Suicide Club was a punk urban adventure club that would do things like climb the Golden Gate Bridge illegally. It was one of the progenitors of Burning Man. That’s where “Leave No Trace” comes from.7
Survival Research Labs staged walloping, genuinely dangerous, and giant performance art with equipment scrounged in Silicon Valley. Like a living, unsupervised guinea pig operating—for real—a tank with a thirty-foot flamethrower. You’d have to sign your life away to attend a show. All of these scenes would play roles in creating the first VR company, but I didn’t know that yet.
“It doesn’t matter if they left. We’re going to do the most interesting stuff ever, and right here.”
“You seem more driven.”
“I hadn’t noticed until you pointed it out.”
On Track
I had been building up a sense of mission for years and it was finally becoming more focused. I would prod the gang to build machines to make social VR possible, and promote VR as a suitably intense source of fascination to compete with the mind-control games and foolishness that Norbert Wiener worried about. VR would be the alternative to AI.
If a high-level strategy was becoming clear, the ground-level tactical game was still vague. Should we try a startup? Try to cajole a university or big company into supporting us in a VR lab? Just earn enough money from games or whatever to make the stuff with no regard for any existing precedent?
All of us wondered what style of institution we were building, but none of us knew. Maybe a fusion of lefty and business ideals? A tech company based on consensus decision making? Would that be a crazy idea? (2017 Jaron intervenes to scream, Yes, that would have been crazy!) But at the time everything seemed possible and everyone was idealistic and young enough to go for nights without sleep to get the latest demo running.
Kevin Kelly visited VPL in the late 1980s and took these pictures of Ann’s early concept drawings that were still pinned on the wall. Upper left: Very early concept drawing for the VPL EyePhone. Like every other team making VR headsets since, we underestimated the ultimate thickness that would be needed. Upper right: An EyePhone in use. Lower left: Children become Punch and Judy avatars. Lower right: A person inhabiting a chicken avatar uses virtual X-ray glass to look inside virtual objects. We used images like this chicken in presentations at places like the Defense Department, and yet they still wanted to work with us.
We became ever more obsessed with building VR projects during the course of 1983.
One thing that was clear was that we couldn’t do everything. I struggled emotionally with the thought, but it became clear that the full-on phenotropic vision was a project for generations, not a few years. VR could be ready in time to run real-time 3-D computers when they started to work, however.
David had been experimenting with one kind of visual programming language at MIT called dataflow. I talked to Chuck and the gang, and we decided on an in-between course that would include a few of the internal tricks we’d developed, like the high-level incremental compiler architecture, but would opt for the already understood dataflow paradigm for VR software, as it would certainly be a match.8 Dave had just finished his PhD and moved out to join us. (Present-day digital artists will probably be familiar with a design tool called MAX, which uses dataflow.) Body Electric was the name Chuck chose for our new VR control program.
We’d also need a 3-D design program. You couldn’t just go out and buy a 3-D modeler. Young took that challenge on and started on a project that would eventually become Swivel 3-D.
We also spent a lot of time on the tracking problem, which is the subject of the next chapter.
13. Six Degrees (A Little About Sensors and VR Data)
The Eyes Must Wander1
When Tom first made gloves, they measured finger bends, but not where the hand was in space, or how it was tilted. (You need six numbers to describe the position and orientation of an object in three dimensions: x, y, z, roll, pitch, and yaw.)
Obviously we’d have to be able to know where your hand was and how it was tilted if we expected your avatar’s hand to be able to pick up a virtual object. Devices that can tell where an object is in space are usually called trackers.
Trackers suitable for human motion were already on sale, though they cost a bundle. Strangely, the state of Vermont used to dominate the tracking industry, such as it was back in the 1980s. Four different tracking companies called a single Vermont valley home. Their customers used trackers in robots, industrial equipment, even flight simulators.
In those days, there was always an external device, a base station that served as the reference point for trackers. For instance, two of the classic Vermont tracking companies (Polhemus and Ascension) specialized in tracking using magnetic fields. There would be a big electromagnet in an impressive enclosure emitting a pulsing field, and then there would be little magnetic field sensors attached to a glove, and eventually to the headset.
There were plenty of other potential ways to accomplish tracking, aside from magnetic fields; lasers, radio waves, on and on. We put a lot of time into ambitious tracking schemes.
Why put a tracker on a headset? Remember the spy submarine? It must probe.
Recall the fundamental principle of vision in VR, which was stated earlier in the section titled “The Mirror Reveals”: “In order for the visual aspect of VR to work, you have to calculate what your eyes should see in the virtual world as you look around. Your eyes wander and the VR computer must constantly, and as instantly as possible, calculate whatever graphic images they would see were the virtual world real. When you turn to look to the right, the virtual world must swivel to the left in compensation, to create the illusion it is stationary, outside of you and independent.”
* * *
Twenty-fourth VR Definition: A cybernetic construction that measures the probing aspect of human perception so that it is canceled out.
* * *
The crucial point I must get across is that the quality of the visual display on its own is not the most important part of the quality of the visual experience of VR. What’s much more important is the tracking.2 How fast and how well does the visual imagery respond to head or eye motion?
The Brain Integrates
* * *
Twenty-fifth VR Definition: A media technology for which measurement is more important than display.
* * *
A universal problem in sensing is that it is a process, so it takes time. If you become a professional in the VR world, you will be asked to use the term “latency” for delays within VR systems.
The primacy of latency was demonstrated dramatically in the early 1980s. A VR lab had sprouted at NASA Ames, run by Mike McGreevy before Scott Fisher arrived.
Mike tried an experiment. He built a black-and-white VR headset that had a resolution of only 100 by 100 pixels per eye. That was the highest resolution possible, given the available display technology. The core rendering was still in vector graphics; a camera looked at the vector images to drive the pixel-based display. At the time, the mere use of pixels in a headset was still novel. This was probably the first example outside of flight simulators.
While 100 by 100 is a plausible resolution for an icon, for a virtual world, it’s preposterous. Remember, the image is spread out over much of what you can see. Therefore, each pixel might look about as big as a wall tile! And yet the effect was amazing.
My jaw dropped when I looked at a simple outline model of a proposed satellite through Mike’s headset. It looked reasonable! You could discern details smaller than the pixels and get a feeling for the 3-D form of the odd object.
The secret was reasonably fast and accurate head tracking. The lower the latency, the better the visual experience! It was as if the resolution had been magically multiplied.
Visual experience is based on integrating all you’ve seen and anticipating what you’ll see next. The brain sees more than the eyes do.3
As you moved your head in Mike’s HMD,4 your brain was sampling how the virtual world looked from each slightly different perspective, moment to moment. As long as the moments were accurate, meaning that the tracker was good, the perspectives were also accurate. That meant that the brain could combine the stream of low-resolution images into a more accurate internal experience of higher resolution.
To the brain, this was nothing special, just another day on the job. The human eye is amazing, but also a squishy, inconsistent, quirky sensor. The brain always sees better than it should be able to, given the nature of our eyes. It is just as happy to guess and cheat in order to make VR look better as it is to do the same for everyday reality.
In the 1980s it used to be incredibly hard to explain this basic quality of VR to newbies, even when they could try a VR demo! I practiced for years to find the language to convey this simple idea, which by now ought to be utterly commonplace.
* * *
Twenty-sixth VR Definition: A media technology that prioritizes stimulating the cognitive dynamics by which the world is perceived over accurately simulating an alternate environment.
* * *
A Moving Experience
If you were one of the early subjects in our eccentric lab at VPL, you would have initially experienced the transition to “believing” in the virtual world. This was called the “conversion moment.”
As VR improved over the years, the conversion moment happened sooner and sooner after a person put on the headset. Around the turn of the century it stopped being a thing.5
Today, the sensation for most people is sudden shock and delight at the quality of the illusion instead of a gradual onset in which there’s time to notice one’s own perception shifting.
This might be an example of better technology not actually being better. There is no greater value than learning more about oneself, and the older, poorer VR equipment might have done a better job at exposing one’s own process of perception.
But that’s no way to think! Perhaps a VR designer will come up with a crafty “slow start” experience on modern VR equipment that highlights the conversion moment even better than the old equipment did.
At any rate, in the old days you would have experienced the conversion moment, but that would only have been the first step. Since our emphasis was on multiperson experiences, you would soon have been introduced to another person inside VR.
You’d have seen this other person as an early avatar: a smooth, cheerfully colored figure with a cartoonlike head, an almost featureless body, and nimble but strangely tubular hands. The visual quality of VR during this period was rather fuzzy.
The face was generic. Everyone had to share avatars. There were few people who could make effective avatars out of the meager available resources, and little variety was possible, for the same reason. The first avatar face in VR was designed by Ann Lasko, and she made it out of twenty polygons; an origami face.
And yet, despite the paucity of visual detail, the presence of a human being came through. The effect was eerie and startling. In the course of everyday life, you don’t particularly notice the difference in your state of perception when you become engaged in contact with another person, but in these crude early VR systems, the difference was highlighted and quite dramatic; a bristling of the skin.
Suddenly another person was present there, in those few polygons! You could feel them, the warmth of human presence.
What was going on? If you recorded the motion of a person and played that motion back to animate an avatar, it would be apparent to people inside the virtual world that the avatar wasn’t inhabited by a real person at that moment. Things were dramatically different when you were interactively engaged with another person, avatar to avatar. You could usually even tell who it was.
These early experiments with avatars paralleled those of a long-standing scientific community that studies the perception of “biological motion.” A canonical experiment in that community is to make a movie of a subject concealed by a black full-body covering, but with a few bright dots stuck on here and there. All the movie shows is spots moving around.
Something remarkable happens when test subjects then look at such movies. They can often recognize individuals, or perceive details about sex, mood, and other qualities in strangers—all from a few dots in motion.
There are still debates about exactly what people can and cannot perceive from videos of such moving dots. Whatever hypotheses one prefers regarding biological motion, avatars probably reveal even more, because they are interactive versions of these experiments.
