Dawn of the new everythi.., p.32
Dawn of the New Everything, page 32
Since phenotropic programming is based on mapping between the human experience level and the bit level, programmers are less committed to particular abstractions. One editor might map a bunch of bits that run as a program in such a way that they look like a maze, while another editor might map the same bits so that they look like a family tree.
Each conventional, source-code-based programming language is inexorably tied to its abstractions, like Fortran’s functions, LISP’s lists, or Smalltalk’s objects. These are all examples from the era when I learned to program. You don’t need to know what they were. The point is that they were concepts that bridged the world of human intent with the flipping bits inside a computer, and each one shone in certain circumstances and became awkward in others.
Phenotropics is an approach that supports different such concepts at different times within one tool. Abstractions can be mixed and matched to suit the needs of the moment.
Variation
This doesn’t mean that abstractions become obsolete.
Imagine living in a future in which VR is programmed in the way my friends and I explored so long ago, phenotropically. In that case, there will be various actions you can perform that cause the underlying bits to change so that the virtual world you are in will suddenly function differently.
What might these actions be? Will you operate a simulated control panel that looks like the bridge of the Starship Enterprise? Or will you pull on chains in a medieval dungeon, or dance like a leaf? Or edit text that looks like the Grace Hopper–style source code everyone uses these days? Any and all of these editor designs might have a place.
But there has to be some design. You can’t accomplish anything without embracing a point of view and a way of thinking. But there is no reason, fundamentally, to be inflexible about which design to use at a given time.
What has happened in Grace’s universe of nonphenotropic source code is that each computer language proposes that certain abstract objects are not only real but mandatory, eternal, and inevitable while using that language. I already mentioned the classic “functions” of Fortran and “objects” of Smalltalk, but I could just as easily add the “bots” in cloud software that are fashionable as I’m writing this book.
Each of these objects is fine and useful at times, but none of them ought to be inevitable. They aren’t real, if “reality” means that which you cannot reject. I find it troublesome that they seem real.
Different abstractions might have been put in place instead of any of the familiar ones, but for the quirks of history. (Whether widely used software abstractions can ever be reconsidered is an open question. In a previous book, You Are Not a Gadget, I examine the way ideas that are expressed in software can become “locked in” by pernicious “network effects,” but for the purposes of this book, I assume that there’s time and hope for change.)
The only things that are fundamental and inviolate—truly real—while you are using a computer are you and the run of patterns of bits inside the computers. The abstractions linking those two real phenomena are not real.
Is it imaginable that a computer architecture could express this philosophy? What if there was a way to swap in and out different editor designs in order to present given bit patterns to you, so that you could understand them and modify them in different ways at different times?
Phenotropic Trial Run
My friends and I built a few generations of phenotropic experiments in the early 1980s. An early one was called Mandala, then came Grasp, then Embrace. (Grasp suggested a glove, while Embrace embraced whole body suits, literally and figuratively.) Some of the major categories of VR applications were first prototyped in VPL’s codeless software.
“Codeless” was not a metaphor; in a literal sense, we didn’t use code. Well, we used traditional code and development tools to get the system going in the first place, but virtual worlds didn’t run on code, only on bit patterns that could, to repeat, be modified by editors that mapped to them.
Editors are fundamentally different from the usual tools used to create software, like compilers and interpreters.
Compilers are the chrysalises in the metamorphosis scheme of traditional code-based software: You edit a text file, the “source code,” and then, only after compiling, you get to see what the code does once it has been changed. Then you go back and forth to debug.5
The phenotropic alternative might sound like an impossibly exotic idea to younger computer scientists who have grown up in Hopper’s shadow. The idea of code is almost universally taken to be synonymous with computing, but it need not be.
Could a phenotropic editor mimic traditional code? In other words, could we edit bit patterns by mapping them to images on a screen that looked like a familiar high-level text-based language? In many cases we could, which meant we simulated code. A phenotropic editor could be designed and constrained to look like text, even though the effect arose from a more general graphical construction. Such an editor could do anything a compiler could do, but as live visual tweaking.6
We played favorites with certain editor designs, meaning with certain visual representations of code, often preferring a principle called dataflow. Dataflow typically looks like wires connecting modules. But it wasn’t fundamental. We could swap in Grace Hopper text–like editors, or other editors.
The experience of programming briefly became a little more improvisatory, a little more like a cross between playing jazz on a horn and drawing mathematical diagrams.
* * *
Fifty-second VR Definition: A way of using computers that suggests a rejection of the idea of code.
* * *
Alas, we eventually had to ask VR customers to develop on a regular monitor instead of from within the virtual world. The main reason was that regular monitors were so much cheaper than VR headsets. More people could be working at once and in more locations.
It still hurts today when I think about that! It hurts worse that everybody in today’s VR renaissance is still using conventional programming languages on conventional screens to develop for VR. It’s like trying to learn a foreign language from a book without ever talking to a native.
Our editor designs on traditional monitors often looked a little like MAX, a visual programming tool used today for experimental computer music and animation.7
We at least peered into an alternate future that will hopefully be explored more thoroughly in years to come.
Scale
A fundamental impulse in computer science is “scaling.” That means computer scientists hope our works will be able to scale up to become unboundedly huge and complicated.
How do you make bigger and bigger phenotropic structures? A phenotropic editor maps the machine’s bits to a user interface so a person can change the bits. But can one editor edit another editor? Could you have towers of editors editing editors, webs of them, giant fungal growths?
Yes, of course, that’s the idea. But in that case, would you need to commit to a set of abstract principles each editor would adhere to so that it could be edited by other editors? Wouldn’t that defy the goal of avoiding commitment to any particular abstraction?
The answer, incredibly, is no! A phenotropic editor need not adhere to any particular abstraction to be operated by other editors.
The reason why is that each editor is a user interface, usable by humans. Therefore, editors can act like simulated humans to operate other editors. An editor can interpret a user interface and use it on the terms of that interface.
For instance, an editor for low-level access to a math library might look like a calculator. A person could use it directly, or another editor could use it through a simulation of user interaction.
A calendar program that needs to call upon arithmetic to calculate the date for a future appointment would simulate pushing the buttons on the simulated calculator.
There need be no shared abstraction dictating how one program is supposed to call another. Instead, each editor becomes responsible for figuring out how to use the human-oriented user interfaces on other editors.8
It might sound like an uncertain and highly inefficient way to get one part of a program to interact with another part, and it is! But only for small programs.
The phenotropic hypothesis posits that once you start dealing with very large systems, with giant programs, a phenotropic principle becomes more efficient than a traditional one, which must enforce abstractions.
You can think of a phenotropic system as a bunch of editors with simulated people sticking out the back of each editor. In a couple of our old designs, you could turn a whole big program sideways to see the collection of underlying editors edge-on, floating in a formation in space like shields in a space war.
Emerging behind each editor was a cartoon-like character in profile, animated as it operated other editors that in turn were backed by other characters. This was all done in the era’s only practical style—8-bit game graphics. We never quite got the whole vision implemented, but we got close. I wish I had an image to show you, but apparently none have survived.
That sideways view was just another editor, of course. Nothing special.
(If you’ve already read the section about my take on artificial intelligence, consider this: Unlike AI, in which a simulated character faces you, in a phenotropic system they all face away from you, toward other editors, but under your control. It is clear they are tools, not equals to you. Same algorithms as in AI, but different conception.)
Motivation
There are varied reasons to entertain the phenotropic hypothesis. Before getting to the nuts and bolts of efficacy, consider usability by people.
It’s always easier to write a fresh program than to understand and modify someone else’s program, but at least, if the program is phenotropic, the pieces you find when you open the hood are always user interfaces designed for people. Because that’s all there is.
A phenotropic system will tend to be made of components that are the right size for human use, since each editor is initially designed for human use. That means phenotropic systems tend to have “coarser chunking” than other architectures.
Instead of a zillion little abstract functions, the organization of a large program will be broken into larger, lucid pieces, each of which is coherent on its own as a user interface. The chunking naturally follows practical human usability instead of an engineer’s idealized scheme, and will tend to be easier to understand and maintain.
In a phenotropic system, it should be possible to watch the animated characters behind each editor doing what they do to get a sense of how an overall program works, but you can also position yourself anywhere in the network of editors to directly play within the program, to experiment.
This observation suggests a fundamental principle. Computers only make sense as tools to serve people. If you make a computer “efficient” but that efficiency makes it harder for people to understand and maintain sensibly, then that computer actually becomes inefficient.
Role Reversal
A great example of this principle is found in computer security. We have created endless layers of abstraction to allow programs to communicate with programs, but those abstractions are hard to understand. Therefore, hackers keep on coming up with unforeseen exploits, and we must all accept an astounding overhead in breaches, maintenance, security software, election tampering, identity theft, blackmail, and on and on.
Would phenotropic software really be more secure? I can’t prove it unless and until there are more tests, but I’m optimistic.
Here is how we build systems today: A bit-precise structure of communication abstractions surrounds the “pay dirt” modules like “deep learning”9 ones that accomplish the most valuable functions.
These critical “AI-like” algorithms are not bit-perfect, but even though they’re approximate, they’re still robust. They provide the capabilities at the core of the programs that run our lives these days. They analyze the results of medical trials and operate self-driving vehicles.
In a phenotropic architecture, the roles of the bit-perfect and approximate/robust components of a program are often reversed.
Modules are connected in a phenotropic system by approximate but robust methods like deep learning and other ideas usually associated with “artificial intelligence.”
Meanwhile, bit-perfect precision is invoked only inside certain phenotropic editors, like the functions accessed by the calculator. Absolute precision is no longer used for communication.
Why would this be more secure? To protect a computer from hackers, we sometimes create an “air gap.” That means that a computer performing a critical function isn’t even online. It’s out of the reach of hackers. A real person has to use it on-site.
Each and every module/editor within a codeless phenotropic network is surrounded by what is effectively an air gap, because they can’t receive abstract messages from one another. There are no messages. Only simulated fingertips pressing simulated buttons. There’s no abstract “press button” message.
Before I get back to the topic of security, I’ll explain more about how the air gap works.
Expression
First, a confession: Back in the eighties there was no way to implement the phenotropic effect without a “press button” event. Machine vision and machine learning were not working well enough yet.
So we needed a little language to describe display and user interface features like onscreen buttons, but we knew that it was only a temporary patch for a temporary problem.10 Moore’s Law suggested that eventually computers would get fast enough to be able to recognize similarity instead of only identity. When that happened, then one editor would be able to watch another with machine vision and operate it with a virtual hand, and there would no longer be a need for an abstract representation of a user interface element like a button.
In the mid-nineties, when computers finally got fast enough to recognize visual similarities in real time, a new batch of friends and I founded a startup, called Eyematic, to perform machine vision tasks like recognizing faces or tracking facial features. (We won government contests run by the U.S. agency NIST in that era for recognizing and tracking faces in difficult real-world situations.)
Most of the scientists on the Eyematic team were ex-students of the formative neuroscientist Christoph von der Malsburg. A few of the original Veeple also reconvened there, including Chuck and some of the same old investors, though the heart of that company was Hartmut Neven. Google bought it eventually.
I have to admit that it was disturbing to work on some of the first effective facial tracking and recognition programs. Were we creating a monster? I used some of the Eyematic prototypes to implement working models of nasty technologies that became scenes in Minority Report, like the billboards that incorporate someone who is running past them as he attempts to evade the police, broadcasting his position to everyone.
The reason I persisted is that I felt there was a benefit that would more than make up for the ugly potential for universal surveillance. If we could get machine vision to recognize faces and track expressions and so on, could we not also apply the same abilities to get editors to use other editors? We could finally get rid of the temporary patch and build a proper phenotropic system, with proper air gaps.
In that case, a phenotropic editor would not support any interface or method of interaction other than its user interface. There would be no protocols, no abstract variables to document, no API.11
Machine vision and machine learning algorithms from one editor would be applied to interpret and operate a virtual hand that would virtually touch another editor. An editor could not “tell” if it was being operated by a person or another editor at a given time, because the interface would be identical in either case.
The nature of the code within an editor that causes it to do certain things to another editor would not be standardized. Nor would the means of programming a given editor.
Some editors might be trained to perform (in the way that we train machine learning algorithms with examples) while others might have to be explicitly programmed. All could interface with all just as a person could.
My belief is that this prize is big enough to outweigh the problems of surveillance. If our information systems can be built on principles similar to the phenotropic ones I have described, then we can eventually come to use tools that don’t require us to accept only certain abstractions universally and forever.
Since from now on our information systems will serve as the molds for many aspects of society—and the guides for how young people become individuals within a society—a move to an information architecture with varying, revocable abstractions couldn’t be more significant. This is how openness and freedom might be encouraged in the far future.
I realize this hope might sound esoteric, and might also sound like a humongous leap of faith, even a utopian impulse, but it’s actually an attempt to outgrow utopia.
Leaving aside the big ideas, it was great fun to finally get avatar faces to track the expressions of human faces. For a while, I experimented with expressive avatar faces with my weird band in clubs like the Knitting Factory in New York City in the 1990s. There was a big screen behind us, and kooky musicians were seen morphing into avatars of the corrupt politicians of the day, for instance. (In retrospect, corruption was pretty tame in those days.)
The Wisdom of Imperfection
Since the modules of an ideal future phenotropic system would be connected through approximate means, using machine vision and other techniques usually associated with artificial intelligence, a lot of the manic, tricky hacking games that go on today wouldn’t even get off the ground.
