AR and VR: Cultivating the Garden

FORUM Short Paper

Presence, Vol. 25, №3, Summer 2016, 253–266 doi:10.1162/PRES_a_00267
a 2016 by the Massachusetts Institute of Technology

Editors’ Note: To celebrate Presence’s 25th year of publication, we have invited selected members of the journal’s original editorial board and authors of several early articles to contribute essays looking back on the field of virtual reality, from its very earliest days to the current time. This essay comes from founding editorial board member Brenda Laurel, who asks the profound question, ‘‘What are VR and AR for?’’ From an opening that places us in the gar- den of a prison farm, we progress to a proposal that aims to help children in the developed world know where food comes from. Such an AR/VR application would have immense value and is eminently achievable.

When I was in graduate school, my PhD advisor and I worked together in a garden plot at Ohio State. We had most of our big conversations there. One day in late summer while we were taking in some of our harvest he told me a story. He grew up on the largest prison farm in Indiana where his father was the Warden. The biggest crop on the farm was peas. When it came time for them to come ripe, his father would go out every four hours or so and sample their sweetness. When he thought the moment was just right, he’d muster 400 prisoners — day or night — to harvest the peas at their peak.

Recent studies have estimated that the domestication of plants and animals in the Americas emerged between 5500 and 4000 years ago (Smith, 1994). We have millennia of agricultural history. A hundred years ago, about 50% of Americans were actively engaged in agriculture. Today that number is more like 2%. For American children, the decline in farming is matched by a decline in awareness and appreciation. “The sad reality is that many children don’t know where their food comes from. In fact, many of them think food originates at the grocery store” according to Margaret Purvis (2012), President and CEO of the New York Food Bank.

In his article “We Are All Noah Now,” Thomas Friedman (2016) alludes to a story (Macfarlane, 2015) about the Oxford Junior Dictionary:

“Robert Macfarlane, in his book Landmarks, about the connection between words and landscapes, tells a revealing but stunning story about how recent editions of the Oxford Junior Dictionary (aimed at 7-year-olds) dropped certain ‘nature words’ that its editors deemed less relevant to the lives of modern children. These included ‘acorn,’ ‘dandelion,’ ‘fern,’ ‘nectar,’ ‘otter,’ ‘pasture’ and ‘willow.’ The terms introduced in their place, he noted, included ‘broadband,’ ‘blog,’ ‘cut-and-paste,’ ‘MP3 player’ and ‘voice-mail.’ While this news was first disclosed in 2015, reading it in Macfarlane’s book still shocks me for what it signifies.”

Where does food come from? Why does it matter? Such questions touch on being in “right relationship” with Earth. How might we employ our new technologies — specifically Virtual Reality (VR) and Augmented Reality (AR) — to illuminate such questions? There is not a simple answer. We will return to these questions after some explorations of our technologies, their capabilities, and the intents to which we might put them.

Let’s begin by examining various flavors of technological mediated “reality.” Where did they come from? What are their powers and affordances? How might they combine?

1. THE “REALITIES”

The term “reality” is complicated in almost any usage. I find its use to describe technological media particularly annoying, but I’m not going to change anyone’s terminology now. I’m using “reality prime” (reality’) in this essay to refer to what others may call “physical reality” or “meat space.” It’s extremely important to understand that reality’ is a human-centric notion as it rests on human attributes that are not the same as those of a cat or a whale or a microorganism. Of course, one of our many great opportunities in VR is to represent as best we can those other species’ realities. But reality’ is tightly coupled with human animals, their capabilities and ideas.

Since humans began making representations, we have been augmenting reality. Some of the oldest known cave paintings can be found in the Chauvet cave located in the Ardèche region of southern France. The earliest art in the cave is about 32,000 years old (Chauvet, Deschamps, & Hillaire, 1996; see also Werner Hertzog’s 3D documentary film, “Cave of Forgotten Dreams,” 2010). The Chauvet art invites us rethink perspective in representations. It may also cause us to reexamine the origins of animation.

Figure 1. Horse panel in the Chauvet Cave

Paleolithic rock art has been strongly associated with shamanic rituals, both in the painting and in the viewing of them.(Whitley, 2011). Chauvet also features a bear altar complete with bear bones. In the Chauvet paintings, we get a sense of movement and mass from the remarkably contemporary-looking style of the art. Shamanic rock art does not generally appear in places where people lived; it is hidden away in caves or lake walls or slender canyons. The sort of rock art found nearer to dwelling sites tends to be more playful, such as the “ochre hands” (pictographs) made by women and children. Imagine now that the art at Chauet and other cave sites can only be seen by flickering torchlight, probably during shamanic rituals (see Robinson, 2013). The sense of movement in the paintings was accentuated, bordering on animation, with the likely intention of inducing altered states of consciousness (Rheingold, 1991).

“New” media types and styles — from cave paintings to cubist and expressionist painting to motion pictures to VR — initially evoke astonishment. Visualize how neophyte film viewers hollered (or ran away) when a bandit points and shoots directly at the camera in the early film The Great Train Robbery (1903). While astonishment may not qualify as a bona fide altered state, the strange reconfiguration of “reality” such as that seen in Duchamp’s “Nude Descending a Staircase 2” (1913) was motivated by physics in its four-dimensional, spatio-temporal trajectory and opens up deep new ways to think about seeing, consciousness and motion (Powers, 2003).

Figure 3. “Nude Descending a Staircase 2”
by Michel Duchamp, 1913.

2. IMITATION MACHINES

In Understanding Media, McLuhan muses that the content of any new medium is always one or more previous media, at least at first — so, for example, the newspaper “ate” broadsides and pamphlets (Stephens, 1988) and early motion pictures imitated theatre. The invention of the computer expanded McLuhan’s notion in the sense that a computer could theoretically imitate any medium — Alan Kay and Adele Goldberg (1977) referred to the computer as a “metamedium.”

What former medium was early Virtual Reality imitating? It seems that VR was striving to imitate reality’ — attempting to enable perceptions and actions as we experience them in specific contexts in the real world. Early uses were for flight training. As early as 1929 the Link flight simulator incorporated aspects of the cockpit environment to train pilots. Later flight simulators included cycloramas, models and films to simulate the view from the plane and motion platforms for more realistic sensory cues. Computer graphics and computer simulations of physics led to vast improvements in flight simulation beginning in the 1970s (Page, 2000).

Ivan Sutherland and Bob Sproull’s (1968) Sword of Damocles was the first head-mounted display, launching an expanded capability space for the development of Virtual and Augmented Reality. In the early to mid-1980s astronauts trained in VR environments that imitated spacecraft (as far as one could with vector graphics) augmented with operations and annotations available from pull-down menus. When polygons began to be used in VR one might encounter a training simulation featuring the blocky dynamic body of a piece of heavy equipment. The goal of first wave of VR (approximately 1985–1995) was to enable people to use more-or-less normal perception and take some sorts of action in environments that were immersive and computationally generated. That’s the virtual part. Augmented reality (AR) was also given a first real start by the Sword of Damocles. The difference is that in AR one is typically interacting with an environment — usually reality’ — that is augmented by overlays that may use vision, sound, or other senses to enhance or change perception.

In 1994 Paul Milgram and his colleagues produced a canonical representation of the “Reality-Virtuality Continuum” that helped to situate the various “realities” (Milgram, Takemura, Utsumi, and Kishino, 1994).

Figure 4. The “Reality-Virtuality Continuum”
created by Paul Milgram et al. in 1994.

Of course, virtual environments are not all based on reality’, as it were. In the influence of artists in VR encouraged designers to stray from imitating the real world, moving into realms of fantasy and dream. Scientists in VR encouraged us to use the medium for seeing and interacting with phenomena that cannot be perceived by the unaided human sensorium. Along the trajectory of the telescope and microscope, now one could see the unseen and interact with it. But one wouldn’t be looking at actual stars and microbes in VR; one would be interacting with representations or models of them. Today, even see-through Augmented Reality applications such as Star Walk allow you to look at the actual stars but interact with models.

The uses of VR and AR in play and games are more strongly emphasized in the second wave. Much of this has to do with vast increases in the capability space. Contemporary delivery systems are hundreds of times more computationally powerful than those used in the first wave. Interaction design is enhanced by major improvements in technologies such as machine vision, video tracking, GPS and Bluetooth, and modeling-and-simulation. The arts and sciences of design and animation have been extended by an efflorescence of new tools. New platforms are affordable to developers and customers. The ubiquity of smart phones, for example, has allowed popular AR apps to be adopted across the globe in the space of a few days; e.g., Pokémon Go.

We are doing well in finding good uses for new forms like immersive video and narrative in immersive environments. They provide emotional and sensory amplification to older media like movies and storytelling. But the hype associated with the second wave has given rise to the inevitable misnomers; e.g., “VR Movies” and “VR storytelling” (linear narrative unfolding in immersive environments) that attempt to piggy-back on the VR buzz. The “principle of action” is a great differentiator.

3. THE PRINCIPLE OF ACTION

Let me step back for a moment to those hazy times of the mid-1970s. I was working for a start-up “home computer” company while working on my PhD. The device, which utilized a standard television screen as its display, featured 2K of usable RAM and was loaded in segments from cassette tape. Memory and computing power conspired to constrain us to designing branching-tree game segments with converging nodes. Working with the unit to author interactive fairy tales, I began to wonder what robust interaction would look like absent the hardware constraints. At that time, most people’s ideas of “interaction with technology” involved things like changing the channel on the TV or turning off the iron before leaving home. As computing power and memory increased through the years, more kinds of interaction became possible. But what form might they take? How might we arrive at a robust sense of personal agency for interactors?

I met Don Norman while working at the Atari Systems Research Laboratory in 1982. He gave me encouragement to pursue my questions and ultimately to publish a chapter in the book, User-Centered System Design (Norman and Draper, eds., 1986). Influenced by Ben Schneiderman’s (1982) concept of direct manipulation, I identified the sense of acting in the first person as important to the sense of personal agency: one could originate action and its consequences would be reflected within the environment. To enhance the life-likeness of choice and action, I identified characteristics: frequency, range and significance.

Figure 5. Frequency, range and significance as characteristics of action

For example, a roller coaster ride (in reality’) maximizes frequency but severely limits range and significance (it’s really hard to jump off). In trekking or hiking all three characteristics are generally high, delimited variously by the affordances of the environment. In single-player first-person shooters, frequency is high, range is typically low, and significance is limited to the game mechanic (e.g., killing enemies or shooting down space invaders). All three characteristics are generally enhanced in multiplayer games where more and different kinds of actions, including social interactions, are involved.

Life-likeness also points to some qualities in the realm of movement and gesture. Early VR systems used semiotic gestures (formal signs) to perform actions like moving forward, stopping, etc. These systems also typically let participants use only one hand (with the DatagloveTM, for example). The Placeholder team (1993) included more affordances for physical action:

1. two sensors, one on the helmet and one on a belt, to separate direction of movement from direction of gaze

2. spatialized audio tightly coupled with movement of the body

3. the use of both hands using simple devices that measured distance between thumb and forefinger to achieve grabbing or gripping objects

4. the ability to take on the action capabilities of a selection of animals (dubbed “smart costumes”) so that if you were a crow, for example, you could fly by flapping your wings

Contemporary and near-future VR systems may do an exquisite job of tracking the body through video or other means, providing potentially greater acuity and expressiveness in movement. They will eventually give interactors tactile and haptic feedback. Such technologies will enhance sensory immersion. But how do we enhance the participant’s ability to take action in a VR or AR world?

The notion of “principle of action” in VR is attributed to Rob Tow, who was first inspired by the theories of psychologist James J. Gibson. In The Ecological Approach to Visual Perception (1979) Gibson posits that human visual perception relies on the movement of the body in space. VR relies on a very close correlation between what one sees in the stereoscopic view of the world and what one hears through spatialized audio. Tow articulated the “principle of action” in VR: “[The] results of action of the body in space and correlations with changes in the sensorium — ‘the principle of action’ — mark the major defining characteristic of VR as a medium.” (Laurel, Strickland, & Tow, 1994). To this we add the sense of personal agency, as characterized by frequency, range and significance of actions one may take. The diagram following shows how the principle of action applies within the larger contexts of perception and agency.

Figure 6. The Principle of Action, Rob Tow and Brenda Laurel, 2016.

The diagram places all three of our “realities” in the region where the principle of action obtains, either through the tight coupling of sensory inputs (as in VR) or through interaction within reality’ (as in AR). The greatest variable is the amount of personal agency afforded to the participant. Rides and theme parks are liminal because traditional theme parks limit personal agency while more innovative theme parks like Meow Wolf invite us to investigate and interact with everything. Meow Wolf has been described as “real-life virtual reality” (Newitz, 2016) and allows much more personal agency than a Disneyland visit or even an “interactive” theatre piece like Sleep No More — still, one cannot change the embedded “story” except to discover or not discover it through clues planted in the interactive environment. Dreams are another interesting liminal case; in some dreams we can act freely — even fly — and in others walking feels like wading through syrup, or we can’t move at all.

Linear narratives are within the realm of perception but outside of region where the principle of action (including robust personal agency) apply. Stories, films, plays and novels have plots that generally cannot be changed by the actions of a reader or audience. There are edge cases. “VR storytelling” admits of some interactivity by allowing a person to experience the virtual environment but typically without experiencing personal agency to any significant degree. Immersive movies provide for an even thinner range of agency until they grow to allow participants to interact with elements in the scene.

One example of how designers are thinking about making “VR storytelling” more robustly interactive is film director Jon Favreau’s VR project Gnomes and Goblins (Murphy, 2016). The project seeks to provide a first-person experience in an environment similar in style to his feature film The Jungle Book (2016). Faveau intends to allow participants’ choices and actions to change the story. When asked what it feels like for a film director to have less control over the narrative outcome, Faveau replies, “You’re still directing, but a whole new language has to be developed to help maneuver the focus of viewers to see what you want them to see while giving them the ability to make their own decisions. You can send clues.” In Computers as Theatre, Second Edition, I discuss various means for what I call “mediated collaboration” between the author/director and the participant to shape an interesting plot (Laurel, 2013). In structural terms, although direct dictation of plot is off the table, the elements available to the director include character, thought (for example, the goblin’s reactions to the participant’s actions), what is visible and audible in the environment, and the use of situational and atmospheric elements.

4. CAPABILITY AND INTENT

What are VR and AR for? A teleological view suggests that we examine them in terms of their intents. Let’s take a step back and have a look at the twin inventions of the telescope and microscope.

The history of the telescope reveals the fascinating interweaving of capability and intent in the development of the technology. The Dutch version of the telescope built upon the capability provided by optical lenses. It was designed in 1608 for the intent of examining faraway things on earth. The Dutch were excited by the idea of being able to see ships approaching from the horizon line. The capability of the telescope was expanded by differing intents. Galileo built his own version with improvements and enlarged its purpose through his own intent, turning the telescope skyward in 1610. The optical telescope worked with visible light. As the scientific understanding of the universe grew, new kinds of telescopes were developed in the 20th Century with the intent of exploring the stars outside the realm of visible light: areas of the electromagnetic spectrum including infrared, ultraviolet and gamma rays. Such inventions increased human capability to see the heavens in ways that an optical telescope could not, revealing more dimensions of the unseen.

Literally the same year that Galileo built his telescope he also built a microscope, based upon work on the notion of a compound microscope in the 1590s by Dutch inventors Hans Lippershey and the father-son team of Hans and Zacharias Janssen. These Dutch inventors were also interested in developing lens technologies, but in this case the intent was to increase our capability to observe things too small to be seen by the naked eye. Galileo studied insects with his invention but soon turned his attention back to astronomy. In the 1660s Robert Hooke made improvements with the intent of studying natural phenomena like plant tissues and snowflakes. In the 1670s Antoine van Leeuwenhoek made great advances in magnification and was the first to observe microorganisms. His developing intent eventuated in a new field of study: microbiology. In the 1860s Henry Sorby developed the metallurgical microscope so as to observe the structure of meteorites, beginning a burgeoning field of intents and the invention of various forms of microscopes that gave us the capability to investigate structures as small as atoms. Here, too, we see a dance between capability and intent that drives the development of new technologies.

I examined the evolution VR technology in Section 1 above, similarly driven by the three-way dance of capability, intent and invention. From early applications in flight simulation and training, the intents of artists and other poetical sorts encouraged the development of new visual styles (and the underlying computational tools) and new kinds of interfaces. Char Davies work Osmose, for example, took the intent of emulating deep-sea diving experiences to create the capability for participants to navigate by their own breathing.

New capabilities and intents are changing the virtual/augmented landscape in many effective and promising ways. In medicine, VR and AR are finding good use in training for difficult procedures as well as in rehabilitation. For example, scientists at Duke University have employed a combination of VR and robotics to help people with spinal cord injuries to regain partial movement (Donati et al., 2016). VR projects address themselves therapy for PTSD (Rizzo, 2005). AR is being used in manufacturing, repair, and even defense from internet hacks (Marks, 2016).

5. INTENT FOR THE GOOD

Our contemporary media landscape accommodates many intents. The promising practical and research-oriented intents such as those described above occupy a tiny corner of that landscape; today’s designers are largely focused on realizing intents for entertainment, and some offer breathtaking views that involve both the natural and made world — for example, Ubisoft’s Eagle Flight. Let’s consider for a moment an intent that is largely absent from today’s mainstream “reality media” content.

What if our intent in developing new ways of seeing and experiencing were to use our technologies to understand and enact the Good? Calls for such intents in the past have sometimes been characterized as quaintly moralistic and frankly not very interesting. But under contemporary circumstances, with decaying political discourse, massive human suffering, degraded educational and learning opportunities, the encroachment of the consumerist spectacle and the power of corporations in our lives, and — most important — the threats posed by ongoing climate change, intent for the Good becomes increasingly relevant.

“What is the Good?” is a question that has motivated philosophical and religious explorations since at least the sixth century BCE and probably much earlier. Studying how various notions of the Good in historical and contemporary contexts can help us define the Good for ourselves and to contemplate how we may enact it. The following is a cursory, non-inclusive look at some of the leading ideas of the Good and the virtues that produce it.

5.1 Evolving Ideas of the Good

Taoism, China’s most ancient philosophical and spiritual traditions, traces back to the writings of Lao-tzu in the 6th century BCE but had origins in even older spiritual practices. The doctrine of wu wei emphasizes harmony with nature and its cycles. It was thought that if one were to behave willfully, things would likely get worse rather than better. The notion of “non-action” meant that one should not meddle but rather yield, as rocks eventually do to water, to a harmonious practice of life (Fasching & DeChant, 2001). Virtue in Taoism is rooted in the “three Treasures”: compassion, simplicity and modesty. These virtues require self-discipline. They emerge in the conduct of everyday life in harmony with society as well as nature. They are not dictated but rather lived.

Confucianism, also born in ancient China in the 6th century BCE, may be seen as fundamentally humanistic in that it emphasizes self-improvability as inherent in humanity. Unlike Taoism, Confucianism places emphasis on a defined social order and the enactment of virtue within its structure. Loyalty, obedience and self-discipline are important aspects of virtue. The “Confucian Gentleman” (junzi) is a leader in government or community whose own virtue serves as an inspiration to others. By talking less about virtue but enacting it, the junzi leads by example (Jurgensmeyer, 2005).

Turning to Europe, Plato’s dialogues stimulate a particular conversation about the Good. The Republic, written around 380 BCE, introduces the idea of the Good as the source of intelligence, truth and knowledge (Plato, 2007b). He compares it to the sun’s role in illuminating the world; in this analogy, the sun is equivalent to the Good (Plato, Phaedrus) suggests that the Good “must embrace and hold together all things,” pointing at harmony as an aspect of the Good (Plato, 2007a).

Aristotle’s notion of the Good was demonstrably influenced by his teacher Plato and Plato’s teacher Socrates. Aristotle’s treatises (especially Metaphysics and Ethics) can be seen as the third act in an intergenerational philosophical dialogue. In the Nicomachean Ethics (written around 340 BCE) Aristotle found Plato’s notion of the Good too abstract and general. Instead, he posited that the Good consists in action and that it varies according to individuals and their situated contexts, ascertained by a combination of rationality and moral virtue, which involves self-regulation. Through good action we experience happiness, which is the ultimate Good (Aristotle, Book 1, Chapter 7, 2002).

Stoicism was founded by Zeno in Greece in 300 BCE, and was developed through other Greek and Roman scholars, culminating in the Meditations of the Roman Emperor Marcus Aurelius between 170 and 180 CE. For Stoics, happiness meant living in accordance with nature as they defined it. The Good resided more in actions than in words, and living virtuously set an example for the larger community. Inner calm and self-control helped people master their passions and excesses. Marriage was considered Good, not only because it was “natural,” but also because it was a social and ethical duty. “The wise man is obliged to marry both for his own welfare, in pursuit of wisdom, and for the welfare of the world community” (D’Elia, 2004).

In North America, diverse Native American cultures flourished before — and survived after — the invasions of the continent by Europeans. Indians including Cherokee, Lakota, Delaware, Squamish, and Oglala Sioux shared the belief and practice of living in harmony with nature. Among the Navajo, the term “hozho” means “natural order” or “beauty,” and a potent Navajo blessing is to “walk in beauty.” Around 1450 CE, the Iroquois Confederacy was founded to bring peace and order among the five tribes: Mohawk, Seneca, Onondaga, Oneida and Cayuga. The confederacy created a constitution and established as its symbol the Tree of Peace.

In 13th-century Italy, Francis of Assisi (founder of the Franciscan order and later canonized) taught of the Good. A life of poverty and charity were among the values he taught and lived. In 1260 Cardinal Bonaventure (2010) wrote of the legend and life of Francis. In Bonaventure’s writing we find Francis’ unambiguous commitment to the lives of animals:

When he bethought him of the first beginning of all things, he was filled with a yet more overflowing charity, and would call the dumb animals, howsoever small, by the names of brother and sister, forasmuch as he recognised in them the same origin as in himself.

The “scientific revolution” during 16th to 18th-century in Europe sharpened focus on reason. It was also a time of great efflorescence in what we can know and how. This was also the time of accelerated invention in making extensions to human perception and understanding, through inventions like the telescope and microscope, as well as great advances in astronomy, biology, mathematics, physics and chemistry. The sense that we “just know” how things work was supplanted by empiricism and the scientific method, including experimentation, in a rigorous and self-regulated way.

Closely following and somewhat intermingled was, of course, the Enlightenment. Reflecting on the scientific revolution, Enlightenment leaders including Voltaire led a philosophical movement rooted in scientific empiricism and experimentation, the replacement of faith with reason, and new theories of government and the public Good. The Enlightenment stimulates a grand discourse around democracy, individual liberty, and separation of powers as aspects of a society that creates, in Francis Hutcheson’s words, “the greatest happiness for the greatest numbers” (2004). The Encyclopedia of Diderot and d’Alembert was meant to disseminate knowledge both to other scholars and to the public at large. Rousseau articulated the idea of the connectedness of all things and reinforces the idea of harmony with nature as part of the Good.

We can find several common threads in Asian, Native American, and European ideas of the Good — as well as many important sources not covered in the previous brief philosophical gloss.

Compassion: Taoism and Buddhism enact the core value of compassion. Aristotle nuances the notion with sympathy and empathy as virtues. The Stoics seek a good life for all in the community. St. Francis revives the ancient idea that compassion extends to other entities with whom we the planet.

Harmony with nature: Bedfellows as strange as Taoism, Greek Classicism, Stoicism, Many Native American tribes, and Enlightenment philosophers emphasize harmony with Nature as a central value of the Good. From McLuhan (1964) to Lovelock (1979) and Margulis (1999), post-Enlightenment thought and science include even stronger notions of the interconnectedness of nature and our belongingness within it.

Virtue as action: Although they achieve it through differing means, Taoism, Confucianism, Greek Classicism, Stoicism, Christianity, the Scientific Revolution and the Enlightenment are in basic accord that virtue exists more in the action than in the word.

Self-regulation: Literally all of the philosophies I summarized involve a degree of self-control, both over our own impulses and toward right behavior in relation to the individual, the polis and the world.

Society and community: from the Taoism to Aristotle to the Iroquois Confederacy and onward through the Age of Enlightenment, ideas about society and community emphasize the happiness and well-being of all as a Good.

Reason: the early Greek philosophers and Stoics see reason as an essential component of judgment, virtue, and the ability to enact the Good. Reason as a means of achieving the Good gains traction throughout the Scientific Revolution and the Enlightenment and continuing to the present day.

5.2 The Good in Practice: A Scenario

I return to the question I posed at the beginning of this essay. How can we help children to know where food comes from and how it is raised? Many schools now have gardens — a great step forward. In my daughters’ grade school back in the early 1990s, all the kids worked in the garden, planting and tending and harvesting and making scarecrows. Award-winning restauranteur and activist Alice Waters started the Chez Panisse foundation on the 25th anniversary of her famous restaurant. With help of her foundation and the principal of Martin Luther King Middle School she developed a program called “The Edible Schoolyard” in Berkeley, California in 1995. It includes both gardening and cooking in its curriculum. In 2000 the concept expanded to supplement cafeteria meals with the School Lunch Initiative. Schools across the US have adopted the ESY approach and curriculum.

Could AR or VR or some mix of realities complement a school or community garden? Here is a three-part scenario.

1. The kids are ready to begin their planting. Through a combination of sensor feedback and AR they examine the soil for its condition and nutrients. They decide to plant the tomato seedlings where they were the year before. With AR assistance they look for good companion plants in the same bed and decide on spinach and carrots. “Remember last year,” one of the older children says, “we got those brown bottoms on the tomatoes. My mom says we should plant them with fish heads this year so look what I brought.” He opens his insulated lunch box to expose three dead but not quite thawed anchovies. Exclamations ensue.

Meanwhile, another team of children is planting corn, beans, and squash — as they have learned in their history class, those vegetables were staples of the Native American diet. They are kneeling over a raised bed considering how to arrange their planting. An Iroquois child from another era appears as an avatar. She tells the Three Sisters legend: that corn is best grown with two companions. “Beans will climb the corn like a natural pole,” she explains, tracing a central circle in the bed with her finger. “We see that beans make the soil more fertile and support the corn. Squash leaves shade out the weeds and preserve the moisture in the ground.” One child asks, how do beans make the soil more fertile? “I don’t know how to answer that question,” she replies, “but you newer people may.”

Time to ask Alice. “Alice, how does planting beans make the soil more fertile?” The display opaques and Alice uses an animation to help explain how bacteria among the bean roots convert nitrogen from the air and make it available to a plant. “Do carrots do that?” a girl asks, holding up her carrot seed package. The display replaces bean roots with carrots. “Carrots are roots,” one of the kids giggles and all soon are laughing. Alice replies, “most kinds of carrots don’t fix nitrogen, but they grow well near beans or peas.” “Because they fix nitrogen,” several children say in unison.

A child asks, “Why don’t we just use a bag of fertilizer to feed these plants?” Alice replies, “we can get a lot of what we need from our compost and our chicken coop.” “Farmers use lots of fertilizer,” he insists. “Fertilizer can have bad consequences when people depend on the industrial kind too much,” Alice replies. “Take a look at this.” The view zooms out from the garden to a nearby industrial farm then zooms high above, following the path of industrial fertilizer down the creeks and rivers then zooms in to the Missisipi River delta. The river mouth is clogged with blue-green algae. “Phosphorus is one of the elements in fertilizer, and it feeds the algae so well that it grows too much.” The scene zooms in on floating fish. “The dying algae suck oxygen out of the river, and that kills the fish.” “Oh, let’s not use industrial fertilizer then!” one of the children exclaims. “Does the river change if people quit using industrial fertilizer?” The display morphs to a healthy river with fish in the water and zooms back above the earth and down into the garden again. The children nod wisely and lean back. After a bit of silence, the kids’ displays return to AR mode. “We’re going to plant the Three Sisters,” one says.

2. During the growing season the children are working in the garden. One spots ants crawling around the corn. She calls the others to check it out. With their displays they magnify the ants 10x and they spot smaller dark insects climbing around the corn’s leaves. “What are those other bugs?” she asks. The AR display identifies them as aphids and provides a close-up view. “What are they doing? Are they friends with the ants?” Displays opaque to show an ant “milking” an aphid. It’s quite an interaction.

Time to go ask Alice. “Alice, what’s going on?” As the display zooms in, Alice replies, “The ants and aphids help each other,” she says. “The aphids make a kind of sticky nectar that feeds the ants. The ants protect the aphids from other predators.” “But,” one child cries, “The aphids are eating the corn!” Alice says, “Yes, they’re feeding on the corn’s juice. Ladybugs will eat the aphids if there aren’t so many ants.” A simulation of ladybug dining accompanies her explanation. Ants and aphids “crawl” around the ladybug. Kids try feeding them to the ladybug, who seems to prefer the aphids. The children lean back wisely. After a moment, their displays return to AR mode. A young boy suggests, “let’s just poison them and wipe them all out. My dad does that.” “We don’t use pesticides,” says a girl with some superiority. “Why not?” asks the boy. “Because they might kill cats,” she replies, “or birds. Or ladybugs.”

Before the argument can continue, another child suggests, “let’s figure out about the ants. Let’s see if we can do it without asking Alice.” Another replies, “Alice said the ants like that sticky sweet stuff they get from the aphids. Let’s put some really sticky stuff around the corn stalk to see if it traps ants.” All agree and head for the honey in the kitchen. The next day, many ants are stuck in the honey and children launch a great ladybug hunt — but that’s another story.

3. The next autumn after the kids have returned from their summer break, the harvest is winding down. A boy steps around to the now-deserted garden from the playground. He notices a dried ear of corn on one of the plants. He moves over to examine it more closely. “What do you do with dried-up corn?” he muses. This time a young Pueblo woman is seen kneeling nearby. “You can grind the corn to make cornmeal. Cornmeal keeps, and you can make all kinds of things with it,” she says as she works away with a pestle in a hole worn away in rock. “You can see holes in the rocks where I come from. They are from generations of grandmothers grinding corn.” “You mean you made your grandmothers do all the grinding?” he says, shocked. But she is gone with a laugh.

A few more children have wandered over to the remains of the garden, kicking leaves around. “This was a good garden,” one of them says. “Maybe we should plant everything just the same next year,” another suggests. “Let’s try a what-if,” the boy replies. “What did our garden look like in the summer?” Through AR they see an image of all their plants as they were at Midsummer. “And what will they look like next year if we plant them the same way?” The view changes — certain plants are smaller and some are healthier and taller. “What if we move them around?” a young girl asks. “Move the Three Sisters to the tomato bed,” one suggests. They remain the same in health and height. “Move the tomatoes to where Alice had us cut off some of the bean plants just when they were flowering,” another suggests. “So that means the nitrogen went back to the soil, I think,” says one girl, and another chimes in, “YES!” as the tomatoes grow higher and greener. A few fish heads are floating around them in the air, seemingly singing. The girls shoo them away. “What about the lettuce? It didn’t give so many leaves — it just got tall and nasty.” “Really fast,” another child adds. “What if we move them more into the shade, up there?” a boy says, pointing to a shady place at the edge of the garden. She moves the lettuce with a gesture to the shaded area and, yes, the lettuce leaves are full and the plants are not too high.

Still, the children aren’t entirely satisfied with their future garden. One of the beds, where there were sunflowers that didn’t do so well, has water running through it from the rain last night. “Will nitrogen stay in the soil? Will something grow here?” The bed lies stubbornly weedy and not too happy-looking in the simulation. “OK,” a boy says. “Ask Alice,” they all say together.

After Alice’s counseling on rerouting the running water that has been leaching the soil, she says, “Sometimes, you need to let the soil rest — you can grow plants with lots of nitrogen and then plow them under.” “Then that would be year after next,” the older boy says. “I’ll be in high school.” The children are attracted back to the playground as our boy still muses. “And I didn’t even get a pumpkin for Hallowe’en — the children cleared them out.” A voice comes from the compost. “Ahem,” it says. He hesitates. “Come over here, if you would be so kind.” He saunters over to the compost. A mid-sized pumpkin has grown up from the edge of the pile in reality’. He grabs the pumpkin, sits down on the dirt and laughs. “Last laugh, compost,” he says. “Always,” the compost replies.

This scenario is achievable, much of it with current or very near future technology: repurposing existing logistics, AI, speech recognition, agents, machine vision, databases of models and simulations, traverses in Google Earth. Challenges remain. What does the delivery platform look like? Smartphones or pads? Glasses? Google’s VR controller (Burns, 2016)? How do we make graceful transitions from AR to model-and-simulation or VR? How do people move elegantly from sharing screens to using their own exclusively? These are problems that can be solved; perhaps some have been solved already. What is lacking is the will to make things with such intents and to allow those intents to inform new capability spaces.

6.3 The Good through the Lens of Love

A few years ago I was giving a talk about another AR-in-nature idea at the New School. An elder (i.e. my age) in the audience responded to my remark that we needed to love our natural world better in the face of planetary crisis. He said, “without love, there is no grief, and without grief there is no action.” His comment stuck with me. He did not say “without fear” or “without protection.” He said “without love.”

The Stoic would rehearse the end of the world, visualizing the loss of all held dearest, in order to control passion and excess. Franciscans would think of how they love all the living things that make up our web of life and send them off to God with blessings. Environmentalists would think of John Muir and his devotion to Wilderness and see it as inevitably passing away. Gaians think of our belongingness to the whole organism that is Earth and feel that we put more than ourselves at risk when we lack actionable love for the whole and all of its parts. And here, I find my notion of the Good. More is at stake than the “inconvenience” of sea level rise in the world’s largest coastal cities or the salinization of rivers on the encroaching tide. We already see these things at work. Upwards of 200 species a day are now going extinct. As nations and corporations achieve short-term “development,” the web of life is tattered by our actions and inactions, our inability to live the Good on a global scale.

Technologies are augmentations of what it means to be human — what we can know, see, and do. Technology is an extrusion of the human spirit. Can we commit to using technology for the Good? Yes, the Good encompasses pleasure. But does empty entertainment lead to happiness? For Aristotle (2002), “…right desire and right reason … [are] the direct and natural response of a free human being to the sight of the beautiful” (Sachs, circa 2000).

We are not without means to point human attention to the beautiful: to create a bond of love with nature (including our own natures); to create a bond of love and grief over ongoing loss (Stoics notwithstanding); to create inducements to action — in the polis and in the world — in defense of the beloved.

If children grow up with early experiences such as those in a school garden, they will have forged a bond with Earth that will not be broken as they grow up. The garden is delight, problem-solving, accomplishment, increased understanding and learning through failure as well as success. They will grieve as they grow older and losses continue. They will become activists for the Gaian cause: to understand the planet as a whole organism populated by multiple organisms (including humanity) — entities within entities in webs of relationships that share Earth as their home. As Gaian gardeners, we exercise canonical notions of the Good: compassion, harmony with nature, virtue as action, self-regulation, the good of society and community, and the application of Reason. We turn our intent to active cultivation of the Good with the remarkable tools that we have at hand, and in so doing, we will discover new capability spaces.

And we just might save the world.

REFERENCES

Aristotle, Nicomachean Ethics, Book 1, Chapter 7 and Book 6, Chapters 3 & 4, circa 340 BCE. In Aristotle’s Nicomachean Ethics, Trans. by Joe Sacks. Focus Philosophical Library Series, 2002. ISBN 1585100358.

Bonaventure; Cardinal Manning (1260). Bonaventure: The Life of St. Francis of Assisi, Chapter 8, sections 6–10. Rockford, Illinois: TAN Books & Publishers, 2010. ISBN 0895551519.

Burns, Chris. “Pizel Android UI expands beyond displays in next-gen Google VR.” Slashgear, 19 September 2016. Retrieved from http://www.slashgear.com/pixel-android-ui-expands-beyond-displays-in-next-gen-google-vr-19456665/.

Chauvet, Jean-Marie, Étienne Brunel Deschamps and Christian Hillaire, Dawn of Art: The Chauvet Cave. New York: Harry N. Abrams, 1996. ISBN 0810932326.

D’Elia, Anthony F. The Renaissance of Marriage in Fifteenth-Century Italy. Cambridge, MA: Harvard University Press (2004): 12–13. ISBN 0674015525.

Donati, Ana R. C. et al. “Long-Term Training with a Brain-Machine Interface-Based Gait Protocol Induces Partial Neurological Recovery in Paraplegic Patients.” Nature.com Scientific Reports, 11 August 2016. http://dx.doi.org/10.1038/srep30383.

Fasching, Darrell J. & deChant, Dell. Comparative Religious Ethics: a narrative approach (Blackwell Publishing, 2001): 35. ISBN 1444331337.

Friedman, Thomas L. “We Are All Noah Now.” The New York Times, 7 September 2016. Retrieved from http://www.nytimes.com/2016/09/07/opinion/we-are-all-noah-now.html.

Gibson, James J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin Company, 1979. Part III, Chapter 10. ISBN 0898599598.

Hertzog, Werner. Cave of Forgotten Dreams. 3D documentary film. [author’s note: must see in 3D]

Hutcheson, Francis. Wolfgang Leidhold, Ed. Francis Hutcheson, An Inquiry into the Original of our Ideas of Beauty and Virtue. Indianapolis: Liberty Fund, 2004. Retrieved from http://oll.libertyfund.org/titles/hutcheson-an-inquiry-into-the-original-of-our-ideas-of-beauty-and-virtue-1726-2004 (1726)

Juergensmeyer, Mark. Religion in Global Civil Society (2005). ISBN 0195188357.

Kay, Alan and Adele Goldberg. “Personal Dynamic Media.” Computer 10.3 (1977): 31–41.

Laurel, Brenda, Rachel Strickland and Rob Tow. “Placeholder: Landscape and Narrative in Virtual Environments.” Computer Graphics, 28.2 (1994): 118–126.

Laurel, Brenda. Computers as Theatre, Second Edition. Addison-Wesley Professional, 2013. ISBN 0321918662.

Lovelock, James. Gaia: A New Look at Life on Earth. Oxford University Press, 1979. ISBN 019217665X.

MacFarlane, Robert. Landmarks. Penguin UK, 2015. ISBN 0241146534.

Margulis, Lyn. Symbiotic Planet. Basic Books, 1999. ISBN 0465072720.

Marks, Paul. “Hololens Augmented Reality to Foil Hack Attacks in Factories.” New Scientist, 16 July 2016. Retrieved from https://www.newscientist.com/article/2097641-hololens-augmented-reality-to-foil-hack-attacks-in-factories/.

McLuhan, Marshall. Understanding Media: The Extensions of Man. New York: McGraw-Hill, 1964. ISBN 8114675357.

Milgram, Paul and Haruo Takemura, Akira Utsumi, and Fumio Kishino, “Augmented Reality: A class of displays on the reality-virtuality continuum.” SPIE Vol. 2351, Telemanipulator and Telepresence Technologies (1994), 282–292.

Murphy, Mekado. “Jon Faveau on ‘Gnomes and Goblins’ and Virtual Reality,” The New York Times. Retrieved from http://www.nytimes.com/2016/08/31/movies/jon-favreau-on-gnomes-goblins-and-virtual-reality.html.

Newitz, Annalee. “Inside Meow Wolf, the amusement park for people who want a weirder Disneyland.” Ars Technica, 2016. Retrieved from http://arstechnica.com/gaming/2016/04/inside-meow-wolf-the-amusement-park-for-people-who-want-a-weirder-disneyland/.

Norman, Donald A. and Stephen Draper, eds. User-Centered System Design. New Jersey: Lawrence Earlbaum Associates, Inc., 1986, 78–79. ISBN 0898598729.

Page, Ray L. “Brief History of Flight Simulation.” SimTechT 2000 Proceedings. Sydney: The SimtechT 2000 Organizing and Technical Committee, 2000.

Plato, Phaedrus, 99c., circa 370 BCE. In Six Great Dialogues: Apology, Crito, Phaedo, Phaedrus, Symposium, and The Republic, Jowett, Benjamin, translator. Dover Publications, 2007(a). ISBN 0486454657.

Plato, The Republic, Book 6, 508c-d., circa 380 BCE. In Six Great Dialogues: Apology, Crito, Phaedo, Phaedrus, Symposium, and The Republic, Jowett, Benjamin, translator. Dover Publications, 2007(b). ISBN 0486454657.

Powers, Edward D. “Fasten your Seatbelts as we Prepare for our Nude Descending.” tout-fait: The Marcel Duchamp Studies Online Journal, 2(5), April 2003. Retrieved from http://www.toutfait.com/issues/volume2/issue_5/articles/powers/powers2.html.

Purvis, Margaret. “Beyond the Grocery Store: Teaching Children Where Their Food Comes From.” Huffington Post, revised January 2012. Retrieved from http://m.huffingtonpost.com/margarette-purvis/beyond-the-grocery-store-_b_1070259.html.

Rheingold, Howard. Virtual Reality. Summit Books, 1991: Ch. 17. ISBN 0671693638.

Rizzo, “Skip,” Principal Invesstigator. “Bravemind: Virtual Reality Exposure Therapy.” USC Institute for Creative Technologies, 2005-present. Retrieved from http://ict.usc.edu/prototypes/pts/.

Kim Stanley. Shaman. New York: Orbit Books, 2013. ISBN 0316098086.

Sachs, Joe. “Aristotle: Ethics.” Internet Encyclopedia of Philosophy. n. d., circa 2000. Retrieved from http://www.iep.utm.edu/aris-eth/. ISSN 2161–0002.

Schneiderman, Ben. “The future of interactive systems and the emergence of direct manipulation.” Behavior and Information Technology, 1(3), 1982: 237–256.

Smith, Bruce D. “The Origins of Agriculture in the Americas.” Evolutionary Anthropology, 3(5), 1994: 174–184. DOI 10.1002/evan.1360030507.

Stephens, Mitchell. A History of News: From the Drum to the Satellite. New York: Viking Penguin, 1988: 131–132. ISBN 0670813788.

Sutherland, Ivan. “A head-mounted three dimensional display.” AFIPS ’68 (Fall, part I) Proceedings of the Fall Joint Computer Conference, 1968, 737–764. DOI 10.1145/1476589.1476686.

Whitley, David S. Introduction to Rock Art Research, Second Edition. Routeledge, 2011. ISBN 1598746111.