Will augmented reality change everything we see? A growing number of Penn alumni, staff, and faculty think so. And even as they bump up against its challenges and limitations, they’re still committed to pulling AR further into our lives.
By Molly Petrilla | Illustration by Roman Klonek
Sidebar: Need a HoloLens? Try the Library
I’m killing scorpions inside Stephen Lane’s office.
They skitter out of openings in the wall, crawling up and down, sometimes even jumping right at me. I try zapping them away but sometimes I miss, blasting holes around Lane’s door and exposing white pipes and other slivers of the building’s bones.
Lane doesn’t seem to mind. In fact, he’s laughing. The computer and information science professor who was just sitting behind his desk, earnestly answering questions about the future of technology, has transformed into a jazzed-up gamer who’s excited to share his toy with a novice.
“Are you getting them?” he asks as scorpions dance around us. “Just keep firing!”
Even as giant predatory arachnids are scurrying in front of me, and even as my fingers trigger laser beams to destroy them, I’m still inside Lane’s real office. If I turn around, I can see his dark wood desk with its stacks of papers and cup of coffee. I can see his crowded bookshelves and his whiteboard. I can see Lane himself—plaid shirt, bristly mustache, belt with colorful little sailboats. But thanks to the Microsoft HoloLens I’m wearing—a headset that wraps around my forehead and hangs down over my eyes, resting on the bridge of my nose—the game is with us too, turning a plain white wall into a scorpion lair.
And that’s augmented reality.
RoboRaid is a simple introduction for the uninitiated. “It’s a game, but it really shows off some of the capabilities,” Lane says just before passing me the HoloLens.
AR isn’t about fully losing yourself in an imaginary world the way virtual reality is. It’s about enhancing your actual surroundings with computer-generated images and objects. At its most ambitious, AR can require programming expertise and a $3,500 headset like the HoloLens. But it can also be as simple as pulling out your smartphone and turning yourself into a bunny with an Instagram filter.
On that morning inside Lane’s office, I’m not the only one at Penn testing out augmented reality’s abilities and discovering its limits. Across campus and well beyond, a number of faculty, alumni, staff, and students are all focused on applying AR to their fields. You’ll find them inside operating rooms and classrooms, heading up teams at Google and the New York Times, and working for a leading AR headset company.
They’re all convinced that augmented reality can change our world—that it will change our world—in every area from medicine and education to research, retail, and entertainment. But they’re also starting to wonder: What will it take to get past the tipping point, and what might life be like after we do?
While I’m chatting with Lane that morning, Brian Park is across campus at the Hospital of the University of Pennsylvania (HUP), getting ready to watch an AR-assisted surgery.
It’s been three years since he got his hands on a HoloLens. Now he’s seeing AR used inside operating rooms at Penn just as he once imagined.
Park has a master’s in electrical engineering from Stanford, and before that he worked on the Xbox 360’s processor at IBM. If anyone’s going to help bring this “hot new field,” as he calls AR, into medicine, it’s not surprising that it’s him.
As a fellow in interventional radiology at Penn, he sees the promise of AR, particularly in his specialty. IR surgeons often access patients’ insides through holes as tiny as a needle tip—and for exactly that reason, accuracy is critical. To help them plan and perform those surgeries, they rely on computerized tomography (CT) and other radiological imaging.
“But once you gown up and are in the operating room, you essentially don’t have those images to view anymore—or if you do, they’re usually on a monitor screen somewhere else in the room, away from the patient,” Park says.
That’s where his vision for AR comes in. By wearing a HoloLens that’s been loaded up with a patient’s radiological scans, “you can see the imaging right on top of a patient, but in 3-D,” he says. “You can see into the patient, essentially.” And just like with the RoboRaid game in Lane’s office, a surgeon can still see everything else around them too.
Park says that with this AR-enhanced approach—which he’s now collaborating on with the NYU-based company Medivis—surgeons can even take new scans mid-procedure and add them to the HoloLens images. Or if it’s all getting too distracting, they can simply say “off hologram” and ditch the AR altogether.
In the last couple of years, he’s clocked more than a dozen surgeries performed by Penn interventional radiologists wearing HoloLenses. His work was featured on NBC10 News locally, and he recently landed a $50,000 grant from the Radiological Society of North America to continue his research this year.
Park may be one of the more visible examples of AR devotion at Penn, but he’s not the only one exploring its medical applications around the University. In the School of Veterinary Medicine, animal surgeon Tom Schaer V’96 is investigating whether loading CT scans into AR headsets and overlaying them on a patient—as Park’s program does on humans—can better guide surgeons in drilling screws into horses’ fractured leg joints at the proper angle and depth.
Marion Leary GNu’13 Gr’14, a resuscitation science researcher who recently became Penn Nursing’s director of innovation, just finished a two-year study using AR to improve CPR administration. Participants performed CPR on a mannequin while wearing a HoloLens that overlaid the dummy with an active, virtual circulatory system. Based on the quality of their CPR, they’d see blood flowing to the brain and organs faster or slower. Leary says her studies are the first of their kind.
Her work also inspired Regina Toto, a fellow in Pediatric Emergency Medicine at the Children’s Hospital of Philadelphia. With funding from the Perelman School of Medicine’s Support for Projects Advancing Research and Knowledge in Education program, Toto has adapted Leary’s CPR application into a training tool for treating septic shock in children. Though still in the pilot study phase, she’s planning for a full-fledged trial, in which participants will administer IV fluids to a mannequin while observing its circulatory system response through a HoloLens.
Multiple people interviewed for this story—both who work in healthcare and also well outside of it—referred to medical applications as “the low-hanging fruit” for pushing augmented reality beyond entertainment and novelty.
Still, there are roadblocks to navigate first.
“It’s super expensive,” says Leary, who received a grant to build her HoloLens application, CPReality. “It can cost tens of thousands of dollars to program things like this through outside companies,” she adds. On top of that, there’s also the steep cost of the headsets themselves.
On the surgical side, Park believes that AR can help make procedures faster and expose patients to less radiation. He’s even helped to prove as much in rats already. But he says no one in medicine knows yet whether AR is accurate enough to make surgeries any safer than they are sans headset.
“One of the things everyone’s working on, us included, is getting the imaging accurately placed on the patient,” he says. “Some people call that the Holy Grail Problem. It’s really hard, just by the specs on the device alone.” Patients move. They breathe. At this point, all of that disrupts the hologram alignment.
“The virtual image has to be in lockstep with the actual patient anatomy,” Schaer notes. “That’s where some of the early problems have arisen. I think we’ll get there, but we have to be very cognizant that right now, these cameras people are using are in the millimeter range of accuracy. Is that enough to always be on the safe side in your margin of error? That’s a discussion I just had yesterday.”
The person on the other end of that discussion? Schaer’s partner in his AR-assisted odyssey, the scorpion-blaster himself: Steve Lane.
Lane stares down at his desk, wondering if the tiny computer in front of him can possibly come to life.
He brought it along with him on the train into Philly this morning, assorted parts and wires tucked carefully inside a black case. By technology standards, this is an artifact—born in between the first Game Boy and the first iMac. And it wasn’t designed for endurance like those. Lane’s was only a prototype, built for a gaming market that, it turned out, wasn’t actually ready for it.
“If I can get it to work, it would be a miracle,” he says, fiddling with cables and plugs, smoothing down a peeling sticker that says Robicon Systems.
“I hear something making noise …” He trails off. A long pause. Then Lane laughs. “It’s booting up!”
He’s sporting a clunky headset that wraps around his forehead. A tiny screen sticks out in front of one eye. It’s printed with a name, The Private Eye, and came from a US company called Reflection Technology in the late 1980s. When it’s my turn to try on the headset, I can see a black background with red text, but I can still see Lane’s office, too. An early example of augmented reality, though certainly not the earliest.
When it debuted in the late ’80s, the Private Eye was meant for people who work with their hands but need to see information at the same time. But Lane and his company, Robicon Systems, had another audience in mind. They created a one-handed keyboard device and a wearable computer in 1993 to complement the headsets, allowing gamers to hold a keyboard in one hand and a gaming glove in the other.
Lane’s creation never evolved beyond visions and prototypes. “We were always ahead of the curve in terms of technology, so we were also ahead of the market,” he says with a laugh. “At that time, we were showing capabilities that people weren’t ready for. So we moved on to something else.”
Long after making a virtual reality sword-fighting game for Disney World in 1998 and founding the Computer Graphics and Game Technology master’s program at Penn in 2001, Lane eventually returned to AR.
As the technology sweeps through campus, he’s often called in to assist. In addition to teaming up with Schaer in the Vet School and having ongoing conversations with Park, he’s also worked on Monument Lab’s Civic Portal project, which created AR monuments for people to interact with and even offer feedback on. And if you’re a Penn student who wants to start making things with AR, Lane’s probably your best bet there, too. His Game Design Practicum course includes multiple AR design projects.
From his years of shuffling between AR creator, educator, and unofficial historian, Lane has a zoomed-out view of what’s happening now and where it might lead.
He compares this moment to when IBM launched its XT personal computer in 1982. “Here we are 37 years later, and we look back and can’t believe [that machine] was so minimal in terms of what it could do,” he says. “In some ways, where we are today with AR is like where we were then with the IBM XT PC—we’re just at the very beginning.”
Even though AR dates back to 1968, when a Harvard computer scientist created the first head-mounted display system—and even though the term “augmented reality” came from a Boeing researcher a full 20 years ago—it’s only bounced into the mainstream in the last few years. (Thank you, Pokémon GO.)
Lane takes a moment to imagine what life could look like when a pair of comfortable smartglasses loaded with 3-D AR technology finally make it onto the market. Both Apple and Facebook, among many others, are reported to be working on exactly that right now.
He describes walking into a friend’s house, admiring their new red chair and instantly seeing where it’s from, what it costs, how it might look in another color, and ordering it for home delivery—all through your smartglasses. He talks about posters inside a travel agency virtually transporting you to Paris. About glasses supplying subtitles when you’re talking to someone who speaks another language.
He runs through a host of other applications, from manufacturing and museum-going to product design and education. He’s particularly interested in that last one, he says, and it turns out there are a number of people at Penn who have already started applying AR to teaching and learning.
The Nursing School has preordered over a dozen HoloLens 2 headsets for its anatomy and physiology lab, where undergrads will use them to study the human body in 3D, with no actual body or skeleton required. “Using cadavers, students don’t get to see different colors or how the circulatory system and arteries and veins work on a living body,” says Ann Marie Hoyt-Brennan, the director of innovative learning and simulation at Penn Nursing. “We are thrilled about the possibilities of AR as a learning tool.”
Instructors in the medical and vet schools have also been experimenting with teaching students anatomy via AR. Schaer imagines someday bringing veterinary students into the operating room to watch doctors perform surgeries, with everyone wearing a pair of AR glasses. “Everyone would see the surgical field exactly the way that the surgeon putting in the implants does,” he adds. “There’s an enormous opportunity here.”
The Penn Museum and the Wharton Learning Lab have both had student researchers testing out the technology, and this past summer, a team of six students worked full-time inside Van Pelt Library to create an AR mobile app tour of campus based on the Penn and Slavery Project [“Gazetteer,” Nov|Dec 2018].
Last spring Peter Decherney, a professor of cinema & media studies and English, convened an entire conference around “Immersive Storytelling” with AR and VR. (“Immersive” is often used as a blanket term to encompass both AR and VR.) He brought in filmmakers, journalists, business strategists, and even a Snapchat lens creator. “There are so many applications for AR,” he says.
Lane saw glimpses of AR’s potential decades ago, back when the technology was mostly limited to prototypes and chunky gray plastic. Now, like so many others at Penn, he’s fully convinced of its power. “Even five to 10 years from now, which isn’t that long, I’m sure there will be many other mainstream applications that we can’t even dream of right now,” he says. “As the computer vision technology gets better and better, whatever you look at, you can potentially augment or get information on.”
But first, he says, some things will have to change.
Lane’s brought a second AR headset into his office today for me to try after the HoloLens. This one’s lighter and its lenses look more like goggles. The trade-off is a disc-shaped wearable computer about the size of a large palm, which clips onto a pocket or belt and connects to the headset. It’s cheaper than the HoloLens: about $2,300 for a “personal” version; more than that for developer kits.
When Emre Tanirgan EAS’15 GEng’15 applied to work for Magic Leap, the start up that makes this headset, he didn’t know much about the company. He’d never even seen their technology. Everything was still a secret back then, “but it seemed like they were doing cool stuff, and the position was interesting, so I decided to go for it,” he says.
A graduate of Penn’s Digital Media Design Program—and one of Lane’s former students—Tanirgan became a prototype developer on Magic Leap’s “Gestures Team” in 2016. The group focused on recognizing people’s motions, testing out ways to let users reach out and grab virtual objects with their hands.
For his first year and a half at the company, Tanirgan couldn’t talk to his friends or family about what he did all day at work. No one even knew what the headset might look like until the company finally unveiled its Magic Leap One design in December 2017. “It was definitely a relief when the device actually started shipping,” he says, “because now we can talk about it a lot more openly.”
He moved into a software engineering role last year, working on an app that allows Magic Leap users to chat with each other via 3-D avatars. “Since we have all these sensors on the device, we also have an idea of what your face and body are doing when you’re talking to someone,” Tanirgan says. “So instead of just a voice call, we can have a 3-D representation of you that moves as if it were you. When you blink, the avatar blinks. When you look around, the avatar looks around.”
As someone immersed in AR development, Tanirgan is also aware of the new technology’s limits and challenges. “I don’t think AR is anywhere close to being mainstream yet,” he says. He believes that two major hurdles have to be cleared before that can happen: price and comfort.
Lane agrees. He acknowledges that for all of his imagined scenarios to come true, we need those AR-ready smartglasses—ones that feel more like wearing sunglasses than computer equipment—on shelves, and people need to think they look good enough to walk around in.
This story started with Graham Roberts EAS’04. When I first heard about him last year, the part that stuck out wasn’t so much the AR work he was doing—although that was notable—but the fact that he was a Digital Media Design alum who’d skipped over the usual tech and animation jobs so many DMDers shoot for after graduation and gone to work at the New York Times instead.
Roberts’ work at the Times is further proof of how many fields are testing out AR right now. As director of immersive platforms storytelling at the Times, he led the 167-year-old media company in efforts to place readers inside its news stories with augmented (and sometimes virtual) reality.
In 2018, Roberts and his team used augmented reality to bring Times readers into the Thai cave that trapped 13 people. They let them examine David Bowie’s iconic costumes up close and showed them what an Olympic figure skater looks like mid-jump.
The cave project in particular sparked “a really strong reaction,” Roberts remembers. “People seemed to really connect with the idea that AR could be more than just a dancing hot dog or a funny lens.”
But over the eight-month span that I interviewed Roberts, things changed both for him and for the Times’s approach to AR.
In our earliest conversation, he acknowledged that AR on smartphones has value—an easy way to reach Joe iPhone and offer some fun features—but he was already seeing its limits. “As much as we’re excited about what phones can do now with AR, the phone is not a native spatial computing device,” he said. “AR is a skill that phones inherited retroactively.”
In a tweet from around the same time, he predicted that AR will become the primary way we interact with digital information “only when a post-mobile-phone device, built from the beginning with spatial media in mind, achieves the right form factor and passes the biggest challenge: fashion.”
We’re not there yet. That’s why, as 2019 continued, Roberts and his team changed their tack. Rather than automatically launching a phone’s camera mode to bring costumes and caves into the room with you, Times stories now make that feature optional, rendering the 3-D images in a browser instead. “AR can add a lot of value on phones,” Roberts says, “but at the end of the day, it is still a phone, and that means there are limitations.”
By mid-August, Roberts was in the midst of an even bigger change: leaving the Times after 13 years. He was headed to Google’s San Francisco office, where he now leads design efforts at Google Brand Studio Creative. Like so many other DMDers before him, he wanted to “move closer to the tech,” he said.
And at Google, that tech has been stuffed with AR lately. In the past year and a half alone, the company unleashed a new AR mode on Google Maps, inserted 3-D AR models into its search results, and began selling a new version of Google Glass, the company’s own take on smartglasses. It also released a software development kit, ARCore, which lets people build augmented reality smartphone apps more easily.
“We’ve really accelerated our efforts in AR,” says Matt Apfel L’90, who’s been Google’s director of AR/VR content since 2015. Noting that the rise of AR is “something I don’t think anybody saw coming,” he predicts that augmented reality will soon slide into our everyday lives “pretty quickly and pretty naturally.”
But even from his tall perch at one of the world’s leading companies, Apfel and his teams are up against many of the same AR challenges that everyone from dabblers to pros are facing right now.
“In technology, there’s this cliché: you’re building the airplane as you’re flying it. I would tell you that in my experience, [when it comes to AR,] the industry is still designing the airplane,” he says. “It’s going to take time for all these enabling technologies to come into place.”
No one can forecast exactly how long that might take. By the time this story appears in print, the HoloLens 2—announced in February 2019—may have finally begun shipping. (In November, the company reportedly began filling orders.—Ed.) Apple may be one step closer to unveiling stylish smartglasses. Google may have rolled out another game-changing update. Or all of that could still remain in-progress, confined to developers’ heads and tech bloggers’ predictions.
But when the technology eventually catches up to the dreaming—if it catches up to the dreaming—will it make our lives better?
Tanirgan thinks a lot about the looming “AR cloud,” which doesn’t exist yet but may someday let multiple people who are wearing headsets (or smartglasses) share the same AR experience. “It’s essentially a whole layer we’d be building together,” he says. “I think it’s exciting, but also a little scary.”
Both he and Lane pointed me to the same six-minute, crowdfunded video online titled “Hyper-Reality.” In three years, it’s been viewed more than three million times. The video reveals an imagined future in which even a simple trip to the grocery store is overcrowded with ads, games, and other digital noise, both literal and figurative.
“People already spend so much time on their phones that the line between what’s real or not is getting thinner and thinner,” Tanirgan says. “Because [AR] is so integrated with the real world, we need to be very careful with how we design it. But I’d say I’m cautiously optimistic.”
Lane, on the other hand, can’t wait to watch AR sweep in. The only downside he can come up with is a hit to problem-solving skills in assembly, maintenance, and repair jobs, since AR will eventually guide workers through processes step by step. “There may be other unintended consequences, I suppose,” he acknowledges. “Like any technology, it has its pluses and minuses. But on the whole, this is an enabling technology.
“I’ve seen a lot of technologies come and go over the years,” Lane adds, “but I believe that AR is for real, and I think it is the next wave of computing after mobile devices. Cell phones have changed the world. Now AR is going to build on top of that.”
NEED A HOLOLENS? TRY THE LIBRARY
In early 2017, Kimberly Eke, the head of Penn Libraries’ Teaching, Research and Learning Services, began to notice a trend.
She kept encountering people across campus who were working with immersive technologies in their individual fields—“and overcoming incredible obstacles to do that,” she adds—but who had never met each other.
So she decided to help. Eke rounded up seven tech-savvy Penn faculty and staff for an AR/VR-focused meetup in the spring of 2017. “They all said it felt like they’d found their tribe, and we thought, maybe there’s something to this,” she says.
At the same time, the Penn Libraries had been consistently adding the newest technologies to their various outposts: 3-D printers and scanners, 360-degree video cameras, top-of-the-line virtual reality headsets. “It was hard for people outside the libraries to understand why we were doing this and how everything connected,” Eke says.
Soon she found herself organizing open houses at Van Pelt for people to meet both the immersive technologies and each other, and launched a PennImmersive website, calendar, and blog.
She says that while the Penn Libraries are interested in helping people create scholarship in new forms—often by loaning out their HoloLenses and VR headsets—they’re also tackling “a wicked challenge” that’s started to surface: How can libraries preserve new types of digital scholarship, like an AR-powered mobile app?
“Over the last 15 years, the library has become so much more than what we think of it as,” notes Peter Decherney, a professor of cinema & media studies who has taught classes incorporating VR. “It’s really at the forefront of using technology in research and teaching.”
“I could talk for hours about the Penn Libraries,” Marion Leary says as we’re chatting about her CPReality work. The PennImmersive initiative “really shows how the libraries are trying to be nimble, change with the times, and focus on the applications that people want to be using,” she adds.
And they even let her borrow their HoloLens for her last study. —MP
Molly Petrilla C’06 writes frequently for the Gazette and other publications and is the magazine’s arts and culture blogger.