[With interesting details about the evolution of CAVEs and Google’s Tilt Brush, and using the language if not the terminology of presence, this article from PNAS (Proceedings of the National Academy of Sciences) describes how collaborations “among artists, coders, and computer scientists” have made and are continuing to make “the barrier between the real and virtual world vanish” to benefit both art and science. See the original version of the article for high resolution versions of the image below and a second image, of Alison Goodyear’s Topsy Turvy. –Matthew]
[Image: Computer scientist and artist Daniel Keefe has been combining virtual reality and art for decades. Here, he creates a three-dimensional image in a “Cave Automatic Virtual Environment,” or CAVE. Credit: Regents of the University of Minnesota, courtesy of Daniel Keefe.]
Virtual reality helps shape and respond to artistic impulses
By Stephen Ornes (Authors Info & Affiliations)
July 21, 2022
In February 2020, weeks before the pandemic made such a thing nearly impossible, the MK Gallery in Milton Keynes, UK, hosted an exhibit called Paint Park that featured works by Alison Goodyear. A visual artist from nearby Bedfordshire, Goodyear works in what she calls “the expanded field of painting.” That means she examines the conventional tools of abstract painting, such as an artist’s palette, using technological ones. And she uses one medium in particular: virtual reality (VR).
To experience the exhibit’s centerpiece, called Topsy Turvy, visitors had to sit in a swivel chair and don a VR headset. They navigated through abstract paintings created by Goodyear. Even though the experience was virtual, it afforded the illusion of being a physical exploration—instead of looking at a two-dimensional surface, viewers encountered digital paint. In one half of the work (Topsy) viewers were plunged into the messy swishes and swirls that oil paint takes on in the physical world. Swaths of paint became swooping, overlapping pathways. The other half (Turvy) plunged viewers into a painting based on digital tools that looked unlike paint, including pulsing neon lights and oscillating graphics.
“When you access these VR paintings, you’re fully immersed, like in a landscape,” says Goodyear. “It’s very otherworldly, and these paintings are like a form of world-making.” By turning an abstract painting into a virtual three-dimensional space, Goodyear says, she began to rethink her ideas of how abstract painting fits into the world. “I started to theorize that painting was a place.”
As VR technology gets better, cheaper, and more accessible, researchers have been experimenting with ways to reach people—everything from treating PTSD to performing operas. Visual artists like Goodyear are the latest to ask how the technology might push the boundaries of their field, seeking new ways to express themselves and share their work. But this is a tricky crossroads, says Goodyear, as it’s difficult to faithfully capture the physical components of painting—the texture of a brush, the pressure on the canvas, a serendipitous accident—within a virtual environment. Even so, the technology has engendered collaborations among artists, coders, and computer scientists who see, at the intersection of their fields, an emerging aesthetic experience, driven by devices and algorithms designed to capture artistic impulses.
Such collaborations are powering the evolution of how people interact with three-dimensional worlds, says computer scientist and digital artist Daniel Keefe at the University of Minnesota in Minneapolis. But this is just the beginning: To improve the experience, he says, new technologies are needed to make the barrier between the real and virtual world vanish. “We want to make the hardware disappear.”
Real Partnerships
These partnerships aren’t only pushing art in new directions; they’re also suggesting avenues for other expression at the intersection of art and science. Computer scientist and artist Carolina Cruz-Neira at the University of Central Florida, in Orlando, sees two ways of approaching these partnerships. “First, what kinds of tools can we put in the hands of artists to help them create their art?” asks Cruz-Neira, who has worked with many artists and galleries to create immersive VR experiences. “Second, how can we collaborate with artists to create an emotional connection between the participants and the art or artist?”
In online platforms such as Discord, artists share requests and coders respond with ways to fulfill them. In one recent exchange, an artist requested the capability to create image layers and edit parts of a brush stroke in a VR program; later the same day, coders responded with updates and ideas for how to implement those functions.
Researchers are also looking to art for bold new ideas about how to represent data. “We want to bring artistic insights into the problem of how you make pictures of these massive, complex datasets,” says Keefe, who has worked at the intersection of the two fields for close to 20 years. His work focuses on both virtual art and data visualization; recent projects include depictions of big data from climate models and artistic explorations of sculpture. His team has also been exploring ways to bring images of real-world materials into a virtual world, creating a kind of hybrid immersive experience that can “blur the boundaries between the digital and the physical world,” he says.
Artists, Gamers, and Cavers
Thirty years ago, when she was finishing her PhD at the Electronic Visualization Laboratory at the University of Illinois at Chicago, Cruz-Neira was one of three inventors of the Cave Automatic Virtual Environment, or CAVE—a cubic enclosure, 8 feet on a side, in which three-dimensional images are projected on white walls, ceiling, and floor. A user wears dark glasses outfitted with thick frames and small antennas. The lenses adjust as the user moves around so that slightly different images reach the right and left eyes. The effect is a stereoscopic, fully immersive image that surrounds the user.
“It was built with scientific and art applications in mind, not as a gaming device,” says Cruz-Neira. She collaborated with molecular biologists and astrophysicists in the design of early CAVE applications; she also worked with digital artists and photographers who were exploring immersive approaches to their work. Cruz-Neira has used CAVE for her own VR artwork. “I approach art as something that can be solitary,” she says, “but it’s also a shared experience.”
One of the few institutions housing a CAVE is Brown University, in Providence, RI, where Keefe was a graduate student in the early 2000s. He helped develop and refine software that enables three-dimensional painting; the works seem to hover in midair around the painter. Painting in a CAVE, says Keefe, can be disorienting, as a user responds to both the empty physical environment and the virtual swirls of color they leave in the air. Although the general approach has remained the same, processing power, image precision, and the sheer volume of data have improved dramatically over the decades, Keefe says.
Something else has remained unchanged: He and his collaborators looked for ways to stay true to the artistic process. In the early days at Brown, “we had physical paintbrushes connected to the computer,” he says. “They were the same tangible, physical items you would hold in your hand.” By attaching sensors to those paintbrushes, they collected data on gestures and speed, then sent those data to algorithms that would generate color and texture. They designed an interface that replicated the stance and gestures of painting as truly as possible, with the palette floating in midair nearby.
While continuing to refine that approach, he’s also been experimenting with tools such as digital scanners. “What if we start with tools that an artist already knows how to use?” he asks. For example: An artist might look for materials with a rich physical texture, such as clay, leaves, shaved wax, or paint. Computer scientists can then scan the image, with a three-dimensional scanner, to create a digital form of those materials. Virtual objects made from these real-world textures might be used in art or in visualizations of data.
“This really opens up possibilities for the visual language we use to depict science,” Keefe says. “We don’t need to replace clay with digital clay in virtual reality. It’s more like our digital environment and our physical world will become one.”
One big challenge is finding and building on the strengths of users and tech. “You want the human to do what the human does well and the computer to do what the computer does well,” he says. “If we have a data glyph designed by an artist, and what we need is a field of 100,000 data glyphs, then replicating it is a great thing to assign to the computer.”
The Rise of Tilt Brush
Tools like a CAVE offer an immersive environment, but most artists working in the medium today don’t have access to the technology. “VR is very consumer oriented,” says Cruz-Neira, “and people expect a low cost of access to the technology. So a multimillion-dollar tool is not easily accepted by the public.” Today they’re more likely to enter the field with VR headsets.
Goodyear’s VR tool of choice is Tilt Brush, a three-dimensional painting program that was acquired by Google in 2015 and discontinued by the tech giant in 2021. After its discontinuation, the company released the code, and the program is now available, freely, as Open Brush. To use it, a person wears a VR headset and waves a touch controller; the program offers an expansive variety of colors, effects, patterns, and materials. One hand controls tools, palettes, and menus; the other paints in the air. A person using Tilt Brush appears as a kind of helmeted cross between an orchestra conductor and an amateur modern dancer.
Coder Patrick Hackett, who together with Drew Skillman designed Tilt Brush, never had art on his mind. The duo started, as many coders do, in gaming. He wanted to make digital games more interactive using devices such as dynamic controllers. They had been inspired by the Oculus Rift, a VR headset, and by the Leap Motion, a device that tracks hand gestures in the air to allow users to grasp virtual objects. “We saw potential for a more immersive type of game,” he says, and immediately began looking for new ways to plunge users into a digital, virtual gaming space.
One night, while developing a virtual chess game, Skillman and Hackett began experimenting with ways to highlight the chess pieces by making them glow. That led them to accidentally design a virtual “light brush”—a tool that would bloom in light, in VR, with the touch of a button—and then to envision a kind of VR experience unlike any other. “We wanted this to be this system where you invite your friends over and say, hey, try this thing,” Hackett says. After Google acquired the Tilt Brush program, it became enormously popular with artists.
And it grew: Those artists began submitting requests and suggestions for ways to bend the program to their artistic impulses, and Google’s coders responded by modifying and adding to the program. Hackett heard from a tap dancer, for example, who wanted the program to record performances into a VR program, and from painters who wanted a greater variety of brushes. “We looked at the decisions we’d made for how you’re interfacing with the art,” he says. Google even established an artist-in-residence program, in which artists spent a few weeks on the San Francisco, CA, campus working with coders to develop new applications and features in Tilt Brush and created works for a VR exhibition. The requests for new expansions and features flooded in. “We couldn’t move fast enough,” Hackett says, who worked on Tilt Brush at Google from 2015 to 2020. “We’d bring in an artist once a week, every other week.”
The projects were dazzling. One of those artists, Estella Tse, used Tilt Brush to recreate famous two-dimensional works of art (Gustav Klimt’s “The Kiss” or Frida Kahlo’s “The Two Fridas”) in three dimensions, in VR, which has been shown at immersive art shows and is now available to download for Open Brush users (see https://www.youtube.com/playlist?list=PLCj7B8MZZOaYI8uOibAT_6jsou6NAgCUB).
Goodyear, in Bedfordshire, first started seeing advertisements for Tilt Brush in 2018 and immediately realized how it might build on visual painting. She didn’t have any gear, though, so she emailed Vertigo VR, a local virtual reality entertainment center that offered VR games, with her thoughts about art and VR. She asked if they offered Tilt Brush.
“Ten minutes later they emailed back, said they wanted to support me,” she says. The owners, it turned out, were interested in exploring other ways to use VR tech. So, much to her surprise, she began a residency at the gaming center, using their gaming kits (and fast game engines) to create art during off-peak hours. That led to the creation of VR works like Topsy Turvy. Goodyear began to see VR not as a replacement for her physical painting practice but instead as a complementary tool that could push it further. Even so, VR will never fully supplant paint, she says. “It’s not messy enough.” She’s only half joking: Artists can discover new aesthetic ideas, she says, through tactile interactions with the paint that don’t occur in VR.
The gaming center stands next to the MK Gallery, and one day Goodyear invited the museum’s curators to her improvised virtual studio at Virtual VR. “They were blown away,” she says, and in February 2020 they opened the Paint Park exhibit.
Future Virtually Wide Open
The best way to virtually create is still a matter of debate. Some experts dismiss big systems like the CAVE because they’re too large and expensive to be practical, Keefe says. But at the same time, he argues that a headset can’t recapture the spirit of a CAVE. “There’s a physical space to [a CAVE], and a potential for collaboration,” he says. “There’s a different feel to the headset. Your whole sense of yourself, relative to the virtual space, is different when you walk into this room versus when you put something on your head.” Cruz-Neira agrees, noting that headsets “can be a barrier between the art and the audience.” She has developed a new version of the CAVE, called the vDen, that, she says, is less expensive and easier to use but still maintains the collaborative spirit of the original.
At the same time, both Keefe and Cruz-Neira say that the advent of VR headsets has made the technology widely accessible to artists and therefore raises the possibility of even more growth and exploration. “We have to try to imagine: What’s the style of interaction we would like to have with computers in the future?” Keefe asks. The benefit of that access, he says, is the potential to push the technology in new ways.
The fate of Tilt Brush may offer some clues. Google’s release of the source code online gave rise to a scrappy online community of coders and artists now united to push Open Brush—the open-source version of the program—forward. On platforms like Steam and Twitter and Discord, users announce their new brushes, textures, and updated features.
“There are all sorts of people asking: ‘Can you do this, work that? What’s a workaround?’ There are developers and designers saying, ‘I got so far with this,’” Goodyear says. “It’s a real embracing of science and tech combined. We want more of that.”
|
