Sphere’s ‘Groundbreaking’ Audio-Visual Technology Could Reshape Live Entertainment

As Sphere, the innovative new Las Vegas venue, opens its doors to the public with the debut of U2‘s 25-date residency on Friday (Sept. 29) and the premiere of the Darren Aronofsky film Postcard From Earth on Oct. 6, it’s doing so with an array of cutting-edge technology — much of which hadn’t been developed when the project broke ground in 2018.

“A lot of this stuff didn’t exist — it just didn’t,” says MSG Ventures CEO David Dibble, who oversees Sphere’s technology and content teams. “Necessity is the mother of invention, and by God, we had necessity.”

Plenty of Sphere’s advancements feel unique to the facility itself, including the geometry of its bowl-shaped theater and its 4D multisensory technology, which can generate effects like vibration, wind, scent and temperature fluctuations. But two key components — the venue’s audio and visual capabilities — could soon have a ripple effect across the concert business and broader live entertainment industry.

Sphere audiences will hear audio via Sphere Immersive Sound, a system created in tandem with the Berlin-based audio company Holoplot. “The problem that we tried to tackle from the beginning was not to build another sound system — because the world has enough sound systems,” says Holoplot CEO Roman Sick, who founded the company in 2011 with the goal of creating “a realistic, authentic audio experience that is not mainly determined by the room you’re having that audio experience in.”

Sphere executives discovered Holoplot after the company deployed its 3D Audio-Beamforming technology in Frankfurt Hauptbahnhof, Germany’s largest train station, in late 2016, allowing multiple audio messages at the same frequency to be sent simultaneously to different parts of the facility.

“If you boil it down, we have two core capabilities,” Sick explains. “The functional level, from our perspective, is you can determine where you want to have sound and where you don’t want to have sound. … And the creative bit is the ability to now move audio objects three-dimensionally across that whole volume [of space in a venue] — from left to right and up and down, but also into the audience back and forth.”

Holoplot dubbed the former aspect 3D Audio-Beamforming technology and the latter one Wave Field Synthesis. With Wave Field Synthesis, Sick explains, Holoplot can make the origins of audio imperceptible to create “a hologram of sound” — an accomplishment he calls “the holy grail of spatial audio.”

To implement these features, Sphere Immersive Sound utilizes advanced hardware and software. Behind Sphere’s 160,000-square-foot LED screen sit hundreds of Holoplot’s X1 Matrix arrays, which combine the functionalities of vertical and horizontal line arrays to allow greater control over the formation and shape of audio waves. Holoplot’s software then utilizes proprietary algorithms and machine learning to synthesize creative input and environmental data, collected by sensors throughout the venue, to further refine the system’s audio output.

Sphere Immersive Sound might sound complicated — and it is — but like many of the venue’s production capabilities, it’s designed to be plug-and-play for visiting artists and their teams.

“You don’t need to be a scientist,” Sick says knowingly. “You just say, ‘Hey, I want sound here and over there, for this configuration.’ And the system says, ‘OK, here it is.'” According to Sick, the system has “a large number of preset formats” — mono, stereo, 5.1, Atmos and so on — for artists to choose from. Once they do, “boom, then it’s the normal workflow,” says Sick, adding that an artist’s audio engineer can even use their normal desk: “From that end, nothing really changes.”

Of course, Sphere’s audio advancements didn’t take place in a vacuum. In implementing Holoplot’s technology, Sick and his team had to consider numerous other stakeholders, chief among them those conceptualizing Sphere’s visual capabilities. “You put something in front of a speaker, it’s going to have an effect — and it’s a negative effect, usually,” says Sick, summarizing the challenge of placing high-end audio equipment behind Sphere’s LED screen. Compared to point source or line array speakers, Holoplot’s matrix array had an advantage — it diffuses energy over a larger surface area, reducing the energy passing through an obstruction, in this case Sphere’s LED screen, at any given point. But Sick’s team still had to find “the best compromise between acoustic transparency and meeting the visual requirements.”

“It’s such a unique and groundbreaking technology that, maybe for the first time ever in the audio-visual world, the images, as incredible as they are, are almost subordinate to the audio experience,” says Andrew Shulkind, senior vp of capture and innovation at Sphere Studios, the Burbank-based entity Sphere launched to develop technology and content tailored specifically for the venue.

That’s saying something: Shulkind became involved with Sphere several years ago to help it create visual content — like Aronofsky’s Postcard For Earth — suited to its massive, high-resolution screen. Initially, Shulkind and his colleagues shot tests using camera arrays, a common but cumbersome filmmaking technique that stitches together video captured from multiple cameras to generate a more detailed product.

“It became pretty obvious quickly that we really need a single-camera solution, for a variety of reasons, for weight and for mobility and ergonomics, and to be able to take all the difficulty of maneuvering something heavy out of the way,” says Shulkind, who enlisted a colleague, Deanan DaSilva, to help create a new camera fit for Sphere.

The resulting device, Big Sky, pushes the boundaries of modern filmmaking technology with a sensor 40 times the resolution of a 4K camera, lenses with high sharpness thresholds and even new data storage solutions to manage the large volume of information it produces. “This was something that [camera makers] weren’t expecting to do for another 10 years,” DaSilva says. “We had to figure out how to move that timeline up.”

Like Sick, ultimately Shulkind and DaSilva had to ensure that their advanced technology was accessible to the outside creators that Sphere wants to court. “We work with external creatives, they come in, they describe what they want to do, they have their support team and then we fill out the capture side,” Shulkind says. “We take the complications of any of these technologies out of the mix, and it becomes about, you know, what story are you trying to tell?”

Filmed by Aronofsky on every continent in conjunction with Sphere Studios, Postcard From Earth, Shulkind explains, was “really designed to take people to another place that they may not have been, or places that they may not have seen in that way before. Darren has been able to marshal all of the different aspects of the venue in service of that goal.”

Shulkind acknowledges Sphere Studios’ myriad technical accomplishments but has a broader view of their implications that transcends the wider implementation of any one of its technologies. For him, Sphere’s format could finally allow filmmakers to “break the rectangle,” or go beyond the rectangular framing of visual storytelling that emerged from rectangular film strips.

“Now that we have all the technology of the minute, whether it’s data storage, whether it’s the fidelity that we’re able to achieve with this high resolution, whether it’s the ability of creating glass that is able to be as sharp as it is, all the different aspects come together to create this greater-than-the-sum-of-its-parts [product], radically rethinking how stories are told and how we experience content,” he says. “You’re looking at all these little composite technologies and all the growing currents of where technology has gone over the last 10 years. How do you apply that all for some creative purpose? I think that’s the real experiential success story.”

And that story isn’t over yet. While Sphere’s teams worked diligently to design and implement new technologies for the venue’s opening, Sick, DaSilva, and Shulkin all note that they’ll continue to iterate and improve their tools going forward.

As creatives “start to really discover how to tell stories in the venue … that’ll very clearly drive the technology evolution,” DaSilva says. “We’ve got a to-do list of all the things to try that we’ve not even scratched the surface on.”

“We constantly keep updating our technology,” Sick adds. “There’s new features that we will deploy over time, even after the venue has opened, that will give new capabilities to Sphere.”

It seems likely that at least some of these technologies will eventually move beyond the walls of Sphere’s Las Vegas facility to other venues — but what shape that proliferation will take remains unclear. After all, the Sphere team has already filed more than 60 patents. “One of the reasons we’ve been so aggressive on our patents is imitation is the sincerest form of flattery,” Dibble says. “But we’d just as soon not be imitated, because we own it.”