NeVRmind: a field test of VR album art

BACKGROUND

My first foray into virtual reality, or VR, began nearly 27 years ago when my family took a summer vacation road trip from Virginia to Toronto. During our stay, the CN Tower just so happened to be hosting a video game exhibit, and the main attraction was a VR shooter—Dactyl Nightmare—in which two people donned clunky headsets and squared off against each other while fending off vicious pterodactyls. The gaming rig, which rendered VR in full 90s polygonal glory, made a significant impression upon me in terms of introducing an exciting technology platform, but at the cost $100,000, I wondered if I’d ever see VR reach my living room.

Interestingly enough, about the same time in 1991, Nirvana exploded onto the music scene with its hit album, Nevermind. Needless to say, I was a big fan and my wardrobe soon primarily consisted of flannel and jeans. Last year marked the 25th anniversary of the album’s release. 2016 also marked my reintroduction to VR as an emerging technology, when I took an introductory digital communications class for my Communications@Syracuse master’s program. My final report for the class centered on Google Cardboard and looked ahead to where the technology was going (Oculus Rift, HTC Vive had already been released and Playstation VR was just on the horizon at the time).

SITUATION

I bring up these past remembrances and events now, as they collided to form the basis of my field test for my Emerging Media Platforms class. In addition to VR, in the course of our study, we’ve examined several new technologies that are transforming the digital communications and media landscape, including chatbots, sensors, 3D reality capture, 360-degree video, and augmented reality. For my final field test, I initially wanted to test the viability of using a Playstation VR—or PSVR—and a drawing application included in a new game, Harmonix Music VR, to mimic the functionality of Google’s Tilt Brush in creating explorable, VR art. Harmonix Music VR is a significantly cheaper version of Google’s Tilt Brush when considering HTC Vive headset and computer startup costs that can total up to $2,300. I already own a Playstation 4, so the investment into PSVR as a platform wasn’t terribly offsetting at a cost of $450. An additional feature of Harmonix Music VR caught my eye, though—it allows you to load sidecar MP3 files to accompany the art creation process. My thoughts soon turned to the music capabilities of the game, and it wasn’t long before I was contemplating the various albums that I would bring to life, including Nirvana’s Nevermind. Some word mapping soon had me focusing on album art. Why album art? Well, as a vinyl enthusiast, I’ve often heard others lament that album art has died with the advent of digital, streaming music; it’s just not as complete of an experience as the physical album art that used to promote a more cohesive, immersive experience while listening. The same axiom doesn’t hold true for albums that it does for books—you CAN judge an album by its cover. In my mind, VR was the natural leap to reinvigorate the musical album experience and return it to the prominence it once had as visual medium.

HYPOTHESIS

I formed a hypothesis that VR environments can be adapted for album art, and that future albums may be released entirely as VR experiences, with either curated videos, photos or art installations that augment a band’s vision, or concept, for an album. As my concentration is Public Relations, VR album art is an interesting promotional piece that bands can use to garner listener interest and engagement amongst a target demographic. Creating an example interface and surveying viewers could yield some initial data to support VR as a viable, and welcomed album art experience. Fittingly, for the field test, I used Harmonix Music VR to recreate album artwork for Nirvana’s Nevermind.

TECHNOLOGY USED

  • Playstation 4 – A powerful, popular gaming platform that runs the PSVR headset and Harmonix Music VR software/game. Cost $299.

Use to media: provides a platform from which to run VR peripherals. Install base of more than 50 million users.

  • PSVR – Released in October 2016, this VR headset is a cheaper alternative to more expensive headsets, yet it has quality features such as OLED display, 1920×1080 resolution, and 3D audio processing. Cost: $399.

Use to media: provides a platform from which to view, interface with and create VR content. Install base of more than 1 million users.

  • Harmonix Music VR – A PS4 game made for PSVR that runs on Playstation 4. Cost: $15.

Use to media: features an “easel mode” similar to Tilt Brush, which allows users to create VR drawings accompanied by music from side-loaded MP3 files.

FIELD TEST

Admittedly, I’m an amateur artist. I haven’t painted since high school. Yet, I didn’t let this deter me from trying my artistic hand and bringing Nirvana’ album to life. I booted up Harmonix Music VR for the first time and was initially impressed with the 3D space of the easel application. Using the PS4’s “Move” controllers, I was able to bring up a limited menu of shapes and animations. I tested which shapes were more suited to the fine detail of drawing a 3D scene and began experimenting with various brushes. The controllers tracked my movements closely, and soon, I was able to create 3D objects such as a chair that really looked as if you could sit in it.

Next, I loaded an MP3 version of Nirvana’s Nevermind on a flash drive and created a playlist within Harmonix Music VR to work alongside of. I turned my attention to recreating Nevermind’s album cover, a photograph of a baby floating through a pool, led on by a dollar bill on a fishing line. I began by creating a rudimentary body using a spherical shape and connecting multiple spheres together to formulate mass. I used a smaller brush to form fingers on the hands. I used a flat brush to create a box, or pool, around the floating figure and then drew the fishing line and dollar bill in.

I experimented with this process multiple times and made multiple versions of the artwork. Each time, I felt my familiarity and ability to manipulate the tools became increasingly more intuitive and natural.

Overall, I was frustrated with the lack of detail that Harmonix Music VR allows. Furthermore, the easel application limits the total number of rendered shapes to under 300, which seriously limits the user’s ability to create a detailed scene. Undeterred, I made version after version until I was at least satisfied that my creation conveyed the concept I was trying to get across. I made a recording of a livestream to Youtube via my PS4’s broadcasting application, and created the following video describing my experience:

SURVEY RESULTS

I formulated a simple survey using surveymonkey.com to collect viewer responses of the field test as well as their opinions of VR as a platform for generating album art. As of writing this, 15 people have taken the survey and provided some key insights into my project and hypothesis.

  • Respondents’ ages range from 18-54, with 60% comprising the 24-34 age demographic (which makes sense as I made the survey initially available to my cohort)
  • 60% of respondents are male.
  • 100% of respondents think media experiences (e.g. album art, music videos, photography) compliment the musical listening experience.
  • 80% of respondents acquire/listen to their music via streaming applications.
  • 33% of respondents have not tried a VR headset, while 40% own a Google Cardboard device.
  • 66% of respondents do not own a VR headset, but intend to in the future.
  • 86% of respondents are interested in experiencing their favorite artist’s music as a VR experience. The remaining 14% said, “Maybe.”
  • 93% of respondents think VR experiences can/will compliment musical albums in the future. The remaining 7% said, “It’s hard to tell.”
  • Some reactions to the demo:
    • “Awesome concept.”
    • “Quality could be better.”
    • “I can see how you could create entire neighborhoods and worlds, even group entertainment associated with bands and genres and interconnect them.”
    • “Great idea.”
    • “Hopefully this idea will spark something, because that would be an incredible experience.”
    • If I’m honest, I felt the visuals fell short. I think if you are going to do VR album art the visuals should be HD and the art design should be just as pristine as it is in Tiltbrush. I think that true artists would want to have a better product. I do think the idea is terrific.”

CONCLUSION

While VR does indeed seem to be a medium suited for album art, my artistic abilities coupled to Harmonix Music VR did not deliver the full quality that many I surveyed—including myself—would expect from a professional album art experience. The game features limited, rudimentary shapes that lack the ability to develop the fine detail that is seen in applications such as Google’s Tilt Brush. Cheaper cost comes at the price of limited quality. I feel that with continued practice and iterations, I could improve the quality of my Nevermind creation. It would take some significant improvement, though. Additionally, I would be interested to see what a more accomplished artist could create using the game’s interface. As of now, though, the this particular technology application just isn’t there yet—the game lacks the ability to export creations, and it has no community sharing feature, which could truly make for an interesting way to explore other users’ interpretations of albums.

Based on the responses of those I surveyed, I am confident that we’ll see VR utilized as a medium of releasing an album in the future. A target demographic of males, age 24-34, who listen to music via streaming services could be an initial demographic to target. Applications such as Spotify and iTunes could use VR side-car album art as a way of increasing listener engagement, which could spell additional display ad opportunities for adverstisers. We’re already seeing music artists such as Foals using 360-degree video for music videos. And true to form, just this week, Gorillaz released a 360-degree animated video for their latest album, Humanz. Ultimately, based on my findings and research, my field test validates my hypothesis that completely immersive VR worlds could be the next iteration of album art, or experiential listening.

Sensors, AR will drive future advertising

In this week’s session of my emerging media platforms class, we were asked to imagine our career’s future through the lens of emerging technologies. In terms of advertising, I believe that many–if not all–of the emerging platforms we’ve had the opportunity to explore and discuss are going to be important mediums to engage consumers and entice them into the sales funnel. Sensors and augmented reality–or AR–are two technologies, specifically, that I think have strong potential for reshaping and strengthening advertisers’ ability to reach consumers. When paired together, these two technologies provide crucial, passive and active interfaces by which to target key segments. In the not too distant future, everything from consumables in grocery stores to restaurant/ entertainment venues will be coded or embedded with programmable sensors that will  convey contextual info such as comparative pricing or user reviews/ratings on the fly to lightweight headsets or visual interfaces like your driverless car’s 360-degree heads up displays (re: car windows). Pairing sensors and AR will obviate the need to utilize smartphones as the go-to research tool of choice. Instead, data will visually flow intuitively and seamlessly above the products and venues of interest to us, culled from customized user preferences and segmentation pulls via big data. Through sensors and AR, our analog and digital worlds will collide, creating an experience that is truly “indistinguishable from magic.”

NeVRmind: Field testing PSVR to generate album art

My emerging media platforms class culminates with an individual field test in which we must utilize one of the many innovations we’ve studied to create a hypothesis and application. My initial idea involves using a Playstation VR drawing app that’s in a new game developed by Harmonix. Truthfully, its “easel mode” is like a primitive–albeit cheaper–version of Google’s Tilt Brush, yet it is unique in that you can load sidecar .mp3s to accompany art creations. My hypothesis is that VR environments can be adapted for album art, and that future albums may be released entirely as VR experiences, with either curated videos, photos or art that augment a band’s vision, solidifying the true concept album. I realize that this application is less journalistic in nature than my course’s focus, but as my concentration is in fact public relations, I think VR album art is an interesting promotional piece that bands can use to garner interest and engagement. I’m looking to employ this particular PSVR paint app to recreate and provide a walkthrough of an album’s artwork as a VR experience, being tentatively focused on bringing to life Nirvana’s Nevermind.

Zen and the art of 3D reality capture

Recently, I was introduced to some incredible tools in my emerging media class that allow you to easily create your own 3D reality capture objects. Using my iPad Air 2, I captured approximately 50 photographs of a Buddha mask that I purchased in Thailand years ago. When photographing the mask, I took care to deliberately photograph it in a systematic way that covered every angle of its surface. I then imported the folder of images into AutoDesk’s ReMake software–free for students–which features an intuitive, easy to use cloud interface that does all of the work for you. The software spat out an .obj file and a texture file that I imported into Meshlab, another standalone piece of software that according to its website, “provides a set of tools for editing, cleaning, healing, inspecting, rendering, texturing and converting meshes.” Ummm, sure. After running the file through a magical filter that did some file size optimization, I imported a .zip file of my project into Sketchfab, a 3D reality capture publishing platform. The software did its magic and voila! A 3D version of my Buddha mask emerged. The Smithsonian has long been capturing its collection of artifacts, from primitive tools to spacecraft, providing a truly awesome way to connect with interactive content. I can imagine 3D reality capture becoming a standard way we digest elements of journalism, such as 3D infographic sidecar content for feature stories, annotated with interesting facts or background info that provide additional context. I’m excited for the possibilities these new tools provide storytellers and look forward to capturing and sharing more artifacts and stories. Shout out to Syracuse Newhouse Prof. Dan Pacheco for the instruction.

There, but not there with 360°, 3D video

“It is not enough for strong photographs to end up in a photographer’s retrospective on the walls of some museum and in the pages of a glossy book–they need to be much more revelatory and useful now.” — Fred Ritchin, After Photography

The advent of 360° and 3D video is quickly transforming the media environment. While video game technology leads the charge into VR, we’re also seeing innovative applications of 360° and 3D video in education, marketing and journalism. Tools to make immersive 360° and 3D experiences–which used to be prohibitively expensive in terms of both time and money–are becoming more prevalent, inexpensive and easy-to-use for the interested prosumer. Take Viar360 for example, a strikingly intuitive cloud-hosted 360° video storytelling tool that can bridge multiple headset, mobile and desktop viewing platforms. Cameras such as the 360fly 4K and LG360 provide affordable options for capturing 360° content with a relatively low learning curve.

Armed with these powerful storytelling tools, creators can add incredible depth and layers to stories that resonate with viewers’ ability to contextualize and empathize with content. Global transboundary issues such as immigration, climate change, pollution, refugee diaspora, natural disaster, terrorism and conflict/war are all potential subjects that are ripe for 360° and 3D video. These storytelling platforms are powerful mediums that can transport viewers into the fold of the realities that affect peoples’ lives and the world writ large. Perhaps the 2D image, in its printed and web form, is detaching by nature in that it limits our ability to truly connect with subjects and grapple with the complexities of situations that are distant from us. When experienced through a headset, 360° and 3D video–coupled with audial immersion–promise to change that, with added layers of sensory perception that allow us to be there, but not there. We’re not detached, we’re non-attached. And there’s a big difference between the two. If done effectively, after taking off a headset following a 360° story, we’re not left with a feeling of helplessness. Instead, we’re left longing to know and do more. What we need, now more than ever, are more meaningful ways to witness the gamut of our human condition–both the tragic and the beautiful–through stories that inform and inspire us to take action. 360° and 3D video may provide that.

 

Building a world wonder in Unity

It’s estimated that Egypt’s Great Pyramid of Giza took 20 years and approximately 100,000 people to build. This week, I harnessed what remaining horsepower my 2011 MacBook Pro has left in it and loaded up Unity–a cross-platform 3D game design engine–to see if I could recreate Egypt’s famous pyramids.

After downloading the free standalone edition from Unity’s site, I was off and running. Having used tools such as Photoshop and Premiere, I found Unity’s interface to be very intuitive, though sometimes frustrating. The biggest challenge was wrapping my head around the 3D environment for simple adjustments. Moving elements around can be a little tricky sometimes, but some mental focus coupled with the X,Y, and Z axis controls quickly rights the situation.

I began my project by creating a simple desert floor with some disparate terrain. I grabbed some free elements from Unity’s Asset Store, specifically the Desert Sandbox Lite package. This kit features sandy textures and simple desert ruins, including the massively important pyramid figure. Through some creative placement and resizing, I was able to put together a rough 3D sketch of Giza’s sprawling complex. As a final touch, I added an epic Skybox to complement the desert tones. Voila.

Despite its tendency to crash on my particular computer (SAVE often!), I’m pretty impressed with Unity; it’s easy to see why so many applications are powered by its very capable 3D engine. My project took just a couple of hours to build. I wonder what I could do with 20 years.

 

 

Old dogs and new tricks: the Innovator’s Dilemma

The Innovator’s Dilemma. According to Clayton Christensen, the prime mover of a sustaining technology can oftentimes fail to recognize and adapt to disruptive technologies that challenge its established position of dominance within a market. The inability to innovate and forecast changes within–and tangentially to–a sector can spell a missed opportunity.

susdir2Sustaining vs. Disruptive technologies, via web.mit.edu

Take the classic example of Microsoft’s Windows, which still–over the course of its 30-year existence–corners a significant majority of the desktop OS market share. Take a look.

img_0538Desktop OS market share, via www.netmarketshare.com

But here’s a question for Microsoft. Do you even mobile? The answer is no, not really.

img_0537Mobile/tablet OS market share, via www.netmarketshare.com

While Microsoft was busy squeezing every last user out of the desktop sector these past few decades, Google and Apple seized upon the bright idea that the future was in mobile. They were right. Mobile has emerged to dominate everything we do in terms of search, media consumption, monetary transactions and engagement.

Now, the OS is increasingly becoming unimportant as peripherals, third-party apps and browsers are able to be ported across multiple OS environments. As nascent technologies such as virtual reality and augmented reality further establish themselves as disruptive media platforms, what OS, app or browser environments will emerge as victors to serve up content to the masses? Oculus, HTC/Valve and Google are all jockeying for position. Interestingly enough, Apple has remained quiet on this front. And Microsoft, perhaps having learned its lesson from its shortsighted approach to mobile, is poised to disrupt them all with its Holographic OS. Who said an old dog can’t learn new tricks?