.4. The Visual – The 360° Immersive Body

VR-visualizations of heritage and bodies in general.
How to bring together the main issues – body, viewer, experience – into a VR-medium?
Visualization of the body experience.

 

” Artists have struggled to generate a space that will provide the viewers or ‘readers’ of interactive work with the complete illusion of being immersed within an interactive story. 360° screens appear to be the next step for the maximization of realism which artists tried to present in prehistoric caves, fresco rooms and painted panoramas”

[1]

Today’s middle-class immersive visuals are 360-panorama-photos and a few 360-videos you can reach via Facebook and YouTube.             
People’s approach to these and other immersive spaces is mostly through games and amusement or short trailers or an unusual experience. But VR-visuals already found their way into entertainment, education, medicine and military usage as well as architecture and other 3D-prototyping branches in engineering.[2] And it is used in the reconstruction of cultural heritage!

Museums and Heritage:
To not lose the grab onto the world’s heritage of historic places, museums and the 3D-objects in it, a lot of museums and other cultural institutions use 360-visualisations to display teasers or more extensive “applications” on the web. But in many cases, they can only function as a “teaser” for the actual place, for example because you only have a photograph on your flat screen and can’t approach the smaller exhibits on the wall. An example are the interactive panorama tours through the Staatliche Kunstsammlungen Dresden (Dresden State Art Collections)[3]. A nice feature here is that you can click somewhere to get further information and to proceed to another room. At another extreme, you find fully animated, high realistic, interactive applications for museum enthusiasts, you can watch with VR-glasses. “The VR Museum of Fine Art”[4] is one of those, providing a level of atmospheric immersion that makes you wonder again, why there are no other people with you.

[5] [6]

 

There are museum trailers, but mostly trailers of certain exhibits, digitalised and brought to motion by 3D- or VR-animations. The shown example, the “Art Plunge: Step Inside Classic Paintings With VR”[7], is highly informative and mostly detached from a main physical frame. This makes it more of a documentation. The next example, the VR-elaboration of Pieter Bruegel the Elder’s painting “The Fall of the Rebel Angels”[8] is more atmospheric. It plays with the border between a scientific art analysis and the space that only can be filled with additional material that is not depicted in the actual painting.

[9]  [10]

With the 360°-video “Dreams of Dalí” [11] the Salvador Dali Museum provides an elevated artwork experience that fills all immersive visual and auditory gaps. Here a distinct, surrealistic atmosphere is elaborated with the use of certain colours, space and object arrangements, and sound. You are moving into a picture and can see something in any direction, because additional details and elements are added. And with “Why Is It So Quiet here?” [12] the artist Jan Rothuizen translates a whole museum’s atmosphere into a short movie. This example is not executed as an actual VR-experience, but it has a lot of its design features.

[13]  [14]

 

The Body:            
Regarding body visualisations, we are living in a highly technical world with radical developments in science and rapid scientific changes. VR techniques are already fixed components of medical education and surgical procedures. For example, the YOU VR[15] and The Body VR[16] are dealing with the implementation of 360-depictions in diagnoses, for the better comprehension of patients, far away from all Latin technical terms. A next step will surely be to make this a common tool and later also implement real-time-body-data of the patients. And as an advancement of haemograms, there may be private mobile applications that show your body information and chemical data designed in real time. There already are good animations and VR-films of imagery in entertainment and popular science – for example the Body VR project provides an interactive journey into a cell. And on the other side there are absolute bodies macroscopied to the real world, in full museum installations like the CORPUS exhibition[17].

[18]  [19]

But I can’t see the cultural “heritage” component being adequately carved out yet. The main approach is a mixture of highly realistic but sci-fi-like depictions of one’s interior. In my opinion as a social oriented illustrator and animator, a more “feeling at home” and “perceiving cultural heritage” in immersive media would bring a more contemporary human value to the world of body interior visualisations and, therefore, to the always current relationship between body and mind.

Therefore, I want to build an immersive museum, using the relationship/connection between the space “museum” and a 360°-visualisation as well as the combination of VR as medium for heritage and VR as a medium for education/entertainment.

The creative liberties, with which the “Dreams of Dalí” animation creates a comfortable and aesthetic experience, will be combined with the physical frame of a museum-panorama-tour, like the Dresden State Art Collections provide. Together this may achieve a satisfying contemporary level of immersion within a positive visual experience.

But to finally bring together the three main issues “Body/Heart”, “House/Museum” and “Immersive experience/360-degree-visualization”, I must translate it into the same language.

 

[1] Oliver Grau, Virtual Art. From Illusion to Immersion, trans. Gloria Custance (Cambridge: The MIT Press, 2003, PDF), 5-6

[2] Lucio Tommaso de Paolis, “Virtual and Augmented Reality Applications.” (paper presented, Department of Engineering for Innovation, University of Salento, Lecce, Italy, 2017, PDF), slide 36-45. Retrieved from
http://avrlab.it/wp-content/uploads/2015/03/lez-1-introduction.pdf

[3] Staatliche Kunstsammlungen Dresden. Art from Romanticism to the Present. Explore the Galerie Neue Meister. 2017. Retrieved from http://www.skd.museum/en/explore/panoramic-tour/

[4] VirtualWorld, The VR Museum of Fine Art para HTC Vive, Filmed 2016, Youtube video. Posted August 21, 2016. Retrieved from https://www.youtube.com/watch?v=6WAC38aZ-hk

[5] SKD. Galerie Neue Meister. 2017

[6] VirtualWorld, The VR Museum, 2016, 00:17

[7] Sputnik. Art Plunge: Step Inside Classic Paintings With VR. Filmed 2016, Youtube video. Posted December 16, 2016. Retrieved from https://www.youtube.com/watch?v=0oWyAPpBfss

[8] Royal Museums of Fine Arts of Belgium Brussels, Bruegel: a fall with the Rebel Angels [Virtual Reality], Filmed 2016, Youtube video. Posted March 14, 2016. Retrieved from
https://www.youtube.com/watch?v=bXR9EEmb-JU

[9] Sputnik. Art Plunge, 2016, 00:43

[10] Royal Museums, Bruegel, 2016, 02:50

[11] Salvador Dali Museum, Inc. Dreams of Dali: Virtual Reality Experience. January, 2016. Retrieved from http://thedali.org/exhibit/dreams-dali-virtual-reality-experience-option-2/

[12] Museum Boijmans Van Beuningen, Jan Rothuizen: Why is it so quiet here? Filmed 2012, Youtube video. Posted October 16, 2012. Retrieved from https://www.youtube.com/watch?v=CXav9qqE78w

[13] Dali Museum, Dreams of Dali, 2016

[14] Museum BVB, Jan Rothuizen, 2012, 01:57

[15] Sharecare (YOU VR), Sharecare website, 2017. Retrieved from https://www.sharecare.com/

[16] The Body VR, The Body VR website, 2017. Retrieved from http://thebodyvr.com/

[17] CORPUS experience, CORPUS experience website, 2016. Retrieved from https://corpusexperience.nl/#!/international-english

[18] HTC VIVE. This is Real – YOU VR. Filmed 2016, Youtube video, 02:42. Posted October 3, 2016. Retrieved from https://www.youtube.com/watch?v=MDhioityjH8

[19] The Body VR, The Body VR: Journey Inside a Cell – HTC Vive Trailer, Filmed 2016, Youtube video, 00:05. Posted September 9, 2016. Retrieved from https://www.youtube.com/watch?v=XKbwmTG8chQ

.4.1

 

Visual Tests and Pre-Constructions

 

With behavior tracking/user experience and tests in between my animation set-up, I researched how to implement a heart-house (experience/heritage) into a VR-space (body/museum):

First I researched what connection people have to their environmental space and 360-visualizations: When we extrapolate the visual information of the totalized little screens most viewers are used to, the viewer is used to get occupied like in the following example. An example are digital news and social media sides, where you always have links to a following related side or article with a diverting character.

But a sphere also brings much more requirements with it then just looking. It is a hot medium[1]. A balance must be created between underload (that creates curiosity and activates imagination) and overload (that gives a “wow effect”).

Focusing:
I analysed three everyday 360-environments. The first is a normal urban one, where most people feel familiar with (https://kuula.co/post/7lpB7 ), the second is a closed space of an underground stop, that could resemble a medical/closed organ (https://kuula.co/post/7lpBN ), and the third (https://kuula.co/post/7lpzT ) is a forest setting, that could resemble an elevated, a transcendent or a nature connected body interior. I reconstructed a normal “look around” and added possible reasons for transitions from one focus point to another. This helped me to place single objects and more complex little “stories/actions” into my final illustrations.

 

Switching:
The viewer already gained a deep understanding of switching realities and zooming. He can easily emphasize to be in another space and can imagine himself in another size, because of his grasp of different sizes in which things can be depicted. The transition between macro- and microcosms lost its metaphysical aura, but became something natural and ordinary.

Imagination:
On the other side illusionism (mimesis) is less and less needed and therefore decreasingly used by the viewer. Especially regarding the fast improvement of 3D-media.[2] Where you before had to consciously assume that especially a two-dimensional image is three-dimensional because of the elaboration of its surface, you more and more consume unconsciously. Pictures get more and more digital and equipped with features that perfectly simulate a real depth. This fact in mind, I played around with the contrast, using e.g. flat images with no completely perfect perspective, but nevertheless presenting them in a 360-sphere. This gives it more the character of an elevated but traditional 360-illustration then a highly modern VR-experience.

Reality and distortion:
Regarding another aspect, though, the viewer seems to lose his feeling for natural depth. Not only in content, but also in a visual way, media devices like smartphones and other screening techniques depict the reality in a distorted way.
In his book “Flesh and Stone”, that also approaches interactions between human and space, Sennett gives a fitting example: Driving through the landscape with a non-human speed, the person changed into a passive body that has no natural resistance/renitence anymore.[3] This thought continued into our age of digitalisation, the body more and more vanishes.

Centre:
So it can become an experience itself to look with your whole body (in a 360 space) instead of looking in the common translation of a flat screen to eye to brain.

In the normal viewer’s perception, when his eyes are open, he focuses on something. In the example, it is a little illustration “on the wall” (my first sketch for a heart in a medical cross-section). Subconscious the viewer measures the distance to the point he is looking at. Therefore, the interaction line between his eyes and the illustration become the centre of perception. But when he is surrounded by the illustration, his head may become this centre. When you completely feel like standing in a scene, even the whole body can turn into it.

Economics:
Taking time, skills and following design decisions into account, the final VR-variant I use for the construction of my final visual will be a 360-animation with a static viewpoint and no extra interaction then the possibility to move around the viewpoint angle. On every device one can click and drag to look around, on touch devices one can swipe, portable devices with motion sensor you can swing around to see every angle, and with VR-glasses you have the most immersive possibilities except motion tracking and a parallax effect (an advanced special vision).

 

VR-techniques bring the viewer into the conflict, that no matter if the depicted worlds might be fictive or representing quite “real worlds”, the viewer gets similarly immersed and therefor excluded from the real environment.
For the experience, I want to create – that is primarily body-focused – it is an advantage to blind out the real world completely. It is like in the meditation experiment, where you close your eyes to concentrate on your body. But in this case I give the images:

 

 

[1] “Nothing is left to the imagination: the picture is complete, full, saturated. … [It] is overfed with information and thereby ‘heated’.”, Definition of hot media, In Arjen Mulder, Understanding Media Theory, 2004, 42-43

[2] Oliver Grau. Virtual Art. From Illusion to Immersion, 19-20

[3] Sennett, Flesh and Stone, 1994, 24-25

.4.2

 

Constructing the Adventure Museum

 

I pick up the viewer in the real world – with the introduction. The animation starts with picking up the moment the viewer puts on the VR-glasses. He sees himself in front, as an abstract figure, and then zooms into his “own” heart.

a virtual reality project

With the introducing heartbeat base that you rather feel than hear and the response of the viewers own heart that he can feel, he is directly addressed and his body does not get completely left outside the VR-screen. Like this the Virtual Reality becomes, at least a little bit, also an augmented reality[1].

[1] In an augmented reality, something virtual is implemented in real space, for example a hologram. 3D virtual objects are integrated into a 3D real environment in real time.
In Ronald T. Azuma, A Survey of Augmented Reality, Research Document (Malibu: Hughes Research Laboratories, 1997), 1

When the zoom in is done, the voice over now gives the impression of a museum guide, and the heart in front of you looks like a huge building complex. And it is clearly pointed out that it is “your heart” that will be entered, through audio, text and image.

The museum as a framework already combines a good connection to the topic of heritage with a body house you can feel at home.

Like this people have a familiar room they can connect with and it gives them a hint what they can do to jump deeper into this or similar topics: Visiting museums, libraries and research on the internet and on the other hand “visiting” their own body occasionally.

The next scenes will give cultural impressions, based on the developments and stories out of the heritage chapter (mainly in the voice over). And the atmosphere is based on discoveries of the receiver chapter, that draw a strong connection to the viewer (mainly in the visuals). And when the viewer’s eyes are closed – what happens with the black transitions between these scenes – the viewer’s heart is in the centre of perception, I termed in the centre experiment.

So, the frame becomes the viewers heart as a museum, as a house, changing from the anatomic heart, over the cultural imperfect, but metaphysical far reaching, to the cultural stuffed complex that you see in the last shot.

Every scene has a little visual story itself, and its own way of telling it. People should discover multiple features of 360-depictions!

In the entrance scene (PIC1), you are provided a special place in a boat-like blood cell that connects you to the scene in a physical way.

In the next scene (PIC2), the forest-like heart muscle is cross-sectioned at the place where you are “standing”.
The following scene, depicting the humorous theme (PIC3) confronts you with a quite flat, early middle age style illustration. But the flowers under your feet makes you part of this environment.

The four religions scene (PIC4) is created as a classical cube, what underlines the different “sides” the heart has in different cultures. Also, it gives the character of cave/wall painting.
The sacred heart (PIC5) even has a moment of tension, when you first only see the hands and long arms healing other organs, follow them and suddenly encounter the eerie but good sacred heart.

The social city (PIC6) gives a classic cause and effect story in a circle. One figure is always connected to another to his left and to his right so you can read a chaotic but linear story.
In the next moment (PIC7) – when the figures vanished – you seem to be the focus of the scene that gives you a powerful feeling and the vast perspectives gives you an enormous feel of distance and depth.

In the pump scene (PIC8) there is a focus object at the spot where the animation started with the figure in front (also in the entrance and exit scene, PIC1 and 10). Other objects around lead you to another side of the sphere. Especially pathways that resemble veins and can lead your view over a long distance. (PIC8). The trail of lights I discovered in the underground station scenario, I almost directly translated into the future city scene (PIC9).

The exit scene (PIC10) is a conclusion, where the viewer only sees things he only saw before, but arranged in one object. The stuffed heart complex stands in high contrast to its vast and almost empty environment. This may suggest that the heart is still only one little organ in the human body, but now at least closer to the viewer then in the entrance shot.

 

Some illustrations in 360°: https://kuula.co/profile/anthea

Video (temporary version):

.4.3

 

Further Alignments during the Design Process

 

The scenes are presented in stills. Like this I don’t force the viewer to look at a certain spot. Then it would be like a common animation, with one focus point at a time. It shall more feel like in a normal room – where the viewer can find rest and have the power to look around when he wants.
Also in my tests, I encountered that people feel as if they would miss something, when there is something moving in the room as well, and thus they get stressed. And one of the primary intentions is that the viewer gets a positive experience!
Also, with changing stills, it feels as if the room would remain, after you leave it. This gives a more realistic impression, as if there is one motion/happening that seems extraordinary. It also supports the house/museum structure, where all the different scenes can coexist next to each other.

The illustration style is strongly connected with this character. The blotchiness gives it a painted character.

Nevertheless, the VR-presentation needs a certain dynamic, next to the viewer’s head turning option. So, I created a time-based experience. The long but limited “times” for each room give it a teaser-character, a certain tension. The colours are luminous and get even more vibrant with the development into a lively complex.

The changing pictures plus sound transport my cultural message: “The heart developed from a holy centred subject to a generally enchanted but still inspirational object.”

Moreover, the practical research about dynamics and voice over brought the result, that every scene needs a clear statement/title: Hence, I narrowed the voice over down to three short sentences at most. Also, they mark the transition between the different scenes, which otherwise wouldn’t have a beginning or end. They use and bridge the empty space and then the viewer has time to look around again.

A light story is explained, the development of the heart in different cultural relationships. It is delimited through the entrance and exit shot of the heart building. It gives the viewer the opportunity to see a wealth of different impressions, without having to deal with an artificial steering or interactivity.
Through the time-based character and the room changes that the viewer can’t influence, he is dragged into a story. Also, like I saw in my tests, most people are already busy enough with looking around and primarily want to enjoy the look around.