Hello everyone. Hi, this is George Giaglis. Welcome to week 9 of the University of Nicosia's
free MOOC on NFTs and the Metaverse. Today's topic is trends in visualization technology.
So we are in the second week of addressing metaverse related issues, and as we also did
in the beginning of the course, before introducing NFTs, we discussed about Ethereum and underlying
technologies providing a foundational infrastructure for non-fungible tokens. We are going to do
the same with the metaverse. So today's lecture is going to be more technical in nature. We
are going to be discussing recent trends in visualization. We are going to see augmented
and virtual reality and try to capture how these things fit into the metaverse vision.
And because this is a very specialized topic, I am honored to be joined by two colleagues
who are experts in the space and will be covering the majority of the presentation today. So
without further ado, let me introduce you to the first speaker who is none other than
Chris Christou, a colleague of mine at the University of Nicosia, associate professor,
and head of our VR lab. So Chris, the floor is yours.
Thank you very much, George. Welcome. Welcome to the first part of this session. I am going
to cover 3D rendering visualization and computer graphics in this part. And then my colleague
George will, talk about its uses in virtual and augmented reality. Visualization comes
in all forms. It is pervasive throughout our lives. It is used to render simulations of
architecture of chemical reactions, crowd simulations, fluid dynamics. So computer graphics
is pretty much everywhere. The origins of visualization come from, I guess,
cave drawings, but more recently from architecture. So if somebody wanted to, for example, create
a building, they would go to an architect and they would create some drawings for them.
These drawings would be orthographic in nature to preserve parallel lines, to preserve the
shape in order for it to be constructed correctly. And if they got it wrong, they would have
to go back to the client, go back to the drawing board as it were. So this is a long
drawn-up process. This has been replaced by computer-aided design or CAD. Everything is
three-dimensional now. We can walk through a model. We can fly through a model of a building
or a city long before it is even created. We can also simulate the lighting that is
available in the building at a particular time of day, at a particular location. So things
have changed an awful lot. Looking forward, we imagine that developments in haptics with auditory
representation or faction even will mean that we don't have just visualization. We will
have perceptualization sometime in the future. These are the enabling technologies that have
helped us along. Primarily the hardware, the graphical processing unit, the GPU, that is
in every device that everyone has in their pockets, in their mobile phones or in their
computers. They can render millions upon millions of polygons per second. And these made it
possible basically for virtual reality and augmented reality to happen. We have high-resolution
displays and this includes the organic LEDs that we have in our VR devices. Computer vision,
AI, machine learning, deep learning are all contributing now to developments in 3D graphics.
So computer vision is responsible; it is the field where you study how to find structure
in the world, whereas graphics is actually the process of rendering that structure and
therefore, forming a happy collaboration. And then finally we have LiDAR and structure
for motion. These are techniques of finding structure, of representing our real world
and putting this into our computer model. I am going to talk about the history of graphics
first of all and explain some of the processes that go into rendering computer graphics to
give the viewers an idea of what computer graphics is. And then I will end with a few
examples of very recent works. So a brief history of CGI, computer generated imagery.
It was very much influenced by Edwin Catmull, Pat Hanrahan and Jim Blinn. Edwin Catmull
was responsible, was one of the co-founders of Pixar which went on to create the short
animation "Luxo Jr" which is available on YouTube even to this day. And this resulted
in computer graphics being used throughout the movie industries and throughout entertainment.
So these guys were also instrumental in the development of the GPU. And as I just mentioned,
this is what's made everything possible on mobile device, how resolution displays on
mobile devices, computer games on mobile devices and very great games on our PCs. So
behind any graphics is the graphics rendering pipeline. So on the left, on the one side,
you have your application. This is your computer game. This is your VR simulation, let's say.
And you want to get the graphics from that app to the screen on the right hand side.
Ok, so somewhere in there you've got the geometry, you've got whatever it is that's
moving, the zombies that are chasing you. And you want to project that onto the screen.
So that involves various stages of occlusion detection and seeing what is visible from
the screen, working out the colors, etc. The rasterization process is the process of
actually drawing something onto the screen. And most of this is done in a scanline order.
So when we talk about scanline, we mean that we refer to the pixels of the screen being
broken up into a rectangular grid. And we usually start at the top left hand side, we
work to the right hand side and we do a zigzag all the way down to the bottom. This is how
we get a 2D image. When we think of the graphics process itself, there is a virtual camera
and there is our geometry. And wherever the virtual camera is, this is what we are projecting
onto the screen. If we're talking about virtual reality, let's say an immersive headset, then
this virtual camera is essentially controlled or moved by your head. So when you move your
head, the virtual camera moves in the virtual environment. A lot of you have heard of ray
tracing. So this is important for a little bit later in what I have to say. So I'll mention
it here. So ray tracing, quite simply, is tracing rays from the eye through each of the pixels
on our screen. And then if these rays don't hit anything, they don't intersect with any
object in the scene, then we just paint the pixel black. If the ray goes through a pixel
and it hits an object, in this case of the point X, then we have to calculate what color
to paint the pixel. Now this color depends on the light. So it's a simple function of
the surface orientation, the surface normal, as we call it at that point, and the angle
that makes with the light source. And this is pretty intuitive. So if the surface is
pointing towards the light, then it gets more energy as brighter. If it's pointing away
from the light source, of course, it receives no illumination and it will be dark. So that's
a very simplistic illumination model. So these illumination models made up the core of computer
graphics research in the last, in the early stages of computer graphics. Okay, so these
researchers were busy coming up with models of how to best represent the various effects
that we have in the real world, the various shading effects that we have in the real world.
One of the earliest models is the Phong model. And this can be explained by the diagram at
the bottom left here. So we can break up the illumination of any object, in this case,
this funny looking shape, into three components. The first one is the ambient component, which
is light from everywhere. It adds nothing to the structure, not into the shading. It
just ensures that the whole object is illuminated even though it's not facing the light source.
The next component is the diffuse component or the immersion component. And this does,
as we saw in the previous slide, this is orientation dependent and it adds the shading that you
can see here. And the final component is the specularity, the shiny highlights that you get on
glass and shiny surfaces. So there's a nice representation on the right-hand side where you
can see that the process is not as straightforward as I have just described as you might
imagine. We have refraction, we have reflection, we have different types of reflection, we have
diffuse reflection, we have specular reflection. So coming up with an illumination model that
actually captures all of this is hard, but the benefits are that you get towards our aim,
which is photorealistic graphics, okay, and post-realism. Another complication is the fact that in the
real world we have indirect illumination and this is nicely portrayed here. So in the image on the
left we have a scene where there is no indirect illumination. On the right we have a scene which
is rendered with global illumination. So let me describe what's going on. The shading patterns
across this image are a function not just of the direct light sources. So this one from the window
for example from here. There's also light bouncing off the floor onto the ceiling, bouncing back
again. So all of this light that's bouncing around in the environment is causing this smooth
shading that you can see, illumination of the ceiling essentially which has no direct light
shining on it. So things are not as simple as we would hope in the real world. More about that
later. So let me describe now just a content generation which is the stuff, the geometry, the
stuff that's actually in our computer game that's in our television commercial. If it's 3D it's
going to have been made in some 3D editor. This is the interface with 3D Studio Max. The first thing
to note is that everything is polygonized. Everything consists of polygons. They use the flats;
flat simple surfaces. We join them all together and not one by one but we join them all together
to make curved surfaces. On the right you may see you may be able to make out that there are basic
primitive shapes and boxes for example and spheres and these are used to create more complicated
objects. You may also note the teapot and this is just the Utah Teapot. I've put a link there. It's
a very special teapot that's been used for computer graphics research for the last 40 or so years.
Characters, avatars, character modeling. There's no difference here. They still consist of polygons.
The special thing about 3D characters or avatars is that they have a biped rig or a biped skeleton
which is the actual thing that does the animation so, if we're talking about animated games for example,
somebody has to create the animations and piece the animations together. This can be done with
keyframing or it can be done with motion capture where a real actor performs the motions and then
these motions are used to move the virtual character. At the bottom, just a brief mention
about this, we can do crowd simulation. This is something from my own work. So we're simulating how
annoyed people get with measuring how annoyed people get when they're surrounded by crowds.
But you can also use it for escape route planning for example, to simulate what happens when there's
a fire in a building. This is a multi-character scenario where we've got many non-player characters
in the scene. Again these are no different from the character models that I mentioned previously.
Probably just a glare resolution. We're looking at current trends now in the last few slides.
LiDAR is used throughout for measuring distance for getting a structure or a special structure.
The principle here is the same as an echo. It takes a while for light to,
projected light to bounce back from surfaces. So we can measure the time that it takes for light
to come back to an emitter. In this case it's available. We're on consumer devices like the iPad Pro.
From this we get a pixelated version of the image in front of us. So a point map.
And the point map can be turned into a depth map which is just an encoding of how far
objects are away from us in relative depth. In turn this depth map can be turned into
a structure, into a 3D model. This is used throughout the modern tech that you will hear about later on.
So all of the devices that for example the meta-quest that uses this to work out where it is in the
room and all the augmented reality and glasses use this to work out the surfaces on which to
project their graphics. This is some exciting work that's done by Meta,
formerly known as Facebook. So here they're actually getting the structure of human beings, of people.
So people would go in here and they would have their head scanned or their body scanned. So this
is a multi-camera rig and a multi-light rig just to ensure that there are no shadows.
It's used to extract the structure of somebody's face, in this case, and also the textures of the
face. And then deep learning can be used to reconstruct expressions that a device such as the Quest
for example of the Future Quest text that the user is making. So if somebody is grimacing
then their avatar will be grimacing in the metaverse.
If you have a, so and light fields and talk about light fields, if you have a Steam
Account or a HTC Vive or MetaQuest it's worth downloading Google's Welcome to Light Fields.
So previously I was talking about getting the structure of the person, now you're getting the
structure of the environment. And this is a wonderful demonstration of how realistic graphics
can be. So what Google have done here is they've mounted a number of GoPro cameras onto this
rotating rig and they're basically sampling the amount of light in the room. And as the rig is
rotating around the calculating, they're sampling the light structure of that room, the so-called
Light Field. Okay, if you store this you can play it back to somebody. And then the feeling is,
is not just one of realism but it's also a way to capture the specular
parts, the specular components of the surfaces in your scene, shiny surfaces, etc.
Light Fields in particular, I would not mention too much about this as we
need to progress. But if you want to capture the full extent of light in the scene, you really
need to use a multi-camera grid. I'm showing in this middle diagram. You can also use a
plenoptic camera. So in the Google case, they would use the multiple GoPro's which were rotating.
But if you want a forward-facing camera, you can use a plenoptic camera,
which as you can see takes small images of the same scene and you can put these together
in order to get motion parallax, in order to get the shine from a glass, for example,
in order to see a little world or represent the world more realistically.
Now, if you had a limited number of these samples, you could use a Neural Network to
actually calculate or to represent the space in between. So this is the principle of NeRFs.
And this is the very recent paper by Benjamin Attal from this year. And this demonstrates,
first of all, the power of occlusion effects in depth perception, the power of motion parallax,
the realism and specularities, and just the overall structure. So these are novel views of
just a limited number of samples of images. This is the basic principle of NeRFs. So the idea is
that you take a small subset of images from an object and you create a volumetric representation
using a convolutional neural network. This is another demonstration of this from Nvidia's
websites, I put a link to this down below. So again, you have limited number of views
and the fly through, which is generated by the neural network. So the thing that you might be
thinking at this stage, and especially seeing the image on the right hand side is, "can we get an
actual model out of this?". "Can we get a computer model out of this three-dimensional neural network
representation?" And the answer is, Yes, and people are actually working on this. So again, this is
very recent work by Munkberg taking Multi-view images representing these within a neural network
and outputting a mesh, a three-dimensional mesh of the object as well as the textures that are used
to paint the detail onto the surface, and also the light probes. So the light probes capture the
specular components, and this can be output straightened to your favorite game engine, Unity,
for example or Unreal or into a 3D editor where they can be further edited.
Okay, so that brings me to the end of what I wanted to introduce you to in my segment.
So, we'll pass over to my colleague, George, and he will take over. Thank you very much, Chris.
That's some fascinating stuff here. So for those of you, the students, I mean, who are interested
in this, and what George Koutitas is going to present in a while, let me tell you that we're
working towards creating a follow-up course that will focus on the upcoming developments in terms of
hardware and goggles and masks and headsets and all that stuff. There's some truly fascinating
developments happening by some big companies and some startups. So we will revisit the space,
post Christmas, and have a special day's course for those that are interested in seeing how our
world will change. So Chris will stay with us until the end and he will be available for questions
if you have any. But before we go to the Q&A let me introduce our other speaker for the day,
my colleague and friend, George Koutitas. George is an executive entrepreneur and academic
with more than a decade of experience in business and R&D. He has a multicultural background. He
has spent six years in Austin, Texas, five years in the UK, and another six years in Greece,
where he's currently based. He has founded a startup company in Austin working on AR and VR
and training of first responders and has a number of publications and a patent in the
AR and VR space. So it's a great pleasure for me to introduce him to the course. George, are you with us?
Yes, hello everyone. Thank you, George, for the warm welcome.
Hello, everyone. Can you hear me? I hope you can.
George, thank you very much for your warm welcome. Just give me an indication that you can hear me
so I can continue. Okay, all right. So I'm very happy to be here with you today and speak to you
and introduce you to the concept of Extended Reality. Some of the things that we're going to
discuss today, probably you might be already aware, but some of them might be new to you and
might help you expand your horizons. So we all hear about VR, AR, Mixed Reality (MR).
Let's understand the difference. Virtual Reality (VR), a simulated experience, okay, in a fully virtual
world, and this is available to you through 3D near-eye displays. So you are fully isolated
from the physical environment. You are in a totally virtual environment with graphics presented to
you from a display in front of your eyes. On the other hand, Augmented Reality (AR) allows you to interact
with the physical world and it overlays digital information and content on top of the physical world.
Mixed Reality, which is now, you know, sometimes we use the same term augmented and Mixed Reality,
is the ability of the digital content to interact with the physical environment. So this means that
if you can see the third circle, the 3D graphic is behind the sofa and it is in the shadow region of
the sofa. The sofa is a physical object in my living room and the robot is a digital content
and I can partially see it. This is called Mixed Reality. Augmented Reality, an example of Augmented
Reality was Google glasses, okay, or even our smartphones that we can have AR applications.
Mixed Reality is more modern applications that can be made usually with Microsoft HoloLens and
other AR devices. From now on, just to not get confused, AR, MR can be thought of almost the same.
In order to experience AR, VR, we need to have a head-mounted device and as you already know,
there is a plethora of devices in the market. The breakthrough in the HMD head-mounted device came
from Palmer Luckey in a kick-starter project. This was in 2012, I think, but started the Oculus, okay,
and there was an excitement there and then an angler of the technology because many developers
used the development kit, DK1, offered by Oculus. So they were able to create applications in the
VR space and people can access them through a marketplace. In the image here on the left hand,
you can see some VR headsets. One is the Oculus Rift. You can see a cable because it required to
be connected to the computer for some processing power. You can see the cardboard that you put your
smartphone in order to act as the VR display and also you can see the latest versions of Meta
Oculus Quest. On the right side, you can see some, a couple of examples of AR headsets. We have
Google Glasses, Microsoft HoloLens and Magic Clip. You can see some of the content here. I'm not
going to read out to you, obviously. I'm explaining in the images. Feel free to use the slides and,
you know, dive a little bit deeper in the terms. So we have AR VR experiences deployed to us through
head-mounted devices. What are the applications? There are numerous applications that we can
experience. Both AR VR has a little bit of struggle in finding the key application areas.
So we have seen VR going very deep in the gaming space. But then other application areas may involve
learning and development, remote collaboration, social networks, or even industrial applications.
AR is the same. But we will see as the time passes that VR is more like on the gaming aspect
and remote collaboration and social networks, whereas AR can be used mainly for industrial
manufacturing, construction applications, or learning and development because it allows us to
interact with the physical environment. This is not 100% true. Obviously, we have VR applications
since they're learning and development or industrial applications. But we have seen these separations
on the application areas. And obviously, the reason is that AR allows you to interact with
the physical world. Gaming is huge. By 2024, it's going to be 2.5 billions.
Remote collaboration, we have companies like Spatial I.O.
Agriculture, it is a recent trend in the integration of AR with Internet of Things.
So we already have augmented reality startup companies that helps farmers
personalize, actually optimize the quality of the growth of their fields by either deploying
sensor networks and taking measurements of the humidity, etc., or by using smart cameras
that allows to optimize where you need to put more water, etc. So there are very, very exciting
applications. Learning and Development, as we will see a little bit later, VR and AR has a very
important advantage compared to traditional, let's say, web training programs. It improves
cognitive learning, but also muscle memory, because you're moving your hands, you can move
in the environment and the brain can remember where items are positioned and what actions you need
to do if it is related to a repetitive work. So very fascinating. Obviously, health, we have a lot
of applications in the health sector, either in the training but also during operation.
Manufacturing and industrial, you don't need to be an expert in order to do a repair. You can download
the instructions and you can do the repair at the same time that you are actually doing the repair
of a machinery or etc. We are not very far away of what we have seen in the movies that you can
download something in your, not brain, on your AR device and execute it without being an expert
in the field, similar to matrix. Architecture and construction, obviously there are numerous
applications there. So the world is fascinating and AR VR will definitely be dominating in our lives
and our work now and in the future. It's really interesting to see how this technology was
evolved. The first HMD head-mount and device started in 1943. Yes, believe it or not, it's so
old. You can see that there was a big gap. Obviously, you know, the technology was not there, user adoption
was not there. Then suddenly in 1960 to 1969, there was a decade of people with the growth of
the computers. They started experiencing different types of technologies in order to
create immersive environments. The most exciting milestone was in 1962 with
an immersive experience called Sensorama. If you see the video now, it's going to be funny,
but for 1962 it was a breakthrough. You will see that there are waves, bursts, let's say, of
evolutions and now we are in the time that the technology, the hardware, is there. We have portable devices
with very great quality of experience and quality of the graphics. The time is now
to exponentially grow the sector. In the VR space, there are a lot of companies that provide
head-mounted devices. Obviously, one of the most well-known is Meta Oculus. They bought a company
some years ago and they focused initially on the gaming aspect and they had more like a B2C
approach, business to consumer. They addressed the consumer market and there was the first, let's say,
exponential adoption of the device. Obviously, there are other companies out there like Google,
HTC Vive, Samsung, etc. Remember, in the VR space, it all started with Sensorama.
I highly recommend to see this video to understand how 60 years ago people created the first immersive
experience. If we focus on one product, the most famous, let's say Oculus Quest, you will see that
it started with a passive VR experience without any type of controllers so, it was more like visualization.
Then we had the Oculus Rift that was connected to a PC in order to provide some required processing
power. Then we had Go and Oculus Quest that work with battery and in a standalone manner,
so you don't need to connect it to a computer. Then we had Meta Quest Pro that was recently announced
that it reduces very cool features like mixed reality. You can see that in the front part of
the display, there are cameras that allow you to perform gestures and you can use your actual hands,
you don't need the joystick. The level of experience and the graphics has dramatically improved
compared to different versions. This evolution is met in all companies.
In the AR space, we have also a lot of companies that provide devices. The most famous one is
Microsoft HoloLens, Magic Clip. Metac is still present in the AR space with what they call Spark AR.
It's a platform that anyone can create AR experiences that are used on the mobile device. They don't have
an AR headset yet, at least available in the market.
Magic Leap is an important company to see because there was an initial hype back in 2014. I think
they raised a lot of money for a huge amount of evaluation. The company was not ready to provide
the product and didn't address the right niche market to penetrate in the market.
That's why there was more like an idle mode for this startup company.
But recently, we see a lot of motion and evolution coming from Magic Leap since they
trimmed their business model to more like enterprise AR and use cases related to health.
So we expect to see a lot of growth and a lot of cool new features from Magic Leap 2.
Obviously, a recent trend is coming from the Metaverse. So, imagine we have AR, VR companies,
we have computer graphics companies and now we have companies in the Metaverse space.
Either by creating 3D environments, either by creating serious games and interactive
environments like Roblox Corporation, Decentraland etc. It's going to be fascinating to see what type of
collaborations, acquisitions or merges are going to happen between the AR, VR and the Metavors space.
So I'm sure that in the next years we're going to see a lot of action in this space.
But let's see what is happening inside a headset. What is inside? What type of electronics do they
have? Obviously, these bullet points do not represent the entire technology but they can give you a
good high level overview of what exists and what are the main components. This is the device from
Meta, Meta Quest PRO. There are front cameras, depth cameras in order to understand proximity
and gesture tracking. So you can put your hands in front of the cameras and by moving your fingers
you can see your virtual hands moving with great accuracy. There are also high tracking sensors
which are important especially when you do like social interaction with another person and the
other person can see your eyes or by optimizing the graphics and the frame rate according to the
place that you focus your eyes. There are devices called IMUs, Inertial Measurement Unit, accelerometers,
orientation and other gravitational forces and include accelerometers, gyroscopes and magnetometers.
They are used for you to accurately measure position of your hands or the rotation of your head.
Time of Flight sensors in order to measure distance, imagine you are entering a room and this physical
room that you enter can automatically become a virtual room in your virtual reality experience.
So you need depth cameras and Time of Flight sensors to do that. There are processors,
speakers, battery (obviously) and controllers.
The controllers of the VR are quite interesting to observe because if you think of the user experience
before VR you have controllers of game consoles. You use both your hands in one device but now
in virtual reality you can actually physically move so you cannot have one controller for both
of your hands. So the UX of every company out there was responsible to convert the controllers
that we had in the gaming consoles to two separate controllers with additional sensory device on
top of them accelerometer, gyroscopes in order to simulate the movement of our hands and provide
the required user experience for us to interact and play our virtual reality games.
And this graph shows how these two different companies created two different joysticks coming
from the concept of the console joystick. But when you don't have a controller
you need to have gestures in order for the device to understand where are your hands
and what are the motions of your fingers. And this is achieved both in AR devices and in the
AMV devices with the cameras that are in front of the headset. So these cameras have the required,
let's say, algorithms that power the cameras to understand the motion of the fingers
and according to the different type of motions that you do, you can interact with a virtual
environment. So for example when you do in a Microsoft HoloLens this movement which is the
movement called Bloom, the main menu appears. If you want to click you need to do this with your
finger not this, this is the gesture. If you want to drag and drop something you click it and you
drop it. So there are different types of gestures in order to allow you to interact with a virtual
environment in an AR or in a VR equipment. What type of delivery mechanism and technologies do we
have in order to experience AR VR? There are numerous let's go each one of them. Where they are
where they are is a virtual reality experience but it is deployed on the browser of your laptop
or your computer or wherever you want. Obviously you don't have all the nice features of VR
it feels like you are playing a 3D game okay but it might be the right solution according to the
program and the application area. So for example if you want to create a training program for students
or people to get familiar with a space a WebVR might be the right place to deploy your
experience because it's already available anywhere everybody has a browser, okay. It's very cheap,
you don't need to buy any new equipment. On the other hand if you need to create a more immersive
environment like a game or a more immersive training then you need a full VR experience
and deploy your experience on the VR headset. Obviously the WebVR is cheap also the VR cardboard
is cheap because the device the cardboard is almost for free it's already very cheap
and you only need the smartphone. On the AR space you can deploy your AR application on a smartphone
I'm sure that you all played Pokemon Go or I'm sure you're playing now AR games on your smartphone
they can be deployed on smart glasses either Google glasses that, I'm not sure if who of you
experienced that in the past, I tried them back in 2015. We now have car manufacturers having smart
glasses in front of the wheel of the car in order to inform the driver about, you know, navigation
or specific alerts and obviously we have AR headsets that you can deploy your AR applications like
HoloLens, Magic Leap etc. So according to the application the level of immersion and the use
case you have a plethora of delivery mechanisms delivery technologies for your AR VR experiences.
Something that is interesting also in the AR VR space is Haptics. So in order to make the experience
even more immersive we now have gloves that can have sensory devices in order to improve the
overall experience, so vibration, so your finger vibration, on a suit that you are wearing.
So imagine that you are playing let's say a game that you are giving a punch to the enemy and you
can feel the punch on your chest or you are in a forest and you can see a bird flying or
the bird is landing on your finger and you can feel it. So all this extra level of immersion
is delivered to you through extra hardware equipment that are obviously in a tactile
manner communicating really fast with the hair with a VR headset and you need to have some extra
hardware to experience it. There are other levels of immersion in order to have a better VR experience
this is an example one of the most famous is the Treadmill that allows you to run in VR this was
one of the main drawbacks of VR compared to AR in AR you can move your hands but also you can
move your body. In virtual reality you can move you cannot move your body you only have the joy
stick in order to navigate in the environment with treadmills (VR treadmills) you are on top
of a treadmill, you can run, you can do all the physical movements and these are translated as
locomotion in the VR space. We also have Flying Simulators we can have Theme Parks etc. in order
to increase the level of immersion. I'm not gonna spend too much time on two of the most,
let's say, commonly used engines to create AR VR experiences, you know, I'm sure that you are
all familiar with Unity and Unreal, both of them are engines that allows you to create a VR and
AR experience. As a very general rule of thumb, Unreal Engine is most widely used in games,
it has very good graphics, where as Unity has a lot of libraries that can help you if you want to
create more like trainings and other type of VR experiences but obviously this is not
a hard rule, it is quite commonly met out there. So if you are a startup and you want to create, let's
say, not a higher resolution graphic VR experience but more related to training and learning and
development, Unity might be the right tool because there are a lot of languages out there and libraries.
If you want to create a very realistic game then probably Unreal Engine might be the right platform
for you but obviously depends on, you know, the use case and the application. So now let's move to
some of the development challenges that we face nowadays. How do you develop a VR experience.
Most probably you are aware of agile development process, so let me give you some
of the lessons I personally learned in my startup career.
Creating a VR or an AR training or game, let's say experience, is time consuming and quite a
difficult thing to do. This is because there is a plethora of platforms, there is a plethora of
devices you can use, there is a plethora of different type of 3D objects and environments that you can
create and in most of the cases that I have seen is that you don't know what really the customer
wants, the user. So one of the most commonly we help you on your development of the experience is what
we call Agile development process that helps you understand what the user needs and what are the
challenges, explore the different alternatives you have, experiment and then materialize. And this is
done through interative cycles with small cross-functional teams so instead of going and creating a
monolithic game or experience that nobody's gonna use, try to make it adaptive and iterative.
So we wanted to create a virtual reality training for first responders and this virtual reality
training should be delivered in virtual reality, Oculus Quest and also a AR experience using
Microsoft HoloLens. So we need to develop two products but we didn't even know what the user
and the customer wanted. So for example and design thinking principles what we did is we
created an MVP with 360 images or 360 videos and we use InstaVR as a platform to let users
experience it. Very easy to do and to tell you the truth, the budget that you need is less than 500
dollars, let's say, or euros, or zero amount of money, you'd go in the place, you take 360 images and
then you program in an InstaVR and experience. You give it to the users and you receive a feedback,
I would like this feature, I don't like that, I would like to add another feature. So with this
iterative process, you know, we started creating progressively experiences in VR and AR, we published
that on a VR store and then we were able to scale it to a large number of users. I definitely want
to give you this advice that, don't go and develop something big, focus on an MVP, MVP stands for a
Minimum Viable Product, and follow agile principles, iterative work in order to, you know, step by step
improve your model and your experience. If you want to see all the development stages of an AR and VR
experience, you know, the most basic steps are the following. Create the theory environment,
design/create the instructional design, let's say the series game and the experience behind it, create
some special effects and immersion levels, you know, some special gestures, define what are going to be
the analytics that you need to keep track in order to understand user engagement, package all of these
in an application file and publish it on a marketplace. On every step there are a lot of questions that
you need to answer, these are just a small tiny portion of the actual questions that exist out there
but it gives you, let's say, an indication of what are the steps involved, what are the main, let's say,
obstacles that you need to bypass. In reality, it's 10x of what you see here.
Another development challenge is the avatar; who owns my avatar, what type of diversity we need to
give to people. It needs to be customizable I want to have my face on the avatar, some other people
want to be anonymized or wear sunglasses, so giving, creating an avatar is not a simple thing in
modern AR and VR experiences and it is something that is gonna, we are gonna see a lot of innovation
in the near future. Another challenge that we met in mainly in virtual reality is what we call motion
sickness. It's an important drawback because, I personally experience it sometimes, because it
doesn't let you experience the entire virtual reality game. After five minutes or ten minutes you
might feel motion sickness and you might quit, abandon the game. It's quite interesting to see
how motion sickness is created. So we have two sensors that detect motion in our body, one is our
ear and the other is our eye. Inside our ear there are some tiny tiny tiny sensors that understand, you
know, motion. Think of it like an accelerometer inside our ear okay. And obviously the eye detects
motion through the visual. When we experience VR, what is happening is that the brain that is
connected to our ear and our eye receives two signals that are opposite. The ear does not feel
any type of motion and it sends a no signal motion to the brain, whereas the eye can see the motion
because I can see, you know, motion in the virtual reality environment, cars are passing by, you know,
I'm flying a plane and the brain does not know which of these two sensors to trust more because it
has an equal trust to both of them; it trusts the ear, it trusts the eye. So in order to defend itself
the brain sends a sickness signal to our stomach and this forces us to stop whatever we do that creates
motion sickness to us. So recent trends now in head-mounted devices in VR headsets is that
they're going to include a magnetic sensor, actually an actuator on the ear side in order to synchronize
the motion that the eye detects, with an actuator on our ear in order to also detect a fake motion.
So motion sickness is something that is not going to happen from now on in many of the new VR headsets.
Another cool challenge that is happening is what we call teleportation. It's not like actual
teleportation but it's very similar to what we have seen in Star Wars movie. The idea for teleportation
is for me to be able to see a 3D full-scale avatar of the person that I'm communicating with.
So imagine that I'm in my room, yeah, you are in your room and you can see my 3D body walking inside
your room and delivering you this lecture. There are different types of technologies to do that
either by transferring a large number of pixels in this 3D environment or by creating a 3D object
and setting, putting a skin of how I look on top of it. Obviously there are different types of
cameras and hardware equipment that need to be created. I'm not an expert about that but I definitely
know that there are a lot of development challenges in that teleportation space. And before I close,
another challenge is how we interact with all these huge networks of internet of things that are out
there. Imagine that by 2025, or it might already be happening, you know non-human centric data,
data that are coming from internet of things are gonna be larger than human centric data,
data that the real human is creating. And one of the key problems that we face now is, how can I
interact with all this big data? We have a dashboard on my tablet or my smartphone but it's too small.
We have NLP natural language processing algorithms that I can speak to as smart device and have
access to this big data, or I can interact with smart devices like this thermostat and I can see the
data. But one of the most expected breakthroughs that is gonna appear is through the use of AR and VR.
I'm gonna be able to visualize big data on the physical world by connecting AR applications with
internet of things networks. So accessibility to data is gonna be an immersive experience to us
instead of having, let's say, a flat screen in front of us.
That's all on my side and obviously there is a list of conclusions that you can see in your slide
and George, we can welcome questions and I hope you found the lecture interesting. Thank you very
much. Thank you very much George thank you very much Chris this was a really packed session but
at least for me because I watched it more as a student because I'm not an expert in these things
I found it very fascinating. Just to let everyone know that this is quite a long presentation you
might have noticed that it's more than 70 slides so we're gonna mint it and have it available for
you to claim as an NFT as soon as possible and obviously both Chris and George will be available
for questions offline as well on Viber or Twitter. So we have a couple of minutes, I think
we can take a couple of questions. One question is, okay, people are naturally confused with acronyms
so George you started by trying to explain the differences between AR VR and MR.
A student is asking about XR which is Extended Reality.
I guess I know the answer to that question but can you clarify the difference on how XR fits with
the other acronyms and what everything is? Yeah, acronyms and abbreviations are always a big
issue and sometimes there is an overlap. Extended reality, mixed reality, AR and VR. I think that we are
gonna have more dominant, let's say, names focusing on VR everything that has to do without any
type of interaction in the physical world so I'm totally isolated in a virtual experience and then
XR, I think, in my personal opinion, you know, that will include all the rest. But this is something
that, you know, we're gonna see different names probably coming in the near future. So me personally,
I use VR AR some other people are using XR so it's up to you to use the name that you prefer.
Chris, any comments on that?
Chris might be able to provide. Well I use VR for everything. (laughter)
Okay case is boy yeah. I personally like to keep it simple and I just say well it's,
I think virtual reality is good enough if it's going to blend with, you know, but who's to say what
real reality is anyway, So keep it simple. Virtual reality is fine, XR I have read papers
which say just treat the X as a variable, just a placeholder. So in the X you can
put whatever, augmented, you can put the glasses you can put immersive and whatever comes next, you
know. So I'd rather not confuse people. I'd rather not confuse people and I would either just go
with VR or go with what George just said, AR and VR are fine I mean it's good enough.
Yeah I agree, I mean the "keep it simple" I think principle applies here. I'm probably older than
everyone around here and I've been around in the early days of the internet, the early days of
mobile, the early days of crypto and I've seen how acronyms are used and abused by consultants
and vendors as they try to position their products and differentiate themselves from
competition. So sometimes we get, you know, bombarded with different acronyms that mostly mean, if not
completely the same, very similar things and tends to be confusing so yeah, I'm all in for simplicity.
And, you know, as it happened with the internet, the things that have real value that the names
will stick. Others like, you know, the intranets we have been discussing back then in the 90s or
everything will just disappear from the foreground. Okay, another question
Both of you, especially George I think, have mentioned a number of devices that are
commercially available, announced, or in the process of being developed, and okay, I guess most of us
know about Oculus and stuff like that, but you mentioned things like haptic interfaces or treadmills
or this actuator in the year that will alleviate the symptoms of motion sickness. Can you give us,
either of you, like a time horizon of when these things would hit the commercial market, when
we would see them. I mean, are they available in the market now? Are we expecting them in 2023 or is
it like a five year horizon thing? Christos, should I go first? Yes go. Okay, the technology is already
here and obviously there is a supply and demand, you know, driver here so the more the demand is
gonna grow from the end users, the technology will accelerate. We have seen cases where the
technology accelerated so fast but the user adoption was not there and this, from the business
perspective, is, you know, sometimes not very sustainable, but for the moment technology is here to deliver,
you know, acceptable levels of immersion and experience so it can be engaging for the end user. So gloves
that can improve, let's say, haptic VR, okay, or treadmills and they already exist. They might be hard to find
because there is no mass production there are no games, you know, still yet out there to let you
experience, you know, with the use of a haptic glove, you know, the level of immersion that you want so
there is. The technology is here, the demand is coming so we are gonna see like a step-by-step
growth. My personal sense is that, you know, 2023 we are gonna see much more evolution compared to 22
and more penetration of this type of technologies in our experiences.
Yeah so yeah I tend to agree but if you if you ask me which one of the AR or VR is going to hit
a use case or a use scenario quicker I think it's going to be augmented reality because of the,
you know, not everybody as George mentioned some people really do not like the sense of isolation
that you get from immersive tech, you know, and I've been working with the tech for quite a long time
and yeah, you don't find me putting on my headset I'd like to kick back and watch a nice flat screen,
but imagine this, you've got augmented reality glasses and you kick back and you turn your room
into a living cinema. Now this is a use case, it flows with larger and larger TV screens for example
that everywhere every Christmas you're buying a bigger TV set, well at some point you don't need
to buy a TV set, okay you can have a shared experience with your family wearing a pair of glasses that
you can take with you from one room to the next. There could be a market for this employment
entertainment in terms of home entertainment. For business uses, absolutely, you know everything is
there currently, it will get better. Haptics has fallen out it's fallen out a little bit because,
you know, it's clunky, the technology is too clunky to be viable at the moment. I remember the
the old haptic device was called the Phantom if anyone would like to go back to the 1990s the late
1990s early 2000s. So this was a little robot, you put your finger into it and you could feel
you could feel stuff and play around with, you know, elastic effects etc.
Now we have haptic gloves but I think, yeah, this will be a while taking off, I think it's a slow
process. Let me let me take you a little bit further in the future then because I have a question that
I really like from one of our students and the question is, what about brain computer interfaces?
How far away is that do you think? I guess eventually we will tap directly into the optical part of the
brain and bypass AR spectacles or goggles. Do you have any views on this? That's already here.
Is it? Okay yeah yeah that's that's already here so okay. There is a principle, I have one just here
in fact, it's a 32 channel BCI with a principal shell so you could you get the 3d model you
could you can print it and you get a pack from a OpenBCI is the name of the company. It was a kick
starter from a few years ago and the perfect use case is as a motion motion device.
Yeah, so you can train, it picks up the skin currents on the head on the on the scalp you can train
it on the on the motor the sensors of the head with repetitive movements and then you can associate
those movements with movement in a virtual environment. And people have been doing that for
a few years now. Yeah, and there's also it's used for paraplegics so if you have a case where you use
it for a paraplegic in a wheelchair to move their wheelchair, and you can for sure take this straight
away and put it into a virtual environment. Fascinating I didn't know we were so advanced in BCI. Next
time I am in your lab, you need to show me this. George, any views on that? I think that
one of the enablers of something really really interesting is gonna be 5G or you know 6G networks
that will allow real-time 360 video transfer. And I remember that a couple of years ago when I
broke my leg, all right, I said I would pay anything if I could click on a person on a map, let's say
on the top of a mountain that he or she is snowboarding with a 360 camera on her head, okay, and I can be
with my broken leg in my sofa of my home in Greece and wear my VR headset and you have the assert
experience real-time, 360 high-definition video and the person is, you know, doing a nice downhill
run for me while I can't, So I think that when we are gonna have content creators real-time,
high-definition, 360 video be able to be transferred and headsets that can allow us to, you know, consume
this type of content, there's gonna be a really really interesting application area,
We are a little bit some years behind because the network and the speed is not already there
in many cases, in some you know denser burn environments it is, but I think this is gonna be
fascinating and obviously we need the 360 cameras in our smartphones.
Awesome, very very interesting. One student is asking, what was the name of the company that you
mentioned, Chris, I think it was OpenBCI, you said? (you're muted, you're muted I think)
OpenBCI, BCI for brain computer interface great, yeah okay.
Another question is, if you were to pick like one or the top difficulty either technical or
adoption related or regulatory or whatever you want to make these things, you know, commercially
viable and adopted by en masse, what do you think that the biggest obstacle or obstacles are at the
moment? Is it that we are, you know, missing technological elements, is it that we miss
applications, that we miss education, what is it that hasn't allowed augmented or virtual
reality to reach their full potential?
I think there are different use cases for each of them.
I believe and, well, I saw a breakdown of 60 to 40 on Meta's expenditure vis-a-vis the
augmented reality expenditure versus virtual reality and I don't think it's the case that
Meta, for example, is building this closed world in the evening you know they expect this closed
world this is not going to be the use case it's not it's not going to be the, you know, what's going
to break it for for this tech. I think this tech will gradually become pervasive
through everything that we do, it's a slow process and I don't think that there is going to be a
massive jump, in my personal opinion. I think what will happen is that we will just, I saw a
visualization of this itself where somebody walks out of their living room and they are bombarded
with augmentation, right, so in the streets where they walk there's information, this information
regarding the street name there's advertising, once the advertisers get in there, oh believe me
things will take off people. I also saw in our cafeteria today a pair of glasses which,
you know, stop the glare from a screen and I think the wearing of glasses like this with a form
factor like this with computer graphics augmented is going to be the clincher. People will start
wearing these, they'll feel comfortable wearing them and we will have, everywhere we go there will be data,
it will be data rich and this will be the metaverse, in my opinion. Very interesting, very
interesting definition of the metaverse as well. George, any final thoughts on this because I
think this is this is our last question for today. I agree that, you know, one of the key obstacles,
you know, to wear the glasses is because now we send all the processing power on top of the glass.
What we need is a glass that it is like the one that we wear for our sun or, you know, to improve
our sight. So having migrating all the processing power to another device, probably our smartphone,
our, you know, our smart watch and having the acceptable level of graphics and experience
delivered in a normal glass will open new horizons in the adoption of the services and then,
you know, wherever you are you can see augmented information everywhere, Advertising, you know, real
time information navigation, everything can make your life much easier. Fantastic we're gonna say
we're gonna see huge changes in the coming years and I agree with Chris that they would happen gradually
and then and then suddenly maybe when advertisers pick up on these and we have a sudden influx of
of applications all around us and then we will be chasing the applications instead of them chasing us.
Anyway, thank you very much this was a very fascinating session thank you for being here thank you for
sharing your expertise with us and looking forward to seeing you again in one of our future courses.
Thank you very much everyone, we'll see you next week with week 10. bye