Synthetic Reality: AI and the Metaverse

AI will play the single most important role in building, operating and governing the Open Metaverse.

Midjourney renderings of the prompt “Artificial intelligence in the Metaverse”

TL;DR

Table of Contents

Midjourney rendering of the prompt “Generative AI generating the Metaverse”

Introduction

Synthetic Reality — A Definition

Midjourney Renderings of the prompt “Synthetic Reality”.

The underlying physical laws necessary for the mathematical theory of a large part of physics and the whole of chemistry are thus completely known, and the difficulty is only that the exact application of these laws leads to equations much too complicated to be soluble.” — Paul Adrien Maurice Dirac, Quantum Physicist (Quantum Mechanics of Many Electron System. 1929.)

The Role of AI in the Open Metaverse

3D Assets and World Building

Source: NeRF project page on Matthew Tancik’s website.
Dr. Karoly Zsolnai-Feher’s Two Minute Papers on 3D Style Transfer (2019)
RigNet — Zhan Xu , Yang Zhou , Evangelos Kalogerakis , Chris Landreth , Karan Singh
Get3D model results produced by Berkeley Synthetic.
University of Texas 3D scene generation via unsupervised object synthesis.

Avatar Control & Animation

AI-Driven Physics-based Character Animation by Nvidia Research
Crafting of a crafting table “zero shot” (i.e. after pre-training only without additional fine-tuning)
Sebastian Starke’s AI4Motion.
DeepMimic demonstration video presented at SIGGRAPH 2018.

Image Generation

Results from TEXTure.
Images generated by Lensa AI — adar_gandelsman

Simulations

James Cameron’s Way of the Water used extensive fluid simulations.
“Beautiful Fluid Simulations .. In Just 40 Seconds!” Two Minute Papers
Accelerated smoke simulation by super resolution with deep learning.
Subscape Neural Physics results from Ubisoft La Forge.
Learning to simulate complex physics with graph networks.
TailorNet: Predicting Clothing in 3D… Results.
This AI Creates Beautiful Light Simulations. Two Minute Papers.

Audio and Communication

Review of MusicLM
AutoFoley results as replicated by the IEEE.
SoundStream: An end-to-end neural audio codec.
Synthesia Tutorial. Setting up a chatbot in 1 hour.

Other Multimedia

EmoG: Supporting the Sketching of Emotional Expressions for Storyboarding.
Google’s Text to Video AI Phenaki
RunwayML currently has Gen-1 in preview for selected invitees.

Marketing & Commerce

Using machine learning to detect money laundering.

Security, Privacy & Governance

Sentiment Analysis: Extracting emotion through machine learning.
Andrew Ng’s CNN course, section on facial recognition.
Descriptive World demo video.

Conclusion

Additional Resources

--

--

Generative AI Researcher | Educator | Strategist | Author | Consultant | Founder | Berkeley Synthetic, UC Berkeley, Open Metaverse Foundation @ Linux Foundation

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Matt White

Generative AI Researcher | Educator | Strategist | Author | Consultant | Founder | Berkeley Synthetic, UC Berkeley, Open Metaverse Foundation @ Linux Foundation