This video is pretty mind blowing. We’re getting closer and closer to real world Holodecks 
Procedural Content Generation (PCG) allows the user to build their own world in game, real time, using a Chat GPT interface. Prompts such as “I want to be on a calm beach” are instantly converted to imagery. The user is placed in the environment that they requested. “Build while you play.”
Moreover, it will be fascinating to observe gaming with both procedurally generated environments and AI NPCs (non-player characters) in shaping an even more immersive gaming experience. A number of AI companies/startups are creating responsive NPC systems that have the capacity to respond in real-time to player queries, as shown in this demo: NVIDIA’s NPC system.
In contrast to traditional NPC interactions, where characters repeat pre-set lines, this approach is adding depth and complexity to storytelling within games. By synthesizing both generated environments and intelligent NPCs, we’re approaching a new era where gaming not only entertains but also offers complex narratives and enhanced user engagement and can potentially have a multitude of different pathways being generated as they are played.
The convergence of procedural environments and AI NPCs in gaming is certainly ‘next level’.
So if I understand correctly, instead of playing the same game over and over, with the same ‘levels’ one after another, the game will generate environments in real time, and be a different experience on each play through. One can imagine that AI NPCs would learn a user’s individual playing characteristics and adapt the game to respond.
Sounds amazing but I wonder if the AI wouldn’t get hung up from time to time and make illogical or out-of-context choices.
Yes, absolutely, every game would be different.
And I think the AI NPCs will be a little buggy at first (sometimes delivering illogical, out-of-context choices), but like everything, those glitches will get ironed out in time. It’s more likely to happen if the player ‘tests’ the AI by asking questions out of context to the game. For example, asking a bartender in a gaming world that has no concept of ‘earth’ what the time in NY is. They may just say ‘I’m not sure if I know where that is, I’ve never heard of a New York’, or it may say something that makes even less sense like “I’m pretty sure it’s the same time as here”.
There are already ‘sandbox’ games like ‘No Man’s Sky’ that have been around for a few years. It’s an exploration game, where you fly through galaxies (that are procedurally generated) and when you choose a random planet, it is randomly generated and filled with its own unique flora, fauna, and minerals. But they’ve come a long way in terms of photorealism since then, here’s where Unreal Engine is at with their procedural environs.
Procedurally generated city:
I think the recent progress in LLMs (large language models) will allow game designers to combine procedural graphics and AI-driven NPCs with role-playing games (RPGs). So imagine being part of an ongoing, immersive story, where there is a crafted storyline, but everything around you is built in real-time.
My prediction (or maybe it’s a bit of ‘wishcasting’) is that over the next few years, RPGs will evolve to have an overarching storyline filled with adventures and mysteries, but you’ll have an infinite number of different paths to explore, much like choosing to go by highway, off-road or by plane, or by foot or anything you can imagine metaphorically speaking.
Picture yourself as a character on a quest, like a lone character searching for a long-lost sibling. You’ll meet other characters who will provide clues and engage in meaningful conversations, driven by natural language processing. You won’t be limited to typing or selecting pre-written questions; you’ll be able to speak through your microphone and the game (and game characters) will understand you and respond in kind.
Imagine landing on a procedurally generated planet, travelling to a procedurally generated city and walking into a virtual bar filled with procedurally generated elements. Everything from the door handle to the the dim lighting to the strangely organic furniture will be created as you enter, with characters that are also generated to fit the story. The conversation won’t be scripted; it will flow naturally, guided by AI.
This means every game will be a unique experience. There will be unlimited scenarios playing out in endless places, all guided by a framework that keeps you on track with the storyline.
That’s where I think gaming is headed. Ever-expanding immersive universes wrapped around great story-telling, shaped in real-time by players’ choices. 
Thanks @Jo-CKC-Studio for the informative post! Helped a lot to understand where things stand. I hadn’t realized that this tech was already out in the world and being used.
As a composer of music for visual media, I wonder how music could interact with spontaneously generated images. Normally, a human composer can experience the visuals themselves, then react to those visuals, translating the emotions from what they see into music. Additionally, the visual media is normally locked, meaning the music can synchronize in such a way as to respond to certain dramatic points.
It is possible to imagine that one day the music score will also be spontaneously created by AI, to react to various forms of input from the player. But I’m not looking for another line of work yet. Composing music isn’t particularly difficult. However doing it well enough to engage the heart of the listener is a different matter all together.
No worries.
There are a lot of AI tools being released constantly, and some of them do generate voice and music.
For example this open source Google research project has taken text descriptions and also (further down the page) famous art pieces and generated music in response to them. Not sure how much the music ‘matches’ the image, but I these are the experiments that I’m seeing.
I’ve even seen something similar to what you have described - called Soundmatch, where you can upload a video and the application creates and recommends a number of AI generated music backgrounds. Watching their demo, I’m not sure how well ‘matched’ these soundscapes are - and how well they are actually ‘syncing’ with moments in the provided video, it might be worth experimenting with though if you are curious. Like all of these emerging AI tools, these are the first drafts and if they find an audience, they will improve.
I do agree that translating emotions into music to capture the imagination of a listener is not that simple. AI can’t code human emotion and creativity itself, but it will be exciting to see what new tools might be added to the process of creation. What will be interesting is tools that artists can leverage, that provide a lot of customization and choices on a very granular level, so as to tweak and shift levers until the soundscape matches what a person is imagining. One where the composer needs to understand the fundamentals of music and sound, but the process of creation may evolve. A new method of composing, perhaps.
Thanks for posting that info and demo about Soundmatch. The function it provides is currently an actual full-time job for hundreds if not thousands of people. The way it has always worked is that when a client (an ad agency for example) wants to find a piece of music for a project, they put out a call to several music publishers to pitch pieces of music that they represent.
Then the client goes through the submitted music to try to find something to license. This process involves a lot of people who have well-paid full time jobs.
The Soundmatch does this literally in seconds. I’m a bit speechless.
The only caveat is what music the AI actually has access to, and whether or not it is of a level of quality that matches what traditional publishers have to offer.
I doubt whether the quality of music accessible to the AI can currently match what traditional publishers offer. At this point in time.
It is important to recognise that we are essentially experiencing the first iteration of these AI tools. While they may still make errors and sometimes lack factual accuracy at this moment, they are in a constant state of evolution through retraining. Mistakes are being corrected, and guidelines are being established to mitigate ‘hallucinations,’ or the creation of incorrect information. Consequently, the content quality is rapidly improving.
It wouldn’t surprise me if the quality of AI-generated music undergoes such rapid advancements that this discussion looks very dated in just a year or two.
