I've been exploring a related concept in a Telegram game. The player can basically create a choose your own adventure-type world by describing a set of interlocking scenes and the segues (buttons/commands) between them.
I'm using Disco Diffusion for the rendering instead of Dall-e 2 but the limitations are similar. It can be frustrating to get consistent results or impose the kind of order on them that is required to make something feel like a real game. Or maybe I'm not thinking hard enough..
One cool thing about Stable Diffusion (the current leading competitor to DALLE) is that you can create images in a similar style by configuring the seed number. Different prompts with the same seed number can be used to create different images with similar visual styles, character design, etc.
For example, this person used SD to generate portraits of the same woman at different ages, from an infant to an 80 year old:
For consistency across generations you'll need something like https://arxiv.org/abs/2208.01618 to carry the concepts across, hopefully nvidia releases the source code soon, or someone that understands the internals of latent diffusion better can implement an open source version.
I'm using Disco Diffusion for the rendering instead of Dall-e 2 but the limitations are similar. It can be frustrating to get consistent results or impose the kind of order on them that is required to make something feel like a real game. Or maybe I'm not thinking hard enough..
If this kinda thing (AI-generated game worlds) interests you, try it here: https://t.me/xiadreamland_bot