World Labs was building an LLM 3d world generation system utilizing gaussian splat technology.
Provided expert developer feedback on the tools and output for a production environment.
Created content with narratives that were built exclusively with their technology stack as proof of concept.
Tested integration into Unity and provide technical user feedback.
Providing developer feedback
This work was to establish what an efficient workflow would look like and how well the tooling covered the needs of a developer building product. This feedback was gathered while doing initial testing and continued throughout the content creation and technical integration. I created my own structure to test and execute within.
Phase 1: Out of the box testing.
What are the current capabilities of the tooling and what does it take to get an acceptable result?
Method:
This test used a standard pattern of prompt check result, iterate, repeat until getting the desired effect
Findings:
When targeting specifically themed or styled environments (Fantasy Castles, Catacombs, Sewers) the tooling performed really well, but when getting outside of these safe areas (Highway going through a forest, ocean floor with sunken ship) the generation started to struggle and iterations became more frequent with sometimes forcing me to abandon an idea.
Phase 2: Using prompt helpers
Does the experience improve if I generate prompts to get more detail?
Internal Tools Only
Started with a simple prompt to render 2d references in the tool. This would generate multiple options with a more detailed description of each. Instead of using the image, I would take the prompt for an image that was close and customize it and run it through the generator again.
Findings:
This method was more successful than Phase 1 with less unfinished ideas.
External Tools Support
Use imagery from online or generated it and ran it through ChatGPT with a prompt asking it to describe the image. Then taking that description, I would use it as a prompt through the World Labs tooling.
Findings:
Similar results to internal tooling, but found this worked better for building a world that had existing imagery available or style.
Phase 3:
Will external GenAI images produce better results than World Labs prompts?
Method:
Use ComfyUI, Gemini, ChatGPT to create images and import them into the World Labs tooling to generate worlds, completely bypassing the LLM portion of the pipeline.
Findings:
This did not work as well as using the internal LLM to generate useable worlds. The generation is very literal to the image output and the external creation did not seem to sync as well with the 3d generation tooling thus resulting in lower quality. This result could probably be improved with more time, resources, and newer GenAI models and would be worth re-testing.
Narrative Scene Creation
Purpose:
Use the tooling to build expansive worlds that tell a story and creatively stress test the capabilities.
Method:
I wanted to create 3 different environments with different subject matters with different tests for the capabilities to get a broad understanding to where the limits of this technology were.
Environment 1: The Goblin Caves
An adventurer exploring the forest comes across a cave with signs that goblins are using it as a base. They move into the cave and find it to be expansive with many forks and a waterfall feeding a stream that is carving out the cave over time. There are signs of unnatural alterations like doors, paths, and stairs. Following one of these paths leads to some rickety rope bridges that connect a living area that has a cliffside exit and a goblin throne room.
Process:
This was intended as being the most complicated environment but using a friendlier subject matter and style for the tooling. With the organic and closed environment, I was able to layer and blend a bit more seamlessly with 20+ generated and merged splats to workout the complications of the production process. My final pipeline was starting with the more detailed locations (Forest, Waterfall, Throne Room, Living Area) then building the connective areas long the path I had mapped out. I added a new way to generate content by taking renders from the world labs environment, painting out an area (cave mouth, cavern, etc.) and using Adobe firefly to generate an extended environment inside of the existing image. I would then take this content and run it through World Labs 3d generator again.
Environment 2: The Breach
A space explorer comes across the remnants of what appears to be the aftermath of a space battle. Seeing one derelict ship the explorer comes in for a closer inspection to assess the situation. They exit move throughout their ship to get to the airlock and enter the vacuum of space. While floating they see the ship framed against a not too distant planet and a nebula. The damage appears to be a hull breach with some debris around the back of the ship. Entering through the hole, damage can be seen throughout the living quarters and trailing to the hangar. Inside the hangar an alien ship can be seen making an escape.
Thought Process:
This was built to test the use of scale creatively. How well can the visuals go from small interior to super expansive exterior and into a new environment that is a multi-room small exterior again. For the style I tried something that was a bit outside of the easy wins with hard surface cell shaded interiors, expansive exterior with a prop of a ship that needed to be detailed enough to go inside of. The exterior ship with the hole was particularly challenging as the it was not a strength of the GenAI gaussian splat system.
Environment 3: The Secret Laboratory
A nosy neighbor is walking through neighborhood and notices something different bout this house, a ladder going underground. The driveway and garage look completely normal in as they walk through, but where does this ladder go? Climbing down the ladder leads to a very sterile looking hallway with a keypad locked door at the end. With curiosity driving them forward they find a laboratory with many tubes and technology lining the wall supporting what appears to be a portal. While investigating, the portal turns on and sucks them through a wormhole to an alien world. What is this crazy sphere, and how will they get home?
Thought Process:
I was trying to test a flow of real to surreal and keeping a more realistic style with real life subject matter. By starting in the real world with identifiable items it was a test to the LLM to see how well it could hold up and transition. Vertical design ended up being a fairly major challenge as the system would build with a priority on a horizontal plane. The ladder ended up needing to be built flat and then rotated into space.
Unity Technical Test
This was testing the import and render process within Unity. With a limited amount of time and budget, I chose a basic flight simulator in a futuristic city environment. Some features were a full flight control system and a button press to load different environment on the fly.
Findings:
The gaussian splats would drop frames if more than one was loaded t a time due to memory consumption.
The quality of the environment would fall apart when moving through it. Optimizations would be needed for some form of LOD loading.
The export/import process was successful, but more advanced features for optimization would be required for improved performance and functionality.