Top > Blender > LLM-Based 3D Modeling in Blender: Trying Out MeshGen/LLaMA-Mesh

LLM-Based 3D Modeling in Blender: Trying Out MeshGen/LLaMA-Mesh

Blender LLM AI Python Game Dev Artwork

Exploring LLaMA-Mesh, an LLM that outputs 3D model data, and its Blender add-on, MeshGen.

In addition, we'll also try using ChatGPT for 3D modeling.

Introduction

Recently, many generative AI models have been developed for creating 3D models.

Looking for something that can be easily tested locally, I found "LLaMA-Mesh," a model recently released by NVIDIA Labs.

A Blender add-on for LLaMA-Mesh, called MeshGen, has also been made available. This allows us to install and run it locally within Blender.

This time, I'll experiment with this in Blender.

Additionally, I'll attempt to generate the same 3D models using ChatGPT and compare the results.

# Environment
Blender 4.2.0
M2 MacBook Air
16GB RAM

Note: This article was translated from my original post.

Understanding LLaMA-Mesh

LLaMA-Mesh is not a generative AI model that directly outputs 3D models. Instead, it extends a text-based LLM to generate 3D data.

LLaMA-Mesh extends text-generating LLMs to output 3D data | Image from paper

The 3D model data is output as text. The model generates vertex coordinates and face positions, which external software interprets and reconstructs into a 3D model.

Vertex and face data are output as text | Image from paper

The paper provides examples of 3D models generated using this approach:

3D models generated using LLaMA-Mesh | Image from paper

The results look promising.

The paper highlights several advantages of this text-based 3D generation approach:

  • Utilizes spatial knowledge that LLMs already possess
  • Enables interactive 3D generation
  • Allows for understanding 3D models

Since the goal of this post is to test the model in Blender, I won't go deeper into the research itself. For more details, see the paper.

3D Modeling with MeshGen

MeshGen is a Blender add-on that allows running LLaMA-Mesh directly in Blender. I'll use it to generate 3D models in Blender.

Installing MeshGen in Blender

First, install MeshGen in Blender.

My environment: Blender 4.2.0 / M2 MacBook Air. Since MeshGen only supports NVIDIA/CUDA GPUs, I'll run it on the CPU.

Download the appropriate asset for your environment from the Release page. Assets are available for Windows, Mac, and Linux, with separate versions for GPU and CPU.

Download the appropriate asset from the Release page

To install the add-on:

  1. In Blender, go to "Edit" > "Preferences"
  2. Select the "Add-ons" tab and click the dropdown arrow in the top-right
  3. Click "Install from Disk" and select the downloaded zip file
  4. After installation, go to "meshgen" add-on settings and click "Download Required Models"

1. Open "Edit" > "Preferences"

2. Select the Add-ons tab and click the dropdown arrow in the top-right

3. Click Install from Disk and select the downloaded zip file

4. After installation, go to meshgen add-on settings and click Download Required Models

The installation of the MeshGen add-on and the model download are now complete.

Now, let's try using it.

How to Use MeshGen

You can use MeshGen by following these steps:

  1. Press the "N" key (or go to "View" > "Sidebar") and select "MeshGen".
  2. Click "Load Generator".
  3. Enter a prompt and click "Generate Mesh".

1. Press the "N" key (or go to "View" > "Sidebar") and select "MeshGen"

2. Click "Load Generator"

3. Enter a prompt and click "Generate Mesh"

Additionally, you can adjust the Temperature parameter, which controls the randomness of the generated result, from the "Options" section below the prompt input field.

Now, let's start generating 3D models!

Generating a 3D Model of a Desk

First, I'll generate a 3D model of a desk using the default prompt.

# Prompt:
Create a 3D obj file using the following description: a desk

Since the model is running on a CPU, it took quite some time to complete (about 2 minutes).

Here is the result:

Desk by MeshGen

Close, but not quite.

It doesn’t seem to generate as well as shown in the research paper.

Generating a 3D Model of a Chair

Next, I'll try generating a chair using the prompt example from the research paper.

# Prompt:
Create a 3D obj file using the following description: 3D rendering of a model chair

Here is the result:

Chair by MeshGen

It looks like some faces (surfaces) were not fully generated. There are parts with vertices but no faces, and even the vertex arrangement doesn’t quite resemble a chair.

Generating a 3D Model of a Bonsai

As a more challenging test, I tried generating a bonsai, which wasn't included as an example in the research paper.

# Prompt:
Create a 3D obj file using the following description: Japanese Bonsai

Result:

Bonsai by MeshGen

As expected, this was difficult.


Since the results so far have been unsatisfactory, I'll try generating the same objects using ChatGPT.

3D Modeling with ChatGPT

Instead of using MeshGen, I also tried generating 3D models with ChatGPT.

  1. Provide a prompt to ChatGPT to generate Blender Python code
  2. Copy and paste the generated Blender Python code into Blender's Scripting editor and run it as is

This time, we won’t refine the code through multiple interactions with ChatGPT.

Generating a 3D Model of a Desk

First, let's try creating a desk.

# Prompt:
Write a Blender Python code for creating 3D mesh of a desk

Here’s the result after running the generated Blender Python code:

Desk by ChatGPT

Impressive! It’s simple, but it’s definitely a desk.

Generating a 3D Model of a Chair

Next, let’s generate a chair.

# Prompt:
Write a Blender Python code for creating 3D mesh of a model chair

Here’s the result:

Chair by ChatGPT

Again, it’s clearly recognizable as a chair.

Generating a 3D Model of a Bonsai

Lastly, let’s create a bonsai.

# Prompt:
Write a Blender Python code for creating 3D mesh of a Japanese bonsai

Result:

Bonsai by ChatGPT

It might not be the most accurate representation of a bonsai, but it certainly looks like a tree.

Conclusion: ChatGPT is Strong in 3D Modeling Too

Conclusion

In this post, I tested 3D model generation using Blender with MeshGen/LLaMA-Mesh and ChatGPT. Under the conditions tested, simply using ChatGPT produced better results than MeshGen/LLaMA-Mesh.

I've seen many cases where leveraging a powerful general-purpose LLM with well-structured task design yields better results than manually trained models. This was another example of that. I might explore how far ChatGPT can go in 3D model generation in future experiments.

That said, models specifically designed for 3D generation can already produce much higher-quality 3D models and textures than what we achieved here.

The rapid progress in AI continues to amaze me.

I hope this article is helpful to someone.

References