Generating Physically Stable and Buildable LEGO Designs from Text

2025-05-09 00:55:00
avalovelace1.github.io

LegoGPT: Generating Physically Stable and Buildable LEGO Designs from Text

Carnegie Mellon University

* denotes equal contribution

LegoGPT generates a LEGO structure from a user-provided text prompt in an end-to-end manner. Notably, our generated LEGO structure is physically stable and buildable.

Abstract

We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models.

StableText2Lego Dataset

(a) From a ShapeNetCore mesh, we generate a LEGO design by voxelizing it onto a $20\times 20 \times 20$ grid and applying legolization to determine the brick layout. (b) We augment each shape with multiple structural variations by randomizing the brick layout while preserving the overall shape. (c) Stability analysis is performed on each variation to filter out physically unstable designs. (d) To obtain the corresponding captions for each shape, we render the LEGO design from 24 different viewpoints and use GPT-4o to generate detailed geometric descriptions. (e) Data samples from 5 categories in our StableText2Lego dataset.

LegoGPT Pipeline

(a) Our system tokenizes a LEGO design into a sequence of text tokens, ordered in a raster-scan manner from bottom to top. (b) We create an instruction dataset pairing brick sequences with descriptions to fine-tune LLaMA-3.2-Instruct-1B. (c) At inference time, LegoGPT generates LEGO designs incrementally by predicting one brick at a time given a text prompt. For each generated brick, we perform validity checks to ensure it is well-formatted, exists in our brick library, and does not collide with existing bricks. After completing the design, we verify its physical stability. If the structure is unstable, we roll back to the stable state by removing all the unstable bricks and their subsequent ones and resume generation from that point.

Step by step generation of LEGO structures from text

“A streamlined vessel with a long, narrow hull”

“A bookshelf with horizontal tiers”

“A backless bench with armrest”

Automated assembly of generated LEGO structures using robots (8x speed)

“A streamlined vessel with a long, narrow hull […]”

“An asymmetrical six-string guitar […]”

Generated Textured LEGO Models

“Rustic stone bench with moss growth […]”

“Hot rod with flame paintwork […]”

“Rustic farmhouse chair built from reclaimed wood […]”

“Live edge walnut table […]”

“Comfortable lounge chair wrapped in Japanese shibori fabric […]”

“Cyberpunk holographic material with neon purple and blue gradients […]”

“Rustic farmhouse armchair built from reclaimed wood […]”

“Vintage floral tapestry with deep reds and golds […]”

“Gothic cathedral bookshelf with arch details, medieval style […]”

“Japanese sliding bookcase with shoji screens, traditional design […]”

“Victorian library shelving with carved moldings […]”

Generated Colored LEGO Models

“Parlor guitar with ladder bracing […]”

“Electric guitar in metallic purple […]”

“Steel resonator with engraved body[…]”

“Sunburst Les Paul with amber finish […]”

Citation

@article{pun2025legogpt, title = {Generating Physically Stable and Buildable LEGO Designs from Text}, author = {Pun, Ava and Deng, Kangle and Liu, Ruixuan and Ramanan, Deva and Liu, Changliu and Zhu, Jun-Yan}, journal = {arXiv preprint arXiv:2505.05469}, year = {2025} }

Acknowledgements

We thank Minchen Li, Ken Goldberg, Nupur Kumari, Ruihan Gao, and Yihao Shi for their discussions and help. We also thank Jiaoyang Li, Philip Huang, and Shobhit Aggarwal for developing the bimanual robotic system. This work is partly supported by the Packard Foundation, Cisco Research Grant, and Amazon Faculty Award. This work is also in part supported by the Manufacturing Futures Institute, Carnegie Mellon University, through a grant from the Richard King Mellon Foundation. KD is supported by the Microsoft Research PhD Fellowship. The website template
is taken from Custom Diffusion (which was built on DreamFusion‘s project page).

Source Link

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Start your free Amazon Prime trial
today and unlock unlimited streaming and more!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!

BITCOIN

Bitcoin Logo