LegoGPT: The AI That Understands Structure

When you look at a pile of Lego bricks, our inner child doesn’t see a mess of colors and plastic, but possibilities: trains, bookshelves, an astronaut — imagination comes easy. However, the real test comes when you assemble the idea. Oftentimes, it doesn’t come exactly as in your mind, not to mention how often it won’t stand upright. Turns out, AI can also look at Lego bricks, but in a different and perhaps better way. A new model, named LegoGPT, can plan an exact build and calculate if it will be physically stable.

Another brick

While previous AI generators certainly included Lego styles, their results were just for the visual appeal and had no real thought into the physics or feasibility of the design. Designed by researchers at Carnegie Mellon University, LegoGPT differentiates itself by not training the model just on images of builds. What it means to the user is that any descriptive input they give does not come back as a mere image, but as a plan to build the idea in real life.

The researchers used the publicly available dataset, StableText2Lego, which contains over 47,000 Lego structures. Of those, over 28,000 are 3D objects paired up with detailed descriptions. This ensures the model not only understands what something looks like, but also how it is made. LegoGPT predicts the next brick in a 3D space, supported by a physics-aware rollback in case the build starts to look like an unstable one. In practice, it is not very different from the steps a human would take.

Real-world practice

The team tested the generated builds in the real world, too. Both humans and robotic arms followed the instructions of LegoGPT, and the results stood, demonstrating the AI’s functionality. A demo is already available for the public, so you too can try out some Lego ideas. Still, the current dataset only includes 21 categories of objects, so anything outside of those is not guaranteed to work.

Also interesting: Unveiling the Power of AI for Freelancers and Entrepreneurs

LegoGPT itself is already somewhat scalable, with researchers mentioning broadening the dataset and introducing new brick variations. However, the implications of an AI that can plan something in a 3D space while taking physics into account are vast. Any industry where physical constraints affect creativity can be affected, from smaller things like prototyping to something as big as architecture. If we think back at how robotic arms built these same Lego plans successfully, a future where you just describe your home and everything is automatically built does not look like science fiction anymore.

YouTube: This AI Creates Realistic Lego Structure – How to Use LegoGPT

Photo Credit: The feature image is symbolic and has been taken by Sebastien Bonneval.
Sources: Ava Pun, et al. (arXiv, ResearchGate) / LegoGPT