0:00 Hi, my name is Alexander Htet Kyaw. Today I’ll be talking about going from text to robotic assembly. It’s about making physical objects with 3D generative AI and modular robotic assembly. I’m currently a PhD researcher in electrical engineering, computer science at MIT and I previously have a background in architecture. Mainly work with the computer science and AI lab there and collaborate with the media lab and design intelligence lab.

0:50 The main challenge that I think we all share is that making physical objects isn’t always accessible. It takes significant amount of technical expertise to bring something into the real world whether you’re using your hands or a machine. So I thought to myself just like a little you know idea what if we could just simply speak objects into existence as just like a preliminary question. So imagine a scenario where you say, “I want a chair.” And within 5 minutes, a physical chair materializes right in front of you.

1:27 And this is kind of what we’re working on right now, where assemble me a table. The person can say, “assemble me a table with one leg.” And then the robot thinks for a while and assemble exactly what they asked for, which is a table with one leg out of these modular block. It can assemble a shelf with two tiers or like more abstract thing like a letter T or just a simple stool for you.

1:54 This very basic pipeline simply starts with like a text prompt from the user which gets converted into AI generated geometry with the help of a texttod model. This gets discretized into assembly components while accounting for fabrication constraints and then we generate a feasible assembly sequence for the robot arm to realize it. It’s accessible because you can say what you want and you get it. You don’t need to know how to 3D model or program a robot and it’s fast because it only takes a couple of minutes.

2:27 But the issue is that if we make something that’s super easy and also fast to make things people are going to start making a lot more and there’s waste and sustainability involved because you know material stuff are limited. So it’s important for the objects to be able to disassemble and reassemble to fit the changing needs of the user over and over again. Hence the modularity helps in this case.

2:53 And we did a lot of trial and error experiments to ensure that the objects that we assemble are feasible. And we came up with a series of heristic based fabrication constraint which is these robotic reachability. The number of physical components that you have in the assembly space that you’re working with the horizontal overhang and when it’s going to fall over the vertical stacked because these are robot the robot arm is moving these components and there’s vibration involves and then just the adjacent connectivity of the components of the generated geometry.

So generative AI meshes doesn’t usually account for these. So this is something that we have to post-process after we discretize the geometry and we come up with a a set values of constraints for each of these factors. And what we do is that anything that goes beyond these factor would cause assembly failure. And here’s just like a very simple example with the letter T where the generated geometry is uniformly scaled to the assembly space and then we run these fabrication constraint and then we the the system would flag that there is a vertical stack of more than five components and it would automatically shrink it down so that it fits the assembly constraints.

4:07 And we did a couple of abilation studies of these different factors just to see if any of these checks meaningfully fixes the assembly. And this is kind of just like a very preliminary pipeline of just going from like text to robotic assembly. But in the real world we’re working with multiple kind of components not just like a you know one kind of component. So we want to also think about how we can start thinking about maybe assembling like multicomponent objects.

4:37 So not just one kind of component one but maybe like another module of component. So the idea is just to start very simple once again just like introducing one other component. We have we imagine we have like the structural component and then we imagine we’re going to have a panel component as just like a surface that can go on these p these objects so that you can have a different kind of functionality.

5:00 The only issue is that there are many ways you can put panels on objects and this really depends on the geometry and the functionality of the object. So we come up with just like a baseline geometry of like just putting panels on the tops horizontal surfaces of these object but that we also realized doesn’t really make sense for all the objects. For example like panels for the chair maybe it shouldn’t be on the top maybe it should be on the seat and the back rest.

5:29 So this really depends on the functionality of these specific object that people might want. So we kind of imagine that we need like a system that is able to think and reason and see the geometry. So we want to leverage a vision language model that can reason over both images and text. And specifically we’re asking the question of how well can a vision language model in this case we use Google Gemini to reason about the geometry and functionality of everyday objects to support these assembly design generation tasks in a functional manner.

6:01 We start with the user prompt. Similar to the previous pipeline. This gets discretized into the structural components and then now we’re asking the question of where can we put these panel components. So essentially what we did was we took a picture of the AI generated geometry. We feed it into a vision language model to ask can you select parts that can you think about parts that in this image that should have the panel components and then the vision language model would return something like oh you should have the panels on the seat and back rest of the chair which is a valid answer but what we need to now do is match where the seat and back rest as like a text response back into the 3D geometry.

6:41 So we come up with a algorithmic way to label each mesh faces of the geometry like how like amusement park map would have like different labels on like different spatial elements and we give this labeled image to the vision language model again saying here is a image with numbers on it. Can you identify parts that should have the seat and back rest on it? And then the vision language model can easily pinpoint in the 3D model and talk to the 3D model saying oh it’s four and six that you should put the panel components on.

7:14 And we also wanted to evaluate the vision language how well a vision language model can actually do this. And the evaluation method that we came up with was essentially just like a user study that we run through different baseline methods. So one is just placing panels randomly everywhere. The other one is like a predefined algorithm that just place panels on top of these objects. And then the last part is the vision language model assemblies.

7:41 And we recruited 32 participants designers mainly to just ask for their preference because we assume this is actually a functional task but also is a pretty subjective task to evaluate. So we just try to get human feedback. Our results say that like the the users prefer the VLM generated results about 90% of the time. But they also kind of express that if they have the option to do it by themselves, they might assign these panels differently.

8:15 And about like 40% of the participants kind of suggested that. And this is just like simply just assigning like panels on a structural frame, which is which we deem like a very simple task that shouldn’t be that complicated. But the complicated part maybe is that people have different subjective ideas of what they think should have panels on them. So in this case a user might be unhappy and they they could be like I actually just want panels on the seat like what the AI provided was great but I actually this is just what I want like I don’t really yeah so this is just what they want.

So we come with like a different way to add user feedback into it where we in the second loop of this feedback. We essentially bypass the first vision language model task that assigns the component and then simply feed the user feedback directly into the second vision language model task where it would just select the label directly based off the user feedback. And here is the robot assembling a chair with I want panels on the seat and then the robot doing just that.

9:23 This are other examples. This is a white bowl with panels for the bottom. And this is assemble a lamp, put panels on the lamp shade. And in the case that the user doesn’t give any extra feedback and in this case it’s just a shelf with two tiers. The robot put panels on where the tears it thinks that it’s appropriate. And yeah, so this is kind of what I’m working on in my research mainly discrete modular assemblies and connecting with the idea of like being able to create reality on demand like the speed and scale is one of the more important part that I also want to tackle.

10:02 And this past year, I’ve had a chance to further push these ideas where this is like a different project where we’re using mobile robots to assemble basically larger structures that are both functional and structural and loadbearing. So this is an example of someone sitting on it and we the the contribution here basically is this octac module that can aggregate into each other to create a structural assembly.

10:27 And then other projects involving like a self-folding modular robot that can change into any shape that you want on demand. So like a step stool or a table and it just folds whenever you need them. So those are kind of the various ideas of how we can kind of get instantaneous shape formation with the help of AI and robotics and with and that’s a wrap on text to robotic assembly. Thank you.