CDFAM NYC 2023 · New York · 14–15 June 2023

Learning to Generate Shapes

Listen as a podcast

Abstract

This recording is part of the CDFAM Computational Design (+DfAM) Symposium and features Karl DD Willis, Senior Research Manager at Autodesk AI Lab.

This session will be particularly valuable for professionals, researchers, and academics in the fields of computational design, additive manufacturing, AI, and CAD. Individuals interested in the integration of AI within CAD for enhancing design and assembly processes will find this presentation highly insightful.

Please note: The CDFAM Computational Design (+DfAM) Symposium is a respected platform for the exchange of insights and innovations in computational design and additive manufacturing. This recording is made available for educational and professional development purposes.

Transcript

From YouTube’s automatic captions, lightly cleaned; expect some errors. Each timestamp opens the video at that moment.

Read the full transcript · 3,037 words

All right, cool, so yeah, thanks everyone for having me here, thanks Juan for the invite. I’m from the Autodesk AI Lab inside Autodesk Research, and today I’m going to talk about some of the work we’re doing using machine learning to generate shapes, and we’re actually focusing on B-rips, so sort of the the 95, 99% of what most people use for mechanical CAD. And I’m going to talk almost exclusively about machine learning stuff and some of the work we’ve been doing, which is essentially publishing papers and and open sourcing data sets and our machine learning models for doing various things with with B reps.

0:50 So first of all, yeah, why do we want to generate shapes? So if you’ve ever used Google or sent text messages, you know text autocomplete is a thing, driven by machine learning. If we type in the word “is Autodesk research”, it comes up with these great suggestions saying “is it legit”, “is it good”, “is it worth it”, 100% definitely is, and you can imagine what we might want to do just in our day-to-day CAD life if we had design order complete, right? So if we had a series of sequence of operations that we did inside of a CAD, all that time we spend pulling our hair out clicking on edges and faces and trying to fill up things, that’s something I think, you know, we want to get that time back in our life and focus on doing the the creative work.

1:34 So we’re trying to look at applications like this, trying to solve problems like this, to make it, you know, basically allow people to achieve their creative potential and and spend less frustrating time and CAD. So but if we back up and look at text autocomplete and look at kind of what’s driving that, and a lot of the innovations around NLP, so natural language processing, we want to sort of first ask the question, what is a language model? So I think a bunch of you probably use GPT3 or Chat GPT, and essentially how it’s trained is to predict the next word and a sequence, right? So if I say “the cat is chasing the”, what it’s being trained to do, the objective function is, is just, you know, come up with the next word, right, so in this case a mouse, but it’s some sort of probability distribution over a series of n words, and that’s essentially how it’s trained.

2:21 And really, I think the excitement in the last two or three years has been driven by scaling laws. So scaling laws, it’s a paper from 2020, and it showed that these large language models, they get predictably better when increasing the compute, the data set size, and the number of parameters, right? So basically, if you don’t, if you, if one of those things is not bottlenecking the other, the model will get better and better and better, right? Just think about this, apply to anything in your life, if you knew that you could just do something and you’d get continuously bitter, boy, that would be cool, but for that, for now, this is kind of the thing that’s been driving, I think, a lot of the performance increases with large language models.

2:58 And what we’re seeing is these sort of new abilities that get unlocked with language models when you cross some certain scale threshold, right? So the bottom x-axis here is model scale, and what you see is that when your model gets big enough, all of a sudden it can solve math word problems, or it gets a certain level and it can do, you know, 8-bit addition, and this is a really interesting area, it’s kind of an area of debate right now about what this emergence is, but I I think it’s sort of quite clear, right, like you cross a certain threshold and you get some really interesting capability to chat to the GPT.

3:36 So however, in the world of design, it’s kind of a different ball game, right? I think the problem, and I think a couple of people have alluded to, Francis before me, you know, design data is limited, right, data in general is limited, and it’s limited especially in these domains where it’s super specialized. Right, medical device, medical imaging, that type of thing, design data is, I think, in that same category, so it’s not only limited, it’s also really challenging, right? So the design data, unlike sort of images or unlike text, it’s high dimensional, so many many layers of information, it’s non-cartesian, so it’s not this nice beautiful grid that we can use, it’s also high variance, of many many different ways to design the same thing.

4:15 And as the previous talk talked about, it’s difficult to label, right, like you can’t, you can’t just ask somebody on the internet to tell you if there’s a cat or a dog in the image, you’re asking questions like, what is the design intent of this, you know, pedal assembly, and no one’s going to be able to really answer that other than the person who designed it. So that’s sort of the challenge, and I’m going to talk about today, like some of the work we’re doing to try and tackle that challenge.

4:44 So I think an easy place to start is actually by reducing a dimension and going back to 2D, and going back to vector shape generation, and that’s interesting for us because that’s kind of where we started at AutoCAD, right? I think some of the clothing has changed a little bit from today, that we don’t really use the printers so much, but actually the underlying representation of how we describe shapes, 2D shapes, is pretty much the same. So 2D vector design data is everywhere, right, so it’s in it’s in fonts, it’s in, you know, emoji, it’s in floor plans, it’s in engineering drawings and electrical schematics, it’s all over the place.

5:18 And so if we wanted to actually work with that representation and train machine learning models on that type of data, we need to approach things differently, and the way that most sort of image training works, and I think Francis gave a good introduction to this, is using things like CNNs, which are convolutions, and they work on this 2D pixel grid, right, and they’ve been very very successful at doing things like classification, so in this case telling you that there’s a boat in that image. But with vector it’s essentially a 1D sequence of curve commands, right, so there’s a line, there’s an arc, there’s a circle, and that basically listed off as a series of types of curves and a series of parameters, and so that’s sort of a different beast.

6:01 And I want to just kind of walk you through kind of where we are at today, not just with our work, but just the the research world in general, and here’s a, here’s a question, right? So this is a vector fonts, and this is a, I’m showing you here some different machine learning models for font generation, and one of these is designed by a human and the other three are designed by, I guess, AI. Any anyone want to take a stab at guessing which is the one designed by a human? Shout it out, fail, you’re not coming back next year. This is the bottom one is the one designed by a human.

So so so yeah, from top to bottom, this is basically like us getting better and better at time at creating fonts, and these are basically the the Neurops, SIGGRAPH, and CVPR, kind of the three of the top machine learning conferences, and this is sort of where we are, right, as of 2021, and if you kind of take a closer look at these you can actually see that the output of some of these models is a little bit hairy, right? I just want to make the point that when you use vector sequence supervision, that’s sort of training on the curves, at a a trained photographer has used, you get this sort of nice topology you see on the far left.

7:17 When you use images as a source of supervision, you get kind of quite nice accurate lines, but the topology is all messed up, like way too many, way too many points in there. When you kind of mix those two forms of supervision, you get better results with the Deep Vec Font, but it’s still, I think, a little bit different from humans. So I think I think we’re getting there on the 2D kind of font font area, this is an example of using that same sort of approach for for vector font generation, but then you sort of double click and you see, well, there’s still some differences, so I think there’s a little bit of work to do in this area.

7:50 This is a paper actually just from this month, which is looking at using it, like, takes prompts to generate icons, and so so the work on the right hand side is is what’s presented in this paper, and it’s showing some different examples of how GPT4 performs on this same task, how stable diffusion performs, and then sort of converting it to vector, but you get this sort of sense of it’s heading in that direction, and I think soon it’s going to be something we’ll kind of tick off.

8:18 So I think there’s this open question about, like, what makes human design human design, and you can kind of see some irregularities we see with this weird output from the neural network, and I think a lot of the things we can talk about here are things like having a minimal number of curves for editability, right, so you don’t want to be basically having to move a hundred of different vertices around to make one small design change, having concentric circles, things like coincident lines, mirroring, symmetry, orthogonal lines, tangent curves, and if these all sound familiar, it’s because these are the bread and butter of engineering sketch constraints.

8:51 If you’ve ever used a sort of sketch and extrude parametric CAD workflow, this is all baked in there, and so what this is, just showing a pretty visualization of how those different constraints are applied, and constraints obviously are important, right, because they maintain this thing we call design intent, right, so basically if you make a change to a design, it’s kind of going to stay looking like you intended to do in the first place. So here we’re changing a couple of the dimensions and it’s sort of maintaining that design intent because we have added those constraints, without the constraints all bets are off, and and things start looking pretty funky, right?

9:30 So having those constraints and being able to predict those constraints is another important thing we want to be able to consider, and ideally we want to get to the point where we can vary those those constraints and have the whole size or the width change, and just somehow our, you know, machine learning model kind of understands that, right, you’re not getting things that are off the off the off the design space you’re trying to target. So this is, this is an example of, I think, probably the best we can do in terms of work right now, this is the work on the right hand side is by DeepMind, and this is from Neurops in 2021, and this is just generated results of sort of random engineering sketches.

10:06 And if you compare them to human, right, like they’re pretty good, and they’re actually, you know, predicted constraints, predicting construction lines, and so I think we’re, you know, getting closer and closer. So that’s kind of where we are right now with with 2D generation, I think if we want to go up one dimension, we have to talk about 3D solid model generation, and that’s generating, you know, boundary representation models, and so I think a lot of the work we’ve seen in, I think Dewan alluded to this, is like you sort of say, see these hyped up articles saying, hey, we can finally do image, sorry, text to 3D, but if you look a little bit closer, you’re getting out these point clouds, or maybe you’re getting a kind of a scrappy mesh, and really what we’re trying to do here is actually, can we go to real solid models that can actually be used in in CAD workflows.

10:58 So that means we have to sort of work with a with a boundary representation, just because that’s that’s the the industry standard right now, and so for those who are probably too familiar with this, but I’ll recap it, consists of, you know, faces and edges and vertices, all of different curve types and different surface types that are bounded together to represent the outer shell of a solid model. And in this work we published last year, called SolarGen, we have this approach where we’re essentially trying to sew together these different edges and faces of a B rep, and we do that by first predicting a series of vertices that are basically going to form form the the kind of the geometry of the shape, and then we try and join these up with what’s called a pointer network.

11:46 We join those up to make edges, and then we join those edges up to make faces, so it’s sort of this hierarchy of stitching things together, we use it, you know, Transformer architecture, which is almost what everyone uses these days, to first predict the vertex vertices one by one, then we use this Transformer network with the pointer network to basically join them all together to form the edges, and then another network to form the faces. And so this is an animation of what it looks like when we’re sort of stitching those things together, so you can sort of see the the model predicting these edges and then filling in the faces around them.

12:26 So this is this is one approach, right, where you’re trying to create a boundary representation where it’s a solid model, and you don’t necessarily care about the parametric history, but it turns out people do care about the parametric history, so we also want to want to have another approach, right, another way to approach the same problem, and we’ve done some work with learning sketch and extrude command sequences. And so here we’re basically trying to predict these modeling operations in a sequence, you know, much like we might do with a language model, and so when when we sort of piece those extrusions together, this is what it looks like.

13:02 We released a data set several years ago called the Fusion 360 data set, that contains the sort of ground truth data for a bunch of these sketch and extrude sequences created in Fusion 360. And if you’re interested in that, I’m just going to quickly flash this up, but Fusion 360 Gallery, you should, you should be able to find that on the internet. So one of our most recent works in this area is called Sketch Gen, and we use this representation, which is, you know, similar but but slightly different to what I talked about before, in that we have these curves that join together to form loops, that join together to form faces, and then instead of kind of joining the faces, we extrude them and perform Boolean operations.

13:48 And these all sort of compile down to a series of tokens that we feed into, basically, what’s, what’s like a Transformer model, or similar to a language model, and basically these are kind of like the words, if you like, in our in our sentence. Maybe I won’t bore you with this complex article, but I think that the point here is that at the end of this we get a CAD construction sequence that was basically a sequence of modeling operations that we play back in a CAD kernel, and that’s that’s what we use to generate the final geometry. And so you can see here on the right hand side, this is what we get for, you know, generating 2D sketches, and in the left hand side, switch that around, on on the left is the 2D sketches, on the right is the 3D generation results.

14:38 So I think that’s probably a pretty good picture of where we are today, I think, with at least the things that are happening in the research community for generating solid models, generating B reps. I think there’s a whole bunch of work going on now as well with different different companies, but I think the main point I want to make here with the talk is that, you know, edible, editable representations really matter, right, like it’s it’s it’s kind of not really helpful to to get something that you just can’t deal with, and some output from the AI model that you just can’t do any editing with. So I think this sort of loop between the human and AI, human and machine learning model, being able to basically take the output, edit it, and then continue on that loop is really important, and design is just such a critical thing, right, we need to have that that iteration, that loop.

15:22 And you know, there’s just so many amazing problems, I think, we can solve with this, whether that’s making amazing buildings, more efficient vehicles, optimized aircraft, I think we’ve heard a lot of those today, and I think machine learning can be a real kind of powerful force to solve some of those in the future. We’re certainly not there yet, don’t believe the hype, we’ve got a lot of work to do, but I think it’s something we can, we can bank on in the future. So thank you very much, thank you.

More from CDFAM NYC 2023

Design Computation Human

Design Computation Human

Onur Yüce Gün · New Balance

Mud y Robots: Keynote Presentation

Mud y Robots: Keynote Presentation

Ronald Rael · UC Berkeley / Emerging Objects

Roadmap to Replicators

Roadmap to Replicators

Neil Gershenfeld · MIT Center for Bits and Atoms

Scaling & Standardizing Patient Specific Design

Scaling & Standardizing Patient Specific Design

Adam Wentworth · Mayo Clinic

Computational Design Ops

Computational Design Ops

Ronnie Parsons · Mode Lab

Overcoming the Design Dilemma

Overcoming the Design Dilemma

Ryan Carter · Johns Hopkins APL

Generative Design & Digital Manufacturing at NASA Goddard

Generative Design & Digital Manufacturing at NASA Goddard

Ryan McClelland · NASA Goddard Space Flight Center

Digital Materials – Geometry and Data Combined

Digital Materials – Geometry and Data Combined

Marek Moffett · General Lattice

Ten Years Teaching Additive

Ten Years Teaching Additive

John Hart · MIT

10 Years of Applying DFAM

10 Years of Applying DFAM

Tim Simpson · Penn State University

MfAM Walks so DfAM Can Run!

MfAM Walks so DfAM Can Run!

Chelsea Cummings · The Barnes Global Advisors

Informatics and AI for Materials Development

Informatics and AI for Materials Development

Wes John Alder · Citrine Informatics

Predicting Fatigue Life in AM Metal Parts Using DigitalClone

Predicting Fatigue Life in AM Metal Parts Using DigitalClone

Behrooz Jalalahmadi · Sentient Science

Designing Material Properties with Advanced Toolpath Generation

Designing Material Properties with Advanced Toolpath Generation

Alex Roschli · Oak Ridge National Laboratory

Design Automation Using Machine Learning & Computational Design

Design Automation Using Machine Learning & Computational Design

Rhushik Matroja · Cognitive Design Systems

Design for Artificial Intelligence

Design for Artificial Intelligence

Chris McComb · Carnegie Mellon University

Register for Updates and Discounts on CDFAM events.