0:00 I wanted to start by thanking Duann for organizing this event, it’s been really lovely so far, and I’ve been really grateful to listen to all of these presentations today. I’m excited to talk to you about a recent research work that my team and I completed, called “From Concept to Manufacturing: Evaluating Vision Language Models for Engineering Design.” So to briefly introduce myself, my background is in mechanical engineering, I hold a bachelor’s and master’s degree in mechi from the University of Maryland and MIT respectively, and I am currently a PhD candidate at MIT in the decode lab, where my research is largely around the interface of artificial intelligence and engineering design and development.

0:43 So without further ado, as engineers we work with data in tons of different modalities. You might be looking at a hand drawn sketch from the early stage conceptual design phase, or a CFD simulation, or even a table of part information for specking parts. Because of this, a useful AI agent for engineers would need to be able to intake these different data modalities, for example, taking in an exploded view of a cat assembly and the prompt instructions, write assembly instructions for this part, interpreting both logically, being able to cross reference them, and then outputting logical and coherent instruction that would be useful for an engineer. This feels like science fiction, but it’s interesting to ask, how close are we to an AI agent that could actually look like this?

1:41 Well, there do exist vision language models, which are multimodal machine learning models, some of them that you might have heard of are GPT for Turbo, or GPT 4V, Claude, Gemini, Lava 1.6, and these are very similar to large language models which have had a lot of traction recently. The difference is that they’re not just trained on text, but they’re also trained on images, and in part of being called large, they are billion parameter models that have been trained on much of the internet. Because of this, there are a couple key qualities to vision language, vision language models, or VMs. The first is that VMs are quite versatile, anyone who’s tried out some of these has seen that you can give it a many different tasks from many different fields and it has some sense of understanding.

2:33 This brings me to the second point, that VMs have some sense of global knowledge, and this, as I mentioned, is because they’ve been trained on much of the, of information available on the internet, and lastly, via their chat bots, there’s a simple and intuitive interface, so the barrier to entry for an engineer using this is quite low. But this beckons the question, can vision language models actually work for engineering design? And this is the question that inspired this work, and that I’ll try to touch on throughout the rest of my presentation. So this is quite a broad question, and in order to scope it down, my team and I decided to scope it via the design process.

3:20 So we all have a mechanical engineering backgrounds, and we used the typical product design process that we’re amiliar with, which starts typically with planning and conceptual design. This is where you might identify user needs, determine your product specifications based on them, and start ideating, coming up with tons of different concepts, moving next into system level and detail design. This is where a lot of the, like, classical engineering work that we think of comes in, so you might determine if your part is going to be an integral or modular structure, what the subsystems will be, material selection, etc, and finally manufacturing and production. And then we also wanted to add this other part to our scope, which is engineering education, and in particular we were curious if vision language models had any sense of spatial reasoning, given that that is such an important aspect of the engineering design process.

4:19 So within each of these categories we developed a number of tasks, and we explored each of these to determine how well vision language models could perform at them. So this is how I’ve, I’ve structured the rest of my talk, and we’ll go through a, a select few of these tasks, given, given the time constraints, and talk about how our findings were shaped by them. As an overview of the experimental approach that we used, we focused mainly on GPT 4V, which is a closed source model by Open AI, and when we first began this work it was only a ailable via the Chat GPT chat bot, so they hadn’t yet released the API for it. We also saw how well Lava 6 1.6 34b could perform at these tasks, and on the contrary to GPT 4V this is an open source model, and we had access to its API immediately, which made it a lot easier to run a large number of experiments.

5:23 We focused only on engineering tasks that required both some visual and textual input given the nature of our work, and we tried to make it as quantitative as quantitative as possible, but a huge disclaimer here is that large language models, vision language models, this field in general is changing rapidly, and so the results that we found could be different today and will most certainly be different in a month as new models are released.

5:49 So first I’d like to give you a little sneak peek into our key findings. Essentially these are loosely categorized as pros and cons. In the pros category we found that there’s promising optical character recognition, so this means it’s kind of a computer vision task, when looking at images they, GPT 4V in particular, was able to pull out characters or text quite well, even for handwritten images. There is definitely a level of high level knowledge, and we could see this through a number of our tasks, and another pro is that when the vision language models were unsure, due to their structure they’re able to request additional information, and so there can be iteration between the user and the model.

6:39 In terms of some loosely cons, there is a tendency towards caution in certain answers, and so this can sometimes present itself as a reluctance to answer a question at all. We see that spatial reasoning is a major area of limitation, and lastly there’s a lack of precision when interpreting images. And so for the rest of the talk I’ll point out how the different experiments that we ran led to our key findings.

7:06 So starting first with conceptual design, here we looked at two tasks, first design descriptions and second concept selection. So for the first task, the data set that we’re dealing with is a data set of around a, sketches that were made by university students where they designed a part, and then they also have a handwritten description at the bottom, and here we wanted to see if vision language models were able to correctly match the description to given the sketch. So we asked the multiple choice question, which text description matches the given design, and then we provided the image both with the handwritten description and with it removed.

7:51 And our results are as follows, as we kind of expected, when the text description was still available there was 100% accuracy in matching the sketch to the actual text description, so this suggests to us that there is promising optical character recognition and there’s quite good sketch interpretation. Then, once we removed the text description from the bottom, we see the results plummet, accuracy drops to 53.3%. % This isn’t so surprising, but it’s still useful to quantify, and this suggests to us that more context, aka the handwritten description, improves results. But then interestingly we noted that when GPT 4V, in this case, was making errors, it was most often erroneously selecting none of the above, so if we didn’t provide a text description it would say that none of the multiple choice options were correct. But when we removed that option and it was forced to choose between three text descriptions, the accuracy improved from 53.3% to 70%, and this starts building the case that GPT 4V tends towards caution, which is useful for understanding as we move forward.

9:16 The next main task that we looked at within conceptual design is concept selection, so what we did here was use a task and try to determine the user needs, and then try to determine a Pew chart, which I’ll explain a bit more in a second. So we prompted GPT 4V with this task, a medical supply company retained a product design firm to develop a reusable syringe with precise dosage control for outpatient use, this comes directly from a product design textbook, and because of this we have some sort of ground truth for both the user needs and the resulting Pew chart, and we asked it to list the selection criteria that should be used for concept selection here. And so here we compare the ones, the selection criteria that were identified from the textbook, with the selection criteria that GPT 4V generated, and what we saw is that for every one of the ground truth selection criteria there was a matching one developed by GPT 4V.

10:24 So this started to give us a sense that GPT 4V had, has some high level knowledge to generate user needs or selection criteria, but then the real test is, can it use this to make any informed decisions? So here we went to making a Pew chart, and for anyone who’s unfamiliar, a Pew chart typically has the row headers as different selection criteria and the column headers as different concepts, and then you subjectively rank your concepts based on their performance for the selection criteria. So we provided seven, yes seven, designs that were also from the textbook, and we said, given these and the selection criteria that you found, can you make a Pew chart. Here we saw that it knew what the format of a Pew chart was, it was able to tell me the rows and the columns and how there would be some reference concept and that you would subjectively rank the other concepts based on their performance against that one, but it refused to fill out the Pew chart, and upon every iteration it would refuse to do this without more information, and so this once again drove home some level of caution.

11:34 So the key findings that these experiments kind of supported were, one, the promising character, optical character recognition, which we saw in the early design description task, that there is some high level knowledge, it understood what a Pew chart was, ability to request additional information, and a tendency towards caution or reluctance to answer sometimes, times.

11:59 Moving next into system level and detail design, here are two tasks, were one, material selection using Ashb charts, and two, engineering drawing analysis and CAD generation. So first for material selection, here we provided an Ashb chart, and for anyone not familiar, this is a chart which has two material properties, one on the X and one on the Y AIS, here we have dens on the x-axis and Young’s modulus on the Y, and then it plots different materials and groups of materials, and an engineer can use this to identify which one, which material they’d like to use given their properties. It’s a bit archaic probably, but as an engineering student this is something you see, and we thought it would be a really interesting task for understanding some of vision language models capabilities.

12:50 So in this case we kept our ass quite broad, we provided this image and we said, I want to select material for a beam and I want the beam to be both stiff and light, and GPT 4V was able to appropriately select a number of different materials that meet these qualifications. But next we wanted to make it a little more challenging and a little more precise, so we said specifically, we want a density from 7 to 10 of megag per meter cubed and a Young’s modulus that’s greater than 100 gigap pascals, and in this case we’re looking for specifically materials in that blue box. We found that GPT 4V was able to sometimes select materials that were in that, but often times erroneously select materials outside of it. Once again this starts suggesting that there is some promising optical character recognition, or perhaps it didn’t use the chart at all but instead just used the LLM capability and used it understanding of materials that expects to be both stiff and light, but we do see that there’s a lack of precision when interpreting images, so despite being given that plot it wasn’t able to precisely identify which images would meet the, the design task.

14:14 Next for engineering drawing analysis, so here what we wanted to see was, can GPT 4V interpret dimensions from an engineering drawing, and can dimensions be used for downstream tasks. And this is kind of funny to me because I made this drawing back in 2017 in my first CAD class, and I never thought that it would be, I never thought that I’d be looking at it so much again, but here we are. So in this case we said, this is an engineering drawing of a part showing four views of it, can you tell me what the part is. And what we were looking for is an answer of, this is a block or a rectangular prism with a blind hole, because as you might have noticed, the hole doesn’t go all the way through the block, it has a depth of four compared to the block’s depth of five. It GPT 4B could only correctly identify this one out of our nine trials, and we ued the trial numbers later on, but essentially we saw the same results, it repeatedly called it a block or rectangular prism with a through hole, so it was unable to interpret those dimensions to accurately understand what the part was.

15:29 But once we told it what the part was, we said, this is a block with a blind hole, can you please extract all of the dimensions and give them an appropriate name, it was actually a lot more successful at this. So six out of nine times it was able to correctly identify the four dimensions given here, and we allowed for any, com, like, pr mutation of width, height, and depth, given that those just need to be accurate within themselves, but what it struggled the most with defining was once again the depth of the blind hole.

16:11 So the next step was, can it use these dimensions to actually generate CAD? In this case we gave it the correct dimensions, and we said, now that you have them and you have the actual drawing at hand, can you use a CAD scripting language, and we used three, CAD query, feature script, and open SCAD, to generate a, a script that encodes the CAD, and so it would answer in the form of a script, and we would run the script ourselves, because at this point there wasn’t any API integration. And if it made it correct immediately we would stop, and if it didn’t we would do up to five iterations where we would say, can you try it again, please refer back to the image that we’ve provided.

16:49 Our results here are as follows, so for CAD query we were able to, or GPT 4V was able to, correctly generate a CAD one out of its three times, so the top three are using CAD query and the bottom are using open SCAD. Overall that was the only time that GPT 4V was able to generate a CAD script that correctly made the part, and you may have noticed feature script isn’t on here, once GPT 4V was never able, ble, to generate a script that ran without syntax errors. So one other thing that we saw is that there was a forgetting of the dimensions when we would iterate, so around generation three of iteration it would often just begin making up dimensions despite having known them in earlier iterations, and we see that iterations didn’t really improve the results.

17:53 This kind of leads to a couple of our other findings, there is some high level knowledge of CAD scripting languages, so as we saw I was able to generate runnable scripts in CAD query and open SCAD, but spatial reasoning needs major improvement, and once again there’s that lack of precision when interpreting images, and this is kind of demonstrated by not being able to extract those dimensions.

18:16 And finally, manufacturing and inspection, we did a number of tasks here, but I’m only going to touch on one, and this was where we were looking at designed for additive manufacturing with a very small scope. So what we did was provide 10 additive manufacturing rules which are available from a company called Hubs, and it’s available as an image, which was useful given our tasks, and then we made 10 parts that are deemed unmanufactured, because they specifically violate one of the rules, and then 10 parts that are manufacturable, and we actually printed these parts just to be sure that they are manufacturable.

19:00 And what we wanted to see was, if given the rules and given a part, GPT 4V could identify if it was manufacturable or not, and if it’s not manufacturable, why. And what we found is that GPT 4V always predicted that the part was not manufacturable, even for parts that are very clearly easy, simple, that we did print them, and then it correctly identified the violated Rule 13 out of 30 trials, but even this number is a bit skewed because it never identified one single rule, even though that would have been correct, it would always choose multiple. So this suggests once again a tendency towards caution, in that it said none of these parts will print, spatial reasoning is limiting again, and then lack of precision, in that it should have been able to, one, understand the rules, and two, use the dimensions provided in our CAD parts to see that something did or did not comply to a rule.

20:03 And so finally, we wanted to drive home one of our key findings, which is that spatial reasoning needs improvement, with a pure test of it. So we used two kind of classic spatial reasoning tests that humans take, the rotation test and the packing test, and they’re shown here, and in general these are slightly difficult. So for the rotation test you need to, you see an object rotated and then you want to do the analogous rotation on a new one, and then for the packing test you’re just choosing which of the many options would pack into the correct shape. Despite being kind of tough, humans usually score around 65 to 70% on these tests, and given their multiple choice statuses, a random guess for rotation would have been 20%, and for the packing test would have been 25%. GPT 4V performed about at random for these, so this really drives home that spatial reasoning needs improvement if we want this to be useful for engineers.

21:12 Now briefly, the comparison with Lava. This whole time I’ve been talking most extensively about GPT 4V, but we also tested this with Lava 1.6 34b, and and across all of our quantitative tests GPT 4V outperformed Lava, so here a higher number is better, and in every case we see GPT 4V outperforming it, which is interesting because Lava has outperformed GPT 4V on some classical multimodal benchmarks, but as we can see those benchmarks don’t necessarily translate into an engineering domain.

21:47 One point that I found quite interesting is that Lava was imperfect in OCR, so if you recall back to my first experiment that I showed, where we had a sketch with handwritten text at the bottom and then we tried to match that sketch to a text description, GPT 4V performed perfectly in this, and that’s in part because all it had to do was interpret that text, but Lava didn’t perform, perform, perfectly at this, which is an interesting finding and suggest that maybe it’s OCR could use some work. And then finally there are a number of NAs here, and that’s because Lava 1.6 34b also can only intake one image as input per thread, because of this many of the tasks that we developed were inaccessible to Lava, and I think in general that vision language models can benefit by being able to intake more images if engineers are to use them, because then they can cross reference and you can provide iterative feedback.

22:55 So as a recap, we looked at the design process and identified many tasks in here in order to test vision language models performance and see where they stood in terms of engineering design tasks. Generally these are the key findings, and if we sort them loosely into pros and cons, there are a couple steps that we can take looking forward. One, we can utilize the existing pros, for example, we saw that optical character recognition is quite good. In addition I found that sketch interpretation and being able to take creative steps after a sketch was given was quite high, and so the project I performed right after this was called sketch to prototype, and here I input just an image of a sketch with a handwritten description, in this case a mug that looks like a backpack, it has handles as straps, and converted that from sketch to text, text to more detailed image, detailed 2D image, to a 3D model, and then I printed it out.

23:55 So there are already some things that vision language models are capable of doing, but given that we’ve noticed many areas of improvement, I would say two points for improving spatial reasoning. One, you can use custom GPTs which integrate Python into a VM, and there you can use the spatial reasoning powers already existing in Python packages to try and supplement the downfall of vision language models, you can also use vision language models APIs and then combine them with existing tools. Another step forward is changing the way that we pre-train these models, so one option is changing the data set that we use, this can be quite tough because they’ve already been trained on a huge amount of the internet, so fine-tuning is, is difficult, but we can also change the pre-training objective to make it more dialed into to spatial reasoning. And in order to do this last point, we need to create meaningful data sets that suit engineers needs, and I see that there’s already a ton of work being done here for that, and I think that will be really important and powerful moving forward.

25:13 So, oh, image citations. With that I’d like to thank my co-authors for this work, and I would like to thank all of you for your attention, I’d be really happy to talk about this more later.