0:16 Hey, hello everyone. Thank you so much for coming. I’m Miguel Amador. I’m a senior research engineer at Enya. My background is in mechanical engineering, but during my whole career I’ve been trying to incorporate physics priors and conservation laws into deep learning models for them to learn physics correctly. And this presentation I’m going to I’m going to present the large hearing models which are aimless aimless aim to reimagine design, simulation, and manufacturing.

0:50 So, first of all, I want to introduce a little bit the field with some some factors that we already know. So, the product development cycle time has been decreased enormously in the last decades. So, first of all, engineers we made very extensive experiments in the lab to validate designs. And with the with the help of computers, we had that tools. We started to get more fancy mathematical algorithms like the finite elements and the finite volumes method in order to get the high fidelity physics simulation and reduce this cycle time dramatically in your computer.

However, simulation is still a bottleneck for engineering as we all know. Very extensive CFD simulations might You might need to wait for days or even weeks for them to be to be solved. And that’s a problem, right? Instead of instead of having like a one-year cycle of your of your cycle time, we are aiming with AI to reduce this dramatically again. So, this will be like the second breakthrough in simulation technology.

2:00 And as for N A I, we are trying to solve this problem with the so-called large engineering models. So, what we call large engineering models, they are LLM technology specifically tailored for physics simulations. And that’s good because we inherit all these nice properties of LLM technology, transformer technology. It’s really tailored for big data sets, big inference data sets. So, that translates into the physics world with a lot of nodes.

2:31 We operate over hundreds and millions of nodes. So, that’s good. That’s an industrial scale simulation. And something important about LLMs is that we operate vertically. So, a foundation model would operate horizontally, right? You are aiming to to solve the physics, like every possible physics or maybe constrained physics, but you’re you’re trying to apply it to several engineering fields. We are not trying to do that. We are trying to take one single field, for example, automotive field, and through that vertical to design a model specifically tailored for that, but it’s able able to handle a huge variability of input spaces.

3:16 So, this is our vision of what an LLM is. But I’m not a technical person. I’m a scientist. So, I’m going to explain what an LLM is. We call it LLM. I’m going to present you the first LLM that we have developed a couple of months ago. We released a nice demo in our webpage. So, I’m just literally going to explain the features of the demo, what kind of input variability we are able to handle with this model, and what are the results and what’s the science behind it.

3:48 Okay, so this first LLM is going to be specialized in all these verticals in in in the very specific use case of the plastic injection molding. So, I’m not sure if you’re familiar with this kind of manufacturing process, but I’m you certainly sure that we all all of us interact with many plastic products in our daily life, right? And most of those plastic products, they are injection molded.

4:17 So, we never think about how they are actually manufactured, but it’s a quite complex process. It pretty easy to explain, but it’s very difficult to simulate. So, first you have an injection unit and you got you got some polymer pellets. You inject that into the injection unit. You heat it up with high pressures and high temperatures, and then you inject it to the second part of the of the machine, which is the mold.

4:43 And once the mold is completely filled, you wait for a little bit till it cools down, and then you release it, and you get the final product. So, this is deceivingly it’s a deceptively complex problem. It’s very easy to to explain, but we have several challenges here. Those are thin-walled geometries. We have free surface flow, so you have a flow front advancing in the in the air inside the the mold.

5:10 We have complex polymer material, so they have very weird rheological behaviors because we are operating on high-pressure, high-temperature regimes. And also we have thermal coupling. So, all this translates into very difficult equations. We have compressible Navier-Stokes equations with nonlinear viscosity and thermal coupling. So, the traditional lifeline here as an injection molding designer would be to create a CAD model of your of your product, right, based on some requirements, mechanical requirements, perhaps perhaps, and then you need to mesh the cards.

5:54 All those mesh cells should comply with the solver requirements, so it should be discretized accordingly in space and time for the solver to to work well and to converge. That would be stage one. Then stage two would be to select the injection parameters, and this is basically based on the injection engineer expertise and the rules of the thumb, basically. They know the best practices of injection molding and which materials are appropriate to each application.

6:24 So, they specifically use the domain expert expertise to set those up. Then the third stage would be to simulate these complex equations, which lasts from several minutes to hours or days if your plastic process is very complicated. And then you iterate over step one and two, right? You go you either change the geometry because you need to maybe make the the the plastic piece thicker because the flow is not flowing correctly, or you change number two, you change the flow rate or the melt temperature, those injection process parameters.

7:01 So, we already know this, right? We have seen already this pipeline in the previous presentations. So, what we are trying to solve with the ALEM, which we call the neural mold, is to basically trim the the cycle time of all these stages. So, first of all, instead of operating in over a CAD geometry, I need to mesh it. We just operate over point clouds, and it’s resolution agnostic, so our transformer-based architecture basically doesn’t It doesn’t care about how fine or coarse your mesh is.

7:34 It operates on point clouds, and I will explain a little bit later how we achieve this. In stage two, yeah, parameter selection, we have a continuous like continuous input of parameters selection, so you can automate that process. For stage three, the AI predictions, they are almost real-time. They are not real-time, but sub-second inference time. So, this is pretty convenient, right? Instead of minutes to hours, we have second predictions.

8:00 And fourth, we can iterate this over. This is everything is differentiable. We can auto grad for this using PyTorch usual blocks. Deep learning blocks. So, so you can use like a stochastic or a gradient best descending algorithm over any KPI engineering KPI that you can think of. Great. So, let me go through the demo of this model to see the capabilities of it. So, we operate over point clouds.

8:35 Because it’s a transformer, we can operate over millions of nodes, tens of millions, hundreds of millions of nodes. Cool. Also, we are using real customer geometry, so they are not simplified or anything like that. We go through the full detailed geometries, and we operate over point clouds. Second, we could select the material. We have 12 different materials based from ranging from low viscosity to high viscosity rheological behaviors.

9:06 This is not a limitation of the model. In fact, we use a continuous representation of materials. We can interpolate between them. So, basically, if an injection molding manufacturer wants to try their custom polymer here, that’s fine. We can we we can make it work. These are the scalar process parameters, so we can select the melt temperature, the wall temperature, and the flow rate of the injection process.

9:32 And we have also safe ranges, which are in the green shades, and outside normal ranges in red. And those those outside of normal ranges, you would probably get out of distribution behaviors, which are quite common, and they are represented therefore if injection parameter it’s good or bad, right? Of if if a design it’s a really manufacturable or not. And then you need to select the injection gates.

10:07 We can select up to any number of injection gates. This is not a limitation of the model. However, for the demo, we are we are limited to only two gates, one or two gates, but we could select a an arbitrary amount of of gates. Once you hit the run button in some seconds, you get a full time resolved output fields of your geometry. So, in this case, we are predicting time on field, temperature, pressure, and shear rate.

10:39 Those are the kind of metrics that an injection molding engineer is interested of. And each gate has also its own prediction of flow rate and pressure. And we we can also post process all those results and get some key process indicators of the of this process. So, in reality, an engineer would look at this and they would say, “Oh, okay. Fill Does it fill completely? Yes. So, that’s that’s good, right?” But it could not fill.

11:11 And that’s not a bad simulation. That means that the input parameters are within outside recommended ranges, or maybe it’s a it’s a bad It’s a bad selection of input parameters. So, we are interested in those in those edge cases. And also, obviously, we have a speed up. So, here in this particular case, it’s 200 x the speed up with respect to traditional solvers. We can have up to 1,000.

11:38 That’s easily achievable with this architecture. And you can try it by yourself. So, you can literally scan this QR code that has a link to our website. You can try to break it. You can try to put as many injection gates as you want in any position with any material. Hopefully, it will work. And also, we have like a I think six or seven different geometries. They are quite complex.

12:07 Yeah, feel free to to to break it and then to to ask me any questions after this. Let me explain a little bit the science behind how do we do this. So, basically, the models that we use, they are based on universal physics transformers, or in short, UPT. And Ankur Brangs UPT. Those are very famous papers from our one of our founders, which is Johannes Brandstetter from the AKU University.

12:36 And basically, they are the transformer adapted to to physics simulations. And we don’t do anything outside of this of these papers. So, everything it’s open source. You can you can see how we do and how we achieve all these nice properties. In particular, we have an encoder processing decoder kind of a structure. They are based on transformer blocks. So, they are specifically designed for to capture long-range dependencies.

13:09 So, this injection molding application, whenever the flow front reaches some part of the mold, the other part of the mold will notice a very high pressure spike. That’s those kind of non-local nonlinearities, they are super common in this application, and for example, GMMs, graph neural networks, they would require a super super small time step for us to capture this kind of behaviors. With our transformer-based architecture, we are able to to solve this problem really easily.

13:43 And also a second very good property of this architecture is that is resolution agnostic. It’s a neural field. So, usual transformers, you have self-attention, in which all the dimensions of the query, key, and value, they are the same length. But there was a very nice paper from DeepMind, which is called the Perceiver IO, which basically decouples queries from key values. So, you can have a very low-dimensional key values, which we call anchors.

14:11 And then you can have a very high-dimensional query, that doesn’t matter, because due to the matrix structure, like the rectangular matrix structure of self-attention, in this case would be cross-attention, everything works fine. So, you can extrapolate to any point in space that you like without being dependent on the anchor discretization. Also, it’s order it’s linear with with the number of nodes. So, it has a very nice scaling loss.

14:43 And some of the features, we have trained this model over hundreds of terabytes of data. It It depends on the kind of model. We have very lightweight models focused for super fast inference. So, and then we have also very bigger models for fidelity. And now we are focused on the filling stage only, but in the future, we’re going to include shrinkage and warpage, which is the next step of injection molding, and we will be most likely to reach the almost petabyte of data set.

15:17 Also, we have included some building conservation laws for the flow rate, for the physical solution to be valid. And also we get consistent 5% relative errors point wise in all the output fields. So, this is just an image of the a time and field prediction of this user geometry. Company geometry Also, I want to mention that the model is open source because it’s published in NeurIPS and ICML, but also the framework is open source.

15:49 So, you can try So, all the building blocks that we have used for neural mode, they are in here. And we call it another from another. And it’s an open software framework. It’s hardware hardware agnostic and it’s meant for modular transformer architectures. And also it has some GNN functionalities if you want to try it with graph neural networks. And it’s ready for scale. It’s meant for really massive data sets and really massive trainings, really massive amount of nodes.

16:23 We have data modules, inference modules. We have model zoo. We have transolver, geo transolver. Yeah, and also we have data pipelines, failure for the transform the transformer architecture, which is very specific. And yeah, so feel free to to try it. So, it’s also in our GitHub of the company. And yeah, a little bit more about Emmy. So, Emmy was founded by Johannes Brandstetter and also Dennis and Mix 1 year ago.

16:56 So, it’s a pretty new startup. We are 30 world class researchers and we are super happy to be here and feel free to reach out to Sophia or me. Just shoot an email or just we can talk. We are till tomorrow here. So, thank you so much. To learn more about the CDFAM Computational Design Symposium, access the archive of previous presentations, interviews with speakers, and information about future events around the world, visit CDFAM.com.