0:00 So our talk today is going to be coming from the trenches of trying to use all these tools in a architecture firm. So our title is form jamming with AI agents. Briefly, you know, form jamming is trying to make sense using these AI tools on on projects. I mean, earlier this summer, we we came up with this term form jamming as a way to u think about these AI tools in in combination with each other.

0:32 I mean, we’ve been experimenting with these tools as we all have for the last couple years, AI coding and cursor and image generation. And we felt like earlier this summer it was the the tipping point where the models have gotten better, the AI coding tools have gotten better and it felt like, you know, we we can use these in in productions in production on our projects and we’ll be talking more about that in a second.

0:59 But Jun Ling and I are from HDR. We’re a large architecture engineering firm. We have a computational design group at at HDR about 15 people. We’re split between software developers, computational designers, a couple people that focus on on AI. We have a couple data scientists that make our group a little bit more unique than different firms out there. So again, we build tools, we work on projects, we we develop services.

1:33 HR overall we’re quite large and so we support mainly our architecture group but we certainly work on transportation water projects all kinds of stuff. So couple parts to to kind of introduction to this this concept of form jamming. And this is where you know earlier this summer when we started thinking about trying to make sense of these tools coming together and actually working on projects you know the things that we were thinking about you know on on one side of this tools and architecture take a long time to produce.

2:10 So along the top here are are algorithms. So you have the pure bezier curve from 1958 and it didn’t get implemented into AutoCAD until around 1980 GIA in 1977. And then you know as all technologies do they get implemented in automotive film faster than they do in architecture. And so the FOA building even though it’s not documented as using desier curves or AutoCAD it seems to be the one of the most clear examples of large project implementing this algorithm in these tools.

2:48 You know it wasn’t until after this that we had Bill Bao larger Zaha projects. And so just just seeing that pace of, you know, the algorithm, 20 years to AutoCAD, 20 years to getting fully implemented on a project. The most clear-cut case of this that I think everyone’s probably familiar with is, you know, Bill Bao, Frankie, and using KIA to do that project. So, the project before Bill Bao was the the Vitra building, and it was famous for having a a kink in the building.

3:20 And from there Frank Gary started using GIA which eventually turned into digital projects. But this is a very specific use case of these kinds of tools because it was about you know his models and the tools achieving exactly what he wanted to do from a a design perspective. So this is in direct contrast to what comes a little bit later. And an example I really like is the Greg Glenn embryological house project.

3:56 Now this was like there’s kind of a genre of this work that happening from from Greg Glenn and then many others afterwards but you know his thesis on this was around mass customization and you know using you know using u the algorithms and coming from animation. So it was about using alias and and it’s it’s actually quite simple. You know he’s all all he’s doing is taking the bezier curve modifying the points and then using the animation software to interpolate between the options.

4:29 Now at the time this was you know nobody in architecture was really using animation software. The I actually like the the the road the image up there is like the first Lucas rendering. Which I’m actually fascinated by as a as a sidebar cuz you know people saw that first rendering and thought you know this was like we should use this to make films and now and you know now it’s like you show somebody a an AI image with one artifact in it and they’re like oh it’ll it’ll never work but this this concept of like the you know the tool and the output of that becoming the work and having a framework around that that Greg did in the embryological house is something we’ve been thinking a lot about.

5:17 So so that’s one part of the conversation around form jamming and the other and that’s about you know tools taking a long time to be implemented in architecture. The other, you know, I really enjoy kind of tracking the effects animation tools, especially as it relates to architecture. You know, it’s just saw how it crosses over. Maybe one one way, one place to start on this is the Wild Robot 2024 film.

5:43 I think this is a a perfect use case of using AI tools even though these film studios don’t talk about this much which I think is something that you know we in in in architecture deal the same way but it’s you know it’s it’s it mainly because they’re they you know they use so many tools that it’s never about just you know using one single tool and 2026 the this critters movie that’s coming out is backed by open AI I, you know, personally, it’s a cool project cuz it’s using AI, but it’s it’s a bit deceiving because the whole point of it is they’re they’re trying to make films for a fraction of the price.

6:29 That is a direct comparison to the way they made Toy Story with John Lacader and Ed Catmull and Steve Jobs. John Lner when he when he pitched Tin Toy to Disney everyone at Disney got excited because and the the head of Disney said okay well you can do it but you have to do it for half the budget and then he said he wouldn’t do it and then that’s how he left Disney and then met Steve Jobs and end up building Toy Story.

6:56 So, I always think like as new technologies come out, you know, speed is one component of it, but saying that you’re going to do it for for less money is is always a a bad route to go. But all these so all these, you know, I always think that these tools in in VFX become interesting as it relates to to architecture and you know, form jamming because then it’s just a matter of like using all these all these tools on on projects.

7:24 So, one one project we we’ve been doing this summer we call block 99. This is work this is a a work that it’s an installation interactive installation and and web app that we installed at the Kucko Museum in in Omaha, Nebraska. So, it’s part you know using AI coding for the web app parts for image generation for building the hardware. It kind of encompasses all these ideas of just like trying to make this stuff happen on a project at like all means possible and as fast as possible.

7:55 We did this whole project in two weeks. So, we bought this TI99 computer. Actually got it working straight out of the box. Played the Pac-Man Munchmon Munchman game. Took apart the keyboard and rewired it to get it working. And so we had this whole idea around using retro hardware to to design the thing and and we rebuilt it to add knobs and sliders. So this is all about like just if we can move at this rapid pace of building our own tools, you know, why not build, you know, physical devices as well with this?

8:37 So this is what the the final result looks like. So on the right there’s we cut out part of the computer where the the old video games go and we added buttons and sliders that you interact with it. So it’s kind of like a Lego building component and it’s actually quite simple because it wasn’t necessarily about the the algorithm even though we had ideas of going much further with this.

8:59 It was just simply about you know it’s almost like building your own modeling application simple block building and then using the computer as a way of like interacting with that. And so the output of this yeah I mean we we you know we have all these little physical model aesthetic out of it. You know we we we kind of see image generation as a you know almost like a a rendering engine.

9:24 We have our own you know it almost kind of seems trivial to talk about image generation at this point but I’d say you know in-house at an architecture firm when you’re trying to build you’re trying to fine-tune an image model to be in production does take a fair amount of work because it’s got to sit in house. You got to fine-tune it. You want to make it be aesthetically pleasing.

9:43 So all that work was great. We did that this summer in two weeks. And but you know the other part of form jamming so that’s like a single project that’s output into a museum. The other part of form jamming is just trying to work with these agents in a quick way. And so Julian’s going to talk a little bit how we’ve built our Rhino MCP kind of the technical framework of that and how that’s kind of fitting into our pipeline.

Hi. Hi. So when we saw the SCP which is the model content protocol, we are really excited especially when we saw the blender SCP, we show a show us a lot of like potential for 3D modeling agent and we immediately put in put it into our radar and start to develop our own Rhino SCP. So from but there is a couple of the challenges we made in our development process.

10:49 Firstly we de a local version randp like blend to run locally in the users laptop but we finally end up with like find it’s really hard to you know do the tin cooperation and when we want to update the feature of this random SAP and the user need to reconfigure it. So we think about deop a cloud first architecture. So we are maintain a centralized Rhinocp manager in the cloud that every single user will have their own rhinoc user instance and their ports that associate with the memory and also the OK geometry data for their own random SCP.

11:37 This one actually make us like more easily you know when we update a new feature the user can have these new features smooth mostly update in their laptop. So here is the we have a lightweight Rhino plugin that you can run the Rhino plugin in the Rhino. Then we need can you know register in our web application part and copy paste this information credential and get your brand own with some you know so here is the sample of the interface.

12:20 So you can use this NCP instance to connect with any NCB client such as our desktop the cursor. So another thing actually this packer introduced another challenge for us how can I you know remote NCP server control local user Rhino. So we actually designed a three layer communication port. So firstly in the NCB client side these will be communicate with the you know remote server with a sinable HTTP protocol.

12:59 So then the the remote server will send the request with the HTTP post with a case session and JSON to the our lightweight Renault plug-in. So in in this architecture the remote NCP every single user will have their own own NCP server that can control and you know interact with the local Rhino smoothly. When it comes to natural language we all know like you know u natural language is quite limited when talk when we want to use natural language to describe things.

13:43 So another thing we made is how we make the you know the NCB server or the agent easily understand the 3D geometry data in the Rhino. So for example the user say want to add two double glaze window on the select worlds. So we actually designed a tool in the random NCP like every time the NCP server generate the geometry to automatically generate the object metadata such as like windows like for example you will have the description you will have the type double glaze and wise and and the view value and then what which is most important the floor which the windows belongs to.

14:29 Actually this give the you know the NC steroid agent a more comprehensive like semantic content. So this actually help benefit by this architecture. This actually help the you know agent or SCP to understand like which object they want to they need to modify and which one they want to change. For example in this case you say change the material of the window at the second floor. So actually we have a automatically filtering tool that can search in the Rhino scene to find find out you know the qualifying object.

15:06 For example, we include the window and the window at the second floor. Actually, we can return all information like qualify windows then back to the NCB server that it can do the next step. And also another challenge for us is how we think about like make the agent or NCP to understand like more users like more comprehensive intention. So for for example when user say I I want to make a previously like rune earlier larger but how the agent understand which room the user refers to and we all know in the you know the traditional NCB client side for example the cloud desktop the cursor they are great they have great like agent memory in the users like conversation prompting but they don’t really if we do the random CP they don’t really know what happened in the Rhino side even you can say you can return a like operation geometry response from Rhino but it’s not enough so we designed a three layer memory system like every user will have their own instance and the memory and operation will store in their instance like in the remote server.

16:28 So in the memory it will include the user prompt like also the agent response and also include a lot of like operation ids. This operation this operation ID will record like the object operation type in the Rhino and also which object they operate. So this actually give the you know NCP or a agent a more comprehensive background like what geometry and make the more the geometry history operation more traceable and here’s example like in a single user like you can how we store this data in our remote server and another thing like if you play with like blender SCP or render SCP before One important tool is extra code which can bring it in infinity like possibility to the your rhino or blender to do the 3D modeling for example any anything you want but we actually in our practice to you know drop in this tool and using in our practice we found really u important the challenge is that these actually this tool take a lot of like arrows also when we because the random syntax is not really popular in the internet which means the agent doesn’t have a lot of like par knowledge about it.

17:54 So we are building a small rack system we get the documentation from the official website of the reno syntax and we use the we use the crow for AI to get the data to the JSON file different kind of JSON file. This JS value will include the method of every single sorry the information of the every single method include the categories method and also and we chunk it into every single method into a into our chroma with the database.

18:32 So every time before the NCP server extrude the Rhino tools, it will look up the syntax make everything more easy make the u response more correctly and we also integrate others with our internal material database to you know generate the material in the Rhino and we drop a NCB NC SQLite NCP server to connect with the Rhino NCP server. We also integrate our air rendering endpoint to wrap that is a tool of part of the NCP server.

19:03 So I will pass to the yeah just to wrap up in just a a minute here. So a couple examples you know in in production I think early on using the MCP server I mean play around with different agent algorithms you know very early on we were doing very basic stuff. This was before a lot of the the the sonnet 3.5 model was out. So it’s it was a lot more difficult.

19:30 But I think what we found from these experiments is just simply like the you know pro like multi-step prompting is extremely important and referencing databases is extremely important which was what Jun Ling was was talking about with the rag system. So in production now we’re we’re we’re we’re building our own databases. So this is for a curtain wall system. So building a databases of of of products and sizes that then the MCP can just reference that and build the curtain wall and and options with that.

20:01 So this is just like our goal of getting this into production and what that takes and you know just finally as a as a final slide the the MCP stuff you know as it relates to form jamming you know these systems aren’t perfect and that’s exactly what we were describing with the the first couple slides and that imperfection of how the tool is operating and what we’re doing with it kind of reminds me of when you know 3ds Max and Maya you know came came out in early 2000s and and the form was generated by applying modifiers and and that as a as a framework and a way of thinking about these AI agents and not necessarily being, you know, this perfect pipeline, this perfect set of tools, but being the framework that we’re constantly iterating and we’re building all kinds of crazy tools for, you know, each individual project. So that’s it. Thank you very much. Too much. Thank you. Thank you very