CDFAM NYC 2023 · New York · 14–15 June 2023

From Simulation to Reality: Computer Vision + Synthetic Data

Abstract

Francis Bitonti, working at the intersection of computer vision, synthetic data, and computational design at Lexset, presentation at the CDFAM Computational Design (+DfAM) Symposium, held in New York City in 2023.

In this insightful presentation, Francis Bitonti delves into the world of synthetic data and its application in training deep learning algorithms for computer vision tasks. With a focus on Lexset’s capabilities, Bitonti highlights the challenges and solutions in the current landscape of deep learning and data annotation. Key areas covered include:

The Hunger for Data: Bitonti begins by emphasizing the insatiable need for data in training neural networks, highlighting the time, cost, and energy involved in collecting and annotating data.

The Bottleneck of Human Annotation: He illustrates the unsustainable nature of human-annotated data, especially with the growing complexity and size of deep learning models.

The Process of Data Collection and Annotation: Bitonti walks through the traditional process of collecting, annotating, and quality-assuring data, pointing out the slow, cumbersome, and security-risk-prone nature of this approach.

The Shift to Synthetic Data: He introduces the concept of synthetic data and its growing importance in filling the gap in data needs, emphasizing Lexset’s focus on enabling developers to build simulation pipelines quickly.

Transcript

From YouTube’s automatic captions, lightly cleaned; expect some errors. Each timestamp opens the video at that moment.

Read the full transcript · 3,487 words

Okay, hi everybody, my name is Francis Bitonti, I’m the CEO and co-founder of Lexset. Given the crowd, you may have also done quite a bit of work in 3D printing, also the author of a book called Design for Additive Manufacturing, so just given, given the crowd, might want to separate those two, I’m here to talk about Lexa and AI today. We are a synthetic data company, we specialize in creating data for training deep learning algorithms that are performing computer vision tasks. So I’m going to talk today about some of our capabilities, some of the things that we focus on, some of the capabilities we focus on providing for our customers, and I’m going to close with a case study. And really the point I want to make is that this isn’t just about getting somebody data, I think this for us is very much about changing the way deep learning engineers work with data, we’re trying to give these teams of people capabilities that they wouldn’t have had before, and create some agility in the process that, that quite frankly isn’t, isn’t there today.

1:15 Let’s just start off and say, neural networks, they, they consume a lot of data, right, just like our friend Pac-Man going ahead munching up little bits, right, they, it needs, they’re constantly hungry for data. If you want to continue to improve the models and iterate on your models, you’re constantly feeding new data into these algorithms, and it’s a, it’s a fundamental, just critical part of, of what you’re doing. And in many cases, well, in almost all cases, this consumes more energy and time and cost than actually training and developing these algorithms.

1:47 I think this is something, unless you’re intimately sort of intimately engaged in the craft of training these models, you may not realize exactly what goes on behind the scenes to get these data sets fed into these models. You know, it takes about, about 20 percent of this process is actually developing the algorithm, the rest of it’s collecting data, annotating data, which we’re going to talk about, editing, augmenting data sets, there’s an enormous amount of data manipulation that happens in order to get the kind of performance that you want.

2:19 And, and I love this quote from the CEO of Drive AI, because I think it, it really drives home sort of like the scale of this issue, right, it’s, it’s for, for one hour of video, and they’re working on autonomous vehicle technology, you’re looking at 800 human hours to annotate that one hour. Now, in creating a computer vision system for autonomous vehicle, you could imagine you’re gonna need tens of thousands, or maybe even millions of hours of video, right, this is a real bottleneck, especially as we start seeing models getting bigger, deep learning is getting more popular, we’re seeing more applications, the, the requirement of human time and labor in annotating these data sets is, is ultimately unsustainable.

3:03 So I’m gonna just walk through this process, because I think this is something a lot of people are often surprised by, what it’s like. So let’s say you want to train a computer vision algorithm, or any deep learning model really, for that matter, first thing is you have to go out and collect a lot of data. We’ve seen some examples today of people using simulation, sometimes you can do that, and you know, that’s our, that’s our focus in emphasis, but most people aren’t doing that, right, they’re going out and they’re, they’re going out and collecting images.

3:29 Once you accumulate all this data, or find some way to collect that data from your users or people in the public, you then have to get it annotated, it’s unlikely you’re going to do this in-house, you’re going to send that data out to some other third party, that’s going to send your data all over the world, you know, and look, this isn’t two people working on it, right, if you’re sending them thousands of images, tens of thousands of images, there’s probably hundreds of people touching this data, looking at your data, putting information in it, it’s going out all over the internet, it’s coming back to you. Then you have to do quality assurance, you have to review the data, maybe run some tests, send it back with your feedback, train your model, and then, oh wait, I have to make some changes, or I didn’t get exactly what I wanted, so I have to repeat this process.

4:12 So this is a slow and cumbersome process that is sort of constantly burdensome for these teams, and these practices, they’re, they’re kind of fraught with some, with some challenges, right. So I kind of touched on a few of these, but one, just amassing more data, especially at a certain scale, you’re not necessarily getting the data points you need to improve your algorithms, so you’re doing a lot of work unnecessarily, and you’re not actually giving the engineers what they need. The other thing is humans have their own biases, they make mistakes, they’re getting tired, they’re doing repetitive tasks that humans weren’t meant to do, so there’s a lot of quality assurance and inspection that has to get done on these data sets.

4:52 These processes often create some security risks, depending on what you’re working on, that may be a really big deal, and then it slows development time, right, because if you want to make changes you have to acquire more data and reduce, repeat this process. A lot of our customers say, before working us, this process can take about six months, and we’ve even had some customers say this takes a year to go and collect these data sets. Bottom line is that artificial intelligence is a very powerful tool, but we need to find some better practices if we’re going to continue to enjoy the pace of innovation that we’re enjoying in the artificial intelligence industry right now, and we’re ultimately heading for a bit of a crisis, right, this is going to eventually just fall off a cliff, because the models are getting bigger.

5:35 You know, this chart’s a little bit old, I’m sure the latest GPT models, you’re probably seeing a couple points way off the top of this chart. So the state of the art is getting bigger, it’s getting more data hungry, at a certain point this really will not be something we continue to crowdsource, and this is why we’re starting to see, and like it’s already been reflected in a lot of the talks that I’ve seen here today, we’re starting to use things like simulation or synthetic data to start to fill in this gap. And there’s been some really aggressive predictions, I mean, I’m, I’m already seeing ratios like this in our customer’s data, where we’re seeing artificial or simulated data outweighing the actual real world collected data.

6:14 And you know what, and again, I’m going to talk about our focus, there’s a lot of people taking different approaches to this, our belief is that we need to enable the developer, we need to give them the tools so that they can build these simulation pipelines quickly and easily, and get their hands on these data sets quickly so they can hit array. It’s about reducing the time to data, we’re not just in this to sell somebody a static data set, the point is, and I’m going to illustrate this through example, it’s about giving you a capability to make your data sets a tool, so that you can enforce certain types of learning in the neural net.

6:47 So to do that, we’ve created a platform that we call Seahaven, I’m going to go through, in just a moment, now, some of the kind of, I think, core values, or things that we focus on providing in the product, but, but again, this is really about giving capabilities to the developer, it’s not just about piping in random data, this is about getting exactly the data you need to accomplish a very specific task that you’re trying to do, or your team’s trying to do.

7:13 So let’s first talk about domain randomization, is this a familiar term to anybody in the room, maybe some, couple, okay, right, no, not ready, okay. So when I say domain, and I say domain randomization, right, let’s just take this, this crowd, right, so I could take a picture of you all, and there’s lots of different people in here, right, so each one of those pixels is, in a sense, a parameter, but we can up level a little bit, you’re all wearing different clothes, your different sizes and shapes, there’s, in, in visual domains there’s millions and millions of parameters, they’re, they’re incredibly complex, but they’re also not random, right, so there is structure there, so depending on what type of signal you want in that data set, you need to be able to architect that structure, and not just randomly throw things around, although that does have utility, and we’ll talk about that in just a moment.

8:07 So one of the things that we’ve built into our platform is the ability to rapidly configure relationships between categories of objects, so what you’re seeing on that side of the screen is, you know, okay, hypothetical scenario, I want to have people standing next to cars for my particular data set, maybe I’m missing or under representing people standing on the right side of those vehicles, so I just want to generate examples of that, a few clicks, I can drag and drop, I can build that structured relationship, and I can save a model of that structure and that relationship, I can easily deploy that, and I can start getting thousands of images that represent that.

8:39 Now, I begin working a little bit more, I realize, ah, now, you know what, in front of the car is being underrepresented, go back to the drop down menu, make a very simple change, and iterate, right. So we want you to be able to continue to modify these models that have structure, which, you know, look, your engineers working in your domain, you are the ones that know the domain best, you know your data set best, so we’re focused on giving the ability to rapidly iterate, make these changes relative to the real data you might be using, or any shortcomings you might see in your model.

9:18 Mouth’s getting dry. So next is sensors and lighting, this is another really big part of what we do, a lot of people doing this work, or you may, you may think, oh, I just scrape the web, I get a bunch of photographs, or hey, there’s photos everywhere, everybody has an iPhone. Well, not all computer vision applications are using simply photos from your iPhone, right, you have very special cameras in the manufacturing space, they put a lot of energy into, you know, advanced optics and special lighting situations, to extract certain types of features from what they’re seeing, so that kind of the myth of data is everywhere, and I just need to go, go get my, my web scraper out, that’s, that’s not necessarily true for a lot of companies. So we put a lot of energy into calibrating that, in addition we’re also simulating other types of sensors, so we have capabilities for simulating lidar, sonar, and now we’re also starting to do radar.

10:09 This is an example of a kind of more, more complete randomization, this is a, this is actually a data set turned into a gif, but we did that so you can kind of see all the variety, right, so in this case we’re interested in tracking these hands, right, we’re interested in, like, tracking these, these poses in American Sign Language, and what you see here is that I have a lot of control again over the randomization of the variables here, I can choose the types of environments that are showing up, I can choose the types of lighting that I’m seeing, in the variety of that lighting, you’re even seeing at the end of this little video, you’re seeing even selecting poses, right, and we can, we can open up that scope, or narrow it as much as we want.

10:48 Yeah, and then the last, the kind of the last sort of prong of this, and the last thing that we need to enable, is that this work all requires that we have really good 3D content, we’ve curated thousands of 3D models of things that you would find all, all around the world, everything from the furniture in your home to, to the cars out in the street, telephone poles, all this stuff. But we need more than just those models, what we also need is, is rich metadata, we need to know what is that made of, right, if I want to switch the sensor modality to radar, I need to know what, what is the material, right, so I can have my software do that. What are some areas on this object where I could, you know, attach other items, or we can interact with, right, so we, we also have, in addition to the 3D content is, is these is this reaction metadata, we also allow customers to upload their own, because you know it’s particularly also in the manufacturing space people have their own proprietary CAD files, and we allow this type of annotation to be done, so that they can interact with our system in this way.

11:54 Okay, so like, let’s try to make some sense of this, and like, what really kind of is special about all this, right, because it’s like, well, they’re just images, but, but again, really what this is about is about control, it’s about operationalizing these data sets so that we can use them in very targeted ways. We’re going to train a really simple, a simple model, it’s going to detect these little bits of hardware and photographs, they’re kind of throwing around the ground, kind of a, kind of a toy problem, but, but it’ll be interesting, trust me, this actually is published, so if anybody wants to reproduce these results, that’s kind of why we chose it as an example. There’s a, there’s a developer blog you can go to, that link there, if I’d known about the QR codes that would have thrown one of those in, but, and then find me later if you want the link, if you can’t make it out.

12:42 Okay, so in a situation like this, as I’ve shown you before, we would start by creating a simulation, in this case we’re just going to randomly drop these items kind of on the ground, really simple physics simulation, we’re going to randomize environments, backgrounds, and lighting, and we’ll produce annotated data sets, no one has to annotate these, you’ll get the, the color images, you’ll get semantic segmentations, you’ll get your bounding boxes, everything you need to train the model in the right format to train the models, so you don’t have to post process it, you’ll run your simulation, you’ll download the data, and then you train your model.

13:19 If my trained AI models, you know, it’s really fast and easy, like this little loading bar, and, and then it works, success. And, and you’ll, if you follow along in the blog post, you’ll see, we got on the first try 98 accuracy against the validation set, and perform really well. But you know, there’s, as a catch, right, we had to deploy this in the real world, real world was a lot more complicated than we thought, it was more complicated than what we captured in our initial validation set, there were conditions that we weren’t accounting for, or couldn’t imagine, because they’re not things that we might encounter all the time, or at least are not aware of it.

13:56 So for example, complex backgrounds like this started causing a problem, we’d get false positives on some of these, these edges that were on the, on the wood embossings, and the accuracy of the model wouldn’t fall by 10 when deployed in real world conditions. So before I show you how we fix something like this, I want to talk about kind of what’s causing this problem, and for those of you working in AI, you know there may be other reasons for this, in this particular case this is what was causing this, this issue.

14:28 Okay, so stepping back, how do these CNNs work? So with these areas, these are visualizations of the filters that are in the different layers, so you’re passing an image through, and then these filters are processing the image data, transforming it so they can extract certain types of features. These filters are formed and learned as they are exposed to data, and they’re optimized as you train, so that they can learn to extract those types of features. I think this is a really great example here, you can, you can see, the like, even like the little dog’s faces are being extracted, right, in the activation, so like, what this is, kind of what they’re able to do, I mean, I’m simplifying things a bit, but this is a good visualization of what’s happening there.

15:13 Now, if our data sets are not representing these edge conditions, right, a lot of the world might, you might have a distribution like this, right, what you see and what you might collect in a three-month period will probably fall mostly in the middle of that bell curve, right, but where your application cannot fail, and, and where you will most, most undoubtedly encounter, is at the edges of that bell curve, right, there’s all these sorts of edge conditions that might occur, if you don’t show that during the training process, the feature extraction mechanisms will not develop in such a way that they can be ready or prepared to extract the right features and make the right prediction.

15:50 So what we can do, and we, this is what’s really, I think, exciting about synthetic data, is that we’re able now to introduce patterns and structures into this data set that could force these, these mechanisms to emerge. So this is a very simple change, we have settings built in for, for randomization, or noise, or certain types of, like, structured noise patterns, and what that would yield is patterns like this, right, this is not something you would most likely, I mean, unless you’re, you’re at a nightclub, you’re probably not going to see lighting patterns like this, right, you’re not going to see patterns like this. But like, even to my eye, it is very difficult for me to find those little pieces of, you know, those wood screws on the, on the table, by, by exposing the neural network to things like this, it learns through the training process the right types of mechanisms that it needs to differentiate those items from the background.

16:42 Adding only 300 images, and a training set of ten thousand, we were actually able to get the accuracy back up 11, which put us in a better place than we were before on the original validation set, and it took really no time at all, versus if I were to try to do this without synthetic data, I’d have to go and collect data, I wouldn’t exactly maybe know what I’m looking for, I’d probably collect more data than I need, spend more money and more time trying to get, trying to get these images, but I was able to very quickly have a hypothesis, test it, train my model, redeploy, and see the results, and I can see that they work.

17:18 So, you know, look, the message I think I really want to kind of leave you with here is that this is something that I think people need to be thinking about as part of their ml Ops ecosystem, right, it’s not, it’s not just about getting a data set that might represent exactly what I see in the world, it’s about getting tooling in place so that development teams can react to challenges and problems with their models quickly, and, and redeploy models and iterate, right, it’s a, it’s a, it’s a necessary tool that could give you superpowers if you embrace it. And what, you know, we really focus on, I think the goals are, is about speeding up development time, and, and really letting people work in novel, in novel ways that they couldn’t before. So that concludes my presentation, I think, thank you all very much for your time.

More from CDFAM NYC 2023

Design Computation Human

Design Computation Human

Onur Yüce Gün · New Balance

Mud y Robots: Keynote Presentation

Mud y Robots: Keynote Presentation

Ronald Rael · UC Berkeley / Emerging Objects

Roadmap to Replicators

Roadmap to Replicators

Neil Gershenfeld · MIT Center for Bits and Atoms

Scaling & Standardizing Patient Specific Design

Scaling & Standardizing Patient Specific Design

Adam Wentworth · Mayo Clinic

Computational Design Ops

Computational Design Ops

Ronnie Parsons · Mode Lab

Overcoming the Design Dilemma

Overcoming the Design Dilemma

Ryan Carter · Johns Hopkins APL

Generative Design & Digital Manufacturing at NASA Goddard

Generative Design & Digital Manufacturing at NASA Goddard

Ryan McClelland · NASA Goddard Space Flight Center

Digital Materials – Geometry and Data Combined

Digital Materials – Geometry and Data Combined

Marek Moffett · General Lattice

Ten Years Teaching Additive

Ten Years Teaching Additive

John Hart · MIT

10 Years of Applying DFAM

10 Years of Applying DFAM

Tim Simpson · Penn State University

MfAM Walks so DfAM Can Run!

MfAM Walks so DfAM Can Run!

Chelsea Cummings · The Barnes Global Advisors

Informatics and AI for Materials Development

Informatics and AI for Materials Development

Wes John Alder · Citrine Informatics

Predicting Fatigue Life in AM Metal Parts Using DigitalClone

Predicting Fatigue Life in AM Metal Parts Using DigitalClone

Behrooz Jalalahmadi · Sentient Science

Designing Material Properties with Advanced Toolpath Generation

Designing Material Properties with Advanced Toolpath Generation

Alex Roschli · Oak Ridge National Laboratory

Design Automation Using Machine Learning & Computational Design

Design Automation Using Machine Learning & Computational Design

Rhushik Matroja · Cognitive Design Systems

Learning to Generate Shapes

Learning to Generate Shapes

Karl Willis · Autodesk Research

Design for Artificial Intelligence

Design for Artificial Intelligence

Chris McComb · Carnegie Mellon University

Register for Updates and Discounts on CDFAM events.