EP 27 1 hr 34 min

Why Defense Gets Humanoids Before Your Home Does: Skylark Labs' Amarjot Singh

With Amarjot Singh (Founder and CEO, Skylark Labs)

EP 27

Episode 27 of the Audrow Nash Podcast.

A four-and-a-half-foot humanoid kicks in a door so a person doesn't have to go in first. Amarjot's case is that work like this is where humanoids pay off now: the tasks vary less than in a home, and working 70 or 80% of the time is still worth it when a person isn't the one getting shot.

I talk with Amarjot Singh, founder and CEO of Skylark Labs, about a model that flags, labels, and learns new things on the device after it's deployed, how they teach a humanoid a skill from a few demonstrations, and why he thinks humanoid hardware becomes a commodity while the value moves to the brain.

You'll like this interview if you work on robot learning or perception at the edge, you follow humanoids, or you're curious how robots get sold into defense. The model architecture stretch (8:51–34:41) is the most technical.

Episode links

Chapters

  • 0:00 Cold open
  • 0:33 Amarjot Singh and Skylark Labs
  • 3:30 Humanoids that sweep for bombs and breach doors
  • 7:23 A brain-inspired model with memory
  • 8:51 Wavelets and Gabor filters as the first layer
  • 13:40 Unsupervised features, supervised on top
  • 17:26 1,000x less data and flagging outliers
  • 21:58 Detecting a camouflaged tank
  • 25:53 Explainability for kinetic systems
  • 28:12 Self-labeling and on-device memory
  • 32:06 From perception to motion
  • 34:41 The robotics stack and sensor language model
  • 36:45 Skills stored as graphs
  • 39:10 Mapping human motion onto a humanoid
  • 40:33 Adapting to a new door handle
  • 43:18 Teaching with Apple Vision Pro and egocentric video
  • 46:58 Simulation and sim-to-real
  • 50:14 What's easy, hard, and not yet possible
  • 52:44 When 70% reliable is enough
  • 55:01 The humanoid Turing test
  • 56:51 Generalization is the bottleneck
  • 1:00:05 From runway debris to robot dogs
  • 1:03:46 Why a humanoid, on two legs or four
  • 1:07:00 Humanoid hardware as a commodity
  • 1:10:56 Why defense comes before homes
  • 1:14:32 Trying markets until someone paid
  • 1:18:40 Building their own humanoid
  • 1:19:48 Manufacturing humanoids in the US
  • 1:22:20 India for data, testing, and talent
  • 1:25:40 Skylark's next three to five years
  • 1:28:21 Advice for robotics founders
  • 1:30:02 Where robotics is going
  • 1:31:57 Final takeaway

In playlists

Audrow Nash Podcast Newsletter

I send one email when a new episode is out: who I talked with, what they're building, and the technical or business takeaways I'm still thinking about a day later, plus the YouTube link.

Or subscribe on Buttondown directly: buttondown.com/audrow

Transcript

Click any line to jump the video to that moment. Machine transcribed and lightly cleaned, so expect some errors. Transcript on its own page, plain text, WebVTT, SRT.

0:00 Cold open

Audrow Nash There's a humanoid kicking in a door with a holding a pistol.

Amarjot Singh Rather than you going inside first, the humanoid can go in. It can go from riot gear to defusing bombs to firing weapons to everything people do. then you replace the human with the humanoid. Can you tell it's an actual human or a humanoid?

Audrow Nash What a world we live in. It's so freaking cool. is kind of the goal you think to make like a super soldier or Hi! Would you introduce yourself?

0:33 Amarjot Singh and Skylark Labs

Amarjot Singh Hi, Audrow. My name is Amarjot Singh, and I'm the founder and CEO of Skylark Labs.

Audrow Nash Hell yeah. Tell me about Skylark Labs.

Amarjot Singh Yeah, so Skylark is primarily a physical AI company, which is focused on building the brain for different kind of machines going all the way from drones to humanoids. And we build it for mission-critical applications all the way from mines to defense. And the key thing which we are focused on is building a brain which has the capacity to learn from limited data. Data is a big bottleneck in all these applications. And then once it is deployed, it's always going to see new variations of the data which you learn from, or totally novel things, and how does the brain identify, label, and learn these new things on the fly. So that's what we focus on, building this self-learning AI brain for machines, for mission-critical applications.

Audrow Nash Shaw my background mm-hmm awesome. hell yeah Before we dive into how you guys actually do that would you tell me some of the details about Skylark, where you're based, how many people you are, your general round of funding and valuation things like this? Just like general details.

Amarjot Singh Yeah, so Skylark is headquartered in Menlo Park in California. We have roughly 70-plus people. So our headquarters is Menlo Park. Then we have an office in Indiana where we do a lot of our real-world testing. Then we have an office in Del Rio in Texas where we deploy a lot of our solutions with DHS. And that will be 23 people total in the U.S. And then we have another 50 people in Pune in India where we do all sorts of real-world data gathering from mines to oceans to everywhere. And then we also have a lot of our engineering and tech stack over there. And we are in our pre-series A. We just finished our pre-series A. We raised a total of 15 million. And valuation is 250 million. We are going for our series A now. We are in the process. We are raising it in two phases. The first one is 60 million and then another 90 million.

Audrow Nash Hell yeah. And can you speak on the valuation for your Series A that you're expecting or?

Amarjot Singh It's going to be close to a billion.

Audrow Nash That's so cool, what a trip. Okay, so with that, mining, oceans, security, and borders and things, some military applications. The common part of all this is the learning on device and correct. Can you tell me about that? Like what is kind of the core approach that you all are using on all these different platforms?

3:30 Humanoids that sweep for bombs and breach doors

Amarjot Singh Yes. So I think the core idea is that for a lot of these applications you want to do certain mission sets. And those mission sets in humanoid can be all the way from scanning for bombs. And I can also show you a video of that. So you can see right up top you can see this humanoid scanning for the bombs. It would pick up the metal detector. It would go and scan. Or we can have share another application. Make it easier. So you can also see our... A little bit jumping around.

Audrow Nash No worries.

Amarjot Singh So you can see our humanoids learning how to engage weapons. They can also... Yeah, you could

Audrow Nash Humanoid with a paintball gun is awesome. That's so funny.

Amarjot Singh You can go all the way from dealing with the shields, which is mainly riot gear, to scanning for bombs to logistics. And then the real world mission sets which we are looking at, this would be one of them. Where you want to basically take human out of danger. So if you had to actually get a person out of this kind of application where you have to engage the weapon. A humanoid can get that mission set. Or this one where rather than you going inside first, the humanoid can go in. Sir.

Audrow Nash Just for people that are listening, there's a humanoid kicking in a door with a holding a pistol. It's a little humanoid. Like it looks like it's like two thirds as tall as people. Which humanoid is this?

Amarjot Singh He is four and a half feet. Just a little guy.

Audrow Nash Yeah, yeah, but it's just, I mean, it's like pistol-butts the door to get the door open in the video, which is awesome.

Amarjot Singh Yeah. But the longer version of this is

Audrow Nash What is the humanoid you're using in these shots?

Amarjot Singh Quite interesting because in this mission set or let's say Where you are holding the shield and you are hitting or when you are grabbing the metal detector or even the box. When you give this task, you are going to be like, okay this is how I use my manipulation to grab these things. But when you pick the pole, it is going to actually apply force on to your elbow because the pole is heavy. So it is going to drag down. Or when you have the shield and the person hits on the shield, that is going to actually have external force. And your shield is going to move and the objective of the humanoid is to keep it in a certain way. So that is where the real world adaption would be needed. Or in this kind of setting where you are pushing the door which could be heavier, lighter. So when the force comes.

Audrow Nash Mm-hm. There's a lot of variation for sure.

Amarjot Singh Yeah. But also when you move.

Audrow Nash Could have stuff behind it. Could stick.

Amarjot Singh Yeah, exactly. And when you move, you have to move a certain way. So you have to go and you have to make space for the other person to come in. So the way the government wants to deploy this is in a human machine tagging fashion. So you are not working on your own and learning. You are also doing group dynamics. So these are some of the mission sets we are doing. So as I explained, in all of these settings, there are new kind of variations which can come up. And then your machine has to first of all estimate error that, okay, there is a deviation from what I am supposed to do. Which kind of ties to self-awareness because none of the AI models actually have that. And once you are able to figure out that, hey, I am doing something which I am not supposed to, then it corrects. So that is the basic idea. And the way we do that is this architecture which you see on the screen. So the basic architecture which people use, that is an end-to-end supervised architecture or some sort of transformer. And you train it on a very large data set and then you deploy it and then you hope to god it works. But in our case, we have a basic architecture, a foundational model which is inspired from the brain. So it is not end-to-end supervised. It is handcrafted and semi-supervised and supervised giving you efficiency on data and compute. But the secret sauce beyond that is the memory. Whatever I am telling you right now, you are able to get right away because of this part of the brain called the hippocampus. And we have an equivalent of that, the memory where all of this magic happens.

7:23 A brain-inspired model with memory

8:51 Wavelets and Gabor filters as the first layer

Audrow Nash Okay, tell me more. So with the, I guess, I'd love to go through that a bit slower to understand how that works. So I, like

Amarjot Singh Sure. Yes.

Audrow Nash Tell me about the foundational models and how you're approaching this, and how you're doing some supervised or semi-supervised. Like I want to understand the full system in some sense.

Amarjot Singh So let's think of a simple example. You want to build a system to recognize the perception side. Everyone understands more or less these image data sets where you can say a dog is a dog or a cat is a cat. if And generally we would take that data set, push it through this black box network and then it will learn these.

Audrow Nash And they're all labeled, so therefore you make distinctions because it says this is a cat, this is a dog, and you propagate it back through the network and everything. Okay, so that's like normal machine learning to me, like neural networks, convolutional neural networks. Okay.

Amarjot Singh Exactly. The standard. Yeah. Yes. So what we do is we would take that whole data set and we will push it through this first layer, which is the battery of Gabor filters, which is extracting edges at different frequencies and scales. And we have transforms. They're called dual-tree complex wavelet transform. So you can go from your input pixel space to the edge space with no loss of energy. So they are completely

Audrow Nash Oh, okay.

Amarjot Singh Transformable. That's a lot of my work, which my professor and I did during my PhD. Once you have that edge space that is nicer.

Audrow Nash And wavelets, I haven't worked with wavelets before, but they're just like a basis set, they're a way of representing things, is that correct?

Amarjot Singh That's pretty much it. So in the image, when you have an edge, it's basically a gradient going from right. And you just want to pick that.

Audrow Nash Yep. And is the key thing of wavelets that they're smooth and thus gradient descent works better for optimizing or

Amarjot Singh You can.

Audrow Nash Why use wavelets as opposed to anything else?

Amarjot Singh So wavelets allows you to capture these edges at different orientations, different scales. But the way you build the wavelet, you can try to see when you want to cut the lobe of your let's say think of it as a Gaussian and you can cut it beyond sigma, two sigma. You can make it fatter. You can make it smaller. Allows you to Yeah. So allows you to capture different kind of edges rather than just an edge. So it is just more

Audrow Nash Mm-hm. Uh-huh. Change your variance, yeah.

Amarjot Singh Diverse if you would. And people design it in different ways to capture different kind of things. But the advantage of the ones we are using is that you can go from input to the edge space with no loss of energy. That's the most important thing.

Audrow Nash Okay. What do you mean by that? I don't fully understand the loss of energy and to edge space. Edge space, do you mean on device?

Amarjot Singh So you take an image. You can push it through a standard edge detector. Let's say Harris or something like that. It's going to it's going to capture some of the edges. It's going to get rid of all the smooth areas, things like that.

Audrow Nash Yep. Yeah.

Amarjot Singh But when you're trying to do the Gabor filter or wavelet filter, whatever that is. You have your initial input signal. You can compute the methods to compute the energy or how much signal is there. And then you transform them onto your edge space or signal space. The strength of the edges which are there, you can infuse that same amount of energy across the whole set. So when you sum that signal strength, the signal sum is same in the input as well as output. It just gets distributed to where the edges are rather than everywhere.

Audrow Nash Edges are. Okay, so what I'm hearing with this is it's an effective compression of your input data effect.

Amarjot Singh Yes. Perfect. Yeah. Yeah. You're redistributing the energy to where it's relevant.

Audrow Nash Okay, so then, yep, so you're keeping it so you could, maybe you can't back out it, but all of the richness of the information is still there.

Amarjot Singh Exactly.

Audrow Nash Okay, and then so from that, it's effective, basically very effective compression of the data that you're getting. How does that relate to the whole on-device learning approach that you're doing?

Amarjot Singh So right now we are building the base model or the foundational model. So the first there is how do I get from my input signal to this edge space where I am basically cleaning the input signal. Once that gets done on top of this very rich edge space, everything in the world is made up of lines. You would put semi supervised learning. You can think of as a convolutional autoencoder.

Audrow Nash Mm-hm. Yep.

13:40 Unsupervised features, supervised on top

Amarjot Singh There are many ways of doing that. But you're taking these edges and multilayer without any labels. You're trying to find common patterns across all of them. So a car. Yeah.

Audrow Nash Patterns and everything for efficient data compression again. Mm-hm.

Amarjot Singh Yeah. So your car would have bonnet, windscreens, tires, things like that. And you find commonalities across the whole data set. And then you do a truncated supervised learning.

Audrow Nash Yep. So, okay, just to me, I'm thinking of when we've done stuff with convolution, or some convolutional neural network stuff that I've seen is quite cool because by hand, initially doing computer vision, you would have your edge detectors and then your feature recognizers and you'd go and look at relationships between features to figure out things. But when you do it with convolutional neural networks, the thing that was cool is you start and you identify like edges. Like if you look at the first images that are made and then eventually they get less and less abstract to eventually be like tires and like full things

Amarjot Singh Yeah. Yeah. Yeah. Yes. Yeah.

Audrow Nash That kind of match in the image. And then when you slide these across the image, you get to see, oh, there's probably a tire here and then you look at the relationships of all the features and that's how they detect things.

Amarjot Singh Yeah. Yeah. Yes.

Audrow Nash What I hear you saying with this, or what I believe you're saying, is you are doing a data transform on the data to make it more efficient and then you're putting it in like a convolutional neural network setup to generate features on your compressed data effectively. Is that right?

Amarjot Singh Yes. Yes. So yeah. So the beauty of convolutional neural networks is we don't have to tell the machine

Audrow Nash You don't have to do your old feature detection. Yeah, yeah. It's much better.

Amarjot Singh What features. What. Yeah. Exactly. So yeah. So the machine can just learn it. But then you also have to like.

Audrow Nash Don't need SIFT or SURF or any of these.

Amarjot Singh Yeah exactly. But then you also have to learn a lot of those additional features. But I think at least on the initial piece where you have low level features. We know if you are able to get to a good enough edge space you don't need to learn them. And then if let's say you're.

Audrow Nash That's what you do with your wavelets and things. Is that right? Mm-hm.

Amarjot Singh That's what we do. Exactly. And then you can combine those edges together to form corners and mid-level features.

Audrow Nash Everything else. Yeah, yeah.

Amarjot Singh Yeah. And then also this is common across all of them because if I tie a label to let's say a car or a truck you're learning a lot of redundant features across all of them which you can later on get by compressed by doing the optimization and all. But you don't need to learn that. There's a lot of commonality. So we do unsupervised stuff there. But then later on where you have to associate that the bonnet is next to the tire it's next to the windscreen and now it's a car there you need supervised learning. So we would bring it in the later layers. But then that gives you massive efficiency on the data and compute. But since we are building them in parallel I will build the wavelet thing then unsupervised learning. I would also compress it to make sure that I don't have any redundancies in my feature maps. Then I would supervised learning and that leads to smaller models.

Audrow Nash Cool. Okay, so what I'm hearing with this is it's a lot of data transforms and compression for very efficient data processing effectively. Okay, so the separation of things you're doing leads to the ability to be very efficient with data and this is kind of, so that's really cool.

17:26 1,000x less data and flagging outliers

Amarjot Singh But we require roughly 1,000x less data and we have multiple papers on this when you're building this thing and this starts to become really really effective when you're trying to do temporal stuff like video or robotics.

Audrow Nash Mm-hm. Oh, very cool. Yeah, yeah. Because then you can, you put your powerful features into something like a transformer architecture or something where it feeds it back and because you have small representations for things, you're able to do a lot and maybe add a temporal dimension. Is that right?

Amarjot Singh Yeah. Exactly. Yeah. Exactly. But here is the thing and here's the thing which is the most important thing. So

Audrow Nash That's really cool. Yeah.

Amarjot Singh When you build your foundation model it's built so someone built it using end to end transformers. I am saying I don't need to learn the low level middle level. I only need to learn associations at the high level. So whether you like this process or not we build this thing. But now when you see something new in the real world rather than let's say having a face which has eyes next to the nose and nose next to the lips now you have a weird kind of face. Somehow someone is trying to spoof you where you have lips replaced with one of the eyelids.

Audrow Nash On the forehead or something. Mm-hm.

Amarjot Singh Yeah. Sorry.

Audrow Nash It's okay.

Amarjot Singh If you do that now if you push that through the network the relationships are not there right now. So you can go through your low level middle level and when you get to the high level it's going to be like well this is not really what I saw. So this is not really a face. I am very low on the confidence score and it will reject it. And a lot of these outliers where you have a tank which is partially camouflaged or you have a new kind of cyber signal where someone is trying to attack you are all of the things which are beyond two sigma. So they still have a lot of the middle level properties but different relationships. So we use our middle level layer to automatically flag a lot of these outliers which then serve as a funnel to push stuff into our memory.

Audrow Nash Cool. Mm-hm. So if I understand, so that when people make a foundational model, they kind of, they go from inputs to outputs and you have nothing, everything is learned in between. Is that,

Amarjot Singh Yeah. Exactly.

Audrow Nash Okay. And then so what you're doing is you're having several divisions through there where you have your wavelet transform and then you have your autoencoders and then you have all these other things. At each step, you're passing things to the next stage and eventually you get to outputs, but you have all these intermediate weights and statistical information and everything about each layer. So then that is very good because you can access it anywhere that you can put it forward into another part of your network or something and then identify this is a camouflaged tank or something.

Amarjot Singh Yeah, exactly. And you're not going to be able to do that from the later layers because in your traditional methods it's input output.

Audrow Nash They would already have failed. Mm-hmm.

Amarjot Singh Exactly, exactly. And that is where you...

Audrow Nash Because you don't have the data

Amarjot Singh Exactly.

Audrow Nash For the new approach. Okay.

Amarjot Singh Yeah, and a lot of our proprietary stuff is how do you design this base architecture which is patented? And then how do you use the mid-level features to flag emerging signals? And we do this all the way from a lot of our work with the intelligence organizations where you can have a new kind of signal which is online, all the way to drones picking up new kind of tanks to a lot of the false positives. That's where a lot of our value has come up because when we started deploying them initially we were deploying them on the drones. And one thing is you want to detect very small targets on a fast moving asset, you will always get a lot of false positives. So what... Sorry, I'm doing something. So what big advantage we got on the memory side initially was how do we get a reliable AI system for these mission critical applications which just doesn't make mistakes. And this memory's first value prop was, we have the lowest false positives in the market.

Audrow Nash Yeah. Tell me how this leads to like, I see sort of how this could work, but I don't fully understand it. Let's go with the tank example. I think it's a cool one. You have a camouflage tank. It's not in your training distribution.

21:58 Detecting a camouflaged tank

Amarjot Singh Okay. Yeah. So

Audrow Nash How do you, how does this work through the network? What are we actually doing when we, oh, there's some tank like things. Like I see, I see tracks, but the rest of it doesn't look right. Or I see that one, the barrel of the,

Amarjot Singh Yeah. So yeah, so let's say you come to me and you say, "Hey, I want to build a tank detector. I'm building a kamikaze drone, real world use case, and the drone is going to fly, it's going to find a tank, it's going to go hit into it. So I'm like, "Wait, I need to first see what a tank looks like." So we get a lot of images of these tanks. We train our hybrid model. It's built

Audrow Nash I don't know what any of the tank parts are called. Mm-hmm.

Amarjot Singh After that, and there's a memory attached to it. Once the drone is flying, if I didn't have my custom architecture, I would push, let's say, a tank which is partially hidden, the AI will miss it, the drone will not hit it, story ends. In our case, the pipeline is two-fold. You take the image, you push it through the whole network, and you say the end of the network, which is supervised bit, does it say it's a tank or not? That's great. And if it says tank, no problem at all. If it says no tank, we put it aside. But there is also a pipeline which is learning from the middle part of the architecture. And when you push an image, you're also doing matching of the mid-level features, which are Class A, Class B, Class C. And you say, "Hey, it has some features of Class A, some features of Class B, or 90% features of Class C." And based on how you decide, you can say, "Hey, it looks like a tank. Let's go and hit it." So there are validation methods on top of that. So you would run these parallel pipelines to do this.

Audrow Nash Okay. And that's where you're saying the two Sigma approach. Like basically you are looking outside of your regular confidence interval. So if it's, if it's under a Gaussian distribution, in the middle, we're very certain it's a tank. But then when it's two Sigma out, it's like, we're

Amarjot Singh Yeah. Yeah. If So

Audrow Nash 93% sure it's not a tank or something like, or like, I don't know, it's over at the 93rd percentile area.

Amarjot Singh If you were 90% or above, your supervised network would have picked it. If it is partially covered or something, you don't have a lot of the features. You have some of the features. So that would push it beyond the mean. And then the only way you can use that is to have some sort of network, which is specifically tuned to pick partial features. And what are those partial features, which are good enough? The mid-level features, which my autoencoder has learned. So then I would do basically template matching. And I would say, "Ah, here's a tire which is glowing. Here's a track which is glowing. Here's a barrel piece." But nothing else is there. So it must be useful. So we flag that. And then you can say, "Out of all the parts you have learned for the tank, if there is a barrel, then you must flag. If it's a tread, you must flag." So you can also put more weights.

Audrow Nash Mm-hmm. Yeah. Cool. Mm-hmm. Yeah, yeah. So Oops. so you can, this is where it's somewhat supervised or you can tune your own weights in some sense when you say, for sure a barrel means tank for this. Like if it's, if you see wheels and you see a barrel, it's like, that's enough. It can be camouflaged,

Amarjot Singh Yeah. Yeah. Yeah.

Audrow Nash But.

Amarjot Singh And we give this power to the user through our platform. So we would say that, "Hey, this is the category." And if you want to put more weight on ABCD, you can do that. You can also do combinations. You can say, "Hey, there's a new kind of thing which has come up, which is partially tank, partially whatever. And if you see two parts of the tank, five parts of this, definitely flag that." So you can also build combinations.

Audrow Nash Yeah. Okay. What's the, was this a challenge from a user interface point of view to make it so you can expose this kind of like mid-level information to users? Or do you have like, it'll be like a picture, like a kind of blurry abstract picture of like a tank tread and a blurry one of like a tank barrel. And they say, oh yeah, this one looks like it and this one looks like it. But I'm imagining that is a challenging thing to get right for users in the field for this kind of thing or how does that work?

25:53 Explainability for kinetic systems

Amarjot Singh So the reason why you want to... So two ways. This can happen automatically as well. The reason why we show this thing to the user is because explainability is a big part of our systems. Because we are protecting kinetic systems, so we don't want to go and eliminate something if we cannot have it.

Audrow Nash Mm-hmm. What do you mean kinetic systems? Just they're moving or do you mean, is that like a battle term or anything? I don't know.

Amarjot Singh So kinetic systems in defense world would be you want to go and hit something. In this case, the drone going and hitting the tank. So if you want to actually want to do kinetic, or if you want to go and actually blow something in this simple term, you want to know what you're blowing and why. So a lot of our models later on also go into decision making, that we are building systems which are right now perception only to action only. Action should be auditable. How do you actually then go and take action? So this decision making chain for us is very, very important. So flag something just because you say a tank or not tank is not useful enough. So this two layer pipeline would also get to say, "Well, I think it's a tank because it has these kind of properties and that's why I'm flagging it." We do a lot of that for viruses on the internet as well. This is the ABCD virus. It has these kind of properties which are underlying. So that's why I flagged it. So that's another reason.

Audrow Nash Okay. Something up. Yeah. Mm. Okay, very cool. So that kind of approach is like the secret sauce of Skylark. Is it correct? Or is there like, or is it one of many approaches? Because there's the mobility, it's the first part. What are the other parts? Where you,

Amarjot Singh The first part. It's the first part. Yeah, be good. Go on, please.

Audrow Nash Yeah, yeah, no, I'm just trying to see how it all fits for all of this. Okay, so that's the first part.

28:12 Self-labeling and on-device memory

Amarjot Singh Yeah, so the final objective is, okay, if I know something, I want to flag it. But if I don't know something, it is something new. I want to flag it, but I also want to label it. I want to learn it. And I want to make sure I can do learning on the device. So most of the other systems which are doing some sort of few-shot learning, someone has to teach them. In our case, there's no one to teach. So the first part of how do you flag something new becomes very relevant, which is the bit I just told you about. The other thing is, how do you label it? Is it a subclass of a tank? Is it a new kind of a tank? Is it a totally novel category that you decide based on these mid-level features as to how many are there in this new kind of signal you have flagged? And that leads to the self-labeling aspect of this thing. And once you have identified and labeled, you will, for the simplest explanation, extract a feature vector for this new thing, store it in the memory, and then next time you see it, you would know what it is. And you can generalize from that. So we have these generalization functions. So the best way of thinking about it is you have a Gaussian. You are in the center. You know how to answer that because of our base foundational model. Then you flag something which is a 2 sigma, and then you want to generalize and extend your generalization from that. And you keep on adding these things, and you keep on extending.

Audrow Nash Pull that.

Amarjot Singh But the beautiful thing here is from that 2 sigma, you can find another 2 sigma, another 2 sigma, and you can keep on doing it. So there is expansion of capability. But you don't want to learn noise, necessarily.

Audrow Nash Yeah, that was gonna be what I was gonna ask next.

Amarjot Singh Yes. Yes, so we have mechanisms to flag what is relevant. That can be done model level, but--

Audrow Nash You like a neural pruning kind of thing where you're like, oh, I keep seeing this one and I don't see this one. Therefore, this one's relevant. Keep it. What?

Amarjot Singh Frequency recency. Yes. Frequency recency. If you don't see something for a while, you can prune it. Whatever you see, you can keep it. But if you see something for a while, your memory is still going to start to clog, and you're on the edge device. And then there's another compression mechanism which happens, which will say, I have n vectors, which are being used again. Let me fit a Gaussian on that. Replace the mean ones with the mean, but the outliers which are beyond 1 sigma. Let's keep all of them so that I don't lose information, but I can also compress.

Audrow Nash Are you constantly compressing to a vector of features for this kind of thing? Or how do you represent all this? Like I'm imagining you have data coming in, you have your distribution, you're one sigma away from the mean, and you say, okay, we keep these feature labels. And then that's very efficient because you're just saying, oh, it's these features as opposed to keeping the original data.

Amarjot Singh Yeah. Yeah. Yeah. We are not keeping the original data. We would keep feature vector only. That's one. And then when you keep on adding more and more, it's again in the feature vector domain. So-- doing

Audrow Nash Yep, and you change your distributions around and you can label them and yeah, yeah, new center of mass, here's a new distribution.

Amarjot Singh Exactly. Exactly. Exactly. When you're doing, let's say, robotics, and we are trying to do this humanoid, trying to learn a new kind of graph. Let's say whether it is trying to pick a pole or trying to do crawling. Someone will show it once, or it can also learn that from its reasoning model. Then you would add a single graph, which is temporal, and you generalize from that.

32:06 From perception to motion

Audrow Nash Okay, I'm not fully all the way there on how this translates into like

Amarjot Singh The body.

Audrow Nash Motion for this kind of thing for robots. I understand it from like a labeling

Amarjot Singh OK. We'll get to it.

Audrow Nash Data, labeling sensor input data. But so that's kind of the key. Your efficient data compression approach where you have the ability to grab from the middling label layers and then kind of redirect them to reevaluate things because you may have only, you may have a camouflage tank. And then the second part is compressing everything down to kind of those mid-level features so you can learn a distribution of what you're seeing in the real world.

Amarjot Singh Yeah. Yeah. All right.

Audrow Nash Does that relate to motion? Or is that kind of, is that the key part that you would make sense that's like the largest problem you're solving? I know that there's a ton of other subdomains that you're interested in, but is that kind of like

Amarjot Singh Look at this.

Audrow Nash The core of most of the things we're going to be talking about today?

Amarjot Singh Yeah. So the same architecture applies to the whole stack. So after this, I can take you to the stack. But if you see the diagram on the screen, so you can see as the architecture builds, it does your-- wavelet.

Audrow Nash Yep. Mm-hm. You get your edge detection

Amarjot Singh Yeah. Wavelets.

Audrow Nash And you have your like middle feature detection and then you have your like object detection from that. So yeah, I see your three layers.

Amarjot Singh And then-- Yeah. And then you have the--

Audrow Nash And I know that that's just a representation. You probably have all sorts of intermediate layers between everything, but.

Amarjot Singh Yeah. Yeah. So coarser level, if you want to see, it's handcrafted, unsupervised, and supervised. So you can see three blocks. And then you can see the memory bit below, where you have a bunch of vectors you add. And then the compression is, let's keep the mean with the outliers so that the memory gets compressed. So that's the idea.

Audrow Nash Mm-hm. Yep. It's a very cool approach.

Amarjot Singh Yeah. But when you get to the stack part of it, so the stack would be, let's say you have this kind of data on which you want to build the model, which is how do you learn motion, how do you learn manipulation. There are a bunch of things

Audrow Nash Mm-hm. Yeah.

Amarjot Singh Happening here.

Audrow Nash Yeah, so tell me how that works. How do you, how do you relate it? How do you do a, like, if you watch a person, maybe you do like a skeleton detection kind of thing. And then you use that as a feature representation and you look at like several frames of that put into one vector and that's how you start to learn a skill or how does it work?

Amarjot Singh Yeah. Yeah. Yeah. Yeah, let Yeah.

34:41 The robotics stack and sensor language model

Audrow Nash I guess you're saying, yeah, so. Yeah, but go ahead. Tell me about this.

Amarjot Singh Let's get into the-- Let's get into the robotics stack. Yeah. So let's get into the robotics stack. Right? So what we are doing in the robotics stack is you give a task, which can be all the way from scanning for the bomb or firing a gun or picking a box, whatever that is.

Audrow Nash Mm-hm. Which is so crazy. Yeah, what a thing.

Amarjot Singh Yeah. So when you give a task, there are a couple of things happening. Right? So it goes to a reasoning model, which could be part of the foundational model or it could be a separate stack, where you say, hey, this is a task. Tell me what to do. What is my high level plan?

Audrow Nash Mm-hm.

Amarjot Singh Now, in our case, generally people will use a standard VLM, a ChatGPT, or a Claude to do that. And they are mainly reasoning over vision. You give a language. You'll give a picture or a small video of what is in front of you, usually a picture. And then ChatGPT will say, do this, do that. And then eventually you'll be able to complete the task. In our case, we are not reasoning just over vision. We are reasoning over vision, audio, gestures, methane gas detectors, gas sensors.

Audrow Nash So various sensors, yeah.

Amarjot Singh Exactly. So we have our custom sensor language model. Let's say that is the first thing we want to build. So we will take the architecture, which I have been explaining to you, which is here. We will take all of that data and we will build this very nice foundational model so that if you give me a language along with a bunch of these sensors, it should produce at the end what are the things you should do. That's pretty much it. Now, if you have a new kind of reasoning, which I have not seen or a new kind of condition, you want to learn that in your memory. And that is where you add vectors corresponding to it. Make sense?

Audrow Nash Mm-hm. Yes.

36:45 Skills stored as graphs

Amarjot Singh Right. And in case of action, once you have understood that, you want to see and you want to act. So the see part I told you, the act part is you're trying to learn these kind of models. You're trying to learn the whole body control and then you're trying to learn manipulation. So let's say the whole body control, we would gather a lot of data for different kind of graphs we have. And then we will train our base model, which is there. And if you ask it to do that, it will do it. But if let's say you tell it to do a new kind of graph where this person is, let's say, doing this kind of action, we will record the skeleton and we will condition the skeleton on our base model so that if you're doing like a loose kind of recording, you can kind of adjust that so that it is sensible. Then you store that to the memory, which is basically a bunch of graphs recorded at 30 frames per second. And let's say your action lasts for five seconds. So you have 30 into 5, 150 graphs. Now, if you say do the same thing, it will just call that graph and it will do it. But let's say now if you put an obstacle in front of it or something like that, then it would pull that graph, the model will predict what actions you should take given this obstacle. It will modify the graph and then execute.

Audrow Nash Mm-hm. Cool. Okay, tell me about the graph and how that relates to the temporal nature of actions for this kind of thing. Like, what is the graph?

Amarjot Singh Yeah. Yeah, so the graph is basically whatever degree of freedoms you have. So in your humanoid, you have whatever 20 plus or 25 plus degrees of freedom. If you look at the example of the hand below, it's a five-finger hand for us. So you have whatever degrees of freedom. So when I'm recording, initially I'm more or less recording what are the motor values for each of them? Yeah.

Audrow Nash Awesome. For everything. So you take the pose tracking, the skeleton, and you infer the joint angles from everything. Some coordinate system that you've established. You map it into that.

Amarjot Singh You're right. Yeah. Yes.

Audrow Nash And then, okay.

Amarjot Singh And map from there to the motor functions so that if I have to do this.

Audrow Nash And map from there to the motor functions. How imp how difficult is that, I guess? Because humanoids, they're shaped kind of like people, but they might be able to move in different ways. Also, if the one is four and a half feet tall, it may have to do the task in a different way than someone who's six foot six or something. how tell me Tell me about how difficult that mapping is.

39:10 Mapping human motion onto a humanoid

Amarjot Singh Yeah. Yeah. It's a very good point. So you can learn that mapping. That's what we do. So for example,

Audrow Nash Yeah. That's such a cool point.

Amarjot Singh Yeah, because like, let's say you're trying to do certain action and we will record your graph and then we will record a machine doing the same thing. And they have that graph and you want to learn this model in between. Yeah, exactly. Minimizes the distance between the graphs and we would do a bunch of that. And that more or less acts as a mapping function so that if you do it, we will push it through that mapping function so that when it goes on to the motor side, it is how it should be.

Audrow Nash Relationship. Mm-hm. How do you... I guess, how does it work if something is I guess, how does it work for slight variations or variations in like, say you've trained a robot how to open a door. Now you get a slightly different door handle than you've ever seen.

40:33 Adapting to a new door handle

Amarjot Singh Yeah. Yeah.

Audrow Nash Does it just throw its hand up and doesn't know how to do it? Or how do you deal... Because you're talking also about learning on the job. Does it always need to see a person do it the first time? Or what would happen? Like, okay, we've only trained on round door knobs. Now it's one with a lever. How does it work? Or that it's like a ship door to open where you have to spin the thing. How would we deal with those?

Amarjot Singh Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah, that's great. That's great. So let's say you teach it how to grab a specific bottle. Here is an example at the bottom right where you are telling it to more or less grab a bottle. So we would tell it grab a bottle and you can either put an object in front of it. So it has to go around it or you can put it at a height. And now there is variation in height. So it is able to do a lot of cool things because we don't learn a pixel to action policy.

Audrow Nash Yep.

Amarjot Singh We learn a geometry to action policy. So we use 3D data, whether it is coming from a LiDAR or a RealSense camera. So if you put a bottle somewhere else and if you recorded one trajectory, I can add and subtract that original trajectory to say, hey, this is where the bottle was. This is where the bottle has moved. So I should do this. And that is what you can see that curve getting formed at the bottom. And since we have this also skeleton, which is in the 3D space, you can also add and subtract to that to adjust it. Now, how much can you do? It's a function of what you have in your base skill. If you have a certain way, the hand is and you want to add and subtract little things to that, the algorithm can do that. It's basically a key point transform. It's basically a graph neural network. So it will modify the original graph. But if you have the need to have a totally new graph, we call our sensor language model, which will then generate a graph which you can add to your memory and then generalize from that point.

Audrow Nash Cool. Nice. Okay. I really like the approach of splitting it up. So you have the trained version and then you have a memory, which is like a local way of executing or perceiving things. I think that's a very clever split. And then you can take the data from the memory and aggregate it on there and then incorporate it into your model, which is quite cool. How do you, for getting the pose of people, are you mentioned needing 3D information for this. I'm wondering, can you use a pose detector on a 2D image and try to, like, are you able to scrape YouTube, for example, for learning how to locomote?

43:18 Teaching with Apple Vision Pro and egocentric video

Amarjot Singh Yes. Yes. Yes. Yes. So. the Yeah.

Audrow Nash Because the pose detectors are quite good lately, I think. And so you can learn from... Mm-hmm. Tell me about that.

Amarjot Singh Yeah. So yeah. So I think the way I mean, one thing is that we want to cut the data bottlenecks. So we want to do few-shot learning. So we want to have a very big base model. We want to have a memory so you can add new graphs to that. And then we want to either do it automatically using our Oracle so it can estimate error, small or large, and produce new graphs automatically. Or you can teach it. Now when you teach it, you can either use

Audrow Nash Mm-hmm.

Amarjot Singh The Apple Vision Pro, wear those glasses.

Audrow Nash That's amazing. I haven't, when I saw it come out, I thought it was such a good product for robotics. Like, it was just like, this, Apple is getting into robotics was my big thought. And then I haven't seen any companies using it really well. And that's really cool that you guys are using it. So yeah, tell me about that.

Amarjot Singh Yes. Yeah. Because then you can have a one to one mapping of how you're looking to the machine. So it's more accurate. But if let's say you deploy this thing in my home and my mom would say, hey, can you do this? Let me show you. She's not going to wear the Apple Vision Pro. So we use a lot of egocentric data, which you can see on the top right where this person is demonstrating.

Audrow Nash Yep. Mm-hmm.

Amarjot Singh And the machine can do the same thing, but also it can adapt to a new graph. It is all one shot. So yes, you can show it. We can get a 2D pose and then we can map the 2D pose to the machine pose and do that. But you need to have a mapping function in between, which is trained. So we have a lot of the demonstrations.

Audrow Nash Hmm. Mm-hmm. Yes.

Amarjot Singh We have a lot of the similar demonstrations the machine is doing. And then we try to minimize that difference.

Audrow Nash That's clever. It's clever to me. I mean, it's something, I guess, you would think about if you were trying to do this, that you need a mapping function of how to go from a human to whatever humanoid or robot you're doing. Does this, so I know humanoid is becoming a bigger focus. Is this also true for like, I saw some robot dogs when you were giving me a tour earlier. Are

Amarjot Singh Yes. Yes.

Audrow Nash You able to map onto different morphologies for everything? Like, can you map from a human to a robot dog? Or how does all that work? I guess you can just map a part of them. So if you have an arm strapped to a robot dog, you can map to the arm of a human or something. Or how does that work?

Amarjot Singh Yes. Yes. Yes. So two parts, right? One is the whole body control and the second is the manipulation. So on a lot of our robotic dogs, we would use a similar kind of arm, maybe shorter, longer. Try to use a five finger manipulator in all of them. So easy to use that same manipulation model. Yeah, and different kind of things. The whole body control differs because humanoids are different in shape, obviously, bipedal versus quadruped. So

Audrow Nash Mm-hmm. So cool.

Amarjot Singh That is where the change comes in the locomotion policy. But you can do the manipulation policy with small modifications.

46:58 Simulation and sim-to-real

Audrow Nash Mm-hmm. Super cool. Tell me how much of a role does simulation play in what you're doing? Like, are you doing a lot of like whole body control tuning and like control figuring out in simulation? How do you leverage real data and simulation? Like, what's the, what's your pipeline look like?

Amarjot Singh Yes. Yes. Yes. So we do quite a bit of simulation. We mainly use NVIDIA's Isaac Sim and Isaac Gym. Sim has more or less the Gym part as well, where we are modeling forces. So if I want to push my humanoid or I want to teach it that if I push you and you fall down, and you should learn how to get up, that is very hard to do all the time in the real world because you're going to damage that humanoid and it's a $70,000, $80,000 machine. So we would build a lot of jumping from one point to the other. How big of a ditch can you jump? Can you actually recover if you fall? So all of that is simulation.

Audrow Nash Mm-hmm. Okay. Mm-hmm. It's awesome. Ah. That makes sense. How, so when you go from simulation to real robots, is there any gap there? Have you guys figured it out? The way that I've heard that people figured it out pretty well is they just vary things in physics wildly. And then the robot's able to generalize fairly well. But, and for whole body control, say, but tell me about that. Tell me about Sim2Real.

Amarjot Singh Yes. I think most of the things which people are trying to do is in manufacturing and home. So your objective is to maybe stand straight or bend down to pick something. You are not making the robot jump from stair to stair or room to room, right? So the space of whole body control in most of the other applications is quite limited. So the idea that you can more or less vary all the physics and hopefully get it. Also maybe you pick something which is a certain amount of weight and then that can pull the center of gravity so that you're going to do something there. I think it's possible to do, we try to do that as well. But still I think the number of variations which can come up are quite a bit because you're moving on uneven surfaces, you're carrying heavy stuff. So that kind of works for us but you still need that adapter piece. And I think without the adapter piece you would not be able to have reliable work in our kind of field. I don't think you can even do that in the homes and manufacturing side as well. I still feel like you would need on-device adaption and people have been doing that like Physical Intelligence but they would more or less do it in an iterative way on a big server.

Audrow Nash Yep. Mm-hmm.

Amarjot Singh Skild has been trying to do few shots but then they go and teach it or show it. While in our case the part about where you can find something new, learn it, that kind of makes it unattended and fully autonomous.

Audrow Nash Mm-hmm. Which is super cool. Okay. let's see. Tell me, so just to understand better where the limits of this technology, like the approach you're taking are at this moment, tell me some of the, like, hard tasks, easy tasks, unattainable tasks, for automating given this approach. Like, you can move a water bottle, you can army crawl, You can bash down a door, but like, so tell me what's easy, tell me what's hard, tell me what's infeasible currently.

50:14 What's easy, hard, and not yet possible

Amarjot Singh Yeah I think like so if you look at from a user's perspective and a lot of our users would be the US military or let's say oil and gas. Oil and gas would say would love for your humanoid to come, go into my mine and do some sort of blast so I don't have to send someone in. Great. You're working in very dusty environments. Sometimes there is oil, sometimes there is water. So your humanoid should be water resistant. It'll be great if it's waterproof. So none of the humanoids have that. So that's one hardware bottleneck. The battery should not catch fire because if you're operating around an oil mine that's a bad idea. So none of the humanoids have that property.

Audrow Nash I was meaning more from the learning approach, but this is interesting too. Like, from the, cause, like, I'm saying what kinds of tasks, but yeah, it is very funny, like, of course, you shouldn't have a robot that lights on fire with its battery in an oil mine, that would be like, the customer doesn't want you back at all, this kind of thing.

Amarjot Singh Yeah fair. I think on the software side of things,

Audrow Nash But.

Amarjot Singh The biggest challenge we have is primarily the simple tasks we are doing are picking things, moving things around which could be a metal detector or let's say pushing the door in. Those are things which are fairly easy to do, how reliable they are, how many times can you run it again as a function of hardware. But from a software point of view, I think you can manage that because there isn't actually a lot of variability. You're doing the same or very similar kind of task with a similar kind of metal detector or you're trying to punch down the door. Very minimal variability there but massive ROI, even if it works 70 or 80% of the time that is still okay because you're not getting shot, someone else is. So yeah, so that has been great.

52:44 When 70% reliable is enough

Audrow Nash What an interesting thing. I hadn't thought about this until you just said it. because one of the things with like, imitation learning is that it's like, a lot of the demos you see, it's like, it's only working 90% of the time, this kind of thing. but the thing is, depending on the industry, that might be okay, because it's not you getting shot or blown up or whatever it is. it's the robot, which is expensive and a bummer if it, whatever, but it can, it's still lower risk and less expensive than a human, I would think. very interesting that because it's like, different interest industries have different standards of reliability that they need. And if this can do a lot of harm reduction, even if it doesn't, or even if it works 70, 80% of the time, it's still fine. Or you can just retry and then it'll get it the next time for this kind of thing. That is super interesting to me.

Amarjot Singh Yeah. Yeah. The other thing which is very interesting and again goes to the software is there are many robotic platforms in the mission critical side but they would just do it one way. You can have like a big robot with a big barrel and it will go and hit the door and everyone is waiting for that to happen. It will go in.

Audrow Nash So specialized. Yeah, yeah.

Amarjot Singh Yeah, but you want to have a machine

Audrow Nash It's a general, do a lot of things.

Amarjot Singh Or a software package. Yeah, mission set on mission set on mission sets. So the foundational models right now we have, they can do breaching, they can do your weapon engagement, IED. But as you start to go up, data is a bottleneck and then software is a bottleneck and then it can start to balloon up. So we haven't tested beyond that. But also the way you move, right, a lot of the temporalness starts to become very, very unique. So that is something which we are working through but right now the biggest point.

Audrow Nash I don't understand. What do you mean? When you move temporal, things become unique. What do you mean? Just the ground being varied or the robot's dynamics changing as it runs or what do you,

Amarjot Singh Oh yeah, yeah. Robot dynamics. Robot dynamics.

Audrow Nash Things heat up. You get mud in the area. There's friction to go through, like all sorts of variety by the environment and the parts wearing, maybe.

55:01 The humanoid Turing test

Amarjot Singh If you just take the example of the human just kicking the door, the way the government wants to deploy this is that it's a group moving and they want the humanoid to be part of the group. So the humanoid should move how the humans move. So you are part of the group. So when you kick the door in, you make space, the other humans move in. So it's formation dynamics and that is basically another temporal dimension on top of just your human. And that is currently difficult and we are working on that side. And they call

Audrow Nash Oh. Mm-hmm. Huh. That's really cool.

Amarjot Singh It the humanoid Turing test. So we haven't pushed that out yet. That if you totally cover it and you are a serviceman and you are saying, "Hey, this is your buddy and he's trained the same way." Go and do breaching and they do it and then you replace the human with the humanoid. Can you tell it's an actual human or a humanoid?

Audrow Nash It's awesome. What a world we live in. It's so freaking cool. okay, I love it. What are, what are some tasks? So moving with the group is very interesting because you don't want to be friction. You don't want these robots to be awkward. To work with.

Amarjot Singh Exactly.

Audrow Nash So you have to realize, and that might vary between groups. Like some groups move faster, some move slower, some whatever. So you have to vary how they do that. So that sounds like a very interesting problem to me. but what are, what are some tasks? That's, so I'm hearing moving stuff, like picking up objects, manipulating them and getting around are kind of like the relatively easy ones that are relatively reliable. Then you have tasks like operating well in the group that are hard and not quite solved yet. what are some other ones that are hard and not quite solved? And then what are some ones that you think are just like our current approach in kind of this imitation learning setup is not capable of solving yet. And then I'd love to hear how you think those will fall. Like when will we get to those? Which ones will be the absolute hardest problems for us for this kind of thing? Moving in the group seems wonderful as a task. That seems like an incredible thing once that's done.

56:51 Generalization is the bottleneck

Amarjot Singh Yeah. I think the biggest challenge in all of this is generalization. And that's the biggest bottleneck of imitation learning, right? How general is it? And can it extend on its own, few shots, one shot? I think that's the biggest challenge. And I think whatever we have deployed, it kind of works. And then we have this VLM model. If your graph is not there, it can generate that graph. But how good it is at generating that graph. Can you really generalize from that graph? Can the modifications be there? And all of that I would put under the generalization bucket.

Audrow Nash Mm hmm.

Amarjot Singh Because we don't want to go down the path of keep on retuning the foundational model. Because if you do that, you will never actually get to a reliable system. Adaption is the key. But then the adaption side, you can say, well, I'm using 15 minutes of data rather than 10,000. But how good is that? And that will all come from testing in the field. Because a lot of the testing we do, we capture these failure points, which the system automatically picks. And then we add it and it generalize from there. But can your mid-level detector or the thing which is looking for an emerging signal, is it able to capture all of that? Do your things fall only on 2 sigma or are they at 10 sigma? So there are bottlenecks on, can you find something new? Can you reliably label it? Can you reliably learn it, which is a function of generalization? And if you don't have that, then do you have to go and retrain your base model because the base model wasn't good enough to do all that stuff? So that we will discover as we go and deploy more. But I think that is the biggest bottleneck in robotics, that generalization is not there. And for our two mission sets, we can more or less still be okay with that because the variability is less. And the other cases, that's quite high.

Audrow Nash Mm hmm. Well, that's really cool that you can find markets that you can that have lower variability in some sense. And then, from that you can go and you can add value, you can improve your foundational model, and then you can go to less structured ones that have higher variability. And you can kind of keep going until you can eat up all the different tasks that have a lot of variability. This kind of thing. What

Amarjot Singh Yeah, absolutely.

Audrow Nash A thing. Okay, very interesting. Now tell me about kind of your hardware path for everything because you guys, you do drones, you do humanoids, you do robot dogs, you do some surveillance. Like it's quite about quite a bit. Tell me how it all fits together and kind of how you went from one to the next. and then I think humanoids is probably your most recent one. And I'd love to see how that fits in kind of the company vision to you.

1:00:05 From runway debris to robot dogs

Amarjot Singh Yeah, so initially we started with markets where there was an ROI, so that if we could deploy, someone would pay for our self-learning model. And the sectors were any asset which is moving very fast. You want to pick targets which are very small. And if you miss the target or if you have a false positive, you're going to actually have a challenge. So we deployed it to detect small debris on tarmac, which can get into fighter jets and can cause damage to F-22 and F-35s.

Audrow Nash Oh, hmm.

Amarjot Singh So we have actually this ground vehicle, which you can see here, which was running up and down the tarmac. And it was, you can see on the

Audrow Nash Yeah, it's so cool.

Amarjot Singh Left side. Yeah, Yeah, that's what we started with. So it's going to...

Audrow Nash So just to describe for listeners, there's a car, it's kind of like a Waymo in some sense where it has tons of sensors on it, a bunch of like security cameras looking things on its front bumper. And then there's a robot dog that's running around with an arm picking up the things that it's identified. Is that right?

Amarjot Singh Yeah, so when we started, we were... d Yeah, the cars were there, right? But the intelligence wasn't. Because the wind would blow up new debris on a tarmac, that's FOD, or

Audrow Nash It's awesome. Mm hmm.

Amarjot Singh You get a lot of oil patches.

Audrow Nash And tarmacs, are they just runways? Like it's a runway for planes? Is that what it is?

Amarjot Singh Yeah, absolutely. It's a tarmac. It's wherever you ha So this is on an aircraft carrier. It's on a military air tarmac, same stuff. And you have these fighter jets, the engine is close to the ground. So they would suck in the debris. So the first thing we did was, can we give you AI, which is not expensive, first of all, because if you use a radar-based solution, that is $2 million, and not everyone wants to buy that. So we were using cameras, and the downside is it'll give you false positive, so it's not reliable. So that is what the self-learning

Audrow Nash Mm hmm. Oh, yeah.

Amarjot Singh Initially corrected.

Audrow Nash Cool. Yeah.

Amarjot Singh Yeah, and that's what we started with. And now some... But initially someone had to drive that car. They will get an alert, get out, and pick that debris. But we wanted to automate that, so now we have a robotic dog, which uses the same perception model, but also has an arm, using which it can pick the debris. So

Audrow Nash Sweet.

Amarjot Singh Automation. But now when it is not doing... the So now when it's not doing the FOD, you can use it to do perimeter security. When it is not doing perimeter security, yeah, it can go to other areas where there is some sort of critical infrastructure. There are gauges. They're not digital. You can go and read the gauge. But even on the FOD side, previously on the car you're just doing the tarmac, but now it can also go to the hangar areas. It can go to the engine testing bay. So whatever you're doing on the FOD side, this expands.

Audrow Nash Nice.

Amarjot Singh And whatever you're doing on the mission set, a single machine can do that. And the reason why we are very excited about the humanoid is, humanoid is one machine based on its degrees of freedom, has the capacity to do many mission sets together. It can go from riot gear to defusing bombs to firing weapons to

1:03:46 Why a humanoid, on two legs or four

Audrow Nash Anything people do. Mm

Amarjot Singh Everything people do. So our bet is that once the hardware matures, we would be able to basically have one machine which can do everything. Now, on the hardware side of things, wherever there is an established player, whether it is drones and dogs, and they have significant market share, we don't want to build that hardware because we can't be the category leader there. So we'll give you our software. We'll partner with you, increase your revenue in the process also, get money. But on humanoid side, since there is no category leader for the mission critical side, that is where we are building our humanoid,

Audrow Nash Mm hmm.

Amarjot Singh Which you can see on the screen. That's a graphic. But ideally, once it is built, it should be waterproof. It should have battery which does not catch fire. But also you can go from bipedal to quadruped because we don't feel like biped is the right design all the time.

Audrow Nash Warm for distance. Mm hmm.

Amarjot Singh Exactly. So you want to put it on quadruped. So a lot of our work is on building this LEGO kind of hardware, but then building control policies so that if you take the top part and put it onto the legs or legs with wheels or treads, the same control policy will work across these humanoid form factors. And that's the work we're doing on the software side.

Audrow Nash Yeah. Damn. That the graphic just to describe for people listening is it almost looks like something out of a sci-fi movie. It's a big black robot humanoid that jumps over something, goes into running like a dog, and then shoots at targets, which is just bonkers. Super cool looking. no. It's like it's even scarier than those the

Amarjot Singh You know, You don't want this thing coming on to you when you're in the battlefield. Yes. Yes. Yes. Exactly.

Audrow Nash Mm hmm. Wow. So what do you think? Yeah, go ahead.

Amarjot Singh We get it. No, no, no. I'm sorry. We're getting there. We're getting there.

Audrow Nash That's bonkers. wow. So where do you, I guess. So what's your read on the rest of the humanoid n companies? Like you have, there's a whole bunch of them. And I guess none of them are really going after this. There are a few, I think, making military humanoids. but, and I don't know, like, I assume Optimus will try to be weatherproof and things like this. But, what's your, it's or it's just too early of a market to say that there will be any leaders. So that's why you guys are betting there as opposed to like competing with DJI on drones or whatever it might be. Where they're well established.

1:07:00 Humanoid hardware as a commodity

Amarjot Singh So I think so I think the hardware market will become commoditized, similar to the drone market. You go on LinkedIn, there's a new humanoid coming out every other day. So value will move to the brain. And I think on our side, we are building

Audrow Nash Mm hmm. Yeah.

Amarjot Singh The brain for specific mission sets on proprietary data on this adaptable brain. So I think that positions us well in the mission critical space. And I think as compar as far as the other players are concerned, whether it is Figure or Agility or Tesla, I think they are doing mainly homes or manufacturing.

Audrow Nash Mm hmm. Mm

Amarjot Singh I think it's great, but I think the variability in that space is so much that it's very hard to get reliability. That's one and then get reliability to the standard of your user. If I put a humanoid in your home and it is breaking the plate one out of 10 times.

Audrow Nash Hmm. Or just falls down.

Amarjot Singh Yeah. And you're pissed off, right? Because you don't even at that point, 90 percent of liability. Exactly.

Audrow Nash Mm hmm. I mean, they're heavy. Yeah. Like a big heavy heart or a metal robot falling over in the house could damage a lot. Like there's real risk in that. Or step on someone's toes or I don't know, squish our dog or whatever.

Amarjot Singh Exactly. Yeah, exactly. And also depends on how much you pay, right? Because in our case, if we get like a Boston Dynamics dog, it's 200K.

Audrow Nash Mm hmm. Mm hmm.

Amarjot Singh If you get a humanoid, we want to sell it for, let's say, maybe double of that or 300 or 350. But then you can justify the ROI, right? Because it can do many mission sets. And you have to train people for all of those mission sets. You can justify the ROI. What are the kind of things you are making a humanoid do in the house, which is really going to get to the ROI? Even

Audrow Nash Yeah. I mean,

Amarjot Singh Manufacturing.

Audrow Nash Manufacturing maybe because bottlenecks in manufacturing really like, I don't know, have you, I read an excellent book that talks about how bottlenecks are unbelievably like they dictate the pace that you can actually produce products. And so if you can automate a bottleneck in a manufacturing process, it could be incredibly valuable. or just make it so you can go. Or data centers. Yeah, maybe. But yeah, in the house, I mean, you're competing to the bottom for this kind of thing for the price because like I would if we could buy a 20K humanoid that would just clean up our house. It's like that's an easy decision. We would for sure do it. But that's a much lower thing than a 400K robot for this kind of thing.

Amarjot Singh Yeah. Yes. I think 20K is okay, but process a function of your acceptance. I think there are some people who are early technology adopters. I buy a lot of things even if they work 50 percent, right? So I like that. But I think for the average user, the market is, look, everyone is going to be using a humanoid. And I think that's good. But then you need to get the product to a point where if you say it's going to do A, it's going to do A. And I think that is where it's going to be a little bit of a time before it gets there.

Audrow Nash Yeah, you do. Yep. Mm hmm. So it seems to me like military because they're so mission critical and the cost is so high of putting humans at risk that it's actually a very good place for humanoids because even if they're not perfectly reliable, you have the ability for them to do a dangerous task. And so it seems to me like this is a very good bootstrapping humanoid approach. And then once you do this, then you move into manufacturing or like even home servicing company like service companies like HVAC and stuff like this, like air conditioning.

1:10:56 Why defense comes before homes

Amarjot Singh I think for us, we try to stay, whether when we did drones or dogs or cars, our objective has always been in places where they are out, remote, expensive to maintain. And if you don't do that, the outcome can be very significant.

Audrow Nash Bad.

Amarjot Singh Yeah. So I think like HVAC and all that stuff are there. But I think as the market evolves, there will be company who would start to capture this thing from the top side and the bottom side, whether it is your home or manufacturing or HVAC or all those kind of things. And one more thing which is very important for us is barriers to entry. So you cannot use a lot of anything Chinese.

Audrow Nash Mm hmm.

Amarjot Singh So and a lot of the humanoid market sits over there.

Audrow Nash Oh, yeah.

Amarjot Singh Yeah. So for us,

Audrow Nash So that's a big barrier to entry because like the what's the company Unitree making humanoids I guess like their spot equivalent is way cheaper. I think their humanoid can be pretty expensive still but like if you get it fully featured with good motors and stuff. But

Amarjot Singh Lot of our. Yeah, exactly. We. Yeah, so we have a lot of Unitrees. We have a lot of Boston Dynamics. So a Unitree would be like 80 K versus 300 K. But I think even if you don't worry about the price, let's say the military can pay whatever they don't want to use Chinese equipment. So

Audrow Nash Yeah. Mm hmm. Mm hmm. Chinese. Yeah, it's a security risk for them.

Amarjot Singh Exactly. But that barrier is not in other sectors. So anyone eventually and the Chinese machines are getting so good. What will happen is in the next one year or so, you'll start to get these Chinese machines, humanoids with these kind of mission sets. And then people who really like the field will start to import from there, modify it, and start to service. So the idea, if you have built a foundational model with a lot of costs, a lot of money, it's really nice. And it may not really be that attractive to a user because at that point price will start to play a role. And if you're.

Audrow Nash The deciding role.

Amarjot Singh Yeah, and if the reliability is more or less similar for both of them. And that's another problem.

Audrow Nash Interesting. So you're you've positioned yourself very well, where if say China produces a ton of cheap humanoids that are high quality, they're still not going to want to be used by the government and therefore you're still in an excellent place for this. Mm hmm. Seems very smart. I feel like just from talking through and kind of seeing some of the progression of your business that it has been very smart strategic like running like a startup into different areas that make good sense and kind of growing your robotics capabilities and positioning yourself well just in general for like long term outcomes. huh. Hell yeah. Can you tell me actually a bit about the journey because I mean doing the tarmac and things like that's wildly different than I guess you still had to do perception on that so you could identify objects and maybe so there is kind of like a common core to that but also doing drones to help military so if they do a kamikaze drone or something like this but then doing humanoids for the military. Application and then it seems like from that video that you shared with the humanoid running over and jumping the thing and then shooting the gun and running away like a dog and just like bonkers. Seems like it's is kind of the goal you think to make like a super soldier or just like a very capable robot for military applications.

1:14:32 Trying markets until someone paid

Amarjot Singh So I think like the evolution from the drones to the cars is relatively easy because we trained on very minimal base data. you're gonna get error anyways and the memory can learn that. So that was relatively easier because otherwise I would have to go through many cycles of updating again and again. And the reason why we did different use cases was we didn't really know what would be useful and where the value chain would be. Everything from finding suspicious people moving in front of a jewelry store so that eventually they will break in. Where

Audrow Nash Mm hmm. Wow. Yeah.

Amarjot Singh No one will pay us, to the tarmacs. So it was just also exploration for us and

Audrow Nash Cool.

Amarjot Singh And the sector.

Audrow Nash You basically you've been trying to make value in all these different industries and seeing what sticks and trying lots of different things.

Amarjot Singh Who would pay us? Where is the actual life? Exactly. And pay us.

Audrow Nash Yeah who would pay you? And also because like because you also need to figure out the go to market strategy and everything and then once you're in with the military and they're using you I assume that's a huge moat because it's probably slow sales cycles and you have to know someone and it's just like it's a hard thing for other startups to do.

Amarjot Singh Yeah. But also a hard thing, not from just the GTM, but also the technology side. Not anyone can come and say, hey, I can also detect ABCD on a camera. If you have to go to the stationary asset, which is the CCTV versus drones or cars, you will get probably hundreds of companies which can make that claim.

Audrow Nash Mm hmm. Mm hmm.

Amarjot Singh When it comes to putting AI, which can work sensibly on a drone or a car, there would be maybe handful. And that also is a big decision because I want to be in a space which is growing. There is active users. There is very clear defined ROI, but less competition so you don't have to fight on price.

Audrow Nash Hell yeah. Yeah and then you can invest and then you can keep doing awesome things like you can keep playing it forward for the future which I think is what you're doing with humanoids now like this is this is a new process for that where that's a big future bet because the market's growing.

Amarjot Singh Yeah. Yeah.

Audrow Nash The world's chaotic and the technology is just about there is what it seems.

Amarjot Singh Yeah. Yeah. Yeah. So my approach is we were doing it on the car. How do we fully automate that? So we went to the dog and then when we are going to the dog, we also want to improve its mission set capability. So it's not specialized. It can do multiple things. And I feel like most capable machine which can do multiple things is the humanoid. And that's the reason why we're pushing more towards that domain.

Audrow Nash Mm hmm. Mm hmm. Mm hmm. Yeah because then you can go adjacent and that helps justify the cost for everything because it can be a soldier it's like your robot dog which can do picking up stuff but it can also do surveillance and then it can also do some other tasks but the humanoid will be a more extreme

Amarjot Singh Is that. Yeah.

Audrow Nash Version of that. And so, hell yeah. What do you think you showed me that you're building a humanoid? Tell me about where you are for that and how that turns into the crazy machine that you showed me in that video.

1:18:40 Building their own humanoid

Amarjot Singh Yeah. I think the two dimensions to the building one is the body and the second are the hands. Well, power system is another one. So we are working with a couple of vendors in Korea. One who can build us the battery packs the way we want. One specializes in hands which are not motor-driven but tendon-driven so you can put the motors on the exactly so you can pull more.

Audrow Nash Cool. Motors are somewhere away. And that probably is better for running like your bipedal running so you don't have heavy weights at the end of the end effectors or that end of the arms I guess. Mm hmm.

Amarjot Singh Exactly. Exactly. So we are working on all of this. I think it will take us probably six to eight months to get the V1 out. But I think once we have that then we are off to the races because that will make a big difference.

Audrow Nash Mm hmm. What are some challenges that you like what are some of the most challenging things with this I guess I my suspicion is manufacturing for any scale will be a huge challenge for this. But like are parts is part sourcing difficult are is designing for manufacturability a big challenge I guess because you're going for high end applications with cross usage you don't have to push down the price as much as like people trying to put humanoids in people's homes. But and then I wonder also from like an imitation learning and control and I don't know just general what are what are some of the challenges of making a humanoid.

1:19:48 Manufacturing humanoids in the US

Amarjot Singh Yeah. Well the first thing is as you said manufacturing. You don't have a lot of manufacturing capability in the U.S. for humanoids right now. The ones which is there. The one that is there.

Audrow Nash What's missing? Yeah, yeah.

Amarjot Singh So I think like for example you have a humanoid, you run, you run wires through the humanoids pin connectors. A lot of that knowledge as to how do you do that is very very specialized. it Most of that sits in China.

Audrow Nash Mm Hmm. Mm hmm.

Amarjot Singh Milling. You want to mill the chassis. The actuators more or less. They have rare earth metals in that. A lot of that is controlled by China which comes from a lot of that southern America. So if you and everything is built on actuators if you don't have actuators that doesn't work. And we are starting to build some but most of them is in Asia or China. So that's the first one.

Audrow Nash Yeah. Mm hmm. Yep. And China does all those rare earth processing to for I think magnets in general like I think they have a huge amount of the market share for that.

Amarjot Singh Exactly. So even if you don't want to buy the whole Chinese humanoid you are more or less somewhere buying Chinese. Exactly. Then the other thing would be the cost. If I want to build something over here it'll cost me X. If I'm doing prototyping I have to do five prototypes which eventually the last one's going to work. So my cost of doing that here versus offshore is extreme.

Audrow Nash Sourcing through them. Mm hmm.

Amarjot Singh And that's the thing. And I don't even know if I put that money if I would be able to actually get that done because that scale is not tried and tested at scale while they have that. So a bunch of those kind of challenges on the hardware side.

Audrow Nash Mm hmm. Mm hmm. What do you think so you have 20 people in the US and then you have seven 50 people or so in India. How do you leverage that? I think in a lot of manufacturing seemed to be moving from China to India. Do you think that will play a strategic role for you where you'll do some I don't I don't know much about India for manufacturing things.

1:22:20 India for data, testing, and talent

Amarjot Singh So India

Audrow Nash But.

Amarjot Singh India is very strategic for us because of two things. One data for robotics. So a lot of this real world data real world testing which we do deploying teams operations, that's way more economical there as compared to India. That's one. The second thing which is very important which is a very very very big bottleneck is talent. If I have to get a robotics engineer in the Bay Area very hard to do and we have people but then they also want to talk to Figure they also talking to Tesla. A lot of options here.

Audrow Nash Oh yeah there's so much competition. Mm hmm.

Amarjot Singh Yeah if I go to New York relatively less if I go to India a lot of very talented people but not that much competition. So we also use India as our talent hub to retrain the policies test these things do breaching there all that kind of stuff. Get data and also manufacture.

Audrow Nash Hmm. Cool. Huh. Do you think that how from your perspective how is AI changing all this like I wonder I don't know like if you have say you say if you go hire an engineer in Silicon Valley. They're great and bring a lot of knowledge. But if you go to India and you hire someone who's probably quite good too but maybe doesn't have quite the education or maybe does. But if they use AI it's probably a bit of a leveler would you think

Amarjot Singh Sure.

Audrow Nash Like I think you could probably leverage it to make your teams even smarter or how do you think about it.

Amarjot Singh We are some most of the specialized end to end autonomy knowledge, VLAs and all that stuff is in the US Menlo Park. We fund university professors at UC Berkeley, Carnegie Mellon. So a lot of our whole body control and manipulation work happens with them through their PhD students. But once that we get the base structure and you want to test it for n kind of applications you want to tweak the different kind of policies. We send all of that back to India and they can then pick that up and they can improve on that. A lot of the people there we have used to work for companies like Dexterity or something like that offshore and they were doing the same thing for them. So we have brought those people on. So refining updating

Audrow Nash Nice. Mm hmm. Cool.

Amarjot Singh All of that we do there and obviously Claude helps because you don't have to reinvent the wheel. You just need to know how to think and that we can either do directly through the.

Audrow Nash You just need to know half the what. Sorry I missed it.

Amarjot Singh Just they just have to think they don't need to know coding and all that stuff which Claude Code has helped with. quite a bit.

Audrow Nash Oh yeah. Yeah it's such a wild world for everything with all the AI going where it's going. What do you I guess so we're starting to wrap up where do you see Skylark going like so I think you were thinking that humanoid robot in three years or so for everything but what do you see kind of as the next. Three to five years for you all like what's where were you guys headed what's the robotics climate look like.

1:25:40 Skylark's next three to five years

Amarjot Singh Yeah. Yeah. So short term we are focused heavily on where we get the money where the business is. So we have drones which are doing primarily detection. So for example we are detecting cracks on a solar panel to cracks on an insulator. And now we have drones which have arms. So the drone is also going and cleaning the solar panel so that the efficiency of the solar panel is there to going and sealing the crack on the insulator. So a lot of our customers are ingesting this aerial robotics application. So that is immediate extension for us. Then same goes on the ground side where we were using cars and now using robotic dogs. So that's the other dimension which we are scaling through our own sales through our new engineering new model class that's that. Then on the humanoid side we are building the brain because we feel like there is there will be significant commoditization on the hardware side and we are starting to build our own as well which we're going to push out in the next eight months or 10 months. But currently we are working with partners from Korea like Robotis, using and through whom we are going and still showing this brain to the end customer. So we are still doing the data flywheel still showing still improving on the humanoid side while we build our own with hardware which is acceptable so that we don't have to bear the cost. That's one. And then on the side where the business is there air and ground over there we are expanding our current business expanding those product suites with aerial robotics or ground robotics and then scaling that. So that's how we see this short term and then long term.

Audrow Nash Hell yeah very exciting. What do you think to me it seems like you're running effectively a few startups at once like all these different areas and I know that there's kind of a common core to it but it still seems like completely different markets completely different products. These kinds of things what advice do you have for I don't know people who want to be robotics entrepreneurs want to do a robotic startup seems like you're chasing the money really good finding good market fit and then deploying but what advice what things would have saved you a bit of learning. If you were to do it all again.

1:28:21 Advice for robotics founders

Amarjot Singh Yeah. Yeah I think it's a tricky market because if you're trying to I mean the reason why we are able to do multiple markets is because we don't really sell anything directly. The go to market motion is with the end customer on whose hardware we are overlaying. So partnerships I think big play a big role in offsetting the price of the hardware or the data or operations. I think that is something which everyone should try to leverage. But I think then finding sectors where your solution no matter how effective it is able to solve a very big pain problem. Which they cannot live without. And then they will buy things even if it is 70 percent efficient or 80 percent efficient. So I think finding something which really where you can get to delivering reliability whatever their definition is as quickly as you can. I think that's the next the next big thing because I think robotics kind of works and kind of is dependent on who you're showing it to. So you get a selective over there. I

1:30:02 Where robotics is going

Audrow Nash Hell yeah. Where do you think robotics is going like if anyone wanted to skate to the puck in some sense like where would you you're obviously placing your bet on humanoid and military. Where do you think it's going.

Amarjot Singh Think it's a very very exciting field. I can easily imagine whatever has happened on the language side and the reasoning side with LLMs would happen on the robotics side as well. Once the foundational models either get adaptability or they get good enough and I think that will happen. I think there will be massive disruption on everything we do which is repeatable which goes all the way from warehouses to manufacturing to hopefully homes. And I think there will be a big shift. We'll see this dystopian world where you have humanoids walking around all the time. So I think that in the next five years. But just imagine right if you're able to build a brain which is able to learn alongside you which is a lot of your personalized agents right. And you can embed them and now we're getting humanoids which also have human like skin. Do you actually get to a world where you can transfer your consciousness. Do you even get if you're talking to a human or a humanoid. I don't even know where the world is going to go right. If you are able to actually build the brain which is adaptable and sensible and the form factor is there which is a lot of the work coming from Japan and China. It's a very it's a it the world's going to look very different in 10 years.

1:31:57 Final takeaway

Audrow Nash Yeah I think that's for sure. Okay hell yeah if there's anything that someone takes away from this episode like one thing you want our listeners and watchers to come away from this episode with what is it.

Amarjot Singh I think the key take away is physical AI is a very exciting field. It's coming. It's coming fast. The core the core thing which is going to get it there is improvements in hardware which is happening very rapidly but also improvements in software which is adaptable. And it's going to affect many sectors. And I think the focus should be on finding sectors where robotics can get value to the end customer quickly and data requirements are less variability is less so that whatever foundation model you're using can be useful and that is for the builder. And I think for the user I would say it's a very exciting world and it's going to solve a lot of people's problems and make their life easy. So yeah.

Audrow Nash Hell yeah. Alright well it's been great talking with you and you guys are doing super exciting stuff.

Amarjot Singh Thank you very much for having us. Thank you very much.

Audrow Nash Bye everyone.