Rendered at 15:45:46 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
alex-moon 6 hours ago [-]
This is an excellent write-up and has successfully changed my opinion on something I think will prove critical in the near future: I always laughed at humanoid robots, assuming they were being built for "comfort" reasons. I now realise this is almost certainly the most reliable way to solve the general robot problem - not just for the reasons the OP lays out, but because the whole world is already built for human-shaped agents anyway.
maeln 6 hours ago [-]
It's not just that the world is built around human-shape, but it's also that humans are very, very adaptable. Our hands are a marvel of evolution allowing us to do extremely precise tasks while still being adaptable to all kind of movement : Screwing in, out, grabbing weird shape in 6dof, rotating along several axes, ... And ours arms, our legs, etc, have similar benefit. As an example, we are not a fast specie, but we can walk on even ground, rough ground, steep inclined, we can climb rock and more. If you see a place, you can probably go there.
This is why human are still used for many, many tasks, including in industry where a lot of things are handle by robot. Yes, you can have an automatic belt feeder, or moving tray system, to move pcb from the pick and place machine, to the furnace, to the assembly, etc. But it means your assembly line is fairly rigid. Accommodating a new process, a different size of pcb or whatever become a costly endeavor. A human on minimum wage can cheaply move things from machine to machine, check if everything is alright in the process, adapt to new processes, new floor layout, very easily and cheaply.
Robot don't need to be exactly human shape, but having robot with the same adaptability has human have a huge advantage for companies where adaptability is a core part of the business.
alexpotato 3 hours ago [-]
> It's not just that the world is built around human-shape, but it's also that humans are very, very adaptable. Our hands are a marvel of evolution allowing us to do extremely precise tasks while still being adaptable to all kind of movement
Not just our hands but really our whole bodies.
I recently taught one of my kids to ride a bike and once they got it I thought to myself:
"Somehow millions of years of evolution have somehow allowed us to ride a machine that was invented <200 years ago"
willhslade 33 minutes ago [-]
Well, it was invented by and for us, but your point still stands.
jacquesm 6 hours ago [-]
The difference is whether or not a factory is designed with robotics as the driving principle or with human labor as the driving principle. They will be completely different, and as a result a lot of those things that would cause issues are simply engineered away.
This only works for high volume / low margin. Humans excel at rapid change - for now.
optimalsolver 4 hours ago [-]
Yeah, for all we know there's a creature at the bottom of the ocean that's just as intelligent as humans, but it's hard to build a civilization with fins.
Symmetry 2 hours ago [-]
Most countries have laws like the US Americans with Disabilities Act meaning that there are enough ramps and elevators around that a wheeled robot can do pretty well. It actually is more about the training factor.
joaquieneCnix 6 hours ago [-]
> but because the whole world is already built for human-shaped agents anyway
not really, there's several non-humanoid shapes that can easily adapt the human shape and to it but it's a little out of the box. you can imagine it without imagination, too, with a little thinking. it's just a few rational hops, really.
pakl 3 hours ago [-]
LLMs have shown a way to absorb large amounts of textual training data and be useful. For sensory processing you need a different architecture to absorb spatio-temporal streams. Bolting vision onto language models is stretching the tech a bit too thin.
The good news is the data is plentiful — it is all around (the robot). The real world — the robot’s umwelt sensed through its sensors and acted on by its actuators — will provide heaps of data.
Interestingly my colleagues propose that embodied learning will lead to grounding, alignment, and AI safety.
(Worked for 4 years at a robotics + machine learning startup, Director of R&D.)
ACCount39 2 hours ago [-]
Embodiment for general purpose goals like "grounding, alignment, and AI safety" just seems to be one of those things that some people want to be true. There's no actual reason for it to be true. There are no findings showing it to be true. People just want it to be true anyway.
I'm very skeptical of that. Wishful thinking is a treacherous thing.
LLMs provide some incredibly solid "general purpose commonsense reasoning" primitives. And the likes of Astra, Opus 5.5 and Gemini 4 have demonstrated generational leaps in "spatial problem solving" - including robotics tasks. Those don't even have dedicated "sensor input embedding" and "action output diffusion" ends rolled into them at model level - but you can give them control over robot arms, and they'll have a decent chance of, slowly, finding a way to complete new tasks you'd previously would have to tune a robotics model for.
Previously, I have considered "generative video models" to be a decent candidate for the foundation of future robotics models - they, by necessity, perceive video, learn the kind of world dynamics required to plan and execute actions in real world environments. And they can begin to learn those dynamics off embodiment-free general purpose web-scale datasets.
But non-robotics LLM generalists have come to show they're no slouches either. They're not at all optimal for things like rapid precise manipulation. But they find more and more use in robotics pipelines anyway. Makes me wonder what something like "Astra reasoning engine but with robotics architectural extensions" could accomplish.
salomonk_mur 2 hours ago [-]
Why would the tech be "stretched too thin"? We have made it work better by adding images, video, audio. Why would other sensory input be any different?
alexpotato 3 hours ago [-]
Sebastian Thrun's talk at Google about winning the DARPA Grand Challenge [0] has a great discussion of training "robots".
Summary:
- the LIDAR had error rates that were not published
- hitting potholes could cause the INS in the car to get confused
- they weren't sure how to fix the errors from the combination of the above
- they ended up just driving the car to generate training data
It worked out well because the computer was predicting if there was an obstacle or not and a split second later the driver of the car would either drive around the real obstacle or the computer would realize that there was no obstacle.
If you've never seen the talk, highly recommend as Thrun is very funny and it's also a time capsule back to when self driving was still a brand new concept.
Is the issue also a lack of available sensor data?
Our own nerves produce a huge amount of data. Our eyes and balance also contribute to a very rich picture to enable movement and kinematics. This is all fused into usable feedback.
Arms on fixed axis with disconnected vision feel like we are still very far away from true robot training.
jacquesm 6 hours ago [-]
That is part of the problem. Yes, humans are absolutely studded with sensors, every hair you've got, every nerve ending. It's amazing that all that information gets processed as fast as it does even though the 'base clock' is rather low. There is a ton of pre-processing done on those signals though (for instance: the optical nerve is pretty much an extension of the brain, how is that for a high speed, high bandwidth interface). Then there are the reflexes, semi-autonomous parts of the nervous system that can react faster than the brain to deal with safety related concerns. For a very interesting biology angle have a look at octopi.
ACCount39 3 hours ago [-]
Human "sense of balance"? Human vestibular system is a sad joke if you compare it against any off the shelf smartphone IMU. Proprioreception is a joke when you compare it against current sensing in every motor and absolute encoders in every joint. Human hearing loses to a half-decent microphone array.
Even human vision, the one perception channel evolution has put an awful lot of weight on, is beat pretty resoundingly by the kinds of camera units that ship in smartphones nowadays.
The two senses robots don't have an edge in are: touch and smell. But I'd struggle to make the case that smell is critical at all, and touch? Hard to beat it directly. In practice, however, it's already partially compensated for.
Sure, a robot with a basic UMI-style gripper doesn't get rich tactile feedback - only a few channels of feedback from arm joints and finger motors. But you know what those actuated UMI gripper get that humans don't? Wrist cams. A robot can see the object it's manipulating - right near the point where the manipulation happens. That matters a lot for precise manipulation tasks.
Humans don't have wrist cameras not because they're undesirable, but because evolution sucks at wiring. We don't suck at wiring.
The fact that humans can pull off general purpose manipulation tasks with the bodies they have, and the best of robots can't? A testament to how much of a gap is there between modern robotics AIs and the evolved wiring of human brains. The intelligence edge is what makes the human body, flawed as it is, work as well as it does. Everything in the human body is half-assed and then salvaged by the brain in post. But that gap is closing as AI tech advances.
Symmetry 1 hours ago [-]
You're overstating how well wrist cams can compensate for a lack of touch. It's really hard to get the instant estimate of weight and rigidity we humans get when picking up an object with a wrist camera, especially for rigid end effectors. They can guide a grasp in ways that would require touch for a human and help detect that you've dropped an object but don't compensate for many aspects of grasp that prevent the drop in the first place.
ACCount39 55 minutes ago [-]
"Estimate of weight and rigidity" is exactly the kind of thing that's performed rather well by low-dimensional sensing of actuators themselves.
Backdrivable actuators know how much current are they sinking - and thus, how much force are they facing. So even the simplest two finger gripper can easily "know" that one object is squishy and the other is not. The response curves are different. Same can apply to arm actuators for weight estimation.
Now, what this doesn't actually get you is an estimate of how hot/cold an object is, and how moist an object is.
Where cameras come in is specifically the "guide a grasp" part. Force sensing alone can tell a robot "the grasp is failing". A camera pointed at the grasped object can tell "why".
Symmetry 18 minutes ago [-]
In theory yes but in practice backlash and stiction make drive currents much less useful than the raw force at point of contact for any realistic gearing. And an important part of weight estimation is that force lateral to the point of contact. Now, your arm motors will be drawing more current as they move so again in theory you can distinguish that, you've still got the weight of the whole gripper added in.
I spent years of my professional life on these topics[1] and from my experience motor drive current usually isn't enough to narrow down the weight of an object in a typical picking task even to within a factor of 2 to detect when you've accidentally picked up two items from a bin instead of one.
Humans have better dynamic range for hearing and vision compared to any sensor I'm familiar with.
Human vision is also extraordinary sensitive, with some people being able to detect individual photons with their rod cells.
ACCount39 24 minutes ago [-]
Human "dynamic range" is a hack - the limits look impressive, but they're only reachable through a lengthy adaptation process. Modern camera sensors with HDR, backside illumination and analog gain controls get pretty close to that - and much, much faster.
In practice, if you want really good performance across an extreme dynamic range, it's usually easier to stack multiple specialized sensors within a single sensor unit. A filter-free reduced resolution sensor for high QE in low light conditions (or at high framerates), and a daylight-optimized sensor that gets high resolution, full Bayer, IR-cut and a polarized glare filter to get you as much data as it can in high light environments. This gives you straight up superhuman range with no adaptation time.
Do the raw low light images look nice? No. But human brain itself takes a lot of postprocessing for "looks nice" to happen. Salvaging good images from series of noisy measurements is an AI problem. We've had some pretty capable sensors for a while now, but it took AI-powered computational photography to truly make low light photography happen on a smartphone.
I'm aware of the "detection of individual photons at above-chance rates" experiments with human vision. But modern backside illuminated CMOS, if plugged into the same setup, would also be able to detect photons at above-chance rates. Not to mention the actual photon event cameras. Very niche items, but they can detect photons pretty reliably - not merely "more likely to be right than wrong".
There are also thermal imagers and image intensifier stages, if you want to take the "niche extreme sensors" route. Not commonly done. But goes to show just how much ass engineered sensors can kick.
The limits aren't in the sensors - it's in making sense of the data they give you.
nkrisc 3 hours ago [-]
Animal olfactory sensing is more or less just detecting the presence of certain molecules in the air.
We already have devices that do that better than humans and even ones that detect things we can’t smell (carbon monoxide, for example).
Though I suppose you could say that “smelling” is the result of the synthesis of the accumulated signals from the olfactory system in the brain, and more than merely sensing their presence.
ACCount39 2 hours ago [-]
Sure, olfactory sensing sensing is "just" sensing of volatile molecules. And we can build plenty of different molecular sensors. Including for things humans can't detect naturally.
But, just like being able to build a load cell and a thermometer doesn't translate to "skin-like sensing", we can't go from "chemical sensors" directly to smell.
For one, things like sensor diversity, density, longevity and manufacturability matter a whole lot. Can you give a robot a weather station type "pressure + temperature + humidity + PM etc" air quality sensor? Yes. Can you give that robot a sensor with enough molecular coverage and fidelity to replace the human sense of smell? Almost certainly not. Can you even get close on a sensible budget? Not with the tech we have.
Getting a system to perform molecular sensing with discrimination against many molecule classes - the kind of operation human smell performs - is very much not trivial.
Now, whether that's worth pursuing at all - I'm still leaning towards a "no". It's a very niche capability.
SwtCyber 6 hours ago [-]
Exactly. I suspect the problem isn't just the amount of training data, but how much of the relevant information is missing from it.
ilikerashers 6 hours ago [-]
I read once analysing EEG (brain electrical activity) data is like standing outside a football stadium and trying to understand what the coach is saying from the roars of the crowd.
adolph 1 hours ago [-]
> Our own nerves produce a huge amount of data.
I wonder if collection of training data is going to be a purpose for Neurallink.
serpicourge 8 hours ago [-]
Here is a good video on the subject by nvidia, very fun prez:
PS: It's sad that robotic does not seems to get a lot of traction on HN :(
dvh 7 hours ago [-]
Feedback loops are too long with anything real worldly
serpicourge 7 hours ago [-]
The feed back loop is getting reduced by those new ways of training, simulation and video generation with realistic physics is kind of changing the game.
Can always skip the real world for training and evolve the controller in simulation.
Currently working on modernizing SIGEL (2000-era project at the University of Dortmund), which uses genetic programming to evolve walking programs for simulated robots in a physics engine.
Some of the 'evolved' walkers so far can be found at youtube.com/@virtualspecies
Differnent kind of beast entirely compared to what NVIDIA is doing, but then again it's fun to see what evolution comes up with - and it works on humble hardware too
shinnc 7 hours ago [-]
robotic is waiting for its chatgpt moment
serpicourge 7 hours ago [-]
You do not think that the chatgpt moment is happening right now ?
thsbrown 5 hours ago [-]
Is it? Genuinely asking.
serpicourge 5 hours ago [-]
Like said by others, the sensor problem is a real one, and personally I don't see a robotic hand matching the dexterity of a human one, within a viable economic cost and sufficient robustness, before a long time (but I could be wrong ).
But for the part that concern physical understanding of the world, adaptability, and basic manipulation, it feels like the field is going through a break through.
The implication might be the apparition of generalist robots, that could perform (in the limit of their dexterity) on a wide range of tasks, commanded by natural language.
Sure, not on pair with human dexterity yet, but as a thought exercise, think of all the things you could do with simple claws, or heavy glove. What if a robot was able to do the same ?
"Matching the dexterity" is a false goal. It's focusing on the wrong things.
As teleoperation readily shows, even a very dumb primitive hand can be salvaged if you get a very advanced control policy to drive it.
Sure, it would still take a lot longer for a well tuned robot with just a pair of UMI-style grippers to tie shoe laces than it would a practiced human. But a robot doesn't have to work faster than a human. "It can do the same task for cheaper" is often good enough.
The right thing to focus on is: AI. We already have robot bodies that are good enough to do many economically valuable tasks. We don't have the AI that would make those bodies perform to their potential. Modern robotics is an AI problem above all.
This is why human are still used for many, many tasks, including in industry where a lot of things are handle by robot. Yes, you can have an automatic belt feeder, or moving tray system, to move pcb from the pick and place machine, to the furnace, to the assembly, etc. But it means your assembly line is fairly rigid. Accommodating a new process, a different size of pcb or whatever become a costly endeavor. A human on minimum wage can cheaply move things from machine to machine, check if everything is alright in the process, adapt to new processes, new floor layout, very easily and cheaply.
Robot don't need to be exactly human shape, but having robot with the same adaptability has human have a huge advantage for companies where adaptability is a core part of the business.
Not just our hands but really our whole bodies.
I recently taught one of my kids to ride a bike and once they got it I thought to myself:
"Somehow millions of years of evolution have somehow allowed us to ride a machine that was invented <200 years ago"
This only works for high volume / low margin. Humans excel at rapid change - for now.
not really, there's several non-humanoid shapes that can easily adapt the human shape and to it but it's a little out of the box. you can imagine it without imagination, too, with a little thinking. it's just a few rational hops, really.
The good news is the data is plentiful — it is all around (the robot). The real world — the robot’s umwelt sensed through its sensors and acted on by its actuators — will provide heaps of data.
Interestingly my colleagues propose that embodied learning will lead to grounding, alignment, and AI safety.
(Worked for 4 years at a robotics + machine learning startup, Director of R&D.)
I'm very skeptical of that. Wishful thinking is a treacherous thing.
LLMs provide some incredibly solid "general purpose commonsense reasoning" primitives. And the likes of Astra, Opus 5.5 and Gemini 4 have demonstrated generational leaps in "spatial problem solving" - including robotics tasks. Those don't even have dedicated "sensor input embedding" and "action output diffusion" ends rolled into them at model level - but you can give them control over robot arms, and they'll have a decent chance of, slowly, finding a way to complete new tasks you'd previously would have to tune a robotics model for.
Previously, I have considered "generative video models" to be a decent candidate for the foundation of future robotics models - they, by necessity, perceive video, learn the kind of world dynamics required to plan and execute actions in real world environments. And they can begin to learn those dynamics off embodiment-free general purpose web-scale datasets.
But non-robotics LLM generalists have come to show they're no slouches either. They're not at all optimal for things like rapid precise manipulation. But they find more and more use in robotics pipelines anyway. Makes me wonder what something like "Astra reasoning engine but with robotics architectural extensions" could accomplish.
Summary:
- the LIDAR had error rates that were not published
- hitting potholes could cause the INS in the car to get confused
- they weren't sure how to fix the errors from the combination of the above
- they ended up just driving the car to generate training data
It worked out well because the computer was predicting if there was an obstacle or not and a split second later the driver of the car would either drive around the real obstacle or the computer would realize that there was no obstacle.
If you've never seen the talk, highly recommend as Thrun is very funny and it's also a time capsule back to when self driving was still a brand new concept.
0 - https://www.youtube.com/watch?v=j8zj5lBpFTY
Our own nerves produce a huge amount of data. Our eyes and balance also contribute to a very rich picture to enable movement and kinematics. This is all fused into usable feedback.
Arms on fixed axis with disconnected vision feel like we are still very far away from true robot training.
Even human vision, the one perception channel evolution has put an awful lot of weight on, is beat pretty resoundingly by the kinds of camera units that ship in smartphones nowadays.
The two senses robots don't have an edge in are: touch and smell. But I'd struggle to make the case that smell is critical at all, and touch? Hard to beat it directly. In practice, however, it's already partially compensated for.
Sure, a robot with a basic UMI-style gripper doesn't get rich tactile feedback - only a few channels of feedback from arm joints and finger motors. But you know what those actuated UMI gripper get that humans don't? Wrist cams. A robot can see the object it's manipulating - right near the point where the manipulation happens. That matters a lot for precise manipulation tasks.
Humans don't have wrist cameras not because they're undesirable, but because evolution sucks at wiring. We don't suck at wiring.
The fact that humans can pull off general purpose manipulation tasks with the bodies they have, and the best of robots can't? A testament to how much of a gap is there between modern robotics AIs and the evolved wiring of human brains. The intelligence edge is what makes the human body, flawed as it is, work as well as it does. Everything in the human body is half-assed and then salvaged by the brain in post. But that gap is closing as AI tech advances.
Backdrivable actuators know how much current are they sinking - and thus, how much force are they facing. So even the simplest two finger gripper can easily "know" that one object is squishy and the other is not. The response curves are different. Same can apply to arm actuators for weight estimation.
Now, what this doesn't actually get you is an estimate of how hot/cold an object is, and how moist an object is.
Where cameras come in is specifically the "guide a grasp" part. Force sensing alone can tell a robot "the grasp is failing". A camera pointed at the grasped object can tell "why".
I spent years of my professional life on these topics[1] and from my experience motor drive current usually isn't enough to narrow down the weight of an object in a typical picking task even to within a factor of 2 to detect when you've accidentally picked up two items from a bin instead of one.
[1]https://patents.google.com/patent/US12612266
Human vision is also extraordinary sensitive, with some people being able to detect individual photons with their rod cells.
In practice, if you want really good performance across an extreme dynamic range, it's usually easier to stack multiple specialized sensors within a single sensor unit. A filter-free reduced resolution sensor for high QE in low light conditions (or at high framerates), and a daylight-optimized sensor that gets high resolution, full Bayer, IR-cut and a polarized glare filter to get you as much data as it can in high light environments. This gives you straight up superhuman range with no adaptation time.
Do the raw low light images look nice? No. But human brain itself takes a lot of postprocessing for "looks nice" to happen. Salvaging good images from series of noisy measurements is an AI problem. We've had some pretty capable sensors for a while now, but it took AI-powered computational photography to truly make low light photography happen on a smartphone.
I'm aware of the "detection of individual photons at above-chance rates" experiments with human vision. But modern backside illuminated CMOS, if plugged into the same setup, would also be able to detect photons at above-chance rates. Not to mention the actual photon event cameras. Very niche items, but they can detect photons pretty reliably - not merely "more likely to be right than wrong".
There are also thermal imagers and image intensifier stages, if you want to take the "niche extreme sensors" route. Not commonly done. But goes to show just how much ass engineered sensors can kick.
The limits aren't in the sensors - it's in making sense of the data they give you.
We already have devices that do that better than humans and even ones that detect things we can’t smell (carbon monoxide, for example).
Though I suppose you could say that “smelling” is the result of the synthesis of the accumulated signals from the olfactory system in the brain, and more than merely sensing their presence.
But, just like being able to build a load cell and a thermometer doesn't translate to "skin-like sensing", we can't go from "chemical sensors" directly to smell.
For one, things like sensor diversity, density, longevity and manufacturability matter a whole lot. Can you give a robot a weather station type "pressure + temperature + humidity + PM etc" air quality sensor? Yes. Can you give that robot a sensor with enough molecular coverage and fidelity to replace the human sense of smell? Almost certainly not. Can you even get close on a sensible budget? Not with the tech we have.
Getting a system to perform molecular sensing with discrimination against many molecule classes - the kind of operation human smell performs - is very much not trivial.
Now, whether that's worth pursuing at all - I'm still leaning towards a "no". It's a very niche capability.
I wonder if collection of training data is going to be a purpose for Neurallink.
https://www.youtube.com/watch?v=3Y8aq_ofEVs&pp=ygUKbnZpZGlhI...
PS: It's sad that robotic does not seems to get a lot of traction on HN :(
- https://www.nvidia.com/fr-fr/ai/cosmos/
- https://dreamzero0.github.io/
- https://generalistai.com/blog/gen-1.5
Currently working on modernizing SIGEL (2000-era project at the University of Dortmund), which uses genetic programming to evolve walking programs for simulated robots in a physics engine.
Some of the 'evolved' walkers so far can be found at youtube.com/@virtualspecies
Differnent kind of beast entirely compared to what NVIDIA is doing, but then again it's fun to see what evolution comes up with - and it works on humble hardware too
But for the part that concern physical understanding of the world, adaptability, and basic manipulation, it feels like the field is going through a break through.
The implication might be the apparition of generalist robots, that could perform (in the limit of their dexterity) on a wide range of tasks, commanded by natural language.
Sure, not on pair with human dexterity yet, but as a thought exercise, think of all the things you could do with simple claws, or heavy glove. What if a robot was able to do the same ?
Here is a set of links to feed your though:
- https://www.youtube.com/watch?v=3Y8aq_ofEVs
- https://generalistai.com/blog/gen-1.5
- https://dreamzero0.github.io/
As teleoperation readily shows, even a very dumb primitive hand can be salvaged if you get a very advanced control policy to drive it.
Sure, it would still take a lot longer for a well tuned robot with just a pair of UMI-style grippers to tie shoe laces than it would a practiced human. But a robot doesn't have to work faster than a human. "It can do the same task for cheaper" is often good enough.
The right thing to focus on is: AI. We already have robot bodies that are good enough to do many economically valuable tasks. We don't have the AI that would make those bodies perform to their potential. Modern robotics is an AI problem above all.