🎙️ Podcast Link 🎙️

Are you one of those people who get excited when a new robot or autonomous tech demo goes viral? 🌐 Me too!

But did you know that not all videos are created equal? It’s crucial to understand the varying motivations behind them and what you can (and can’t) infer from them, and why.

In this video, I break it down into some key points:

1) The motivation behind releasing a demo video 🎯

2) The variations in what you can infer from the video, and why 🔍

Some videos are released purely out of excitement or in association with academic research papers, while others have financial motives or even aim to de-hype certain technologies. It’s essential to understand the context before drawing conclusions. 📚

Tech demo videos can also fall into different categories: staged, composited, or representing a (theoretically) mature technology stage. Knowing which category the video falls under can help you understand its reliability and significance. 🤔

The bottom line: Tech demo videos are a fantastic way to showcase the capabilities of new technologies, but it’s important to approach them with a discerning eye. 🧐

Please reshare if useful 🙏

🕒 Timestamps are as follows:

📌 (0:00) Tech Demo Videos Are Everywhere!
📌 (0:07) They Get a Wide Range of Reactions
📌 (0:17) Demo Videos Are Impactful and Reach a Wide Audience
📌 (0:37) Videos Can Be Misinterpreted with Consequences
📌 (0:52) Helping People Interpret Videos is Important
📌 (1:09) The Why and What of Tech Demo Videos
📌 (1:24) Two Common Motivations for Video Demos
📌 (1:32) Excitement-Driven and Academic Videos
📌 (2:14) Commercially Motivated Videos
📌 (2:27) Financial Incentives Affect the Context
📌 (2:38) Many Videos Have a Combination of Motivations
📌 (2:46) A Third Type: De-Hyping Videos
📌 (2:51) De-Hyping Videos Can Help Ground the Discussion
📌 (3:20) De-Hyping Videos Should Also Be Viewed Critically
📌 (3:42) Three Primary Types of Demo Video
📌 (3:47) Highly Staged Demo Videos
📌 (4:01) Context Sometimes Dictates Staging
📌 (4:29) Composited Demo Videos
📌 (4:47) Disclosure is Important
📌 (5:06) Research Maturity Informs Interpretation
📌 (5:14) Example: Walking Robots
📌 (5:42) Videos of Deployed Robot Technology
📌 (6:08) A Critical Viewing is Still Required
📌 (6:17) You’re Seeing a Tiny Fraction of What’s Happening
📌 (6:44) Reliability is Much Harder Than Doing it Once
📌 (7:00) Tech Failure Case Video Segments
📌 (7:11) Failure Authenticity Can Vary a Lot
📌 (7:41) Videos Should Be Interpreted in the Application Context
📌 (7:55) Imperfect Reliability is Sometimes Acceptable
📌 (8:28) Some Applications Require Near Perfect Reliability
📌 (8:38) Short Videos Present Insufficient Evidence
📌 (8:50) Autonomous Vehicle Example
📌 (9:21) Hypothetical: Two Videos From Different Companies
📌 (9:48) More Information Beyond the Video is Needed
📌 (9:57) Weight Expert Advice Appropriately
📌 (10:51) The Great Reason Why Demos Sometimes Go Backwards
📌 (11:16) Example: Drones With and Without External Localization Systems
📌 (12:16) Less Impressive Demos Can Be Very Important
📌 (12:28) Other Scenarios: Less Sensing, Compute and More Learning
📌 (12:53) Good Videos Will Explain the Context
📌 (13:02) Tech Demo Videos are Important But Need Careful Interpretation

Full Video Notes

• On an almost weekly basis you see a new demo video of a robot, a drone, an autonomous vehicle, that goes viral online. Inevitably, it’s shortly followed by reaction – from the “this is so amazing”, to the “is this real or CGI” to the “sure, but show us all the bloopers”.
• These videos are incredibly influential – in the technology development space it’s arguable that they are more influential in terms of reach than the original research papers, in person demonstrations or conventional media coverage. They’re also seen by a wider audience – not just people in the field, but the wider public.
• It’s very easy to have the wrong take on a video, which can be extremely damaging. Taken at face value, people may then become incredibly disillusioned with the technology when the promise of the video fails to live up to reality. 
• That’s why it’s super important to work with people more broadly so that they can be excited by these videos but also interpret them appropriately. This is the purpose of, somewhat ironically, this video. A video about videos, so to speak.
• I want to cover two key issues. The first is the motivation behind shooting and releasing the video in the first place. The second is the huge variation in what you can and can’t infer from what is shown in the video, and the key reasons why. 
• There are lots of reasons for filming and releasing a product demo, but they typically break down into one or more of two categories. The first is purely excitement: the researchers or engineers who’ve created the technology are genuinely excited about what they’ve done, and want to show it off. Sometimes this is done in conjunction with an academic paper submission, sometimes as a stand alone release. Although the video still needs to meet basic criteria in terms of an honest representation of what the technology can do, you can give videos released in this context a fair amount of tolerance, especially if it’s a university research lab – the intent is usually not to explicitly deceive or misrepresent anything, they’re just genuinely excited about what they’ve done.
• The second category of reason falls into a much more deliberate bracket – a startup or major company releasing a tech demonstration video, to shore up their case for further investment or other explicit purposes. These videos need to be given much more scrutiny because of the hugely influential underlying financial context, and I’ll explain how you can think about them shortly. That’s not to say the people involved in this second category are not also excited about the technology they’re creating.
• A third less common but very important category of motivation is de hyping. When a particular technology field is super hot, the public perception of what the technology can do can be way out of touch with the reality. A de-hyping video, often showing the technology failing in all sorts of ways – can be very effective at bringing the narrative back down to earth around what the tech can actually do. A great example of this was the video from one of the DARPA grand challenges of robots falling over and attempting to turn imaginary valves. Again, the motivation here isn’t always completely clean – sometimes there can be a commercial angle to slowing down the enthusiasm for a particular technology. And without context, showing failures can also be misrepresentative of a technology that works sufficiently well to be viable in some contexts, and not in others.
• The type of video itself also typically falls into one of three categories. The first category concerns completely staged videos – in the robotics and autonomous vehicle space, this could mean a supposedly autonomous robot or vehicle that is actually fully controlled by a human operator, often off camera. The context matters here – there are legitimate reasons why this might happen, including a journalist wanting a particular shot that is not immediately safely achievable in fully autonomous mode, often outside of the intended operational domain of the platform. Maybe the journalist wants to lie down in front of one of the robot’s wheels. Most high profile robotics labs that do a lot of media have had to deal with this situation. 
• A slightly less contrived but still sometimes misrepresentative video demonstration is when the tech demo has been shot many times, with the final video being reconstructed from the best bits of each demo run. You can sometimes detect this in almost imperceptible cuts in the video. Again, there are legitimate reasons for the video shoot to occur this way, but disclosure is important. This point is particularly important because it helps the viewer understand whether this is a technology that runs flawlessly every time, or one that can work but is not yet reliable.
• Whether a video that shows a robot doing something once but perhaps not reliably is notable depends on the context. A couple of decades ago, showing a bipedal or quadruped robot that could walk reliably over even slightly undulating terrain was something that was still very challenging for the robotics field, and hence was notable even if it wasn’t entirely reliable. Nowadays, for a few thousand dollars, you can buy an off-the-shelf quadruped robot that is quite capable of doing this – so showing a robot walking around, in of itself, is no longer very notable.
• The third type of video shows robots or autonomous systems out and about in their operating domain, either in full commercial operation, or in large scale long term pilot trials. These videos are much rarer, simply because the scale of resource and development needed to get to this stage is often substantial, ruling out many of the smaller scale demo videos that a typical university lab might produce. But even those these videos often represent a more mature technology stage, there are a few key things to remember.
• The first is that you’re seeing a few minutes of video of platforms that may have been operating for, accumulated across the fleet, many years of operation. You’re only seeing a tiny, tiny fraction of their total operational time. Often the video will focus on showing the most impressive “feats” of what the platform can achieve in a problematic situation – but you’re not seeing what happened the other ninety-nine times the system encountered a similar problem. This is an issue because, while it’s still very hard, it’s far, far easier to make a robotic system capable of occasionally doing impressive things in a certain context, than reliably doing impressive things in all contexts.
• For balance, sometimes these videos will show examples of failure points, more often in research lab demonstrations where there is more expectation to show failure cases. Again, you’re seeing the failures that the creators of the video have deliberately chosen to show you, not necessarily a balanced representation of the actual failure modes and frequency of failure. Sometimes these videos of failure will be an authentic and representative account of the current system limitations, and sometimes they will be disingenuous representations of failures that aren’t really substantial problems, whilst omitting the most significant failure modes. 
• The application context is super, super important here. The significance of a robot doing some task once, but perhaps not super reliably, depends on the context and application domain. A consumer toy robot that does some really impressive human-robot interaction but doesn’t always work perfectly, is potentially acceptable in that domain. A consumer might accept that extra level of awesome interactivity even if it comes at the cost of some reliability. The same goes for say a service robot like a home robot vacuum cleaner – if it gets stuck once a week or fortnight and needs rescuing, it may still be a commercially viable product that people buy, at least until something better comes along. On the other hand, an on-road autonomous vehicle that mostly navigates a tricky situation but occasionally runs into a pedestrian is completely unacceptable. From the short demo video that shows the system doing it once or a handful of times, you really can’t make any strong conclusions about the viability of the tech – you simply don’t have sufficient evidence. To take the autonomous vehicles example further: it is most definitely impressive from a technical standpoint when an AV is shown to navigate a very tricky situation on city streets, and a lot of development and innovation has had to occur for it to reach that capability in the first place. But there’s still a substantial gap between showing that it is capable of doing that a few times in a video, and it being capable of doing that at a reliability level that is sufficient for ongoing operations.
• Another way of thinking about this is to imagine a hypothetical scenario where there were two autonomous vehicle companies, one of whom had a fully viable and reliable AV capability, and one which was advanced but still a long way from a sufficiently reliable service. Their two minute videos showing a few snippets of their vehicles doing impressive navigation through crowded city streets could very easily look near-identical. You’d need more information from other sources – such as quantitative performance reports – to make a more informed assessment. 
• Tech demo videos will sometimes have important or influential people talking, often very enthusiastically, about the technology they’ve just witnessed. It’s important here to somewhat decouple that person’s profile and achievements from the words that they are saying – they may have been one of the most successful entrepreneurs or researchers in the field, but that doesn’t mean their predictions or statements about the technology are particularly informative. This is especially true in certain very-hard-to-predict fields, like for example autonomous vehicles, where very clever, very experienced and highly accomplished people have in good faith made predictions that have often proven to been wrong. The takeaway here is simple: moderate the level to which the endorsement of a high profile person influences your take on the technology. 
• At the opposite end to highly contrived videos is another phenomenon, when the demos seem to get notably worse, rather than better. There is often a very good reason for this – the researchers and engineers behind the tech have gone back and solved a hard problem in robotics or automation in a new way that is cheaper, more generalizable or needs less equipment.
• A classic example of this would be drones. For many years, the fancy videos you saw of drones doing acrobatics were achieved with the aid of external localization systems – a network of sensors distributed throughout a room that, combined with markers placed on the drone, fed very accurate, very low latency information to the drone about where it was located in three-dimensional space. With that information, those acrobatics were much more achievable. A major step forward, even though it may have looked like the tech was going backwards for a while, was when researchers switched to so-called “onboard-only” drone navigation systems, where the drone no longer had access to this external information. Instead it had to rely entirely on the sensors and compute it carried itself. This understandably led to a momentary backwards step in terms of what the drones could do, which has rapidly improved since then.
• So superficially what look like backwards steps in the tech can actually be important steps forward to making the technology more widely applicable and economically viable. Other common situations when this situation can occur are when a robot achieves something with a much more limited or cheaper set of sensors or computational power, and when a robot achieves a task with less explicit or implicit human involvement: for example, having genuinely learnt how to do the task, rather than executing a program largely hand crafted by a person. If the video creators have done a good job, they’ll have some explanation at the beginning that explains why this is a notable step forward.
• Tech demo videos are here to stay and serve a valuable purpose in communicating the broad capabilities of a new robot or autonomous technology in a very short period of time, to a much wider audience than would be reached by an academic paper or a technical report. They should rightly generate excitement, but this excitement should be informed by an understanding of what you can and can’t infer from those couple of minutes of footage alone.