How Self-Driving Cars Think
A self-driving car perceives the world through sensors, builds a model of its surroundings, predicts what others will do, and plans a safe path step by step.
A complete interactive classroom, not just a preview.
Start when you are ready to enter this Stage's 11 scenes and explore, respond, and learn as you go.
When a self-driving car sees the road, how does it actually decide what to do next?
- sensing
- Cameras, LiDAR, radar, and ultrasonic sensors each capture a different slice of the world around the car.
- perception
- Raw sensor data is turned into labeled objects like cars, pedestrians, and lanes.
- localization
- The car matches its sensor view to a high-definition map to know exactly where it is.
- prediction
- The car forecasts where other road users are likely to move in the next few seconds.
- planning
- A safe trajectory is chosen that follows the road, obeys rules, and avoids predicted hazards.
- control
- Steering, throttle, and brake commands translate the planned path into real motion.
Self-driving cars just react to whatever is directly in front of them.
Show that they build a 3D model of the full scene, predict others' motion, and plan a path before reacting.
Cameras alone are enough for a car to see everything on the road.
Demonstrate that LiDAR and radar fill in gaps cameras cannot handle, especially distance and bad weather.
The AI drives with full attention at all times without ever getting confused.
Show that perception is uncertain and the planning system continuously weighs confidence levels.
- basic understanding that AI can recognize images
- deep learning math
- specific company products
- legal and ethical policy debate
- V2X and connected infrastructure
- Learner can name the three sensor families and what each is best at.
- Learner can label the six stages of the self-driving pipeline.
- Learner can adjust a planner to avoid a predicted pedestrian in a mini game.
- Explain to a friend why a self-driving car might still brake for an empty plastic bag.
Curious learners with no engineering background who can use a mouse and sliders.
- 01How Does a Self-Driving Car Think?slideOrientationObserve
Introduce the question and frame the six-stage pipeline the car follows every fraction of a second.
- The car runs a loop, not a single decision
- Sensors in, driving commands out
- Six stages: sense, perceive, localize, predict, plan, control
- 02Compare the Car's SensorsinteractiveModel buildingObserve
Switch between camera, LiDAR, and radar views to feel the strengths and blind spots of each sensor.
- Camera gives color and shape
- LiDAR gives precise 3D distance
- Radar sees through fog and measures speed
- No single sensor is enough
- 03Detect What Is on the RoadinteractiveMisconception repairChoose
Tune a detection model's confidence threshold and see how it labels cars, pedestrians, and cyclists from raw camera input.
- Lower threshold catches more but adds false alarms
- Higher threshold is precise but may miss a pedestrian in shadow
- Every label carries a confidence score
- 04From Pixels to a 3D WorldslideModel buildingObserve
Bridge perception and planning: explain how detected objects and a high-definition map combine into a live model of the road.
- Detections are dropped into a 3D coordinate frame
- HD maps add lanes, signs, and crosswalks
- Localization snaps the car onto the map within centimeters
- 05The Self-Driving PipelineinteractiveModel buildingObserve
Walk through the full six-stage loop as an explorable diagram, with each node revealing what it does.
- Sense, Perceive, Localize, Predict, Plan, Control
- Output of one stage feeds the next
- The loop runs many times per second
- 06Predict the PedestrianinteractivePredictionPredict
Given a short clip, predict whether the pedestrian will cross, wait, or step back, then reveal the car's forecast.
- Prediction is about probability, not certainty
- Body pose and speed give strong cues
- The car considers several futures at once
- 07Plan a Safe PathinteractiveApplicationConstruct
Drag waypoints to plan a route around a stalled car while obeying the speed limit, then watch the controller follow it.
- Planner balances safety, comfort, and progress
- Waypoints must respect lane geometry
- Control translates the path into steering and throttle
- 08What Happens When the AI Is Unsure?slideMisconception repairExplain
Address the misconception that the AI is always certain, showing how low confidence triggers safer behavior.
- Every prediction has a confidence number
- Low confidence leads to slowing down or pulling over
- Uncertainty is a feature, not a bug
- 09Drive a Full City LoopinteractiveApplicationApply
Take the role of the planner across a busy intersection, adjusting speed and lane choice to keep the ride safe.
- Multiple agents must be predicted at once
- Trade-offs between speed and safety are continuous
- The same pipeline runs every tick
- 10Build Your Own Self-Driving PipelineinteractiveSynthesisConstruct
Drag the six modules into the correct order, then test the assembled loop on a mini scene.
- The order matters: sense before plan, plan before control
- All six stages are required for a working system
- Your mental model is now the real pipeline
- 11You Now Know How a Self-Driving Car ThinksslideSynthesisExplain
Summarize the six stages, the role of uncertainty, and where to look next in the field.
- Sense, perceive, localize, predict, plan, control
- Multiple sensors, multiple predictions, one safe plan
- Open frontier: edge cases and long-tail situations
Discussion threads for a Stage aren't available yet.
