Engineering Notes
What we learned building robot simulation
Physics, RL, sensors, navigation, standards — engineering notes from the imSim team. 69 posts grounded in real runs, commits, and measured data.
AMR Navigation Basics
12What Is an Occupancy Grid — Drawing the Sensed World as a Grid
The occupancy grid — how a mobile robot represents its world as a 2D probabilistic grid of 'wall / free / unknown': the resolution trade-off, why it must be a *probability*, the limits of 2D, and the difference between how a real robot (SLAM) and a simulator (ground truth) build it.
From Static Map to Nav2 Costmap — A Map That Knows Walls vs. One That Doesn't
How an occupancy grid flows through Nav2's map_server into a static-layer costmap that the planner can actually use, and why substituting an all-free empty map makes the robot believe the world has no walls.
Costmap 2D Dissected — How the Static, Obstacle, and Inflation Layers Compose
Nav2's costmap is not a single map but a composite of static, obstacle, and inflation layers stacked together. This post covers what each layer contributes and the real bug where removing the obstacle layer caused robots to pass through each other.
What Inflation Does — Bloating Obstacles So the Planner Can Treat the Robot as a Point
Inflation wraps a cost gradient around obstacles to compensate for the robot's physical width, letting the path planner treat the robot as a dimensionless point. This post explains the principle and why a single inflation radius breaks down in heterogeneous fleets.
Global vs Local Costmap — Dividing the Work Between Long-Range Planning and Instant Avoidance
Nav2 splits costmaps into a global one for whole-map path planning and a local rolling-window one for real-time obstacle avoidance. This post covers their roles, dynamic sizing, and why multi-robot deployments must populate both.
Nav2 Lifecycle and Behavior Trees — How the Navigation Stack Wakes Up
Nav2 servers activate in a strict lifecycle order, and bt_navigator orchestrates planning, tracking, and recovery through a Behavior Tree. This post explains both and covers the real failures that occur when activation order goes wrong.
Path Planners NavFn and Smac — Finding the Lowest-Cost Route Across the Costmap
The planner treats the global costmap as a graph and solves for the shortest path. NavFn uses Dijkstra/A*-style potential field propagation; Smac adds kinematic constraints. The most common planner failure is not algorithmic — it is the goal falling Off Grid.
The RPP Controller — How Pure Pursuit Converts a Path to Wheel Velocities (and the 0.05 Rotation Bug)
Regulated Pure Pursuit tracks a lookahead point on the path to produce cmd_vel. This post covers the principle and a real debugging case where odom acceleration clamping permanently pinned angular velocity at 0.05 rad/s.
Recovery Behaviors — How the Robot Gets Itself Unstuck
When path following fails or the robot stalls, recovery behaviors like spin, backup, and wait change the situation and retry. This post covers how the Behavior Tree triggers them and how fleet-level stops must cancel Nav2 goals — not just halt the wheels.
VFH+ Reactive Avoidance — Finding a Gap with a Polar Histogram, No Costmap Required
Vector Field Histogram reduces a dense occupancy grid to a polar density histogram and picks the most open steering direction. VFH+ adds robot-width compensation via angular enlargement — useful as a lightweight fallback when Nav2 is not running or dynamic obstacles dominate.
How Nav2 Connects to Isaac Sim over ROS2 — TF, cmd_vel, and Namespace
Isaac Sim and Nav2 form a closed ROS2 loop: sensor data (scan/odom) flows from the simulator into Nav2, and commands (cmd_vel) flow back out to the simulated wheels via DriveAPI. Multi-robot isolation uses namespaces, and sim odom lag affects closed-loop controllers.
AMR Safety Simulation — From Navigation Stack to ISO 3691-4 Reference KPIs
The navigation concepts covered in this series — occupancy grid, costmap, planner, controller, and avoidance — ultimately answer one question: is this AMR safe to operate alongside people? This post shows how those components map to ISO 3691-4 safety KPIs measured in simulation.
How Isaac Sim Works
10Isaac Sim Boot Order — SimulationApp Must Come First
Isaac Sim's omni.* modules do not exist until the Kit runtime is running. SimulationApp must be instantiated before any omni.* import, and world.reset() is a second hard boundary where physics state is locked in. imSim always uses Fresh mode — one boot per run — for reproducibility.
Composing a Scene with USD — Stage, Prim, and Reference
An Isaac Sim scene is not a single monolithic 3D file. It is a tree of prims on a USD stage. Environments and robots are inserted as references; runtime obstacles are defined as primitive geometry. The prim path is the address used to control every object later.
PhysicsScene and the Solver — How Physics Attaches to a Scene
Gravity, collision, and solver iterations are stored in a single PhysicsScene prim that broadcasts its settings to every rigid body in the scene. PhysX is the default — kinematic mode is only a shortcut for quick tests, not a substitute for real physics.
Why Physics Runs at 200 Hz and Rendering at 60 Hz — Substeps
Why a simulation loop steps physics multiple times per render frame (substeps) — decoupling timesteps to hit stability and performance at once. This is also why legged robots fall apart at 60 Hz.
What It Means to Drive Wheels with DriveAPI — Torque, Not Teleportation
Wheels and joints are articulation joints; DriveAPI (target velocity + stiffness/damping) lets PhysX compute the torque that actually spins them. The fundamental difference from kinematic position assignment.
Bridging to ROS2 with OmniGraph — A Visual Graph That Publishes Sensor Topics
How sensor-to-topic publishing and /cmd_vel-to-actuation are wired up as OmniGraph (visual compute graph) node chains — and the common pitfall when receiving external ROS2 commands.
Attaching Sensors — PhysX LiDAR vs RTX LiDAR, Camera, IMU
Sensors are created as prims in the scene; RTX LiDAR extracts a point cloud from the render pipeline. The fundamental difference between PhysX LiDAR and RTX LiDAR.
Why Is Rendering On If It's Headless? — headless ≠ no-render
Even without a display window (headless), HLS streaming, video recording, RTX sensors, and synthetic data generation all require rendering — the common misconception that 'headless means render off.'
Fresh Mode vs Daemon Mode — The Temptation of Fast Reuse and the Problem of State Contamination
Fresh mode boots a new simulator process per run; Daemon mode reuses a running process. The latter looks faster, but cross-run state contamination destroys the reproducibility that validation depends on.
The Life of a Simulation Run — From Submission to Video and KPIs
The complete lifecycle of a simulation run triggered by a single button press: queue, worker, runner, outputs. What each stage does and where artifacts come from.
Scenario DSL (OpenSCENARIO)
4What Is OpenSCENARIO — Borrowing an Automotive Standard for Robot Simulation
Why scenarios are written in a declarative language (DSL) rather than code, what imSim borrows from ASAM OpenSCENARIO and what it doesn't, and the three-layer design that adopts the language but builds its own domain actions.
Anatomy of a Custom OSC Action — setup, tick, terminate
How domain actions like robot spawning, firing, and classification are built. The internal structure of a single custom action called by an OSC scenario — setup (once), update (every tick), terminate (cleanup).
Scenario Logic Must Live in the DSL — The OpenSCENARIO Self-Containment Philosophy
If half the scenario lives in OSC and half lives in a separate script, it's a one-off demo. An end-user must be able to reproduce the same result using only the domain language (OSC) — that is the bar for a deliverable system.
How an OSC Scenario Becomes a Behavior Tree — Compiling serial and parallel into a Tree
How a human-readable OSC scenario text becomes an executable Behavior Tree: serial maps to Sequence, parallel maps to Parallel, and each action becomes a leaf that emits SUCCESS/RUNNING/FAILURE.
Data Contracts (Spec & Schema)
4Adding a Robot = One JSON File, Zero Code — The Robot Manifest Promise
The platform's core value proposition: any robot runs in simulation from a single manifest (JSON). The single robot abstraction that makes it possible — and where the promise leaks.
I Created the Environment But It Doesn't Appear in Runs — The Single Source of Truth Trap
You built a new environment and registered it in the catalog, but it doesn't show up in run selection. The cause: registering only in the 'display catalog' table while omitting the 'runtime registry' the orchestrator actually reads — a classic SSOT failure.
RuntimeConfig — One Contract to Describe an Entire Run
RuntimeConfig bundles environment, robots, scenario, sensors, and options into a single declarative object. It separates *what to run* from *how to run it*, keeping the contract clean and runs reproducible.
We Were Writing the Same Thing Five Different Ways — Schema Fragmentation and Consolidation
The same entity (vessel specifications) was described in five different schemas. Starting from 0% canonical adoption, this is the story of consolidating to a single authoritative schema and attaching standards-based validation.
Standards & Compliance
3What 'Measuring' Robot Accuracy Under ISO 9283 Actually Means — AP vs. RP
The key concepts of ISO 9283, the performance standard for industrial manipulators — the difference between accuracy (AP) and repeatability (RP), the standard cube workspace measurement protocol, and real numbers measured in simulation.
How Collaborative Robots Avoid Injuring People — ISO/TS 15066 Contact Force Limits
For a cobot to work alongside humans without a safety fence, the forces it can exert on contact must stay within human pain thresholds. ISO/TS 15066's contact force limits, speed and separation monitoring (SSM), and the role simulation plays in verification.
The NATO 4-Domain Standards Stack — Unifying UAV, UGV, USV, and UUV
A 3-layer architecture that integrates air, ground, surface, and subsurface unmanned systems via STANAG 4586/4677/4817. Regardless of domain, all platforms share a common safety, communications, and metadata layer for joint operations.
Physical AI Basics
5What Is Physical AI — Intelligence That Steps Out of the Digital World Into the Physical One
Physical AI — intelligence that perceives and acts in the physical world, beyond the digital AI of text and images: why it is emerging now, how it tackles its biggest wall (data), and why simulation is the core infrastructure.
OpenUSD Scene Composition — Assembling a World with Layers, References, and Payloads
USD is not a single 3D file format — it is a non-destructive layer-compositing system. How the same asset can be reused across many scenarios without conflicts, through layers, references, payloads, variants, and LIVRPS strength ordering.
Synthetic Light for Real Sensors — RTX-Based EO/IR/LiDAR/Radar/Sonar
How multi-domain sensors — EO, IR, LiDAR, Radar, Sonar — use physics-based rendering and ray or acoustic tracing to generate synthetic data, and what makes each modality different.
A Digital Twin Is Not a 3D Model — It's a Living, Synchronized Replica
A digital twin is not a pretty 3D visualization — it is a virtual counterpart that stays synchronized with the real system. Sensors update the twin; the twin predicts and feeds back control. The bidirectional loop is the definition.
From Simulation-Trained Policy to a Chip on the Robot — The Reality of Edge AI Deployment
Getting a policy trained on massive GPU clusters onto a small edge chip for real-time inference. The constraints of quantization, inference optimization, latency, and power — and how simulation validates the bridge between them.
Field Notes
4ISO 10218 — A Major Overhaul After 14 Years (2025), Bringing Cobot Safety into the Main Standard
In January 2025, ISO 10218-1/-2 was comprehensively revised for the first time in 14 years, absorbing collaborative robot safety (ISO/TS 15066) into the normative text. What changed, and why it matters for simulation-based safety verification.
ROS 2 Distribution Map (2026) — Jazzy LTS and Kilted: Which One to Use
ROS 2 releases a new distribution every May, with even-year releases receiving 5-year LTS support. A practical guide to choosing between the recommended LTS Jazzy Jalisco and the non-LTS Kilted Kaiju in 2026.
Robot Simulator Landscape 2026 — Isaac Sim, MuJoCo, Genesis, Newton: When to Use What
In 2026 the robot simulator market is not dominated by a single winner — it is a division of labor by use case. Isaac Sim/Lab, MuJoCo, Genesis, and the newly announced Newton from NVIDIA, DeepMind, and Disney: what each is for and when to reach for it.
Humanoid Robots in 2026 — Between the Highlight Reel and Shop-Floor Reality, and Where Simulation Fits
In 2026, humanoid robots have crossed from research demos into early commercial deployment. The current state of Figure, Optimus, and Unitree — and the gap between impressive highlight videos and operational reality that simulation exists to bridge.
Debugging War Stories
5The One Line That Sank Seven Ships to the Mariana Trench — The Substring Matching Trap
A surface buoyancy setup used a single identifier check to decide whether buoyancy was needed. Because the submarine's identifier didn't contain 'UUV', the entire fleet's buoyancy was switched off and all seven vessels sank to -2989 m.
The Joint Was Running 40× Over the Speed Limit — What Peak KPI Hid: 12,078 dps
A peak KPI that only measured maximum lift height kept awarding PASS even when the arm dropped the object mid-transfer. Once trajectory KPIs were added, cells that had been marked PASS turned out to have joint velocities 40× over the recommended limit.
Static Analysis Said 'Impossible' — But the Actual Run Rendered It: Gaussian Splatting in Isaac Sim
Code analysis concluded that Gaussian Splatting was impossible in this build. Then we actually ran it and got a photorealistic interior. A textbook case of empirical measurement overturning static reasoning.
One Vehicle Rolled Over at -63° — A Physics Bug That Was Actually Environment Version Drift
A vehicle convoy scenario that had already passed validation suddenly flipped one vehicle to -63° roll. We spent a long time suspecting physics, but the real culprit was a batch job that had silently swapped the environment DB from the validated flat terrain to a steep version.
Vehicles Are Passing Through Each Other — The Price of Confusing ORCA Avoidance with PhysX Collision
Vehicles drove straight through each other and through obstacles. A record of tracking down the bug — rooted in confusing two distinct collision concepts: ORCA proactive avoidance vs. PhysX rigid-body contact.
Maritime & Subsea Physics
4The Sonar Equation — How to Calculate Whether a Submarine Can Be Heard, and Why Active and Passive Are Different
The sonar equation — the dB accounting ledger for underwater detection. How the active (two-way) and passive (one-way) forms differ in their terms, why transmission loss dominates detection range, and what happens if you mix the two up.
The Pipeline for Finding Mines Underwater — CAD → CAC → ATR
A UUV sweeps the seafloor with side-scan sonar and hunts for mines through a three-stage pipeline — detection (CAD), classification (CAC), and recognition (ATR) — and the fundamental trade-off between detection rate and false-alarm rate.
How a Submarine Holds Its Depth — Neutral Buoyancy, Depth Control, and the z=NaN Trap
Surface ships float at the waterline; submarines actively regulate buoyancy to hover at any desired depth. The principle of neutral buoyancy, the depth-control feedback loop, and a subtle simulation bug where z becoming NaN produced false mission successes.
Why Ships Don't Sink — Buoyancy, Stability, and Radiation Damping in Maritime Simulation
The three physical forces that govern surface and subsurface vessel behavior in six degrees of freedom: buoyancy (staying afloat), metacentric height GM (righting moment), and radiation damping (settling oscillations). How simulation solves each one.
RL & World Models
5The Sim-to-Real Gap — Why Policies That Work in Simulation Break on Real Robots, and How to Measure It Honestly
The root causes of the sim-to-real reality gap, domain randomization as the primary countermeasure, and why honest measurement — not just simulation KPIs — is the only way to close it.
Generating Synthetic Data with a World Foundation Model — NVIDIA Cosmos and the Data Flywheel
Real robot data is scarce and expensive. World Foundation Models — video-generation models that understand physics — mass-produce synthetic data for training perception and policy networks, forming a data flywheel that compounds with each iteration.
The End of Training a New Policy for Every Task — VLA Foundation Models and GR00T
From training separate policies for every task to a single model that sees, hears, and acts: the VLA foundation model paradigm. NVIDIA GR00T's dual-system architecture and the role simulation plays in it.
Training Thousands of Robots Simultaneously on a GPU — Isaac Lab Parallel Reinforcement Learning
What reinforcement learning is and why it's slow, why you can run thousands of environments at once on a GPU (it's all tensors), and the truly hard part — reward design. The basics of parallel RL.
RL Policy or cuMotion — They All Plug Into the Same Socket
The 'brain' driving a robot arm can be an RL policy, a classical motion planner, or a foundation model. The PolicyAdapter Protocol defines a common interface so any brain plugs into the same socket — the runner never needs to know which one it is.
Co-simulation & Rendering
4Adding a Second Engine — Why a Dedicated Unreal Simulation Track Exists
Isaac for high-fidelity physics validation; Unreal for cinematic visuals, large-scale terrain, and real-time interaction. The product-track strategy of splitting engines by purpose while reusing a shared orchestration infrastructure.
The UE Stream Only Gets to 'negotiated' From Outside — A WebRTC TURN Relay Story
Unreal Engine Pixel Streaming worked perfectly on the internal network but stalled at 'negotiated' with zero media from the outside. The cause: ICE candidates were all internal IPs. The fix: TURN relay.
Viewing and Controlling a Headless UE From a Browser — The Three Pillars of Pixel Streaming
The three pillars that let a browser see and control a remote GPU server running Unreal Engine: headless rendering, WebRTC streaming, and real-time bidirectional interaction. Plus: version compatibility is half the battle, and unattended GPU operation needs an auto-stop guard.
What Is Co-simulation — Why ZMQ, Why Leader/Follower
Rather than demanding one engine handle both precise physics and photorealistic visuals, co-simulation routes only state values across a lightweight message bus to connect two engines that each excel at different things.
Combat Modeling
4What It Means to Simulate Combat — The World of Multi-Resolution Modeling
From the ballistics of a single tank round to a corps-level week-long operation, no single resolution can capture everything. The fundamental idea behind multi-resolution combat modeling — and how physics-level (T1) and abstract-level (T2) models calibrate each other.
Weapon-Target Matchup by the Data — Why an ATGM Scores 0.90 Against Armor but 0.20 Against Infantry
A weapon's effectiveness is target-dependent, not absolute. Anti-tank guided missiles are lethal against armored vehicles but wasteful against infantry. Encoding that matchup as catalog data — not code — lets a single line extend the engagement engine to any domain.
One Anti-Ship Missile and a Layered Defense — The Structure of Naval Engagement
The engagement chain from ASCM launch to ship impact, and the layered defenses — hard-kill (SAM, CIWS) and soft-kill (chaff, decoys, jamming) — arrayed against it at each phase. Why saturation salvos are the real threat.
Lanchester's Laws — The Mathematics of Concentration and the Attrition Combat Model
The mathematics of attritional combat formalized by Lanchester in 1916 — the Square Law (advantage of concentration) versus the Linear Law (area fire), and the abstract model foundation for solving large-scale engagements fast.
Infra & Orchestration
4Mounting s3fs Over an Existing Directory Makes Your Data 'Disappear' — Shadowing, Not Deletion
One day, every video under a directory vanished. No reason to panic — mounting s3fs over a non-empty directory had shadowed the originals. The files were never deleted.
The Run Is Stuck at 'requested' — Reverse-Tunnel Zombie Sockets and the Half-Open Trap
A simulation run dispatched to a remote GPU node stayed stuck at 'requested' forever. The culprit was a half-open zombie socket left behind when a VPN idle timeout dropped the reverse tunnel — process alive, forward dead.
Cancelled the Run but the GPU Is Still Pegged — cancel ≠ kill
Setting a run to 'cancelled' in the database does not kill the Isaac process the worker already launched — it keeps holding the GPU. How to kill exactly the right process safely, and why pkill can shoot your own session.
Delivering Assets to Isolated Remote GPU Nodes — Why s3fs Instead of NFS
Remote and air-gapped GPU nodes cannot use NFS. The solution: mount an object-storage bucket (MinIO, S3-compatible) on each node via s3fs (FUSE) so USD assets look like local files. Choose the wrong write path, though, and node views diverge permanently.