BUILD A WORLD AI Labs · Spatial intelligence · Benchmark
VLGE transforms how 3D worlds are built, played, and explored into synchronized environments, telemetry, event streams, physics-grounded scenarios, and behavioral micro-signals, all anchored in real human interaction. Together, they form the human behavioral data layer for AI systems learning to perceive, reason, and operate in 3D space.
The benchmark
A benchmark is the reference set a lab measures its own agent and world models against. Ours is a real person's full trajectory through a captured space, route, dwell, interaction, hesitation and attention, and the human-grounded agents that reproduce that behavior at scale. The benchmark itself is published free, the way DROID, LIBERO and Open X-Embodiment are. Scoring against it requires the human reference data, which is ours.
How far the lab's agent sits from the real human distribution, per task, per world, per metric.
Real people build and play inside the world while every channel records: position, kinematics, hand and finger articulation, grasp episodes, gaze, dwell, hesitation and decision points. This is the reference distribution, not a scripted demonstration.
LLM-based agents are conditioned on that observed behavior rather than on hand-written policies, then run the same worlds and tasks. Because they inherit the human distribution, they generate interactions, decisions and trajectories at a scale human capture alone cannot reach, without drifting from the people they model.
A lab's agent is scored against both: the human reference and the grounded agents reproducing it. Every metric is a number, and every number is traceable back to the synchronized streams it was computed from.
What gets scored
First public release in preparation. Reference worlds, tasks and the metric definitions above are fixed, baselines are being measured now. Request early access →
The session record
Every session is both a navigation trace and a manipulation demonstration. It becomes a continuous event feed of pose, velocity, state changes and dwell, time-aligned with source timestamps, carrying the layers below: per-finger articulation, the full arm and shoulder kinematic chain, complete grasp episodes with their failures, labelled interaction scenarios, physics-driven motion, a LiDAR channel, and a per-frame behavioral layer. Exports preserve the raw data material and can be adapted to the buyer's training purpose. Below are four real captures, replayed and interleaved as they're stored.
Per-finger flexion and abduction every frame, three joints per finger, fifteen per hand, at the same frequency as full-body movement. Fingertip position and velocity through grasp and release, and confirmed contact point, normal, distance and target, recorded in the object's own frame so a contact site is reusable across sessions as a grasp affordance.
Closes the gap between full-body motion capture and the fine motor control object handling needs.
Shoulder, elbow and wrist angles and angular velocity; requested versus resolved wrist pose with the overrun in metres when a target sits beyond comfortable range; and the whole-body compensation, torso lowering, forward lean, shoulder assist, when the arm alone cannot reach. Local rotations retarget cleanly to robot arms with different link lengths.
Separates a reachable target from a strained one, for workspace design and reach planning.
Every grasp is a complete episode, not an event flag: grip taxonomy (power, pinch, palm) with the pre-contact hand pose, object mass, dimensions, compliance, break force and torque, closure progress, grip-volume occupancy, opposing-finger contact count, object pose in the hand frame every frame, and release linear and angular velocity. Failed attempts get their own record.
The engine's per-frame validation fields are a ready-made grasp-success classifier, with no annotation, and the failures give a policy its negative examples.
A labelled library of pick-up, place, push, pull, open and close. Each scenario pairs a pre- and post-interaction snapshot to isolate the effect on the object; push and pull carry applied force direction, displacement and object velocity; open and close carry articulated state such as hinge angle or slide position. Every discrete interaction is also a semantic event with sub-second precision.
Trainable manipulation and object-affordance data straight out of the log, without annotation.
When the engine takes over a participant's movement, the full physical event is recorded at locomotion fidelity: fall height, impact velocity, contact point and angle, resulting body trajectory, object collision response, and the frame-by-frame recovery path back to controlled movement including residual instability.
A continuous, physically accurate record of body dynamics under uncontrolled conditions.
A dedicated channel from virtual sensor rigs mounted on participants, with the full sensor configuration and a raw scan payload up to 360°. Each frame adds per-ray 3D hit point, surface normal, distance, material tag, reflectivity and object identity, plus aggregated semantic and material counts.
Every frame is at once a geometric measurement, a semantic scene description and a material classification, at a scale that would be prohibitive to acquire physically.
The BrainRecord pipeline adds 48 per-frame features and session-level analytics across world creation and world interaction: velocity decomposition, turning, acceleration and curvature; frame-level action labels; gaze, attention and spatial coverage; and hesitation, decision points, micro-corrections, commitment and control effort.
Outputs are analytics-, visualization- and training-ready, including engagement, hesitation, novelty and goal-directed-versus-wandering heatmaps.
Full field-by-field dictionaries available under MNDA.
The corpus
Each point is one timestamped state. Depicted are 61,736 points across 30 sessions, a browser-sized sample of a much larger captured corpus. Enough to read the structure, synchronization and signal quality for yourself. Drag to orbit; hover any point to read its raw JSON.
Process layer · decision-making
Where a person hesitated, showed intent, weighed it, and committed to a direction, read straight from the deliberation, hesitation and movement channels of one real session. This is the cognitive layer that no robotics or driving dataset captures.
Process layer · interaction
Movement is half the story. The other half is the build itself: Every time the user places, moves, rotates, scales, or lights an object, we capture it. Below is one real Edit-Mode session replayed from its event log. Scrub the timeline, or switch builders. Each tick is a real recorded action.
Two People, One Space,
Two Completely Different Paths.
What data is available
The console plays one real captured session end to end, with every panel reading the same playhead. As you scroll the streams, it reiterates that session through each one.
Production-grade interactive 3D scenes, from interiors and streets to towers and stores, authored to be walked and played inside online.
Gaussian-splat captures of worlds built and recorded from reality: photoreal radiance fields for training perception and novel-view synthesis at the fidelity the physical world actually has, with derived perception labels for registered geometry.
The structured layer beneath the geometry: labelled zones, object semantics, adjacencies, sightlines and coordinates, the grammar a model needs to reason about a space. Buyer exports define metrics in fixed time windows so fingerprints compare cleanly across sessions.
Timestamped behavior at inferred source cadence, with buyer-cadence resampled exports: pose, gaze, velocity, dwell, build and interaction events. This is the record that turns a static scene into how it's actually used, and it's the JSON streaming on the right.
Realistic physics drive every interaction, enabling object manipulation, accident scenarios, creative puzzles, and emergent behaviors. Every action is paired with full-body motion capture, from body pose and hand articulation down to individual finger movements, creating a complete behavioral record stored alongside the event stream.
The same streams, packaged as a scoring set: a fixed reference world and task, one real human trajectory as ground truth, the human-grounded agent runs that reproduce it, and the metric definitions above. A lab runs its own agent on the world and gets a number per metric. How the benchmark works ↑
Need a specific environment, task, population or modality? We build the world, run the collection, and deliver the dataset to your spec, ready to scrub, sample and export.
Delivery package
The live instruments prove synchronization; the benchmark proves performance. VLGE turns the same human-anchored streams into benchmark-ready datasets with structured exports, enrichment layers, and standardized splits, supporting quantitative evaluation through benchmark metrics and qualitative evaluation through synchronized behavioral, event, telemetry, and 3D streams.
Open any component for what it contains, the fields it ships, and where to see it running on this page.
Quantitative scoring against the human reference. A per-task number for how far an agent's behavior sits from the real human distribution.
Scored on the attributes the data actually carries, which is what makes it behavioral rather than task-completion. Route and dwell say where the agent went, decision and attention say how it chose, manipulation and reach say how it acted. Each is a distribution comparison against real human sessions in the same world on the same task, so runs are comparable across labs.
Objects react through collision, gravity, constraints and force, never scripted animation. What the data records is a physical consequence, not a keyframe.
The same action produces different outcomes under different conditions, and both are captured at full fidelity. When the engine takes over a participant's body, the whole event is recorded: fall height, impact velocity, contact point and angle, the resulting trajectory, and the frame-by-frame recovery back to controlled movement.
Source cadence runs roughly 90 to 280 Hz depending on session and channel, resampled on export to whatever fixed rate you train at. One clock, one coordinate convention, field dictionary included.
Pose, video, events and sensor frames arrive already aligned, so there is no reconciliation pass on your side. We match your axis convention, unit system and frame rate rather than asking you to match ours, and every export carries its own source cadence per channel so nothing is inferred.
What a camera would have seen. Calibrated views with labels that are exact rather than annotated, pixel-aligned to known 3D truth.
Because the geometry is authored rather than reconstructed, depth is the true depth buffer, masks are true object identity per pixel, and boxes are projections of actual object bounds. Intrinsics and extrinsics ship per frame, so views are reprojectable for novel-view synthesis, depth, segmentation and detection from one capture.
The BrainRecord layer: 48 per-frame features plus session analytics, across both world creation and world interaction.
This is what separates a trajectory from a decision. Velocity, turning, acceleration and curvature are decomposed per frame; hesitation, decision points, micro-corrections, commitment and control effort are measured; every frame carries an action label. Outputs are heatmap-ready, including engagement hotspots, hesitation zones and goal-directed versus wandering behavior.
Instruction, execution and outcome for vision-language-action training. What the person was asked to do, what they did, and whether it worked.
A session run against a written instruction has intent attached to it rather than being untargeted exploration. Audio and transcripts sit alongside the action stream, and each attempt carries an outcome label including the repair, the second try after a first attempt fails, which is the part most instruction-following datasets are missing.
Every manipulation, movement and environmental change as a replayable event stream, including the attempts that failed.
A grasp is stored as a complete episode, not a flag: the grip formed before contact, the object's pose in the hand frame every frame, closure progress and opposing-finger contact, and the velocity at release that separates a placement from a drop. Scenarios pair pre- and post-interaction snapshots, and failures get their own record, the negative example a recovery policy needs.
Full-body pose, the complete arm and shoulder chain, and per-finger articulation, all at one frequency.
Conventional motion capture gives you the body and loses the hand. We record flexion and abduction for each digit, fifteen joints per hand, at body-movement rate, with fingertip position and velocity through grasp and release. The arm chain adds the IK residual between requested and resolved wrist pose, and the whole-body compensation when the arm alone cannot reach. Local rotations retarget cleanly to robot arms with different link lengths.
A dedicated LiDAR channel from virtual rigs mounted on participants, with full sensor configuration and a raw scan payload up to 360°.
Beyond the point cloud, every frame carries per-ray hit point, surface normal, distance, material tag, reflectivity and object identity, plus aggregated semantic and material counts. Each frame is a geometric measurement, a scene description and a material classification at once, at a scale that would be prohibitive to acquire physically.
Several real people in one world on one clock, for social navigation, collision avoidance and crowd modeling.
Co-present sessions are one synchronized capture, not separate traces stitched afterwards, so inter-personal distance, yielding, path crossing and mutual gaze are measurable frame by frame. Humans and grounded agents can share the same world and are recorded identically, so a mixed population is available where pure human capture would not scale.
Example applications
Ground generative world models in how real spaces are built and changed.
Populate simulators with real human trajectories, not scripted agents.
Navigation & manipulation priors from human behavior in 3D.
Agents that learn to perceive, plan and act in physical space.
How people read, cover and reason about space, measured per session.
In depth
Explorable 3D and 3DGS environments of any real space, a museum, a street, a store, a tower, a game, authored to be walked, built and played inside online. Watertight geometry, materials, navmesh and floor plans, or a photoreal Gaussian-splat capture of the place itself.
The environment and 3DGS streams ↑Every session becomes a synchronized record on one clock: pose and kinematics, hand and finger articulation, grasp episodes, semantic events, physics, LiDAR, and a per-frame behavioral layer covering dwell, hesitation, attention and decision points.
What a session looks like ↑LLM-based agents that create a world and play inside it, perceiving, reasoning, acting and changing the world as they go. Conditioned on observed human behavior rather than hand-written policies, they scale human-believable data far past what human capture alone can reach.
How grounded agents run ↑Agents grounded in real human ground truth close the sim-to-real gap for embodied AI. The benchmark built on that data is free for any lab to run, and scores how far their agent sits from the real human distribution, per task, per world, per metric.
The benchmark framework ↑