ai/cosmos3-edge repository overview

⁠Cosmos 3: Omnimodal World Models for Physical AI

Model Collection⁠ | Code⁠ | White Paper⁠ | Website⁠

NVIDIA Cosmos™⁠ is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.

⁠Model Overview: Cosmos3-Edge

⁠Description

Cosmos3 is a collection of Omnimodal world models capable of generating dynamic, high-quality video, image, audio, and action commands from combinations of text, image, video, and action trajectory inputs. It serves as a foundational building block for a broad range of Physical AI applications and research spanning world understanding, world generation, simulation, and embodied policy learning.

This model is ready for commercial and non-commercial use.

Model Developer: NVIDIA

⁠Model Versions

Released on: 07/20/2026

• Cosmos3-Edge:

  • Given multimodal inputs including text, images, video, and action trajectories, generate coherent text, images, video, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.

• Cosmos3-Edge-Policy-DROID:

  • Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.

• Cosmos3-Super-Image2Video-4Step:

  • Given one or more input images and optional text instructions, generate temporally coherent video sequences that are consistent with the provided visual content. Distilled from Cosmos3-Super-Image2Video using Improved Distribution Matching Distillation (DMD2)⁠, enabling high-quality generation in 4 steps.

• Cosmos3-Super-Text2Image-4Step:

  • Given text input, generate high-fidelity images that are consistent with the provided description. Distilled from Cosmos3-Super-Text2Image using Improved Distribution Matching Distillation (DMD2)⁠, enabling high-quality generation in 4 steps.

Released on: 05/31/2026

• Cosmos3-Nano:

  • Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.

• Cosmos3-Super:

  • Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.

• Cosmos3-Nano-Policy-DROID:

  • Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.

• Cosmos3-Super-Image2Video:

  • Given one or more input images and optional text instructions, generate temporally coherent video sequences that are consistent with the provided visual content.

• Cosmos3-Super-Text2Image:

  • Given text input, generate high-fidelity images that are consistent with the provided description.

⁠License

This model is released under the OpenMDW1.1⁠

⁠Deployment Geography

Global

⁠Use Case

Physical AI: Encompassing robotics, autonomous vehicles (AV), and smart space environments, including industrial and factory-scale applications.

⁠Release Date

Hugging Face 07/20/2026 via https://huggingface.co/collections/nvidia/cosmos3⁠ GitHub 07/20/2026 via https://github.com/nvidia/cosmos⁠

⁠Model Architecture

Architecture Type: Transformer

Network Architecture: Mixture-of-Transformers (MoT)

Cosmos3 is an Omni-modal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation. During inference, text is generated through standard next-token autoregressive decoding, while non-text modalities, such as images, video, audio, and actions, are synthesized through iterative denoising. This unified architecture enables Cosmos3 to model heterogeneous modalities within a single framework while preserving generation mechanisms best suited to each modality.

This model was developed based on: Cosmos Framework⁠

Number of trainable model parameters:

Released on: 07/20/2026

• Cosmos3-Edge: 4B
• Cosmos3-Edge-Policy-DROID: 4B
• Cosmos3-Super-Image2Video-4Step: 64B
• Cosmos3-Super-Text2Image-4Step: 64B

Released on: 05/31/2026

• Cosmos3-Nano: 16B
• Cosmos3-Super: 64B
• Cosmos3-Nano-Policy-DROID: 16B
• Cosmos3-Super-Image2Video: 64B
• Cosmos3-Super-Text2Image: 64B

⁠Input/Output Specifications

• Generator Input
  • Input Type(s): Text, Image, Action Trajectory
  • Input Format(s):
    • Text: String
    • Image: jpg, png, jpeg, webp
    • Action: JSON — 2D array shaped (T, D), where T is the number of frames and D is the embodiment-specific dimensionality
  • Input Parameters:
    • Text: One-dimensional (1D)
    • Image: Two-dimensional (2D)
    • Action trajectory: Two-dimensional (2D)
  • Other Properties Related to Input:
    • Image input is RGB color (8 bits per channel, sRGB color space); grayscale inputs are not supported.
    • Action input is a per-frame sequence of robot/agent state or control values (e.g., joint positions, gripper state, camera pose). The full input is a 2D array shaped (T, D), where T is the number of frames and D is the embodiment-specific dimensionality listed below.
    • Input action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (10D).
  • Input Size and Length limits:
    • Text: 4096 tokens
    • Image: 256p and 480p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16)
    • Action: 16 – 400 sequence length
• Generator Output
  • Output Type(s): Image, video, action, text
  • Output Format(s):
    • Image: JPG
    • Video: MP4
    • Action: JSON
    • Text: string
  • Output Parameters:
    • Image: Two-dimensional (2D)
    • Video: Three-dimensional (3D)
    • Action: Two-dimensional (2D)
    • Text: One-dimensional (1D)
  • Other Properties Related to Output:
    • The generated video is an MP4 file, with the resolution, frame rate, and duration specified in the input.
    • Video generation supports 256p and 480p resolution, 12–30 fps, and 50–150 frames. These are set per request via the size, fps, and num_frames fields.
    • The generated action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (10D).
    • Video: mp4 at the FPS specified in input
    • Image: JPEG
• Reasoner Input
  • Input Type(s): Text, Text+Image, Text+Video
  • Input Format(s):
    • Text: String
    • Image: jpg, png, jpeg, webp
    • Video: mp4
  • Input Parameters:
    • Text: One-dimensional (1D)
    • Image: Two-dimensional (2D)
    • Video: Three-dimensional (3D)
  • Other Properties Related to Input:
    • Video inputs are recommended at a frame rate of 4 fps.
    • Long-context inputs supported up to 256K tokens.
  • Input Size and Length limits:
    • Text: Up to 256K tokens (context window).
    • Image: Standard input image formats; passed as file or URL.
    • Video: mp4 at the recommended 4 fps.
• Reasoner Output
  • Output Type(s): Text
  • Output Format(s):
    • Text: string
  • Output Parameters:
    • Text: One-dimensional (1D)
  • Other Properties Related to Output:
    • Default max_tokens=4096+ is recommended for reasoning outputs; longer outputs may be requested.
    • Reasoning outputs may include structured chain-of-thought, 2D/3D point localization, and bounding-box coordinates for vision-based tasks.

The video content visualizes the input text description as a short animated scene, capturing key elements within the specified time constraints.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

⁠Software Integration

Runtime Engine(s):

• vLLM-Omni⁠
• vLLM⁠
• PyTorch⁠
• Hugging Face Diffusers⁠

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper

Operating System(s):

• Linux (We have not tested on other operating systems.)

Note: Only BF16 precision is tested. Other precisions like FP4, FP8, and FP16 are not officially supported.

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

⁠Training, Testing, and Evaluation Datasets

⁠Dataset Overview

• Total Size: 1.3B data points
• Total Number of Datasets: 393 dataset entries
• Dataset partition: Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
• Time period for training data collection: 2024–2026
• Time period for testing data collection: N/A (standard public benchmarks)
• Time period for validation data collection: N/A (standard public benchmarks)

Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.

Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.

• For more information about the datasets used to train this model, please see the Public Summary of Training Content⁠.

Data Modality and Training Data Size

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Human, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

⁠Public Datasets

⁠Private Datasets

⁠Synthetic Datasets

⁠Evaluation Datasets

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Human, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

⁠Benchmarks

For detailed evaluations of the base model, see our technical paper⁠.

⁠Overall

The table below summarizes Cosmos3-Edge across reasoning and generation. Each reasoning column (General, Robotics, Smart Infrastructure, Driving) reports the average score over that capability's benchmarks. For generation, Image2Video is the PAIBench overall score and Policy: Robot is the RoboLab success rate. In each column, the best result is in bold and the second-best is underlined. * denotes post-trained Cosmos3 variants: Cosmos3-Nano-Policy-DROID⁠ and Cosmos3-Edge-Policy-DROID⁠.

⁠Reasoning Benchmarks

⁠Generation Benchmarks

⁠Image-to-Video Generation

All models are evaluated on image-to-video generation at 480p, 24 fps. Throughput is the number of frames generated per second, measured in eager mode on a single NVIDIA H100 GPU. Cosmos3-Edge delivers the highest generation throughput while achieving competitive quality across PAIBench, RBench, and PhysicsIQ.

⁠Action

The Edge model is a strong initialization for downstream action tasks. For example, post-training it on the DROID dataset produces a policy whose RoboLab success rate is reported in the Cosmos3-Edge-Policy-DROID model card⁠.

⁠PBR (Performance Benchmark Reporting)

The following tables report single-GPU or single-platform inference performance for the Cosmos3-Edge Generator and Reasoner towers.

Generator results are measured using end-to-end or generation latency in seconds; lower is better. Reasoner results include serving and token-generation metrics, such as time to first token, request latency, and throughput.

All results were measured using a single GPU and a batch size of 1.

⁠Generator

Unless otherwise noted, visual-generation benchmarks use 480p resolution. Image-to-video benchmarks generate 189 frames.

⁠vLLM-Omni

⁠PyTorch

⁠Reasoner

The following tables report Cosmos3-Edge Reasoner performance. Reasoner workloads produce autoregressively generated text and therefore use different metrics from the Generator workloads:

• Time To First Token (TTFT): Time from request submission until the first output token is produced. Lower is better.
• Request Latency: End-to-end latency for the complete request. Lower is better.
• Request Throughput: Completed requests per second. Higher is better.
• Output Token Throughput: Generated output tokens per second. Higher is better.

⁠vLLM serving benchmarks

These measurements use the nvidia/Cosmos3-Edge checkpoint with one GPU. Metrics were collected at client-side concurrency levels of 1, 64, 128, and 256.

The workload notation is input sequence length / output sequence length / video FPS.

⁠RTX PRO 4500 Blackwell Server Edition

⁠RTX PRO 6000 Blackwell Server Edition

…(truncated — see the full README on HuggingFace)