Astra Beyond Tabletop: Into the Home

In September 2026, OpenAI released GPT-6 Astra (OpenAI, 2026), a general-purpose multimodal model built for reasoning, software engineering, and task automation. Although not designed for robotics, Astra has been shown in evaluations to act as a capable robot policy with no task-specific training (Zhang et al., 2026; Yang et al., 2026, Jedlicki et al., 2026; Galbot Team, 2026; Robocurve, 2026).

Unlike established VLA- and VLM-based approaches, which rely on large-scale robot training data or hand-engineered interfaces, Astra issues actions directly from observations, with the action representation, horizon, and context specified in the prompt at inference time. VLA models such as RT-2 (Brohan et al., 2023), OpenVLA (Kim et al., 2025), and π0 (Black et al., 2024) fine-tune a pretrained backbone on robot demonstrations, fixing the action space and horizon at training time. VLM-based planners (Ahn et al., 2022; Driess et al., 2023; Chen et al., 2026) typically decompose tasks into subgoals executed by predefined skills or learned policies, enabling flexible task composition while remaining constrained by the capabilities of the underlying controllers. Code as Policies (Liang et al., 2023; Fu et al., 2026) generate executable programs that combine control logic with calls to perception and control APIs, but reliable execution can depend heavily on the human-designed abstractions provided by these interfaces. Astra thus combines the direct control of VLAs with the zero-shot generality of prompted models, but its reliability under real-world mobile manipulation settings is untested.

In this work, we evaluate Astra’s zero-shot performance on 24 household tasks using the Hello Robot Stretch 4 mobile manipulator, prompting the model to output relative joint and base movements directly. Through task evaluations and ablations, we characterize the capabilities and limitations of this approach and examine how design choices shape its control performance.

2 Experiment Setup

2.1 Robot and environment

We use a Hello Robot Stretch 4, a mobile manipulator with 9 degrees of freedom. It has a three-wheel omnidirectional base, a lift that moves the arm along a vertical spine, a horizontally telescoping arm, a three-axis wrist, and a parallel-jaw gripper. Stretch has an RGB-D camera on its wrist, three cameras on its head (a center camera plus two fish-eye cameras on the left and right), and other lidar/ground sensors that were excluded from our experiments. A computer onboard the Stretch handles the lower level control logic while a seperate workstation sends relative movement commands across a wireless network.

The evaluation takes place in a mock apartment environment in our lab. The setup has a kitchen (counter, sink, toaster, fridge, a Keurig coffee maker, drawers, and cabinets), a dining table with chairs, a living area with a sofa, and a play area with toy bins next to a bedroom area with a dresser.

2.2 System overview and control loop

Figure 2 illustrates the main control loop. The system operates in a synchronous observe → reason & command → execute loop. At each turn, Astra receives sensory observations, robot joint states, and previous execution feedback. It may also choose to access wrist depth information via a provided API. Astra then outputs joint relative position commands, which the robot executes. After execution, updated observations and execution feedback are provided to Astra, and the loop continues.

Wrist depth API. We provide Astra an API that takes in a set of coordinates in wrist camera image space and returns the corresponding depths in centimeters in wrist camera frame. During each turn, Astra decides whether or not to use the API, and if so, what coordinates to query.

Model output. During each turn, Astra is asked to either output each joint’s relative position command or decide to finish the task. The complete action schema is provided in the appendix.

Execution feedback. Each joint can return one of four types of execution feedback:

1. completed: The joint reached its target position.
2. stopped_short: The joint stopped moving before reaching its target position.
3. timeout: The joint continued attempting to move but did not reach its target position within the allotted time.
4. rejected: The target position was outside the joint’s limits.

2.3 Our implementation

We access Astra (gpt-6-astra) through a Codex CLI instance. It retains no information across experiments, but within each episode, its context preserves the sequence of image and text inputs. This allows Astra to refer back to what it has seen or done earlier in that episode.

For every task, we use a uniform system prompt that specifies the overall objective (controlling a Stretch 4 robot to complete the user’s task) and outlines the control loop, including its inputs and the required JSON output format. The prompt also includes additional details that may be helpful, such as robot embodiment information and safety guidelines. At the end, the one-sentence task instruction is appended. Importantly, there is no task-specific information, hints, or human priors embedded in our prompt. See the appendix for the full prompt. We also use the high reasoning level and provide the model with input from all cameras (head-left, head-center, head-right, and wrist).

Zero-shot claim. We regard this configuration as genuinely zero-shot: there is no robot-specific training, fine-tuning, demonstrations, in-context examples, or hand-written skills such as a grasp planner, navigation stack, or object detector. The prompt remains identical for every task and contains no task-specific hints; only the task instruction changes, except in cases where we explicitly test prompt variations.

2.4 Tasks and evaluation protocol

Inspired by prior work “Evaluating $\pi_0$ in the Wild” (Wang et al., 2025), we adopted a “vibe-checking” approach, improvising tasks rather than relying on a standard benchmark. For each task, we defined success criteria, measured relevant metrics, and documented failure modes.

Example selection. We include illustrative examples of Astra’s Strengths and Weaknesses only when the relevant behaviors are observed multiple times across independent runs. This criterion helps distinguish recurring capabilities and limitations from isolated successes or failures that may arise by chance.

Starting conditions. Because we aim to test Astra in a real household environment, we varied the initial states of the robot and objects within the bounds of a reasonable test case, unless otherwise noted.

Stopping and interventions. An episode ends when Astra declares finished, the operator intervenes to prevent potential damage, the model produces an invalid output, or the time limit for the task is reached. We set different timeout thresholds for each task.

3 Strengths

What stood out was not any single task Astra completed, but how much coherent robotic behavior emerged from using a general-purpose VLM zero-shot. Across unfamiliar household tasks, it could often infer what to do, translate that intent into a coordinated sequence of physical actions, and adapt when its first attempt fell short.

3.1 Zero-shot generalization over diverse tasks

Deployed zero-shot in a household environment, Astra can complete a diverse range of tasks, suggesting that a powerful VLM can transfer its knowledge to the physical world and generalize across robotic tasks.

For some tasks, while Astra may not consistently achieve full completion, it reliably demonstrates the correct motion intention in nearly every case.

3.2 High-level reasoning and planning

Even though VLMs are known for strong semantic understanding and are widely used for high-level task planning in robotics, we were impressed by how Astra grounds these capabilities in physical actions, enabling active perception, strong understanding of user intent, and robust planning and replanning.

3.2.1 It actively gathers information it needs

3.2.2 It understands ambiguous task instructions

3.2.3 It plans over long horizons and replans when necessary

3.3 Coordination of mobile base and arm

We found that Astra can adjust its whole-body pose by controlling the mobile base and arm together, enabling it to perform tasks that require coordinated movement.

3.4 Strong language steerability and adaptation

We found that Astra can generate motion commands that closely follow detailed natural-language instructions specifying exactly how the robot should move, such as precise arm positioning, gripper orientation, or base adjustments.

3.5 Failure recovery

When Astra encounters difficulty completing a task, it autonomously attempts alternative strategies. This works particularly well when the source of the difficulty lies within the current step. Its recovery is less effective when the problem stems from an earlier step; see Fails to reason about dependencies across steps.

4 Weaknesses

Although Astra demonstrated impressive capabilities, our experiments also revealed recurring limitations. Most of the failures we saw were not isolated or random. In the following sections, we group these failure modes by theme, describe the behaviors we observed, and discuss the factors that may have contributed to them.

4.1 Fine-grained manipulation

We found that Astra struggles with fine-grained manipulation: contact-rich actions in which millimeter-scale errors in pose or small differences in grip force can determine success. It has particular difficulty in confined spaces, where limited clearance makes it easy for the gripper or the object being manipulated to become wedged against the surroundings. Three factors appear to contribute to this. First, Stretch has no force feedback, so Astra must infer contact from images alone. Second, the wrist camera is often the only view that provides enough detail for precise manipulation, yet its view of the relevant contact point or surrounding clearance is frequently occluded by the gripper or the object itself. Additionally, the telescopic arm frequently occludes the center head camera’s view of the workspace, further limiting the visual information available for precise manipulation. Third, Astra often fails to exhibit the “physical commonsense” described by Zeng and the Generalist Team (2026).: the reactive, closed-loop intuition for forces, friction, compliance, and uncertainty that supports small corrections during manipulation.

4.2 Fails to reason about dependencies across steps

Astra struggles to reason about dependencies across steps. It often fails to anticipate how the execution of an earlier action will affect later ones. When a later step becomes difficult, it tends to treat the problem in isolation rather than trace it back to an earlier decision and revise that decision. As a result, it may repeatedly try alternative approaches to the current step even when the problem can only be resolved by revisiting a previous one. This differs from its ability to recover from failures within the current step, where trying a different approach can resolve the problem.

4.3 Scene awareness and collisions

Astra often lacked sufficient awareness of its surroundings and spatial context during task execution. As a result, it frequently made unintended contact with objects in the environment, especially those that were not the current target or directly involved in the immediate step. These collisions included bumping into nearby furniture, appliances, or other obstacles that were outside its main focus of attention. Sometimes operator interventions are required to prevent damage.

5 Ablation Studies

We conduct ablation studies to assess how different components affect Astra’s autonomous control in executing a representative household task: toasting a bagel.

5.1 Experiment setup

Instruction: put the bread in the oven and rotate the top knob to 300 deg

Time limit: 13 minutes

Task Decomposition: To complete the instruction, Astra must accomplish the following five sub-goals in sequence:

1. Open the oven lid
2. Pick up the bread from the dish on the table
3. Place the bread in the oven
4. Close the lid
5. Rotate the knob to the 300 mark

Ablated Components and Variants: Table 1 shows all system components we evaluated. For each ablation, we varied a single system component while keeping all other settings at their default values.

Metrics. We record the metrics in Table 2 for each rollout. Per-run results for each ablation are tabulated in the appendix: reasoning effort (Table 6) and wrist depth (Table 8).

5.2 Reasoning level

We evaluated Astra’s performance at three reasoning levels: low, high (default), and ultra, comparing task metrics as shown in Figure 3. Our main findings are:

• Lowering the reasoning level from high to low reduced both the success rate and completion rate. With low reasoning, Astra frequently forgot or omitted important steps, such as closing the lid before setting the oven temperature. In one instance, it dropped the bread onto the robot base, causing the human operator to end the episode.

• Raising the reasoning level from high to ultra did not lead to better performance. Notably, in one high-level run, the robot was unable to properly pick up the bread, failed to place it inside the oven, and ultimately caused a severe collision that required human intervention. While such failures were rare, when they did occur, Astra often struggled to reason about dependencies across steps and could not recover. In practice, we found that high and ultra reasoning levels performed similarly.

• Interestingly, adjusting the reasoning level had little impact on inference time or token usage.

In summary, we observed no systematic benefit to increasing reasoning above the default high setting, while lowering it degraded performance on this task.

5.3 Wrist depth

We evaluated three configurations for providing wrist depth information: (1) no depth input, (2) a color-mapped depth image, and (3) an API allowing Astra to query depth at specific image coordinates (the default setting). The task metrics are shown in Figure 4, and our main findings are as follows:

• Omitting depth information resulted in similar overall performance but required more Codex turns, greater total time, and increased token usage. Without access to depth data, Astra had to estimate distances from RGB images alone, which led to wasted turns, time, and tokens on empty reaching and grasping. However, Astra consistently retried and recovered from these failures, resulting in comparable success and completion rates overall.

• Providing a color-mapped depth image produced the weakest performance. Missed grasp and retries were as common as when no depth information was provided, so the duration of the task completion also suffered. We suspect this is because the colorized depth image is not perfectly aligned with the RGB image, unlike the API option where each pixel directly corresponds to a depth value. As a result, the additional information was not only unhelpful but may have distracted Astra and degraded its effectiveness.

6 Conclusion

We evaluated Astra’s zero-shot control of a mobile manipulator on diverse household tasks, without robot-specific training, demonstrations, or hand-written skills. We show that an off-the-shelf general-purpose vision-language model can translate household instructions into multi-step plans, coordinated movements, and adaptive behavior. We also identify key limitations in fine-grained, contact-rich manipulation, reasoning about dependencies between steps, collision avoidance, and efficiency. Our ablations further reveal how common system design choices affect performance. We hope these findings provide useful insights for the community and help advance the development of general-purpose robotic systems.

7 Appendix

7.1 Timing limits

7.2 Prompt

7.2.1 System prompt

Identical for every task and sent once at the start of each Codex session. The task instruction is appended at the end as Task: <instruction>. Shown with the default configuration.

You control a Stretch 4 mobile manipulator to complete the task specified by the user.

Stretch 4 has 9 degrees of freedom: a three-wheel omnidirectional base that drives in any direction and turns in place, a lift that moves the arm up and down the mast, a telescoping arm that extends out to the robot's front, a three-axis wrist with yaw, pitch, and roll, and a parallel-jaw gripper.
The robot has the following cameras:
- The head left camera is a fish-eye camera mounted on the mast. Its image only changes when the base moves.
- The head center camera is mounted on the mast. Its image only changes when the base moves.
- The head right camera is a fish-eye camera mounted on the mast. Its image only changes when the base moves.
- The gripper camera is mounted on the wrist and looks out from the arm. Its image changes when the base, the arm or the wrist moves.
Call get_gripper_depth to read the wrist stereo depth for this turn's gripper image. Pass points [{x, y}, ...] of integer pixels (origin top-left, x right, y down; max 64) number of points. Each turn's text states the gripper image size. The tool returns Euclidean distance from the camera origin in centimetres, or null if that pixel has no reading or is outside the image. You may call it at most 32 times per turn. Calling is optional. The final output is still one JSON object.

The robot runs a synchronous observe-predict-execute loop: it observes, you predict an action, it executes that action to completion, then it observes again. The next observation is sent only after execution has finished (completed, stopped_short, timeout, or rejected) and every camera has published a frame newer than that result.

Input. Each turn you receive the latest gripper image, head-center image, head-left image, head-right image, and joint state. After the first turn you also receive the previous command result: completed, stopped_short, timeout, or rejected.
Head cameras are fixed on the mast and only move when the base moves. The gripper camera is on the wrist and moves with the arm and wrist.
Joint state is lift_cm and arm_cm in centimeters, wrist_swing_deg, wrist_tilt_deg, and wrist_twist_deg in degrees, and grip_pct in percent (0 closed, 100 fully open). It does not include base pose.

Output. Return one JSON object and nothing else. Write reasoning first, then kind and the motion.
{"reasoning":string,"kind":"move|done","joint_relative_position_commands":[{"drive","slide","lift","arm":{"direction":"hold","cm":0},"turn","wrist_swing","wrist_tilt","wrist_twist":{"direction":"hold","deg":0},"grip":{"direction":"hold","pct":0}}]}.
kind "move" outputs exactly 1 waypoint.
- drive: {direction:forward, backward, or hold, cm:number}. Motion is in the robot body frame at the observation. forward moves the base forward, backward moves it backward, hold with cm 0 leaves it.
- slide: {direction:left, right, or hold, cm:number}. Motion is in the robot body frame at the observation. left slides the base left, right slides it right, hold with cm 0 leaves it.
- turn: {direction:left, right, or hold, deg:number}. Motion is in the robot body frame at the observation. left rotates the base in place left, right rotates it right, hold with deg 0 leaves it.
- lift: {direction:up, down, or hold, cm:number}. up raises the arm, down lowers it, hold with cm 0 leaves it.
- arm: {direction:extend, retract, or hold, cm:number}. extend lengthens the telescoping arm, retract shortens it, hold with cm 0 leaves it.
- wrist_swing: {direction:left, right, or hold, deg:number}. Motion is in wrist frame. left swings the gripper left, right swings it right, hold with deg 0 leaves it.
- wrist_tilt: {direction:up, down, or hold, deg:number}. Motion is in wrist frame. up tilts the gripper up, down tilts it down, hold with deg 0 leaves it.
- wrist_twist: {direction:clockwise, counterclockwise, or hold, deg:number}. Motion is in wrist frame; clockwise is looking along the arm toward the fingertips, counterclockwise is the opposite way, hold with deg 0 leaves it.
- grip: {direction:open, close, or hold, pct:number}. pct is percent points on the same 0-100 scale as grip_pct in joint state (0 closed, 100 fully open). open increases the opening this turn, close decreases it, hold with pct 0 leaves it.
Each waypoint is a displacement from this observation's joint state. hold is only valid with 0; any other direction requires a positive amount.
A command that would take lift, arm, a wrist, or the gripper past a soft stop is rejected in full and nothing moves. The next observation includes the previous command result, one command status then a line per commanded joint:
completed: that joint became idle at the commanded target.
stopped_short: that joint became idle before the commanded target. For grip this usually means the fingers closed on an object (a grasp). For other joints this usually means a collision, and a safety guard stopped the joint.
timeout: that joint was still moving when the time limit expired; the executor stopped the robot.
rejected: the whole command was refused and nothing moved; the named joint is why.
kind "done" only when this observation shows the task is complete. The robot does not execute it. Still include exactly 1 waypoint, all hold 0.

Avoid collisions with obstacles, people, the environment, and the robot itself.
HINTS (YOU SHOULD PAY ATTENTION TO THESE):
- Prioritize base movements over arm movements. Extending the arm too much can cause the robot to be unstable.
- The gripper finger ends are 10cm away from the wrist camera, and the gripper finger tips are 20cm away from the wrist camera. Adjusting the gripper pose so that the target grasping points are around 15cm away from the wrist camera enables stable grasping.
Task: <instruction>

7.2.2 User message

Sent with the camera images each turn. Previous command result is omitted on the first turn. Example:

Live observation.
Images follow in order: gripper, head_center, head_left, head_right.
Gripper image is <width>x<height> pixels (origin top-left, x right, y down).
Joint state:
{"lift_cm":90.0,"arm_cm":10.0,"wrist_swing_deg":0.0,"wrist_tilt_deg":0.0,"wrist_twist_deg":0.0,"grip_pct":0.0}
Previous command result:
completed
lift: completed
arm: completed

7.3 Action schema

Astra returns one JSON object per turn. Example:

{
  "reasoning":"Orient the fingers vertically for the horizontal drawer handle and lower toward it, staying in front of the drawer.",
  "kind":"move",
  "joint_relative_position_commands":[{
    "drive":{"direction":"hold","cm":0},
    "slide":{"direction":"hold","cm":0},
    "turn":{"direction":"hold","deg":0},
    "lift":{"direction":"down","cm":20},
    "arm":{"direction":"hold","cm":0},
    "wrist_swing":{"direction":"hold","deg":0},
    "wrist_tilt":{"direction":"up","deg":30},
    "wrist_twist":{"direction":"clockwise","deg":90},
    "grip":{"direction":"hold","pct":0}}]
}

• Amounts are non-negative. hold is valid only with amount 0.
• Each waypoint is a displacement from the joint state in the current observation.
• drive, slide, turn are in the robot body frame. Wrist keys are in the wrist frame. Left and up are the robot’s, not the camera’s.
• grip changes the opening by percentage points (0 closed, 100 fully open).
• A command that would push lift, arm, a wrist joint, or the gripper past joint limit is rejected in full and nothing moves.
• done must still include one waypoint with every key set to hold 0. It is not executed.

{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "reasoning",
    "kind",
    "joint_relative_position_commands"
  ],
  "properties": {
    "reasoning": {
      "type": "string"
    },
    "kind": {
      "type": "string",
      "enum": [
        "move",
        "done"
      ]
    },
    "joint_relative_position_commands": {
      "type": "array",
      "minItems": 1,
      "maxItems": 1,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": [
          "drive",
          "slide",
          "turn",
          "lift",
          "arm",
          "wrist_swing",
          "wrist_tilt",
          "wrist_twist",
          "grip"
        ],
        "properties": {
          "drive": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "cm"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "forward",
                  "backward"
                ]
              },
              "cm": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "slide": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "cm"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "left",
                  "right"
                ]
              },
              "cm": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "turn": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "deg"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "left",
                  "right"
                ]
              },
              "deg": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "lift": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "cm"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "up",
                  "down"
                ]
              },
              "cm": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "arm": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "cm"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "extend",
                  "retract"
                ]
              },
              "cm": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "wrist_swing": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "deg"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "left",
                  "right"
                ]
              },
              "deg": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "wrist_tilt": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "deg"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "up",
                  "down"
                ]
              },
              "deg": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "wrist_twist": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "deg"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "clockwise",
                  "counterclockwise"
                ]
              },
              "deg": {
                "type": "number",
                "minimum": 0
              }
            }
          },
          "grip": {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "direction",
              "pct"
            ],
            "properties": {
              "direction": {
                "type": "string",
                "enum": [
                  "hold",
                  "open",
                  "close"
                ]
              },
              "pct": {
                "type": "number",
                "minimum": 0
              }
            }
          }
        }
      }
    }
  }
}

7.4 Wrist depth API

The get_gripper_depth tool returns wrist-camera depth values for specified image coordinates.

{
  "name": "get_gripper_depth",
  "description": "Look up wrist stereo depth on this turn's gripper image. Pass integer pixel coordinates (origin top-left, x right, y down). Returns Euclidean distance from the camera origin in centimetres, or null if that pixel has no reading or is outside the image. At most 64 points per call.",
  "inputSchema": {
    "type": "object",
    "additionalProperties": false,
    "required": [
      "points"
    ],
    "properties": {
      "points": {
        "type": "array",
        "maxItems": 64,
        "items": {
          "type": "object",
          "additionalProperties": false,
          "required": [
            "x",
            "y"
          ],
          "properties": {
            "x": {
              "type": "integer"
            },
            "y": {
              "type": "integer"
            }
          }
        }
      }
    }
  }
}

7.5 Case study and ablations full results

7.5.1 Reasoning effort

We conducted 5 trials per reasoning level. All other settings at default.

7.5.2 Wrist depth results

We conducted 5 trials per setting. All other settings at default. api = get_gripper_depth tool, image = color-mapped depth image, none = no depth…