Y: An experiment on myself

Background

"I mostly got what I wanted by ignoring advice." - Sam Atlman

As an engineer, I deeply believe that context is everything. And nobody has more context on my life than me. I tend to be extremely cautious in taking advice.

I think of my life as the dependent variable (Y), a product of my environment, conversations, information diet, etc.

My Hypothesis: tracking all these independent variables will push me towards greater intentionality.

I recently met Andrea, an incredible founder working on an app that listens to him 24/7, turning his day into a story. I decided to join &

Scope

I decomposed the system into two specialized models:

1. Planning: Vision-language model generates semantic actions from screenshots
2. Grounding: Specialized model maps semantic actions to pixel coordinates

Given a screenshot $s_t$ and history $h_t = \{(s_i, a_i)\}_{i=0}^{t-1}$, the planner outputs action $a_t$:

The grounding model converts semantic actions to coordinates:

Example output:

{
  "action": "click",
  "target": "Submit button",
  "coordinates": [834, 672],
  "confidence": 0.94
}

Cost Analysis

Vision-language models tile screenshots for processing. A 1920×1080 screen generates 6 tiles at 170 tokens each:

For a 100-step workflow:

The optimization strategy is to cache UI elements between actions and route simple interactions directly to the grounding model. UI state changes slowly—most elements remain at fixed coordinates across consecutive actions.

Training

I trained the grounding model via GRPO (Group Relative Policy Optimization) with binary rewards:

where $\hat{c}$ is the predicted coordinate, $c_{\text{true}}$ is the ground truth, and $\tau$ is the hit threshold.

Training uses trajectory augmentation: one recorded workflow generates multiple training samples by varying UI states and timing. For a trajectory of length $n$, I extract $O(n^2)$ sub-trajectories as training data.

Adaptive Routing

For simple UIs, the grounding model executes directly without planning. I route to the planner only when confidence is low:

where $H(p)$ is the entropy of the grounding model's coordinate distribution, $\theta$ is the entropy threshold, and $\gamma$ is the confidence threshold.

This achieves ~50ms latency on simple actions (A100), escalating to 2-5s only for ambiguous cases.

Future Work

The current implementation processes discrete states. Moving to streaming would enable continuous perception-action loops at 5-10 Hz, handling dynamic interactions (drag, hover, scroll) more naturally.

For repeated workflows, policy distillation could compile trajectories into specialized models:

This converts the planner from runtime dependency to training-time teacher, enabling local execution of routine tasks.

Related Work