gaze-pane: Focusing iTerm2 Panes With My Eyes

I run a lot of terminal panes. A Claude session in one, a dev server in another, tests in a third, a shell in a fourth. The friction is never the work, it's the constant Cmd+Option+Arrow (or worse, reaching for the mouse) just to put my cursor where my eyes already are. So I built gaze-pane: a webcam watches my eyes, a small model maps my gaze to a screen coordinate, and the iTerm2 Python API focuses whichever pane I'm looking at. No clicking, no shortcut gymnastics.

The bet was that I don't need cursor-grade precision to pick a pane. I need to know which quadrant of the screen I'm looking at. That turns out to be very doable with a plain laptop webcam.

The pipeline

Every frame goes through the same path:

1. Capture. OpenCV pulls frames off the webcam in a background thread at ~30 fps.
2. Landmarks. MediaPipe's FaceLandmarker returns 478 face landmarks (irises included) plus a 4x4 head transform matrix per frame.
3. Feature vector. I distill those into 9 numbers: left/right iris offset normalized by eye width, head yaw and pitch from the transform matrix, left/right eye openness, and inter-iris distance as a proxy for how far my face is from the camera.
4. Map. Z-normalize the features, then a ridge-regularized affine model turns them into a top-left-normalized screen coordinate.
5. Pane hit-test. Figure out which iTerm2 pane that coordinate falls in.
6. Activate. After a brief dwell, hand that pane focus.

The two pieces that took the most thought were the mapping model and the pane hit-test, so those are the ones worth talking about.

Why ridge, not plain least squares

The obvious move is to fit an affine map from features to screen position with ordinary least squares: show 16 calibration dots, record the feature vector at each, solve. I did exactly that and the model behaved terribly at inference time, occasionally predicting coordinates like (-5, 4) for a gaze that was clearly on-screen.

The culprit is collinearity. My left and right irises move together almost perfectly. When two input features are nearly linearly dependent, least squares is free to put a giant positive weight on one and a giant negative weight on the other, and they cancel during fitting. The fit looks great on the 16 calibration points and then explodes the moment a new sample breaks the cancellation by a hair.

Ridge regression (L2 penalty on the weights, λ=0.1) fixes this directly. It refuses to hand out enormous offsetting weights, so the model stays well-behaved on inputs it has not seen. The residual on the calibration points goes up a touch; the behavior on real gaze gets dramatically more stable. That is the trade you want for a control input.

Calibration that doesn't lie to itself

Calibration is a fullscreen black canvas with a grid of dots. Look at each dot, press SPACE, it averages 12 frames. Then it fits the model and prints the RMS residual.

The part I'm happiest with is the validation phase, because it's designed to not fool me. After the initial fit, it shows 5 fresh test points (four corners and center) and captures my gaze passively, with no prediction dot on screen. That last detail matters: if you render the green "here's where I think you're looking" dot during validation, your eyes snap to the dot and you validate a lie. Hide the prediction, capture the truth, and only then show the error.

The summary screen gives a real choice: take the refined fit (initial plus validation samples), keep the initial fit and throw the validation data away, or abort and save nothing. Every calibration is archived to a history folder before being overwritten, so reverting is a cp.

Hit-testing a pane iTerm2 won't give you a rectangle for

Here's the annoying part. iTerm2's API gives you a Session per pane, but a Session doesn't expose its pixel frame. I have a gaze coordinate in screen space and no rectangles to test it against.

What iTerm2 does give you is the splitter tree: tab.root is a tree of horizontal and vertical Splitter nodes with Session leaves, and each node reports a grid_size in character cells. So I reconstruct the geometry myself. Start with the window's content rect (the window frame minus the title and tab-bar chrome), then walk the tree recursively, splitting each rectangle in proportion to the child subtrees' cell counts. Every leaf comes out with a proportional pixel rect, and now hit-testing is just "which rect contains the point."

The chrome offset is a fudge factor (--chrome-top, default 52 points, auto-reduced in fullscreen), but everything below the title bar is computed, not guessed.

Dwell, overlay, and not fighting the user

Raw gaze is jittery, and you don't want focus flickering between panes every time your eyes saccade. Two things smooth it: an exponential moving average on the feature vector, and a dwell timer, your gaze has to rest in a new pane for --dwell-ms (350 by default) before it actually switches. Glance away and back and nothing happens; settle in and it commits.

There's an optional translucent overlay too, a click-through always-on-top dot that's green when your gaze is inside a pane and red when it's outside. AppKit wants the main thread for that borderless NSWindow, so the asyncio iTerm2 loop moves to a background thread and the two talk through a small lock-protected dict. It's the kind of thing that's invisible when it works and maddening when the threading is wrong.

The voice detour

Once focus follows my eyes, the next obvious thing is to not touch the keyboard either. So there's a --voice mode: mic to silero-vad to MLX Whisper, listening continuously, watching for a wake-and-end phrase sandwich.

"hey claude git status send it"

Whatever lands between the phrases gets typed into the currently-focused pane plus Enter. The honest caveat: Whisper is excellent at English prose and mediocre at shell syntax. "list files" gets typed verbatim, it does not become ls. You learn to dictate the way you'd actually type: "ls dash l a", "git pull". And the recipient is whichever pane is focused when you finish speaking, so you look at the target pane before you say the end phrase. Eyes pick the pane, voice fills it.

What it is and isn't

This is webcam gaze, not a Tobii bar, so precision is roughly 2 to 3 inches at laptop distance. That's perfectly comfortable for a 2x2 or three-up layout and frustrating with a dozen tiny panes. It's macOS-only (the AppKit overlay and iTerm2 API both pin it there), single-monitor, single-tab. Big posture changes want a recalibrate, which is why I made calibration fast (~30 seconds).

But within those lines it genuinely works, and there's something quietly delightful about looking at a pane and watching the cursor already be there. It's MIT licensed and on GitHub if you want to point a webcam at your own terminals.