obs = gw.observe(sid, max_width=1280, mark=True)print(obs.text.summary if obs.text else "")print(obs.changed) # False only when DXGI dirty available + emptyprint(obs.capture_mode) # "dxgi" (preferred) or "gdi_fallback"print(obs.dirty) # None when dirty unavailable — assume changedprint(len(obs.jpeg), "bytes")print(len(obs.targets), "targets")
obs.text is a compact structured summary — focused window title/role plus
readable target lines like [3] Filename (edit, focused) — so your model can
often decide without spending image tokens.Password and masked fields are redacted in that text (masked, value always
[redacted]). See Safety and consent.Pass your own targets to get a Set-of-Mark annotated frame back:
rects = gw.dirty_rects(sid)if rects: # Encode only the changed region when it is small: x, y, w, h = rects[0] frame = gw.screenshot(sid, x=x, y=y, w=w, h=h, max_width=960, quality=60)else: ... # nothing changed; skip the expensive call
Or use observe and check obs.changed / empty obs.dirty["rects"].
Pair dirty rects with ROI observe (x,y,w,h): encode only the changed region
and skip full-frame model calls.
Cadence
You do not need a fresh observe between every click. Verify after each
meaningful step — after a short batch of predictable actions, or after a
single action when the next screen is uncertain. Prefer image=False for
verification when text/targets are enough.Target ids from observe / list_targets are valid for that response only. Do not
persist them across sessions, and re-observe before clicking when the screen may
have changed (lists, file browsers, tables).See Vision for the concepts and Agent
loop to put it together.