79 lines
3.1 KiB
Markdown
79 lines
3.1 KiB
Markdown
# Stealth Browser (Camoufox) for pi
|
|
|
|
Bypasses bot detection / fingerprint blocking that the built-in `web_scrape`
|
|
tool hits (Cloudflare, DataDome, 403s, captcha walls). Includes a **manual
|
|
CAPTCHA-solving WebUI** and **admin log panel**.
|
|
|
|
**Deployed:** http://192.168.0.65:8010 (login: admin / admin — change it)
|
|
**Health:** http://192.168.0.65:8010/health
|
|
|
|
## Architecture
|
|
|
|
```
|
|
pi (any machine) --HTTP--> stealth-browser container (192.168.0.65:8010)
|
|
web_fetch_stealth tool Camoufox patched Firefox, headless
|
|
│
|
|
├─ captcha? → WebUI challenge (login admin/admin)
|
|
└─ logs → /api/logs (admin panel tab)
|
|
```
|
|
|
|
pi extension `~/.pi/agent/extensions/stealth-tools.ts` registers
|
|
`web_fetch_stealth`, pointing at the server by default. Override with
|
|
`STEALTH_BROWSER_URL`, add basic auth with `STEALTH_BROWSER_USER` /
|
|
`STEALTH_BROWSER_PASS` (server uses `ADMIN_USER` / `ADMIN_PASS`).
|
|
|
|
## WebUI (admin + captcha solving)
|
|
|
|
- **Challenges tab** — live queue; when a fetch hits a captcha the job pauses
|
|
and appears with ⚠. Click *Open & solve*: live screenshot, click-to-interact,
|
|
type into focused field, reload/back/scroll, goto URL, then **✔ Done —
|
|
extract page** (or ✖ cancel).
|
|
- **Logs tab** — every fetch: start/captcha/action/done/failed/cancelled with
|
|
status, duration, error. Stored in `./data/logs.jsonl`.
|
|
|
|
## Redeploy after changes (on Docker host, 192.168.0.65)
|
|
|
|
```bash
|
|
cd ~/stealth-browser # synced from /blackbox/.../stealth-browser
|
|
docker compose up -d --build
|
|
curl localhost:8010/health
|
|
```
|
|
|
|
## Proxy (important)
|
|
|
|
Fingerprint evasion alone gets blocked on datacenter IPs. For Cloudflare/
|
|
high-security sites, set a rotating residential proxy:
|
|
|
|
```yaml
|
|
environment:
|
|
- PROXY=http://user:pass@residential-proxy.example:port
|
|
```
|
|
|
|
## API
|
|
|
|
- `GET /health` — liveness
|
|
- `POST /fetch` — `{"url": "...", "extract": "text|html", "selector": "css?", "pause": bool}`
|
|
Blocks until done (or captcha solved via WebUI). Returns
|
|
`{status, title, url, text, outcome, job_id}`.
|
|
- `GET /api/jobs` — job queue
|
|
- `GET /api/jobs/{id}` — job + result text
|
|
- `GET /api/jobs/{id}/screenshot` — live JPEG
|
|
- `POST /api/jobs/{id}/action` — `click|x,y|img_w`, `type`, `key`, `scroll`, `goto`, `reload`, `back`
|
|
- `POST /api/jobs/{id}/continue|cancel`
|
|
- `GET /api/logs` — last 200 log entries
|
|
|
|
## Limits
|
|
|
|
- Defeats fingerprint bot detection. Does NOT defeat:
|
|
- IP reputation blocking (needs the residential proxy above)
|
|
- CAPTCHAs it can't flag — detection looks for known captcha iframes
|
|
(Cloudflare, hCaptcha, reCAPTCHA, Turnstile, Arkose); pass `"pause": true`
|
|
to always pause for manual review.
|
|
- `MAX_CONCURRENT=3` browser instances; each fetch boots a fresh browser (~2-5s).
|
|
|
|
## Deploy notes / gotchas fixed
|
|
|
|
- Port 8000 was taken by Nextcloud → service runs on host port **8010**.
|
|
- Camoufox needs `headless=True` explicitly (defaults to headful → DISPLAY error).
|
|
- Docker blocks Firefox's user-namespace sandbox → `MOZ_DISABLE_*_SANDBOX=1` env.
|
|
- GTK/X11/font system packages required in image. |