//DOCS Semantic Web

Semantic Web Runtime

Punk's semantic web runtime turns web pages into Semantic Object Model snapshots so agents can use structured, low-token, replayable web context.

What SOM Is

SOM is a JSON representation of page meaning:

• Page URL, title, language.
• Semantic regions.
• Semantic elements.
• Stable element ids.
• Available actions.
• Useful attributes.
• Byte counts for token-savings estimates.

SOM removes most presentation noise from raw HTML while preserving the information agents need.

Fetch Flow

POST /api/v1/web/fetch runs:

1. SSRF/public URL guard.
2. SOM cache lookup unless bypassCache=true.
3. Plasmate-compatible AWP/SOM engine if PLASMATE_URL is configured and healthy.
4. Built-in TypeScript compiler fallback.
5. Diff against the latest stored snapshot for the URL.
6. Store snapshot.
7. Store SOM cache entry.
8. Return SOM plus compact prompt context.

Sources

The built-in compiler is good for local/dev fallback. A configured Plasmate-compatible AWP engine should be preferred for richer browser behavior.

Web Sessions And Actions

The public HTTP surface currently exposes SOM fetch through /api/v1/web/fetch. The @punk/web-core package also carries the session/action primitives used by Punk's agent-native web runtime.

Current action model:

The built-in session is intentionally protocol-level. It can follow links, type/select into forms, and submit GET or urlencoded POST forms, then recompile the resulting page to SOM. It does not execute arbitrary browser JavaScript. A configured Plasmate-compatible AWP engine can provide richer browser-backed page.act behavior while keeping the same action intent contract.

Every session fetch/navigation should use the same safety posture as SOM fetch: URL guard, timeout, body-size cap, traceability, side-effect classification, and policy gates for user-visible writes.

Security Defaults

Punk protects web fetches from SSRF:

• In open dev mode, private/loopback fetches are allowed so local demos work.
• When PUNK_API_KEY is set, private/loopback/link-local destinations are blocked.
• Set PUNK_ALLOW_PRIVATE_WEB_FETCH=true only when the deployment is intended to fetch internal resources.

Upstream bodies are capped at 5MB and fetches time out after 10 seconds.

SOM Cache

Default SOM TTL is 5 minutes.

Cache key currently uses tenant and URL. For production-sensitive authenticated pages, include subject/auth scope in higher-level routing decisions and avoid sharing cached private page context across subjects.

Semantic Diffs

When Punk has a previous snapshot for the URL, it returns a semantic diff. Diffs are useful for:

• Detecting page drift.
• Deciding whether a web artifact remains valid.
• Monitoring pricing or product changes.
• Filtering footer/layout noise.
• Updating pattern confidence.

High-impact changes include pricing, form, CTA, and primary content changes. Low-impact changes include boilerplate and footer noise.

Prompt Context

The API response includes context, a compact prompt-ready rendering of the SOM. Use it instead of raw HTML when asking a model to reason about a page.

SDK Usage

const result = await punk.fetchSom("https://example.com");
console.log(result.context);
console.log(result.tokensSavedEstimate);

Use bypassCache: true to force a refetch and produce a diff against prior state:

await punk.fetchSom("https://example.com", { bypassCache: true });

Operational Notes

• If the configured AWP engine fails, Punk backs off for 60 seconds and uses the built-in compiler.
• The latest snapshot per URL is stored for future diffs.
• SOM snapshots are tenant-scoped.
• Session/action support is package-level until a public API route is added.
• Web fetch errors return a generic 502 with category timeout, too_large, or upstream.