Observability alerts

APCO Cloud can email (and optionally webhook) the project owner when operational incidents occur. Alerts are disabled by default — enable them per project under Settings → Observability alerts.

Channels

• Email — sent only to the project owner's verified account email.
• Webhook — optional HTTPS endpoint with HMAC-signed JSON payloads (apco.alert.v1).

Fixed thresholds (v0)

Alert	Opens when	Recovers when
Deployment failure	Any failed deploy, rollback, promote, or start	N/A (one-shot)
Backup failure	Manual backup fails or housekeeping marks a stuck backup failed	N/A
Scheduled job failure	A logical run exhausts retries as failed or timed out	A later run of the same definition succeeds
5xx spike	Production traffic: 5 min window, ≥25 requests, ≥10 5xx, ≥20% rate	Condition clears
Health check	3 consecutive probe failures (every 2 min)	2 consecutive successes
CPU / memory	2×5 min ≥90% of plan limit	2×5 min below 75%
Quota warning	Monthly requests or bandwidth ≥80%	Below 80%
Storage / database	≥80% of plan limit on fresh sample	Below 80%

Health monitor limitations

• Probes the canonical production URL (https://<slug>.apco.space + manifest health path).
• Password-protected projects are skipped in v0.
• Stopped and suspended projects are skipped.
• Probe traffic uses APCO-HealthMonitor/1.0 and is excluded from analytics and quotas.

Webhook verification

Headers:

• X-APCO-Event-Id — stable across retries
• X-APCO-Timestamp — ISO-8601 UTC
• X-APCO-Signature — HMAC-SHA256(secret, "<eventId>.<timestamp>.<rawBody>") hex

Verify the signature over the exact raw JSON body. Payloads never include secrets, env vars, or logs.

API scopes

• alerts:read — GET /api/v1/projects/:id/alerts
• alerts:write — PATCH settings, webhook configure/rotate/remove
• GET /api/v1/deployments/:id/timeline — logs:read

Self-hosting email

Alert email uses RESEND_API_KEY and EMAIL_FROM (same as auth mail). If RESEND_API_KEY is unset, delivery retries then marks the attempt failed — configure Resend for reliable alerts.