Idempotent webhooks for agencies: signature, duplicates and retries

Each delivery is a POST with a JSON envelope with the fields delivery_id, event_id, version, instance_id, instance_name, event and data. The headers carry X-ZapZap-Id (the delivery_id), X-ZapZap-Event and, when the instance has a secret, X-ZapZap-Signature in the format t=<unix>,v1=<hmac>.

The secret (prefix whsec_) is generated when you set the instance webhook and comes back in the configuration response. The signature is an HMAC SHA-256, with that secret, over the text t, a dot and the exact request body. That is why the body must be read as bytes, before any JSON parsing.

Recompute the HMAC and compare it with v1 using a constant-time comparison. Reject with 401 if the signature does not match, and also reject when the time t is more than 5 minutes from your clock, to stop the replay of a captured old request. The tolerance recommended in the product is 5 minutes.

A common mistake in Express is using express.json() before the route: the body is reserialized and the HMAC does not match. Use express.raw() on the webhook route, as in the code below, which was run locally with a valid signature, an expired time and a tampered body, and returned true, false and false.

import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";

const SECRET = process.env.ZAPZAP_WEBHOOK_SECRET; // whsec_..., returned when you set the webhook
const TOLERANCE_S = 300;                          // reject anything older than 5 minutes
const seen = new Set();                           // production: table with a UNIQUE key (see SQL below)

export function validSignature(rawBody, header, nowS = Math.floor(Date.now() / 1000)) {
  const parts = Object.fromEntries(
    String(header || "").split(",").map((kv) => kv.split("=").map((s) => s.trim())),
  );
  const t = Number(parts.t);
  if (!Number.isFinite(t) || !parts.v1) return false;
  if (Math.abs(nowS - t) > TOLERANCE_S) return false;
  const expected = createHmac("sha256", SECRET).update(t + "." + rawBody).digest("hex");
  const a = Buffer.from(expected), b = Buffer.from(parts.v1);
  return a.length === b.length && timingSafeEqual(a, b);
}

const app = express();
// raw body: the HMAC is computed over the exact bytes ZapZap sent, so do not use express.json() here
app.post("/webhook", express.raw({ type: "*/*", limit: "2mb" }), (req, res) => {
  const raw = req.body.toString("utf8");
  if (!validSignature(raw, req.get("X-ZapZap-Signature"))) return res.sendStatus(401);
  const envelope = JSON.parse(raw);
  const key = envelope.event_id || envelope.delivery_id; // pick one key and keep it
  res.sendStatus(200);                                    // answer fast, work afterwards
  if (seen.has(key)) return;                              // duplicate: ignore
  seen.add(key);
  setImmediate(() => console.log("process", envelope.event, key));
});
app.listen(3000);

/* production dedupe (any SQL database):
   CREATE TABLE webhook_events (
     instance_id text NOT NULL, dedupe_key text NOT NULL,
     received_at timestamptz DEFAULT now(), payload jsonb,
     PRIMARY KEY (instance_id, dedupe_key));
   INSERT ... ON CONFLICT DO NOTHING; if 0 rows inserted, it was a repeat. */

The envelope carries delivery_id and event_id. Pick one key and always use the same: the simplest option is event_id when present, falling back to delivery_id. Store the key in a table with a uniqueness constraint per instance and key. If the INSERT inserts no rows, it is a repeat and you ignore it.

The API's webhook test uses a null event_id and a delivery_id starting with test_, so your code must not treat a null event_id as an error. For an agency, the composite key with instance_id avoids collisions between clients and gives isolation: one client's event never mixes with another's.

Answer 200 as soon as you validate and store the key, and process the rest in the background. A slow response becomes a timeout, and a timeout becomes a repeat. On ZapZap's side the first attempt is immediate, and failures go into a retry queue with growing intervals (about 2 minutes, 10 minutes and 1 hour in the product's code); after the last attempt the event is no longer resent.

That is why your dashboard needs a recovery path: the API lists recent deliveries (GET /instances/{id}/webhook/deliveries) and webhook errors, which helps you find what was missed after an outage. See the reference.

One secret per instance, stored encrypted; one endpoint with a route per client or resolution by instance_id; a log of the key, the event and the result; an alert when the 401 rate rises (a sign of a changed secret or an attack); and tests with POST /instances/{id}/webhook/test on every deploy. To build this with a code agent, see build with AI and the page for AI agencies.

Remember that received content (text from end customers) is untrusted input: never execute it or pass it unvalidated to an agent with send permission.