Webhooks for screenings and alerts

The Smart Oversight API lets your own software create clients, start screenings and review alerts. Until now, that software had to go looking for news. It called the API at regular intervals and compared each answer with what it already knew, which developers call polling.

Webhooks reverse the direction. When a screening ends, an alert is resolved or a client is created or updated, Smart Oversight sends a notification to your server itself.

The trouble with polling

A screening takes a little while, since it checks sanctions lists, PEP lists and negative media. If your system rarely asks how it is going, it learns the outcome late. If it asks very often, most of its calls come back with nothing new.

An onboarding process that is waiting for that result to move on copes badly with either. With a webhook, your system is told when the screening ends.

Registering an endpoint

The endpoint is the HTTPS address on your server that will receive the notifications. You register it with POST /v2/webhook_endpoints, giving the address in url and the event types you want in enabled_events. The API key used needs the webhooks:write scope.

The response contains the endpoint’s signing secret, and this is the only time it appears. Store it straight away. We keep it encrypted and GET /v2/webhook_endpoints never displays it. If it is lost, create a new endpoint, then delete the old one with DELETE /v2/webhook_endpoints/{id}.

The address must be public and use HTTPS. Local addresses and private network ranges are refused. Smart Oversight does not follow redirects, so a notification never ends up anywhere other than the address you registered.

The events

All events share the same shape. Each one carries an id starting with evt_, a type, a created_at timestamp and a data field, which holds the resource exactly as the API returns it when you look it up directly.

• screening.completed: every search in the screening succeeded and the result is final. The onboarding process can move on to its next step.
• screening.failed: at least one search did not succeed and none can progress any further. The client has not been fully screened and someone needs to pick the file up.
• alert.resolved: a decision was made on an alert, through the API or in the Smart Oversight app. Your CRM can release or hold the file without waiting for an email.
• client.created and client.updated: a client was created or updated through the API.

Checking the signature

Every notification carries a signature header in the form t=<unix timestamp>,v1=<hex>. The v1 value is an HMAC-SHA256 of the timestamp, a dot and the raw request body, calculated with the endpoint secret. This is how your server makes sure the notification really comes from Smart Oversight and was not altered on the way.

Because the timestamp is signed together with the body, a captured signature is no use for any other message. You still need to reject notifications whose timestamp is too far from your own clock, which stops an old message from being replayed. The example below does this.

const crypto = require('crypto');

// rawBody: the request body exactly as received, before any JSON parsing
function verify(rawBody, header, secret) {
  const [, t, v1] = /^t=(\d+),v1=([0-9a-f]{64})$/.exec(header) || [];
  if (!t) return false;
  if (Math.abs(Date.now() / 1000 - Number(t)) > 300) return false;

  const expected = crypto
    .createHmac('sha256', secret)
    .update(`${t}.${rawBody}`)
    .digest('hex');
  return crypto.timingSafeEqual(Buffer.from(v1, 'hex'), Buffer.from(expected, 'hex'));
}

The calculation uses the raw body. JSON that has been parsed and re-serialised will not give the same signature, even if it looks identical.

To try this check before a real alert depends on it, POST /v2/webhook_endpoints/{id}/test sends an event of type webhook_endpoint.test to that endpoint alone. It has the same shape, signature and automatic retries as a real event.

Retries, duplicates and order

Your server should reply quickly with a 2xx status. If it does not respond, or replies with an error, Smart Oversight sends the notification again automatically. The safest approach is to acknowledge straight away and do the processing afterwards, in a queue or background job.

The same notification can arrive twice, because delivery is at-least-once. The event id stays the same every time it is sent, so you can use it to set duplicates aside.

Order is not guaranteed either. Two events about the same client can arrive out of sequence. Sort them by created_at and read each payload as the state of the resource at that moment, not as a change to apply.

After an outage

GET /v2/webhook_endpoints/{id}/deliveries lists an endpoint’s deliveries, newest first. For each one you see the event type, its status, the HTTP status your server returned, the last error and when the next attempt is due. If the problem was on your side, say an expired certificate or a faulty deployment, fix it and call POST /v2/webhook_deliveries/{id}/replay. The delivery goes out again straight away.

Smart Oversight also keeps a history. Every event of a type one of your endpoints listens for is recorded there, whether or not the notification reached you. GET /v2/events returns it newest first and can filter it by type. After an interruption, your system reads it back to the last event it processed.

The evt_ identifier is the same in the notification and in the history, so an event received both ways need only be processed once. For a full reconciliation, your system can also read your clients and their alerts directly from the API.

The integrations page lists what the API covers today. To see how these events fit into a complete onboarding process, read how to connect KYC & AML screening to your CRM.