How should an AI email-to-CRM workflow prevent duplicate records and duplicate follow-ups?

The three things that people call deduplication

The word “duplicate” hides three different questions. They need different keys, tables and failure handling.

Do not use the email body, subject line, generated reply or an LLM result as the event key. A customer can legitimately send the same words twice. Conversely, delivery systems can redeliver one inbound event after a timeout. Use the stable delivery ID supplied by the email or automation provider, namespaced by the source and mailbox. Preserve the raw provider event or a protected reference to it for audit and replay.

Likewise, do not turn a loosely extracted name and phone number into a universal identity. A family may share a phone number, a sales inbox may be shared, and a misspelled name may describe either an existing or new person. A business should define what “one contact” means: for example, one opted-in person per tenant and canonical email address, or one account per verified customer ID. That is a product policy, not a model judgment.

Why a CRM lookup and delay do not solve concurrency

Consider two different emails from alex@example.com that arrive within a second. Worker A asks the CRM whether Alex exists. Worker B asks before A creates the record. Both receive “not found,” and each creates one. A delay makes this less likely, not impossible. A retry can reproduce the same race days later.

The invariant must live where competing writers are serialized. In a relational integration store, create a unique constraint over the tenant and approved canonical identity, then use one atomic upsert. PostgreSQL documents that INSERT ... ON CONFLICT DO UPDATE guarantees either an insert or an update under high concurrency, provided no independent error occurs. PostgreSQL INSERT documentation, current on 2026-09-01. Other databases and CRMs have equivalent primitives under different names, such as a unique external-ID field and upsert API. Verify the vendor's concurrency and uniqueness semantics before relying on it.

The integration store is often the right source of truth for this workflow even when the CRM is the system of record for sales users. It holds the inbound-event ledger, matching decisions, idempotency keys, attempts and outbox rows in one transaction. The CRM sync is then a managed downstream operation. If the CRM has a reliable, tenant-scoped unique external ID, send the integration contact ID to it. If it does not, keep the mapping in the integration store and treat a CRM create as an operation that must be reconciled after failure.

Separate the model from the control plane

An email is untrusted input. It can be incomplete, contain instructions aimed at the model, or describe a person who should not receive marketing. Limit the model to a bounded extraction task. It receives the relevant, minimized content and returns a schema, not an unrestricted CRM command or message-send tool call.

Structured output can enforce the shape of a response. For example, OpenAI's JSON Schema structured-output option is documented to ensure a response matches the supplied schema. OpenAI API reference, accessed 2026-09-01. That is valuable for parsing and required fields, but it does not prove that an extracted email address is correct, that two people are the same, or that a follow-up is permitted.

Use a deliberately small record such as this. Confidence means the model's stated extraction confidence, not a probability that you can trust without validation.

{
  "sender_email": {
    "value": "alex@example.com",
    "confidence": 0.98,
    "evidence": "Please contact me at alex@example.com"
  },
  "contact_name": {
    "value": "Alex Morgan",
    "confidence": 0.82,
    "evidence": "Regards, Alex Morgan"
  },
  "company": {
    "value": null,
    "confidence": 0.0,
    "evidence": null
  },
  "intent": "product_question",
  "possible_opt_out": false,
  "needs_human_review": false
}

Validate this result in ordinary code: parse the schema, validate address syntax, apply a tenant-specific identity-normalization policy, restrict which fields may update an existing contact, and retain the source evidence. Do not automatically normalize provider-specific dot or plus-address conventions unless the business has explicitly decided that they identify the same person. Use the mailbox envelope or authenticated sender as a corroborating signal where appropriate, but do not assume it proves ownership.

Route to a person when the critical identity field is missing or invalid, several eligible contacts match, an allowed field would overwrite conflicting data, confidence falls below an empirically calibrated threshold, the message suggests an opt-out, or the reply would be high impact. A model can draft a response for approval. It should not silently classify an ambiguous email as permission to contact a person.

A reference architecture with explicit boundaries

The following is product-independent. A workflow platform can run portions of it, but the durable ledger and constraints should remain in a database or service that supports transactions and unique indexes.

Email provider or webhook
        |
        | immutable delivery event ID
        v
Ingress API
  verify source, store raw event reference
        |
        v
Integration database
  inbound_events  -> event uniqueness
  contacts        -> business identity uniqueness
  match_decisions -> explainable manual or automatic match
  followup_intents -> message-send uniqueness
  outbox          -> durable work to perform
        |
        +--> extraction worker --> schema validation --> review queue when uncertain
        |
        +--> CRM sync worker --> CRM external ID or upsert --> mapping and audit
        |
        +--> send worker --> consent and suppression gate --> email provider
                                                       |            |
                                                       +--> attempt and provider result ledger

The important boundary is between a database transaction and an external request. In the same transaction, write or update the contact, create the one allowable follow-up intent and add an outbox row. Do not call the CRM or email provider inside that transaction. The transactional outbox pattern exists to avoid a database update and an event notification diverging after a failure. AWS describes storing the entity change and outbox event in the same transaction, then having another service send the event. It also warns that downstream delivery can happen more than once, so consumers still must be idempotent. AWS Prescriptive Guidance on the transactional outbox pattern.

This is not a claim of magical exactly-once delivery. It is a design for a correct observable effect: a retry can repeat a request, but it cannot create a second event ledger entry, second logical contact or second follow-up intent. The final provider call needs a documented retry behavior, a stable idempotency key and a way to reconcile ambiguous outcomes.

A concrete data design

The names below are illustrative. Encryption, access control and retention rules should match the organization’s privacy policy. Store only the email content and personal data necessary to operate and audit the workflow.

The source event ID is an idempotency key for ingress, not for a contact. The contact's canonical identity is not an idempotency key for a send, because the same person may receive a later, valid follow-up for a different event or rule. A follow-up intent key should include the triggering event and a versioned rule or campaign. Changing the wording of a template does not automatically justify sending a second email, so decide explicitly whether a new rule version is eligible to create a new intent.

For example, generate an opaque idempotency key from the immutable follow-up intent ID, such as a random UUID persisted with the intent. Do not put an email address or other personal data directly in a header or provider idempotency key. Stripe's API documentation provides a useful example of this pattern: repeated requests with the same key return the first result, and it advises against using sensitive personal identifiers as keys. Stripe idempotent requests documentation, accessed 2026-09-01. An email provider may offer different semantics or none at all, so read its documentation rather than assuming Stripe's behavior transfers.

The processing sequence

This sequence uses a relational database for illustration. A CRM with a transactional upsert API can reduce some reconciliation work, but it does not eliminate the need for the inbound-event and send ledgers.

1. Verify the webhook signature or provider authentication. Persist the provider event's immutable ID with INSERT ... ON CONFLICT DO NOTHING. If the insert conflicts, acknowledge the delivery and return the prior processing state. Do not run extraction again just because the webhook was retried.

2. Extract only allowed fields. Validate the structured record and apply deterministic business rules. In the same database transaction, create a match-decision record, upsert the internal contact by its unique business key, create a follow-up intent if policy allows, and write its outbox row. If an uncertain match requires review, commit a review task instead of a follow-up intent.

3. A CRM worker reads the committed outbox. It uses the integration contact ID as the CRM's external ID when supported, records the provider response and retries only with the same operation identity. The integration contact is not considered fully synchronized until the CRM result is recorded.

4. A send worker claims one unprocessed intent with a lease. Immediately before requesting delivery, it reads the current consent, suppression, contact status and approval record. It then submits the provider request using the intent's persistent idempotency key and stores the response. A transient failure leaves the intent retryable. A permanent failure or ambiguous outcome becomes visible to operations.

5. Do not acknowledge an outbox row merely because the process began. Mark it complete only after the durable result update commits. Use bounded retries, backoff and a dead-letter or review queue. Replays should start from stored events and retain all original identifiers, not synthesize new ones.

Here is pseudocode for the decision boundary. The syntax is intentionally generic.

onInbound(providerEvent):
  begin transaction
    event = insert inbound_event(providerEvent.source, providerEvent.mailbox,
                                 providerEvent.immutable_id)
            on unique conflict do nothing
    if event was not inserted:
       commit
       return already_processed

    extraction = validate_schema(extract_allowed_fields(providerEvent.content))
    decision = deterministic_match(extraction, tenant_policy)
    if decision.is_ambiguous or extraction.requires_review:
       create_review_task(event, extraction, decision)
       commit
       return review_required

    contact = atomic_upsert_contact(decision.canonical_identity, allowed_fields)
    intent = insert followup_intent(contact, event, rule_version, approval_state)
             on unique conflict do nothing
    if intent was inserted and currently_eligible(intent):
       insert outbox(intent) on unique conflict do nothing
  commit

sendIntent(intent):
  if not currently_eligible(intent): mark_suppressed; return
  result = email_provider.send(payload, idempotency_key=intent.key)
  begin transaction
    append_send_attempt(intent, result)
    mark_sent_if_provider_accepted(intent, result)
  commit

The two failure stories to test before launch

Run these cases with real retry settings, parallel workers and an intentionally killed process. A happy-path integration test is not enough.

If the provider has no idempotency facility, inspect whether it accepts a durable custom message marker and whether its API can reliably search by that marker before resend. If it can, reconcile first. If it cannot, the system cannot honestly guarantee that the prior request did not send. Choose a documented trade-off: place the intent in manual review to avoid a duplicate, or retry and accept a possible duplicate. Do not conceal this uncertainty behind an “exactly once” label.

Test a second scenario too: two different delivery IDs, E-43 and E-44, carry messages from the same normalized identity at the same time. They should create two event records but contend safely on the one contact key. The unique contact constraint and atomic upsert choose one row. Business policy then decides whether each message can legitimately create its own intent, whether a quiet period applies, or whether the second one is appended to a single draft for review.

Approval, suppression and legal boundaries

No deduplication key makes an unwanted email acceptable. Keep consent, opt-out, suppression, do-not-contact, complaint and account-status state outside the model. The send worker must read the latest state at send time, not merely the state that existed when extraction ran. An inbound opt-out should immediately write a durable suppression record and cancel queued marketing intents for the applicable identity and scope. If language or identity is unclear, do not send until a person resolves it.

For United States commercial email, the Federal Trade Commission says the CAN-SPAM Act applies to commercial messages, including business-to-business ones, and requires a clear opt-out mechanism. Its guidance says opt-out requests must be honored within 10 business days. FTC CAN-SPAM compliance guide, accessed 2026-09-01. Other jurisdictions and message categories can impose different consent, data-protection, retention or timing requirements. This article is an engineering guide, not legal advice. Have counsel and the organization’s privacy lead define the policy that the deterministic gate enforces.

Use approval deliberately. Common policies include automatic replies only to a known customer inquiry with an approved template, manager approval for a new prospect, and no automated send when the model inferred a sensitive attribute or an opt-out. Give reviewers the original message context, the extracted fields and evidence, matching candidates, proposed recipient, proposed content, suppression result and reason for approval. A reviewer should approve the specific intent, not a vague future capability.

Audit history that can answer a real incident

When someone asks “why did this person receive two replies?” the system should reconstruct the answer without depending on model logs alone. Capture the provider source and delivery ID, receipt time, protected content reference and content hash; model and prompt version; validated extraction; confidence and evidence; matching candidates and rule used; contact changes; intent creation; eligibility decision; approval; suppression checks; idempotency key; provider request and message IDs; response; retry count; and operator actions.

Protect this history. It can contain personal data and message content. Apply role-based access, encryption, least-privilege service accounts, retention limits, redaction for operational views and a deletion or access-request process where required. Auditability is not a reason to retain raw email forever.

Common failure modes and their replacements

A practical rollout checklist

• Define an immutable ingress identifier from the provider and verify it survives retries and replays.

• Write the exact tenant-scoped contact identity rule. Test shared inboxes, shared phones, aliases, malformed addresses, address changes and potential merges with business owners.

• Add database uniqueness constraints before enabling parallel processing. Test them with concurrent workers, not just sequential requests.

• Persist event, contact, intent and outbox records in one transaction. Make the outbox consumer idempotent because at-least-once delivery is normal. AWS guidance explains why duplicate downstream delivery must be tolerated.

• Select an email provider with documented idempotent submission or a reliable reconciliation mechanism. Test the crash after provider acceptance, not only timeouts before the request.

• Put consent, scope-specific suppression and approval checks immediately before sending. Test that an opt-out cancels queued intents.

• Limit the LLM to schema-constrained extraction. Maintain a labeled test set of real redacted emails for extraction accuracy, ambiguous matching, opt-outs and prompt-injection-like instructions.

• Create alerts for unique-constraint conflicts, ambiguous matches, provider response mismatches, intents stuck in sending, duplicate provider message IDs and suppression blocks. Review the audit trail after a deliberate replay exercise.

Useful alternatives and their limits

For a small, low-volume workflow, a CRM with a documented unique external-ID upsert and a provider with idempotent sends may be sufficient. You still need an inbound-event table or equivalent durable store, because webhooks can redeliver and the CRM alone usually cannot explain every received event. Avoid a distributed lock as the primary guarantee. Locks expire, workers crash and operational failures can turn an availability mechanism into a duplicate-creation mechanism. A unique constraint is the final arbiter.

A queue with first-in, first-out ordering can reduce contention for one key, but it does not replace uniqueness constraints or send idempotency. Messages can be replayed, workers can fail after a side effect and two different queues may still touch the same contact. Similarly, making a model call deterministic with a low-temperature setting may improve reproducibility, but it does not make record identity or external side effects safe.

If the correct identity cannot be represented by a stable deterministic key, use a candidate-match queue rather than automatic merge. This costs more operational time, but it prevents a more damaging failure: merging or emailing the wrong person with high confidence. For high-risk communications, require a human approval step and use the system to prepare a draft rather than send it.