Fundraisers ask Copilot in Outlook or Teams to draft a reply to a donor and get a personal email built from that donor’s real gifts and notes in the CRM. I took the agent from a single-customer pilot to a multi-tenant design: one Marketplace package and one backend, with each organization connecting its own CRM key.
“Estimate” numbers are engineering estimates to confirm with the vendor. All others are counted from the code, tests and project documents.
The first version served one pilot organization, and its CRM API key was a single backend setting read at startup. When the CRM vendor tested the install, they found the obvious consequence: one package meant one backend meant one key, shared by everyone who installed it.
A self-serve Marketplace listing therefore needed a separate build and hosted backend for each customer. That was not acceptable, and neither was one shared key across organizations.
single / multi mode switch kept the pilot running unchanged while the new path shipped.The diagram is wider than the screen: swipe sideways to see all of it →
Two organizations mean two tenant ids, two rows and two keys. A request can only ever use the key that matches its verified identity claim. Admin routes and key entry are gated to org admins, and keys never appear in logs, where donor emails are redacted too.
The CRM key moved from a deployment secret to runtime configuration keyed by the Entra tenant id. A Teams package is a static file everyone downloads, so it could never carry a customer secret. Trade-off: the Outlook surface had to move from no auth to Entra SSO so its calls identify the organization.
Drafts start from deterministic templates that insert explicit placeholders when data is missing. The gpt-4o rewrite only adjusts tone and length, must keep placeholders verbatim, and falls back to the template if it fails.
Microsoft supports custom engine agents only in Teams and Copilot Chat, and declarative agents in Outlook and Office. Running both on one backend reaches users everywhere at no extra infrastructure cost. Trade-off: two Teams packages to maintain.
Any organization with a valid key can connect, because the key itself is the gate. An approval flag stays in the record so the vendor can turn on allow-listing later without a redesign.
Problems that came up in testing with the vendor, and how they were closed:
The vendor’s own install test showed that the backend’s single key setting made every installer share one key, or forced a custom build per customer.
A per-tenant key store and resolver behind one package. Back-compat mode kept the live pilot untouched while the change rolled out.
The declarative agent’s plugin used no auth, so the backend could not tell which organization was calling and could not pick a key.
Moved the plugin to Entra SSO and added JWT validation middleware on the action routes, replacing the shared action key.
With self-serve setup, a typo in a key would only show up later, as a failed draft for a fundraiser.
The key is validated at save time with one read-only probe and no retries. Invalid keys are rejected and nothing is stored.
Agents are cached per tenant for speed, so a rotated key could have been ignored until a restart.
The cache entry remembers which key built the agent. A different key rebuilds it, and the admin API invalidates the entry on every change.
The chat card would have let any member replace the organization’s key.
Admin-only gating through the Teams roster (owner or admin), plus encryption at rest and redaction of keys and donor emails in logs.
Bulk drafts hit 429s and occasional 5xx responses from the CRM.
Up to 3 retries with full-jitter exponential backoff, honouring Retry-After, plus one token refresh on an expired session.
A 0.x version, a non-canonical schema URL, a non-GUID agent id, a remote spec URL and an unsupported confirmation flag were each rejected.
Fixed each one and added build scripts for the deploy zip and for per-tenant packages. Nine packaging rules are recorded in the docs.
An illustrative replay with synthetic tenants and donors. It contains no client data or code. Each scenario shows the checks the backend runs.
| Measure | Before | After | Source |
|---|---|---|---|
| Deployments for N organizations | N (one per customer) | 1 backend + 1 package | derived |
| Onboarding an organization | Hand-built package + backend, ≈1 day | Admin pastes a key, ≈5 min | estimate |
| Key per organization | One shared key | One encrypted key per tenant | code |
| Outlook surface identity | None | Entra SSO, validated JWT | code |
| Bad key detected | At first failed draft | At save, nothing stored | code |
| Automated checks per push | — | 96 tests + lint, types, build, evals | code |
Source: code = in the repository; derived = follows from the design; estimate = to confirm with the vendor.
Free scoping call · 30 minutes · No commitment
Write a few lines: the process or system, what hurts today (time, errors, cost or an AI pilot that stalled) and who uses it. After a short call you get a written plan and a fixed quote within ≈3 days.
Opens a pre-filled email · reply within one business day
The model reads. The code counts.
All projects