Somebody in security sends a polite reminder that the API key for your CRM connector is ninety days old and due for rotation. An engineer pastes the new key into the config, restarts the service, and goes to lunch. Three hours later the nightly sync has failed, the warehouse is missing a day of orders, and nobody can say which of the six jobs that used that key stopped working first.
Rotation itself is rarely the hard part. The hard part is that data integrations scatter one credential across schedulers, retry workers, webhooks, and a script somebody wrote two years ago. This article walks through a rotation routine that keeps syncs running through the change and tells you quickly when something did not pick up the new value.
Why integrations break on rotation in the first place
An application with one database talks to one place for its secrets. An integration usually talks to several systems, and each of them issues credentials in a different way. One vendor gives you a static key, another hands out OAuth tokens that expire hourly, and a third signs requests with a shared secret that both sides must agree on at the same moment.
Most outages come from two assumptions. The first is that there is exactly one place where the credential lives, when in practice there are copies in environment files, CI variables, a cron host, and someone's notebook. The second is that the old credential can be revoked the instant the new one is created, which only works if every consumer switches in the same second.
Start with an inventory of who uses what
Before you rotate anything, write down every consumer of the credential. That means scheduled jobs, long-running workers, webhook receivers that verify signatures, ad hoc scripts, and any dashboard that calls the same API. If you cannot list them, rotation is a guessing game.
A spreadsheet is fine for this. For each credential, record the issuing system, the scope it grants, where it is stored, which processes read it, and who owns each of those processes. The act of building the list usually turns up one forgotten consumer, and that is the one that would have caused the outage.
Photo by Brett Sayles on Pexels
Prefer two live credentials over a hard cutover
The single most useful pattern is an overlap window. Create the new credential while the old one still works, move every consumer to the new one, confirm nothing is still using the old one, and only then revoke it. For a short period both are valid, and that gap is what absorbs mistakes.
Many providers support this directly by letting you hold two active keys at once. If a vendor allows only one, ask whether they offer a grace period after regeneration, and if not, treat that integration as high risk and schedule its rotation inside a window when a failed run costs little. Plan the hard cutover with a person watching, not as an unattended job.
Keep secrets in one store, not in many files
Rotation is far easier when consumers read the credential from a single secret store at runtime instead of baking it into configuration. Tools such as HashiCorp Vault and AWS Secrets Manager exist for this, and both support versioned secrets so a consumer can fetch the current value on each run or on a short cache interval.
If a full secret manager is more than you need today, the same principle still applies in a smaller form. Pick one place that is the source of truth, make every deployment pull from it, and delete the copies. A credential that exists in five places will be rotated in four of them.
Handle OAuth tokens differently from static keys
OAuth access tokens are built to expire, so a well-behaved integration already refreshes them. The credential you rotate in that case is the client secret and, occasionally, the refresh token. The framework described in RFC 6749 is explicit that refresh tokens can be rotated by the server on use, which means a worker that fails to store the newly issued refresh token can lock itself out.
The practical rule is to persist the new refresh token atomically before using the new access token, and to serialize refreshes so two workers do not race. When two processes refresh at once, one of them can end up holding a token the server has already invalidated, and the failure looks like a random authentication error that clears itself on restart.
Treat webhook signing secrets as a two-sided change
Signing secrets are the trickiest case because the sender and receiver have to agree. If you change the secret on the vendor side first, every incoming event fails verification until your receiver is updated. If you change it on your side first, the same thing happens in reverse.
The safest approach is to let the receiver accept signatures from either of two secrets for a while. Verify against the new secret first, fall back to the old one, and log which one matched. When the log shows only the new secret for a full day, drop the old one. It is a few extra lines of code that turn an outage into a non-event.
Scope credentials so a rotation touches less
Every integration should get its own credential with the narrowest permissions it needs, instead of all of them sharing one broad key. A shared key turns a routine rotation into a coordinated change across every team that uses it, and a leak of that key exposes everything it can reach.
Separate credentials also make the usage log readable. When the old key shows activity, you know immediately which integration it belongs to, rather than guessing among a dozen unrelated jobs. The OAuth 2.0 overview is a useful reference if a vendor offers scoped tokens as an alternative to one all-powerful key, and it is usually worth taking that option when it exists.
Splitting credentials has a small upfront cost, because there are more of them to track. That cost is exactly what the inventory is for, and it pays back the first time you rotate one integration's key without anyone else noticing.
Make the sync tell you when it used a stale credential
Silent failure is what makes rotation scary. If a job fails loudly with an authentication error, you fix it in minutes. If it catches the error, logs a warning, and exits with zero processed records, you find out when a stakeholder asks why the report is empty.
Make authentication failures a distinct, high-priority category in your alerting. A 401 or 403 from an upstream system should page someone with a message that names the integration and the credential, not get lumped in with transient network errors that retry on their own. A job that reports success while processing nothing is its own class of problem, and a simple expected-minimum row count catches it.
Photo by jim jorjani on Pexels
Run a verification step before you revoke anything
After you move consumers to the new credential, do not revoke the old one on faith. Check the vendor's usage or audit log for activity on the old key. Most platforms show the last time a credential was used, and that single field tells you whether a forgotten consumer is still out there.
If the old credential was used in the last hour, find out by whom before you revoke it. Matching the timestamp to a scheduler entry or a host usually takes a few minutes. If it has been quiet for a full cycle of your slowest job, a weekly report or a monthly export, then it is safe to revoke.
"Rotation breaks things when it is treated as a one-off event instead of a routine. Every integration should be able to survive its own credentials changing, and the way you find out is by doing it on a calendar, in daylight, with the old key still alive." - Dennis Traina, founder of 137Foundry
Rotate on a schedule so the process stays boring
Teams that rotate rarely get bad at it. The runbook goes stale, the engineer who knew the quirks moves on, and the next rotation happens under pressure after a leak. Teams that rotate quarterly tend to have scripts, checklists, and monitoring that work, because the process gets exercised often enough to be trusted.
Put rotations on the calendar with an owner and a named backup. Tie each one to the inventory so the checklist always reflects the real list of consumers. The OWASP Cheat Sheet Series includes guidance on secrets management that is a good starting point for deciding how often each class of credential should change.
Plan for the emergency rotation too
Scheduled rotation is the easy case. An emergency rotation, such as a key accidentally committed to a public repository, has no overlap window and no calendar. You rotate immediately and accept some breakage, which is why the inventory and the monitoring matter so much when it happens.
Rehearse it. Once or twice a year, pick a low-risk integration and pretend its key leaked. Time how long it takes to identify every consumer, issue a new credential, deploy it, and confirm the old one is dead. The first rehearsal is almost always embarrassing, and that is the point of running it.
A short rotation checklist
Keep the final routine small enough that people actually use it. List every consumer of the credential. Create the new credential while the old one is still valid. Update the secret store and redeploy consumers one at a time, confirming each one authenticates. Watch error rates and row counts for a full cycle of your slowest job. Check the vendor's usage log for activity on the old credential, and only then revoke it.
Write the result down afterward, including anything that surprised you. The surprise is next quarter's checklist item.
Where to get help
If your integrations have grown past the point where one person knows every credential and consumer, it may be worth bringing in a second set of eyes. 137Foundry's data integration team helps companies map their connectors, move secrets into a single managed store, and add the monitoring that makes rotation uneventful. You can see more about the work on the 137Foundry services page, or start from the 137Foundry homepage to get in touch.
Photo by Ann H on Pexels
