Why Your Webhook Handler Will Fail Without Idempotency (And How to Fix It)

Here's the uncomfortable truth: If you're building webhook handlers without idempotency, you're building a ticking time bomb. Payment providers like Stripe don't send duplicate events by accident—they send them by design. And when they do, your carefully crafted integration will either crash spectacularly or silently corrupt your data.

The key insight here isn't just about preventing duplicates—it's about understanding that webhooks operate in a fundamentally unreliable network environment where retries are the norm, not the exception.

The Real Problem: Webhooks Are Fire-and-Forget by Design

Most developers treat webhooks like regular API calls, but that's a fundamental misunderstanding. When Shopify sends you a webhook and doesn't receive a 200 OK within 5 seconds, it doesn't politely wait—it retries with exponential backoff. Network hiccup? Retry. Your server had a momentary CPU spike? Retry. You returned a 201 instead of 200? Retry.

This means that same "payment successful" webhook might hit your endpoint 3, 5, or even 10 times. Without idempotency, each hit could trigger another database insert, another email, or another API call to your fulfillment system.

The ASP.NET Core Solution: Event ID + Distributed Lock

The most robust pattern I've seen combines event ID extraction with distributed caching. Here's the core implementation:

1public class IdempotencyService
2{
3    private readonly IDistributedCache _cache;
4    
5    public IdempotencyService(IDistributedCache cache)
6    {
7        _cache = cache;
8    }

Notice the critical details here: we only mark events as "processed" after successful completion, and we use separate keys for locking versus tracking completion. This prevents the nasty edge case where a partial failure leaves your system thinking it processed an event it actually didn't.

Integration Pattern: Middleware + Background Processing

The cleanest integration wraps this in middleware that handles the boilerplate:

1[HttpPost("webhooks/shopify")]
2public async Task<IActionResult> HandleShopifyWebhook([FromBody] dynamic payload)
3{
4    var eventId = Request.Headers["X-Shopify-Webhook-Id"].FirstOrDefault();
5    
6    if (string.IsNullOrEmpty(eventId))
7        return BadRequest("Missing event ID");
8        

Storage Strategy: Cache vs Database

You have two main options for tracking processed events, each with trade-offs:

Distributed Cache (Redis):

• Pros: Blazing fast, automatic TTL cleanup, perfect for high-throughput scenarios
• Cons: Data loss on cache eviction, no audit trail
• Best for: E-commerce sites processing thousands of webhooks daily

Database with Unique Constraints:

• Pros: Durable, queryable for debugging, provides audit trail
• Cons: Slower writes, requires cleanup jobs for old records
• Best for: Financial applications where you need to prove processing history

For most applications, I recommend starting with Redis-backed distributed cache. You get the performance benefits immediately, and you can always add database logging as a secondary concern.

The Hash Fallback Strategy

Not all webhook providers send reliable event IDs. For those cases, generate your own:

1public string GenerateEventKey(string webhookBody, string timestamp)
2{
3    var combined = $"{webhookBody}:{timestamp}";
4    using var sha256 = SHA256.Create();
5    var hash = sha256.ComputeHash(Encoding.UTF8.GetBytes(combined));
6    return Convert.ToBase64String(hash);
7}

This approach uses the webhook body plus timestamp to create a deterministic key. Same content = same key = automatic deduplication.

Testing Your Idempotency

Here's how to verify your implementation actually works:

1[Fact]
2public async Task Should_Process_Event_Only_Once_On_Duplicate_Calls()
3{
4    var eventId = Guid.NewGuid().ToString();
5    var processCount = 0;
6    
7    var processor = async () => 
8    {

Why This Actually Matters

Every developer thinks they'll remember to handle duplicates "later", but here's what happens when you don't:

• Financial chaos: Double-charging customers, duplicate refunds, inventory miscounts
• Data corruption: Duplicate database records that break foreign key relationships
• Cascade failures: Downstream systems receiving duplicate events they can't handle
• Debug nightmares: "Ghost" transactions that appear randomly in production

I've seen companies lose thousands of dollars from duplicate payment webhooks and spend weeks untangling corrupted order data. The 30 minutes you spend implementing proper idempotency will save you from career-threatening production incidents.

Your next step: Audit your existing webhook handlers right now. If you see direct database operations without idempotency checks, you're sitting on a time bomb. Start with your payment webhooks—those are usually the most critical and the most likely to cause expensive problems when they fail.

The beauty of this pattern is that once you build it right the first time, you can reuse the same IdempotencyService across all your webhook handlers. Your future self will thank you when the next provider integration "just works" without any duplicate-related surprises.