Service Interruption

Incident Report for Rebuy

Postmortem

GCP Outage Summary

What Happened:
The disruption stemmed from an invalid automated quota update in GCP's API management system, leading to increased 503 errors across multiple services. While most regions recovered within two hours, some areas experienced longer recovery times due to infrastructure overloads.

Impact on Our Services:
As a GCP customer, we were directly affected by this incident. Many of our services experienced intermittent access issues during the outage window. We understand the inconvenience this may have caused and appreciate your patience as we worked through the disruptions.

Current Status:
All affected services have been fully restored, and normal operations have resumed.

Posted Jun 13, 2025 - 09:10 EDT

Resolved

Rebuy is fully operational again.
Posted Jun 12, 2025 - 17:50 EDT

Update

Rebuy services are restored across storefronts.

We are continuing to monitor all services.
Posted Jun 12, 2025 - 17:27 EDT

Update

We are seeing Rebuy services continuing to be restored and Rebuy Admin is accessible.

Continuing to monitor all services for any further interruptions from Google Cloud.
Posted Jun 12, 2025 - 17:16 EDT

Update

Services are looking much better currently. We're seeing Rebuy services restored across many store fronts and Rebuy Admin is able to be accessed again.

Continuing to monitor all services for any further interruptions from Google Cloud.
Posted Jun 12, 2025 - 16:46 EDT

Update

We're seeing improvements in Google Cloud Services and as a result, Rebuy services are beginning to come back online also. We're still actively monitoring as neither Google Cloud Services nor Rebuy by extension are back to 100% yet, but moving in a positive direction.
Posted Jun 12, 2025 - 16:28 EDT

Update

Google Cloud Service is still working to restore full service to users. We're continuing to monitor our systems and preparing health checks to run once that service is restored.

https://status.cloud.google.com/
Posted Jun 12, 2025 - 15:51 EDT

Monitoring

We're seeing that some Google Cloud Services are beginning to respond again but they are intermittent, not at full capacity. Our teams are actively tracking the changes so that Rebuy services can be restored as quickly as possible.
Posted Jun 12, 2025 - 15:01 EDT

Investigating

Rebuy services are currently degraded due to an incident with Google Cloud Services that we are reliant on to function properly. Our teams are monitoring the situation closely and will begin testing and validating Rebuy services as soon as the Google Cloud Services interruption is resolved.
Posted Jun 12, 2025 - 14:28 EDT
This incident affected: API, A/B Testing, Admin Portal, Checkout / Post Purchase, Discounting Functions, Merchandising Widgets, Smart Cart, and Smart Search.