On April 25, 2025, between 10:20 and 10:55 UTC, TicketCo experienced a significant service disruption across its core platforms, including ticketco.events, ticketco.shop, and key backend APIs. During this 35-minute window, the majority of user-facing systems were largely unavailable, with only a small fraction of requests—approximately 10–20%—being processed successfully.
Users attempting to access the platform encountered extremely slow loading times and widespread request failures, primarily due to timeouts. One component (discounts) had a long running query on one of our production database read replicas. The query created severe internal locks, effectively blocking all subsequent read operations on that replica.
As web servers began to stall while waiting on database responses, request queues rapidly saturated, leading to degraded performance across all major services. This made the platform largely unresponsive for most users during the incident period.
Once the problematic query was terminated and traffic was rerouted away from the affected replica, services gradually recovered to normal levels.
In response to the incident, several corrective actions are being undertaken. These include optimising the discount query logic, implementing safeguards to fail fast in case of long-running database operations, and improving system monitoring and alerting around read replica health and load. Additional measures to better balance database read operations are also being evaluated.
We sincerely apologise for the inconvenience caused to our users and partners. Ensuring the reliability and resilience of our services remains a top priority, and we are committed to addressing the root causes of this incident to prevent recurrence.
Kjetil Sørtun
CTO@TicketCo