Issue with ticket reservations on TicketCo.Events.

Postmortem Report: Ticket Reservation System Disruption

Incident Date: March 15, 2025

Time: 18:42 – 21:30 UTC

Duration: 168 minutes

‌

Direct impact on users: 8 minutes (18:42 – 18:50 UTC) – isolated impact on a single organizer.

Widespread impact: 160 minutes (18:50 – 21:30 UTC) – affected all users.

‌

Summary of Impact

Between 18:42 and 21:30 UTC, users experienced difficulties purchasing tickets due to an issue with our ticket reservation system. The incident initially impacted one organiser before expanding to all users as the issue escalated.

‌

Root Cause

A database transaction became stuck during a background process, causing a lock that prevented reservation updates for a specific event. This triggered a backlog in our task processing system (Scheduling system), eventually affecting ticket purchases across multiple events.

Resolution

• Identified and cleared the blocking database locks.
• Restored scheduling processing to resume ticket reservations.
• Communicated updates to affected organisers.
• Monitored system stability to ensure normal operation.

‌

Preventative Measures & Next Steps

To mitigate similar issues in the future, we are:

✅ Implementing improved transaction handling to prevent same type of locks.

✅ Enhancing alerting mechanisms to detect and escalate similar issues faster.

✅ Reviewing task processing limits to avoid bottlenecks affecting multiple organisers.

‌

We apologise for any inconvenience caused and appreciate your patience while we worked to resolve the issue. Thank you for your continued trust in TicketCo.

For any questions, please reach out to our support team.