Postmortem Report: Service Interruption

Summary

Today at 15:52 EDT, a service disruption affected the MeloTel Customer Portal and MeloSIP Click2Call services due to an unintended misconfiguration during email service updates. This incident resulted in an automatic email retrieval issue that required emergency maintenance, temporarily impacting customer access.

Incident Timeline

• 15:52 EDT – Issue Identified:

  • An issue was detected affecting Click2Call, stemming from emergency maintenance on the MeloTel Customer Portal.
  • The team began investigating the root cause with urgency.

• 16:09 EDT – Investigation Ongoing:

  • Support engineers were engaged, and further analysis was conducted.

• 16:25 EDT – Issue Identified:

  • The cause of the customer portal outage was determined, and recovery efforts began.

• 16:31 EDT – Partial Restoration:

  • Customer Portal and MeloSIP Click2Call services were restored.
  • Maintenance continued to ensure full resolution.

• 16:59 EDT – Full Resolution:

  • All affected services were reported restored.
  • A secondary maintenance session was scheduled to address a support ticketing system bug, though this is not expected to impact customer services.

Root Cause

• A misconfiguration occurred while attempting to reconnect email services to our Customer Portal.
• The incorrect email connection triggered automated email retrieval, resulting in the unintended deletion of emails and an impact on our Customer Portal.
• To prevent further data loss, the server was temporarily shut down, leading to a brief service disruption for the Customer Portal and MeloSIP Click2Call.
• Once the situation was secured, services were restarted, and minor post-recovery adjustments were made.

Impact

• Customer Portal and MeloSIP Click2Call were unavailable for approximately 40 minutes.
• Several pending support tickets were unintentionally closed.

Next Steps & Preventative Actions

• Email Service Reconfiguration: Ensure correct reconnection of email services to Customer Portal.
• Process Improvements: Implement additional verification steps for email service changes.
• Automated Monitoring Enhancements: Strengthen monitoring to detect unintended email retrieval events before they escalate.
• Support Ticket Review: Assess and restore any support tickets that were mistakenly closed.

We sincerely apologize for the inconvenience caused by this incident and appreciate your patience. Our team remains committed to improving our systems and preventing similar occurrences in the future.

For further inquiries, please contact our support team.

Melo Technology Team