Phone Registration Issues
Incident Report for MeloTel Network Operations (NOC)

This postmortem is related to the service interruption which affected MeloTel customers this morning between (8:10 EST) and (10:03 EST).

The event affected registration of devices which used secure encryption TLS and was the result of a broken SSL chain in our Kamailio server. Since the SSL chain was broken, the devices which renewed registration could not authenticate to our servers. This means those devices were not able to make or receive calls.

Once our engineering determined the specific problem, we immediately procured a new CA SSL Certificate chain. The process took approximately 30 minutes to install the certificate.

At that time many devices were able to register and we believed the issue was resolved.

Unfortunately the resolution notice was sent prematurely and we quickly identified a large amount of customers who continued to face registration issues.

Upon further investigation we were able to identify the issue was being caused by some left over PEM certificates which were interfering with some devices getting a secure encryption. Which ultimately prevent them from being able to register their device to make calls.

The final solution was implemented by clearing old certificates and reissuing the new CA SSL certificate once more, which restored all services and allowed devices to register once again.

We are sincerely sorry for the interruption of service today and recognize the major impact this has on our very valued customers and their businesses. We do not take these situations lightly, learn from these experiences and a promise to use all data from this event to make improvements to our monitoring and put measures in place to expedite an event resolution such as this in the future.

We wholeheartedly understand that our customers put a significant amount of faith in our ability to prevent these issues. We are working every day to continually improve our technology and engineering to make MeloTel a more reliable and stable service.

Thank you for your patience and we sincerely appreciate your understanding during this event. We value your business.

Posted Dec 05, 2019 - 21:39 EST

This incident has been resolved.
Posted Dec 05, 2019 - 21:17 EST
We are continuing to monitor for any further issues.
Posted Dec 05, 2019 - 21:17 EST
All reports are that phones are now registering without issue. We will continue to monitor this situation closely. We are truly very sorry for the inconvenience this morning. An RFO will follow.
Posted Dec 05, 2019 - 10:12 EST
Phones are now registering. If not, please reboot. We are monitoring closely.
Posted Dec 05, 2019 - 10:07 EST
We are still working on this with the highest urgency. Updates will follow. We do not have an ETR at this time.
Posted Dec 05, 2019 - 10:04 EST
We are very sorry for the inconvenience. We have all hands on deck to see this issue is restored as soon as possible. We have our engineers and switch vendor assessing the situation.
Posted Dec 05, 2019 - 09:26 EST
It appears there are still some issues with clients being unable to register devices. Were investigating and will update shortly.
Posted Dec 05, 2019 - 09:11 EST
This incident affected: VoIP Services (SIP Registration).