This postmortem is related to the service interruption which affected MeloTel customers this morning between (8:10 EST) and (10:03 EST).

The event affected registration of devices which used secure encryption TLS and was the result of a broken SSL chain in our Kamailio server. Since the SSL chain was broken, the devices which renewed registration could not authenticate to our servers. This means those devices were not able to make or receive calls.

Once our engineering determined the specific problem, we immediately procured a new CA SSL Certificate chain. The process took approximately 30 minutes to install the certificate.

At that time many devices were able to register and we believed the issue was resolved.

Unfortunately the resolution notice was sent prematurely and we quickly identified a large amount of customers who continued to face registration issues.

Upon further investigation we were able to identify the issue was being caused by some left over PEM certificates which were interfering with some devices getting a secure encryption. Which ultimately prevent them from being able to register their device to make calls.

The final solution was implemented by clearing old certificates and reissuing the new CA SSL certificate once more, which restored all services and allowed devices to register once again.

We are sincerely sorry for the interruption of service today and recognize the major impact this has on our very valued customers and their businesses. We do not take these situations lightly, learn from these experiences and a promise to use all data from this event to make improvements to our monitoring and put measures in place to expedite an event resolution such as this in the future.

We wholeheartedly understand that our customers put a significant amount of faith in our ability to prevent these issues. We are working every day to continually improve our technology and engineering to make MeloTel a more reliable and stable service.

Thank you for your patience and we sincerely appreciate your understanding during this event. We value your business.