Between the hour of 11:27am and 5:39pm on Thursday September 30th, 2021, MeloTel users encountered issues with specific Polycom model devices failing to register if they were rebooted. The issue escallated to major service inturruption for many customers. The issue was ultimately resolved by issuing a new TLS certificate provider. Total major impact was 2.5 hours.

Here is the timeline of events.

• 11:27am - The event was detected by a customer who had relocated a telephone to another workstation and then failing to register their extensions afterward. The problem was reported to MeloTel Support and our team started working on the customer’s issue.
• 11:32am - We received a second complaint from a customer reporting a similar problem.
• 12:10pm – MeloTel support attempted to replicate the problem internally and confirmed a problem with re-registration.
• 12:18pm – MeloTel announced on status.melotel.com that we were investigating troubles with devices re-registering. (At this time, the issue had only known to impact two devices, but we knew it would impact more if a customer’s phone was restarted)
• 12:51pm – Our engineers believed they narrowed the issue to our provisioning platform and focused efforts there.
• 2:22pm – After exhaustive troubleshooting our provisioning platform we involved our switch vendor engineers to investigate a possible Certificate issue on our PBX platform. (At this time the issue had only impacted approximately 15 customer phones)
• 3:06pm – In an attempt to re-issue our TLS encryption certificate, our switch engineers restarted a service in our cloud which resulted in many of customers Polycom devices becoming unregistered. While calling on the network still worked. Without a device to register, it was impossible for many customers to make or receive calls from their Polycom phone.
• 3:09pm – Since the incident was impacting more than 1,000 customers, the incident status was updated to critical status. Remediation measures were being offered to customers to re-route their services to cell phones and cloud softphone apps.
• 3:44pm – Our engineers identified the issue was related to kamailio TLS registration service using Let’s Encrypt SSL IdentTrust DST Root CA X3 certificate.
• 3:58pm – MeloTel engineers work in corporation with Polycom Engineering Support, our switch vendor, and a security expert to determine the only solution would be to issue a new security certificate from DigiCert, a Polycom confirmed trusted certified authority.
• 4:46pm – We generated a new CSR and placed the order with DigiCert. Unfortunately, the activation is not instant and for security, MeloTel had to be independently verified by DigiCert before the certificate would be issued.
• 5:28pm – The certificate was issued to MeloTel by DigiCert.
• 5:38pm – The certificate was installed by our engineers and devices immediately began re-registering.
• 5:38pm – All clear was given to customers. Service status closed.

IN CLOSING: While MeloTel had known about Let’s Crypt DST Root CA X3 Expiration this since May 7, 2021, we were assured that Polycom Firmware versions would survive this without failing. While it was true that many Polycom devices were not impacted by this, there were some Polycom VVX series devices which were impacted and ultimately put out of service.

We have learned from this. MeloTel stand committed to stop using Let’s Encrypt SSL Certificates for TLS device registration moving forward. Our new certificate issuer is DigiCert.

We sincerely apologize for the inconvenience that you had experienced due to this incident. We thank you for your patience and your support.