Yesterday (Wednesday) we experienced a problem with voip devices losing registration. The root cause was not identified immediately but we were able to get the services back to operational.
Last night we performed some maintenance tasks on the environment in an effort to mitigate the problem.
This morning again, devices were flapping registration causing customers to lose calls and prevent them from being able to make calls.
Once we involved our switch vendor, they were able to clearly identify the problem being related to a security protocol we have in place, related to our intrusion and toll fraud IP blocking processes.
Technically, a cache was building too large and needed to be pruned. Since it was exceeding capacity, the firewall would stop all connections as a fail safe.
The log was pruned and services were restarted which restored service.
Engineers are confident the adjustment to log pruning rotation has remedied the problem once and for all.
We learned an important lesson from this event and stand committed to our customers to always strive for a very high standard of reliability.
We sincerely apologize for the inconvenience caused during this event.
From the technical support perspective, this ticket has been resolved. If anything related to this ticket changes, please reply to this email to re-open. Otherwise, please open a new ticket by emailing us at support@melotel.com.