Bell Canada TFN Outage: Incoming Toll Free Number Routing Problem
Incident Report for MeloTel Network Operations (NOC)
Postmortem

Below is the official RFO (Reason For Outage) from Comwave;

Dear Valued Customer,

We wanted to update you on recent toll-free issues you may have experienced.

Comwave uses several backbone networks for Toll-Free, and two weeks ago we added Bell Canada. While balancing traffic, unfortunately, Bell inadvertently left several hundred phone numbers in limbo between networks and without a carrier which ultimately resulted in out-of-service phone numbers. Bell also had issues with toll-free origination from the United States. As of Friday afternoon, we have reinstated all toll-free numbers back to the original Comwave network and have halted all further plans with Bell until further notice. After a decade of reliable toll-free service that has never given us a reason for second thought with our core network, we are not impressed with Bell.

Should a change be contemplated in the future, we will be consulting with you.

We understand how important phone numbers are to the businesses you support. Please accept our sincere apologies and let us know if we can help further.

Regards

Jiban Luitel

Comwave NOC

Posted Apr 29, 2019 - 13:57 EDT

Resolved
Please be advised Toll free numbers have been activated the USA coverage as well.
We ask that you please re-test your DID and let us know if you encounter any residual issues.
We are working on obtaining an RFO from underlying carrier which will be posted once we receive it.
Posted Apr 29, 2019 - 09:28 EDT
Monitoring
Please be advised Toll free numbers outage is now resolved and corrected for Canada Coverage only.

We ask that you please re-test your DID and let us know if you encounter any residual issues.

Please allow us 2 business days to activate the USA coverage.
Posted Apr 26, 2019 - 20:28 EDT
Update
We have not yet received the final all clear from Bell Canada. However we have verified each number tested is functioning properly. This ticket will remain open until further updates come. Ticket status downgraded.
Posted Apr 26, 2019 - 20:08 EDT
Update
Our carrier confirmed Canada coverage TFNs will be fixed in next 30 mins. It might take up to 2 business days to allow USA coverage.

We appreciate your patience on this matter.
Posted Apr 26, 2019 - 19:25 EDT
Update
Our carrier working on high priority to re-provision impacted numbers at their end. They are also waiting for the USA portion to load up. No ETA provided as of yet. Our team will continue to follow up with them. We will keep you posted.
Posted Apr 26, 2019 - 17:58 EDT
Update
Our carrier working on high priority to re-provision impacted numbers at their end. They are also waiting for the USA portion to load up.

No ETA provided as of yet. Our team will continue to follow up with them.

We will keep you posted.
Posted Apr 26, 2019 - 16:50 EDT
Update
We just got off the phone with our carrier and they have advised they require some more time no ETR. We are keeping a close eye as they are slowly repairing all your affected numbers and will be continuously testing.

Once we have a confirmation, we will advise.
Posted Apr 26, 2019 - 16:03 EDT
Update
We just received a confirmation from ULC (Bell Canada) this issue is expected to be resolved by by 4pm EST.
Posted Apr 26, 2019 - 15:19 EDT
Update
Bell Canada has found the root cause for this issue and currently working to fix.

According to Bell Canada, the ETA according to Bell Canada is 2 hours from 12:50pm EST.

We are sincerely sorry for the inconvenience caused to our customers.

We will update this ticket as more information becomes available.
Posted Apr 26, 2019 - 12:52 EDT
Update
We have received an update the Manager of the Network Operations Centre about the problem.

He states: "We believe we have located the issue and we are hoping this be fixed in the hour." (posted: 11:35am EST)

We are continuing to press our underlying carrier to provide us with frequent updates.
Posted Apr 26, 2019 - 11:42 EDT
Update
We have identified the problem with routing toll-free calls is due to an undying problem with Bell Canada. The nature of the outage is not known at this time. Bell Canada Network Operations has been engaged and we expect to get further updates shortly.
Posted Apr 26, 2019 - 10:58 EDT
Update
Our underlying carrier has acknowledged the ticket regarding the issue, however we do not have any official update from them on the matter at this time. We are escalating the matter now. Updates will follow shortly.
Posted Apr 26, 2019 - 10:52 EDT
Identified
We have identified an underlying carrier problem with some of our customers toll-free numbers reaching MeloTel. This is resulting in an automated message stating the number is unavailable. Although this does not affect all of our customers it does affect a significant portion of them which is why we are opening this incident.

This problem only seems to affect toll-free incoming calls.

We have a critical ticket open with our underlying carrier and will update our customer shortly.
Posted Apr 26, 2019 - 10:14 EDT
This incident affected: VoIP Services (Toll Free DIDs).