Issue affecting multiple Carriers
Incident Report for MeloTel Network Operations (NOC)
Postmortem

The last 24 hours has been extremely challenging for everyone in British Columbia, and the effects of the severe weather events have had far reaching implications. Extreme weather events in southern British Columbia led to torrential rain and massive landslides across the province, which has had massive effects across the Telecom Industry.

As a direct result, outages are being felt by virtually every large telecom provider across Canada, as major network facilities common to several major network providers have been physically impaired by the weather, and restoration efforts are impaired by emergency services.

Distributel and Iristel have experienced a massive and unprecedented impact to their national core network including the redundant connections and systems they had in place that tie together their network.

Here is a summary of events, and what they are doing right now to fully restore essential services and re-establish business activities.

Summary of Events
Distributel & Thinktel have mulltiple core network sites in Vancouver, BC. Ssites are in independent carrier grade Data Centers. The network sites house several key platforms that are critical in delivering your voice and data services.

These carriers have multiple independent diverse paths from multiple carriers in place from both Vancouver core sites to connect back to the rest of the network. These paths traverse a diverse set of major telco partner networks serving southern BC and the US.
On Sunday night they lost one of the core connections because of a landslide in BC, however there were no customer impacts due to the engineered redundancy in the network.

On Monday afternoon, several landslides in Southern BC took out major highways and railways, resulting in major cuts to two more of their diverse fiber paths provided by our other major telco partner. This resulted in initial set of outages for both our customers and our telco partner’s customers.

The engineering team worked tirelessly to restore many services using alternate routes from another major telecom provider. Almost immediately upon restoral, at approximately, another set of extreme weather events in the region caused an outage in alternate carrier partner paths. Removing all links to Vancouver connecting our core sites to the rest of our network.

Distributel & Iristel both have have additional paths through the US for select traffic which also lost connectivity as it comes into Vancouver through the northwest US, which is facing similar weather conditions.

The biggest challenge we continue to face is to restore physical connectivity with the Vancouver core site, which requires access to the regions directly affected by the landslides. Emergency Services have been restricted from accessing the area, prioritizing evacuation and rescue operations.

Our team has worked throughout the night to restore as many services as possible using temporary solutions, identifying alternate paths to our core network site in Vancouver. While it’s important to us that we are able to provide you access to your services, we do fully recognize that these services are not being delivered at the service level we expect, until our carriers can fully restore the primary routes to the Vancouver core site.

During the process of restoration, we are working closely with our telecom partners to bring back services on our collective networks.

What’s Next?
We are continuing to monitor our telecom partners’ efforts for the physical restoration of the fiber circuits in areas directly impacted by the landslides. We do not have an ETA yet, as Emergency services remain a priority.

Our engineering team is actively working to bring back online all services with maximum service levels as quickly as possible. Our Business Technical Support team together with your Account team will keep you updated on service restoral efforts, specific to your account.

Once we are fully back in operations, we will be reviewing our network design, disaster recovery plans, to see how we can improve our response to these unforeseen natural weather events.

Your business is important to us, rest assured we are doing everything possible to get you back up in full operations.

Posted Nov 17, 2021 - 14:30 EST

Resolved
We have recieved a new update from Distributel/Thinktel. All services have been fully restored with the exception of DID service in the following rate centers. Kamloops . Cranbrook. Vernon. Penticton. This service status update will be closed for now until we recievan official RFO.
Posted Nov 17, 2021 - 11:16 EST
Update
We have received an update from Comwave.

“We have made significant progress recovering the majority of services that have been affected by the severe weather events in British Columbia. At this point, we have most of our routes back up and running on permanent or temporary facilities, while we await the remaining routes to be permanently restored by our telco partners from fiber cuts resulting from the landslides.

For next few days, you may experience intermittent call completion or voice quality, as we continue to fine tune our network, and bring it back to full capacity and redundancy.

Due to severity of impact from weather events, and lack of alternate routes available, we currently have four CLEC regions that have not been fully restored. We are working with our partners to fully restore all services as quickly and safely as possible. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.

• Kamloops
• Penticton
• Vernon
• Cranbrook”

We apologize for any inconvenience and thank you for your understanding.
Posted Nov 17, 2021 - 06:36 EST
Update
We have received an update from Iristel.

“We have completed the deployment of our network changes. At this time, our internal stats are showing optimal traffic patterns and all systems are functioning per design in normal parameters. Our engineers at this time are not executing any additional changes. While we are confident that the current status will remain stable without further interventions, we continue to monitor our entire network. Our team will revert with a detailed "Reason for Outage" in the following days after all the details of the event have been gathered.”

We are still waiting on updates from other carriers.

We thank you for your patience and understanding.
Posted Nov 16, 2021 - 18:27 EST
Update
Iristel has provided an update: “ While we continue to monitor the full restoration of our network and services back to normal operation, our stats are showing better and stable improvement.”
Posted Nov 16, 2021 - 17:10 EST
Update
We have to received an update from ThinkTel/Distributel.

“We continue to face major network outages as result of severe weather in BC. Our Engineering teams have been working throughout the night to provide alternate options as this network event has impacted multiple service providers in the region.

At this point, certain services have been temporarily restored using alternate paths. During this time you may experience intermittent call completion or voice quality.

We are working with our partners to fully restore all services as quickly and safely as possible. Emergency crews are attempting to repair fiber cuts to multiple circuits affected by landslides in the region. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.”

We apologize for any inconvenience and thank you for your understanding.
Posted Nov 16, 2021 - 16:36 EST
Update
We received an update from Iristel. “While we continue to add more measures to complete the full restoration of our services to normal operation, our stats are currently showing good improvement. ETR remains 16-Nov-2021 16:45 EST. “

We kindly request you patience and understanding.

We would provide more updates soon.
Posted Nov 16, 2021 - 15:36 EST
Update
Latest Update from Iristel: Please be advised that we continue to work towards full resolution and maintain our ETR of 14:00 EST. Currently, we are facing intermittent issues with audio, call drops and post dial delays. These issues are related to this incident and are expected to subside by 14:00 EST. No updates from other carriers at this time.
Posted Nov 16, 2021 - 12:31 EST
Update
Future updates will include details from only the primary channel providers we are monitoring.

Voicemeup appears to be getting their information from Iristel. So we will drop Voicemeup updates and stick with direct updates from Iristel.

Here are the latest updates from each carrier.

Distributel: We are working with our partners on providing alternatives options as this is a wide-spread carrier network outage, impacted multiple providers. We are working with our partners to restore services as quickly and safely as possible.
Estimated Time of Recovery (ETR) is not confirmed yet but be rest assured that all hands are on deck to resolve the issue as soon as possible.

Comwave: Please be advised that we are currently experiencing DS3 outage for below rate centers. Calls are failing with fast busy. We have escalated this issue to our upstream carrier. We will keep escalating with them and provide updates accordingly. Most impact is being felt in NANAIMO, VICTORIA, ABBOTSFORD, KELOWNA, SOUTH KAMLOOPS, SOUTH KAMLOOPS, PENTICTON, VERNON, VANCOUVER, SOUTH KAMLOOPS, NEW WESTMINSTER.

Bell Canada: Wireless calls and data services are being restored for mobility customers in B-C & Alberta. Calls to landlines from mobile phones and vice versa are still impacted. Some customers will continue to experience service interruptions as we work with our partners to restore services.

We are currently attempting routes thru some of our secondary providers as a possible mitigation.
Posted Nov 16, 2021 - 10:23 EST
Update
The ongoing issues being reported are being expanded to involve several national telco carriers in different ways.

Comwave: Currently reporting ongoing issues with CLECS in multiple local exchanges in Alberta and British Columbia.
Comwave was previously reporting calling issues dozens local exchanges in Ontario and Quebec. They claim it is resolved, but we are seeing it is not resolved yet.
Telnyx is currently providing service status updates for service interruptions and some calls may be experiencing audio degradation.
Telnyx was previously reporting DDos Attacks which their maintenance window has closed.
ThinkTel is reporting network issues affecting customers in all Provinces in Canada caused by weather issues in British Columbia.
VoiceMeUP in Montreal is reporting issues effecting incoming cand outgoing calls.
Iristel is reporting issues with incoming and outgoing calls expected to be resolved by 2pm.

Regretfully these are some serious situations happening across many high-profile carriers across the country that it makes it very difficult for our customers to experience the ideal calling scenarios.

At this time we are facing reports from customers experiencing the inability to place outgoing calls to some areas and in some cases their dedicated DIDs are failing as well.

We have found that Mobile calls thru Bell and Rogers will behave differently as well.

We have escallated this ticket to "partial outage".
Posted Nov 16, 2021 - 10:09 EST
Update
We have received an official update from Iristel NOC this morning. "Please be advised that we are working towards resolving the issue today by 2PM EST. We will keep you updated as we make more progress on this incident and its mitigation." Although MeloTel is not directly impacted by Iristel outages, we feel it is important to keep our customers informed about this issue as we have received reports of calling problems by our customers for any calls transmitting over Iristel. Furthermore, this issue appears to be multi-pronged as there are several major telcos reporting issues last night and today. We will keep you informed as we get updates.
Posted Nov 16, 2021 - 09:06 EST
Update
The upstream provider has confirmed services are being restored. MeloTel is observing improved connectivity on local Canadian calls at this time. We are continuing to work with them and will monitor this situation closely for further updates.
Posted Nov 16, 2021 - 08:34 EST
Update
We are informed that Iristel is the victim of a high-profile DDoS attack. While they have yet to respond with more details or update their network status page, we have had confirmation from their larger customers.

We thank you for your patience while awaiting further information on this situation. We will continue to update our network status as soon as we have another update.
Posted Nov 15, 2021 - 20:56 EST
Update
We have also confirmed that Comwave is experiencing major outage on multiple legs of their network caused by severe weather events in Western Canada that is affecting multiple Telco’s across Canada. This appears to be impacting calling accross the country in many ways. We are working on this case and will provide an update shortly.
Posted Nov 15, 2021 - 16:36 EST
Identified
We have identified a problem affecting our inbound calls and are currently awaiting details from CLEC Iristel. CLEC senior engineers have implemented a fix around 13:51 EST. However, their services are still noticed failing intermittently and they are working on resolving the pending issues. No accurate ETR has yet been provided.
Posted Nov 15, 2021 - 16:19 EST
This incident affected: VoIP Services (Canadian DIDs).