DaFeeder Slowness
Incident Report for MeloTel Network Operations (NOC)
Postmortem

I would like to address the recent issues experienced with MeloTel DaFeeder platform.

 Growing pains would be a simplified explanation of the challenges we are facing with the DaFeeder.

 This wonderful platform has served customers since 2010 and we have continued to grow steadily in users and feature implementation.

 Over the past 1 year we have undergone some of the most significant feature enhancements, integrations and customization we have ever implemented.

 Many of our enhancements have performed significant time saving and tracking improvements for our customers.

As MeloTel popularity has grown, with it has come a growing user base for DaFeeder, resulting in hundreds more users in a short period of time (6 months).

 The growth has become exponential and we have been working around the clock to make improvements and infrastructure enhancements to support the growing demand.

We have faced challenges with large imports, large exports, feeding speed and our engine stalling due to overloading.

Please know that the challenges with DaFeeder is not taken lightly. Our customers interruption of service is not something acceptable to us.

Over the years, MeloTel has faced growing pains with many of our platforms and we have always persevered and become stronger and more stable because of it. DaFeeder is no different.

Over the next few days we are continuing to work on finalizing our lead feeding performance enhancements. During this time, it is possible we may ask users to refresh their browsers to accept maintenance updates. Further down the road, we will be coming some of our larger customers to dedicated environments as well.

We sincerely appreciate your patience and understanding during this growth spurt. We stand committed to making DaFeeder fast, reliable and no compromises on functionality as soon as possible.

Thank-you for choosing MeloTel.

John M. Meloche

CEO

MeloTel

Posted Sep 19, 2018 - 15:45 EDT

Resolved
This incident has been resolved.
Posted Sep 19, 2018 - 15:44 EDT
Update
Our DaFeeder maintenance was completed last night and all systems are running optimal. The user experience during feeding and disposition is much faster.

This ticket will remain open as we monitor performance today. The ticket will close with a postmortem later today.
Posted Sep 19, 2018 - 09:15 EDT
Update
We will be performing last nights update again tonight to implement improvements before start tomorrow. For today we maintain the service is functional, but responding slowly. We apologize for the inconvenience and stand committed to resolving the slowness. thank you for your patience during this time.
Posted Sep 18, 2018 - 11:23 EDT
Update
Due to some campaigns still not being able to be loaded for customers, we have made the decision to roll back some recent updates. It the speed of feeding is not expected to be as fast as today, but all campaigns will be loading. Feeder should be loading again.
Posted Sep 18, 2018 - 11:20 EDT
Update
DaFeeder engine is being restarted. May be down for 1 minute.
Posted Sep 18, 2018 - 11:15 EDT
Update
Shortly we are going to restart the engine for the feeder which will require users to refresh the browser.
Posted Sep 18, 2018 - 10:42 EDT
Update
We are aware of some users having difficulty feeding leads at the moment. This should be resolved shortly. A more detailed update will follow.
Posted Sep 18, 2018 - 10:14 EDT
Update
We are currently planning an update to the Feeding engine to improve the MySQL pooling. We are fighting capacity issues and strategizing improvements to accommodate DaFeeders rapid growth. We sincerely apologize for the inconvenience. Systems are functional, but may be delayed. This is high important for our development team.
Posted Sep 17, 2018 - 12:24 EDT
Monitoring
Our engine stalled. It has been restarted.
Posted Sep 17, 2018 - 11:46 EDT
Investigating
Were investigating slow DaFeeder. Updates ot follow.
Posted Sep 17, 2018 - 11:34 EDT