Global DNS Issue
Incident Report for BitCloud
Resolved
Our DNS services have returned to normal after a month-long struggle to re-populate our global name servers. BitCloud has changed a great deal of its systems and we're offering customers a massive discount to return now that we're stable again. If you were affected by the outage, get in touch to get 1 year of free hosting!
Posted Apr 21, 2019 - 13:01 PDT
Update
We continue to monitor our DNS propagation throughout the world. Our new nameservers are now recognized globally, but domains that were previously on the old nameservers are still only resolving 50% throughout the world. DNS takes 24 to 72 hours to fully propagate. We expect our system to be fully healed by Sunday, March 10, 2019.
Posted Mar 07, 2019 - 07:33 PST
Update
We are upgrading our status to Degraded Performance. Our servers and systems are online and accessible by more than half of the Internet. Phone systems and hosting systems have been restored. Please ensure that you're updating your hosting accounts to use our new name servers as discussed here: https://bitco.ly/TOUPC
Posted Mar 07, 2019 - 00:43 PST
Update
We're upgrading our status to: Partial Outage. We have published detailed instructions on how to restore your website to normal operation. You can find the detailed instructions here: https://bitco.ly/TOUPC
Posted Mar 06, 2019 - 13:08 PST
Monitoring
Our domain name registrar has again corrected the issue that caused the DNS outage. It can take up to 72 hours for the domain name system to recognize this modification. We will continue to provide updates as more is known.
Posted Mar 06, 2019 - 11:17 PST
Investigating
Though yesterday we identified the issue with the Global DNS systems and a problem at the registry level, which was corrected and began to repair itself over night, the domain registrar involved has caused a total repeat of the incident. The NS servers are no longer propagating. BitCloud is changing the status of this issue back to investigating as there is no current resolution.
Posted Mar 06, 2019 - 09:52 PST
Update
We are continuing to monitor for any further issues.
Posted Mar 06, 2019 - 09:34 PST
Update
We continue to monitor the repropogation of our DNS nameservers through the global network. At present, we are still experiencing a significant outage, which will continue to manifest in customer websites and web apps being down. Our API system is also degraded as a result, so many hosted mobile app web services remain down. Telephone support is currently unavailable as the DNS issue also effects our VOIP systems. We will continue to monitor the DNS issues and will provide an updating in the coming hours.
Posted Mar 06, 2019 - 09:32 PST
Update
We are continuing to monitor for any further issues.
Posted Mar 06, 2019 - 09:29 PST
Monitoring
We have resolved the DNS network issue, which occured at the domain registrar level. At this time, all BitCloud systems appear offline except in the Paris region. DNS systems will need to propogate the changes at the registry level. We're updating the status to monitoring but leaving our system at Major Outage until such time as more than half the world can access our systems. A full announcement will be made in the coming hours and days to explain what happened more fully.
Posted Mar 06, 2019 - 00:42 PST
Update
For internal and external Bit users, our telephone system is currently not functioning as a result of the global DNS issues. BitNet is online if you're within a BitCreative office, but offline for freelancers and any telecommute workers. VPN services are also offline. We continue to investigate.
Posted Mar 05, 2019 - 22:55 PST
Identified
We are investigating a global DNS outage that is causing all BitCloud servers to be offline. We are working hard to resolve these issues and will update as soon as we can.
Posted Mar 05, 2019 - 22:47 PST
This incident affected: BitCloud (Hosting 1, Hosting 2, DNS 1, DNS 2, DNS 3, BitCloud API Hosting Services) and Bit Corporate (BitNet, SBC, Media Gateway, and Phone System Infrastructure).