Rackspace hosted Exchange suffered a devastating outage starting December 2, 2022 and is still ongoing since 12:37 AM December 4th. At first referred to as connectivity and login issues, the guidance was ultimately updated to announce that they were dealing with a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the problem was, much less an ETA of when it would be resolved.
Consumers on Buy Twitter Verified reported that Rackspace was not responding to support emails.
This has been rather the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours approximately. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client independently messaged me over social networks on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Unsure the number of business that is, however it’s substantial.
They’re serving a 554 long delay bounce so individuals emailing in aren’t knowledgeable about the bounce for a number of hours.”
The official Rackspace status page used a running upgrade of the interruption but the initial posts had no information other than there was a failure and it was being examined.
The first official upgrade was on December second at 2:49 AM:
“We are investigating a concern that is impacting our Hosted Exchange environments. More details will be published as they appear.”
Thirteen minutes later on Rackspace began calling it a “connection issue.”
“We are examining reports of connectivity issues to our Exchange environments.
Users might experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates explained the ongoing issue as “connectivity and login issues” then later that afternoon at 1:54 PM Rackspace announced they were still in the “examination phase” of the interruption, still attempting to determine what failed.
And they were still calling it “connection and login issues” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
Four hours later on Rackspace referred to the circumstance as a “considerable failure”and started offering their consumers free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround up until they understood the issue and might bring the system back online.
The main guidance mentioned:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further concerns while we continue work to restore service. As we continue to overcome the origin of the problem, we have an alternate service that will re-activate your ability to send and receive emails.
At no charge to you, we will be supplying you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notification.”
Rackspace Hosted Exchange Security Event
It was not until nearly 24 hours later on at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was experiencing a security event.
The statement further exposed that the Rackspace service technicians had actually powered down and disconnected the Exchange environment.
“After additional analysis, we have identified that this is a security incident.
The recognized effect is separated to a portion of our Hosted Exchange platform. We are taking required actions to examine and secure our environments.”
Twelve hours later that afternoon they upgraded the status page with more info that their security team and outdoors experts were still working on resolving the blackout.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not released information of the security occasion.
A security event generally includes a vulnerability and there are two extreme vulnerabilities currently in the wile that were patched in November 2022.
These are the two most current vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to read and alter information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assailant has the ability to run harmful code on a server.
An advisory published in October 2022 described the effect of the vulnerabilities:
“An authenticated remote aggressor can carry out SSRF attacks to escalate privileges and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mail box server, the assaulter can possibly gain access to other resources through lateral movement into Exchange and Active Directory site environments.”
The Rackspace blackout updates have not indicated what the particular issue was, just that it was a security occurrence.
The most present status update as of December fourth mentioned that the service is still down and customers are motivated to move to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in attending to the event. The accessibility of your service and security of your information is of high value.
We have actually dedicated comprehensive internal resources and engaged first-rate external proficiency in our efforts to lessen unfavorable effects to clients.”
It’s possible that the above noted vulnerabilities belong to the security occurrence affecting the Rackspace Hosted Exchange service.
There has been no statement of whether consumer details has actually been jeopardized. This event is still continuous.
Featured image by Best SMM Panel/Orn Rin