MonsterMegs Status - Incident history

Mysql Offline

Mon, 20 Jul 2026 19:10:22 +0000

Type: Incident

Duration: 12 minutes

Affected Components:

Jul 20, 19:10:22 GMT+0
Investigating - We are aware that mysql is offline and databases are not loading. We are working on the issue as we speak..

Jul 20, 19:15:02 GMT+0
Identified - The cause has been found and it is still flushing from memory on a restart. It is taking much longer than it should, but once it is back up, we will review the issue at hand..

Jul 20, 19:22:27 GMT+0
Resolved - Mysql is now back online and everything is performing as it should..

Investigating issue with Lightning server

Fri, 10 Jul 2026 02:01:06 +0000

Type: Incident

Duration: 6 minutes

Affected Components:

Jul 10, 02:01:06 GMT+0
Investigating - We are currently investigating an issue with the server Lightning. Our engineers have been alerted and further details will be provided if necessary..

Jul 10, 02:07:35 GMT+0
Resolved - The issue with the Lightning server has been resolved and all services are running as normal. If you are still facing issues, please submit a ticket to our Support department..

Router Maintenance Window

Tue, 3 Feb 2026 07:00:00 +0000

Type: Maintenance

Duration: 1 hour

Affected Components: ,

Feb 3, 07:00:00 GMT+0
Identified - Estimated Maintenance Window: 10 minutes Description: We will be performing a scheduled hardware upgrade on the SLC core routers to support increased network demand and capacity requirements. This maintenance is necessary to ensure continued performance and reliability of the network. Impact: • Brief service interruption affecting traffic traversing the SLC core routers during the maintenance window Action Required: No customer action is required. Network teams will monitor the upgrade and restore services immediately upon completion..

Feb 3, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 3, 08:00:00 GMT+0
Completed - Maintenance has completed successfully.

Dropped Packets

Wed, 21 Jan 2026 15:19:10 +0000

Type: Incident

Duration: 12 hours and 44 minutes

Affected Components: ,

Jan 22, 04:03:01 GMT+0
Resolved - We have not seen any further network impact. We are considering this issue closed..

Jan 21, 15:19:10 GMT+0
Identified - We are noticing dropped packets on our US servers. We have contacted the datacenter and they have confirmed that they are seeing an extreme 20 billion packet UDP amplification flood. They have been mitigating the attack and stated that they just need to resolve the random small packet loss. You may notice some random slowness or dropped connections..

Jan 21, 17:09:32 GMT+0
Monitoring - For the last hours, we not seen any further dropped connections. We will continue to monitor the network traffic over the next few hours and if we see no further issues, we will close this incident..

Lightning DDos Attack

Tue, 13 Jan 2026 16:15:15 +0000

Type: Incident

Duration: 6 days, 21 hours and 14 minutes

Affected Components:

Jan 13, 18:23:08 GMT+0
Identified - We have contacted the datacenter and they confirmed the attack is ongoing. Their mitigation system is blocking the attack to keep the server online, but there may be dropped connections from time to time. We also feel this attack may be coming from a Cloudflare IP address to a site that is hosted under Cloudflare. So if you are seeing Cloudflare error pages, you might want to disable the proxy service on that domain for the time being..

Jan 13, 21:16:05 GMT+0
Monitoring - It looks as if the attack as stopped. We are not seeing anymore dropped connections. We will continue to monitor this over the next few hours and if there are no changes, we will close this issue..

Jan 13, 16:15:15 GMT+0
Identified - There has been an ongoing ddos attack on the Lightning server. The datacenter has already began filtering the attack, but this may lead to random dropped connections until either the attack stops or the filtering completely mitigates the attack. We will keep a close eye on this and will update as new information comes in..

Jan 20, 00:59:11 GMT+0
Identified - The ddos attack continues. The datacenter is doing all the filtering they can without completely taking the server offline. Unfortunately now it is a waiting game for the attach to subside. We will continue to update as we monitor this situation..

Jan 20, 13:29:11 GMT+0
Resolved - The ddos attack is now over and there is no more mitigation being performed on the server. We are considering this issue closed..

Jan 14, 04:46:59 GMT+0
Resolved - It has been several hours and there has been no further ddos activity. We care considering this issue closed..

Jan 19, 20:42:38 GMT+0
Identified - We are reopening this incident as it seems the server is under another ddos attack. The datacenter is filtering the attack to keep the server online, but you may notice some dropped connections/packets..

US Network Disruption

Mon, 17 Nov 2025 02:52:53 +0000

Type: Incident

Duration: 36 minutes

Affected Components: ,

Nov 17, 03:28:31 GMT+0
Resolved - The network is now back up. There was an issue with a core spine switch and has been replaced..

Nov 17, 02:52:53 GMT+0
Investigating - We are currently investigating a network outage in our US datacenter. We will update as more information comes in..

Nov 17, 03:05:48 GMT+0
Investigating - Fiberstate is aware of the issue and they are investigating the cause. It does look to be a network disruption..

Server Offline/Locked

Tue, 17 Jun 2025 20:43:45 +0000

Type: Incident

Duration: 9 hours and 15 minutes

Affected Components:

Jun 17, 20:43:45 GMT+0
Identified - Today we received an abuse complaint from the datacenter, giving us one hour to remove the mentioned content. Within that hour timeframe we removed the content and reported back to the datacenter, stating it had been removed and the account terminated. Later we received notice that the content was still loading, although the account was terminated. After that 1 hour timeframe, the datacenter locked the server, causing it to appear offline. The mentioned content continues to load even though the account was removed and the server being locked. Proving that the content is not being hosted on our server any longer. We are working with the datacenter to get this resolved and the server unlocked/back online..

Jun 17, 21:57:26 GMT+0
Identified - We have filed all the necessary documentation to prove the content (account) was removed and filed an unlock request. Now we are just waiting to hear back from their support/abuse department. .

Jun 18, 00:35:05 GMT+0
Identified - We are still awaiting a response from the datacenter. Unfortunately they do not allow multiple unlock requests, so we are on the mercy of their response time. We are hoping that it will not be too much longer. .

Jun 18, 05:58:38 GMT+0
Resolved - The datacenter has confirmed that the content was properly removed from the server and the block has been removed. All network connectivity has been restored to the server..

Investigating issue with Storm server

Tue, 13 May 2025 07:27:05 +0000

Type: Incident

Duration: 2 hours and 41 minutes

Affected Components:

May 13, 07:27:05 GMT+0
Investigating - We are currently investigating an issue with the server Storm. Our engineers have been alerted and further details will be provided if necessary..

May 13, 07:59:02 GMT+0
Identified - We are noticing some major packet loss on this server. We have contacted the datacenter to investigate further..

May 13, 08:29:15 GMT+0
Identified - The datacenter is aware of the issue and working to restore full network connectivity..

May 13, 10:07:38 GMT+0
Resolved - The network has stabilized and everything is back to normal..

Failed Hard Drive - Storm

Sat, 15 Feb 2025 23:35:20 +0000

Type: Incident

Duration: 23 hours and 11 minutes

Affected Components:

Feb 15, 23:35:20 GMT+0
Identified - We have been informed by our monitoring systems, that one of the hard drives on the server has failed. We are going to proceed immediately with a request to have the hard drive replaced. We anticipate the drive to be replace within the next couple hours. During this time, the server will go offline for about 30 minutes. After the replacement is done, there will be a few reboots required to rebuild the raid and update the firmware on the drive..

Feb 15, 23:43:36 GMT+0
Identified - The datacenter has already taken the server offline to replace the drive. We will update as things progress..

Feb 15, 23:55:06 GMT+0
Identified - The hard drive has been replaced and the server is back online. Now we will proceed to rebuild the raid and update the firmware on the drive. Expect 1-2 reboots and then the replacement will be complete..

Feb 16, 00:51:52 GMT+0
Monitoring - We found the newly installed drive has the latest firmware, so there is no need for further reboots. We are awaiting for the raid rebuild to complete and then we will consider this issue closed..

Feb 16, 01:05:01 GMT+0
Investigating - Something has happened during the raid rebuild and has caused several parts of the server to become inaccessible. We are investigating and will update as we have more information..

Feb 16, 02:00:02 GMT+0
Investigating - We are still investigating the what happened during the raid rebuild. At this time we don't have any further information, but hope to have more details very soon..

Feb 16, 02:46:13 GMT+0
Monitoring - We found that after the reboot, one of the servers security programs was triggering some extreme ddos protection that was freezing the server and causing processes to fail. We believe we have disabled the features that have caused this and have also reached out to the vendor to find out why this happened. If this does happen again, we will disable the service completely, until we can work this out with the vendor. We are going to continue to monitor over the next few hours..

Feb 16, 03:35:20 GMT+0
Investigating - It appears that was not the cause as we had thought. Again we are seeing services falling off after a period of time. We are continuing to troubleshoot the issue. This is looking to be a bit more complicated than we anticipated. At this point we do not have an accurate timeframe to offer of how soon this will be resolved..

Feb 16, 06:26:46 GMT+0
Investigating - After further investigation, it looks like the second drive might have failed during the resync of the raid array. This is extremely rare, but it has been seen to happen. Before we jump to the extreme and reprovision the server, reconfigure it, and start restoring backups...we are gonna still work to try and recover the data without a complete rebuild. If we do have to resort to a full rebuild, we will be email all customers with further details. While we work to try and recover the data as it stands, the updates to this incident will be limited. As we wan to put full focus on attempting to recover the data. If it does come to the point that we need to rebuild and restore backups, this could realistically take a few days to fully restore the server. So we want everyone to be aware that will not be a quick process and will take time to restore everything correctly without rushing and causing more errors..

Feb 16, 07:10:47 GMT+0
Monitoring - We hopefully have some good news, but we are not out of the woods yet. We had the datacenter swap the drives around from their original mounting location. This many times, will refresh the bios to see the drives again. This could have been a case of the main drive getting bumped and having a loose connection or the bios just having issues reading the drives. Now that the main drive is showing up, the server booted correctly and the raid has been resyncing for the last 20 minutes or so. Once the raid resync completes, everything should be good to go. We will update once the resync is complete or if there are any further issues along the way..

Feb 16, 08:02:49 GMT+0
Investigating - Unfortunately, the resync of the raid array failed at 82%. So we are continuing to explore our options and see if maybe the new hard drive has issues itself..

Feb 16, 14:04:13 GMT+0
Identified - We are currently talking with the datacenter to try a motherboard replacement. The fact that the drive does show up fine for a period of time and then disappears, makes us think the motherboard may be malfunctioning. This will be our last attempt to rectify this issue and if this does not work, we will have to proceed with disaster recovery..

Feb 16, 15:00:28 GMT+0
Monitoring - As you may have noticed, the server is back online. Please note that is most likely temporary. The datacenter did not feel that the problem was with the motherboard, so we have booted the server with just the one drive that has all the data. We are gonna let this run for a couple hours and see if it remains stable. If it does, our next plan of action is most likely to setup a second server and then migrate the data to this second server. If the hard drive does fall out of connection or throws any errors, then we will be forced to rebuild the server and restore backups. We will update this post, once we have our next plan of action..

Feb 16, 17:39:43 GMT+0
Investigating - After talking with the datacenter, we feel we might have a lead on what is causing the drives too show failed or fall out of connection. It appears this server setup uses a PCI adapter when there is more than 1 hard drive installed. This morning we had the datacenter bypass the PCI adapter and plugin the min drive directly into the motherboard. The server has been running stable now for nearly 3 hours. So we feel that the PCI adapter might be failing. Within the hour we are going to take the server offline once again to plug the second drive back in and use a new PCI adapter. If this does snot resolve the issue, then we will have to migrate to a new server, but we have high hopes that it is the PCI adapter that is at fault..

Feb 16, 18:14:14 GMT+0
Monitoring - The PCI Adapter has been replaced and the second hard drive has been added back in. We will continue to monitor the server closely over the next few hours and will update this incident if there are any further issues..

Feb 16, 19:32:47 GMT+0
Monitoring - We are happy to state that the raid rebuild did complete and everything does look to be stable at this time. We do feel that the PCI Adapter was the cause of the hard drives showing as failed or disappearing from bios. We are going to keep this incident open for a few more hours and will closely monitor the server for any further issues. .

Feb 16, 22:46:02 GMT+0
Resolved - The server has now been running for over 4 hours and there have not been any errors. We are now confident that the issue is resolved. .

Hetzner Emergency Router Maintenance

Tue, 3 Dec 2024 03:38:04 +0000

Type: Incident

Duration: 1 hour and 54 minutes

Affected Components: ,

Dec 3, 03:38:04 GMT+0
Investigating - We are currently investigating a network outage in the Hetzner datacenter. We will update when we have more information..

Dec 3, 03:45:50 GMT+0
Identified - It appears they are performing emergency maintenance on several routers. Here is a snippet from their status page: During the above-mentioned time frame, we will be carrying out urgent maintenance work on the access router fsn1-dc13-ex9k2, fsn1-dc13-ex9k1, fsn1-dc11-ex9k2, fsn1-dc11-ex9k1, and the connected ToR switches. During this work, network traffic will be interrupted, meaning that affected customer systems will not be accessible during this time. This also affects vSwitches for Dedicated Root Servers. Private Switches and Custom Solutions are excluded from the maintenance, unless explicitly stated..

Dec 3, 04:13:45 GMT+0
Identified - The Lightning server is back online. We are currently waiting for connectivity to be restored on the Hurricane server..

Dec 3, 04:54:26 GMT+0
Identified - Hetzner has listed the maintenance as complete, but still the Hurricane server has no connectivity. We have contacted Hetzner support to find out why and are awaiting an update. We we post back as soon as we have more information..

Dec 3, 05:32:24 GMT+0
Resolved - After a server reboot, the server is back online. It appears for some reason, the server did not recapture the incoming network connection and a full server reboot was required..

Hard Drive Adapter/Cable Replacement

Fri, 11 Oct 2024 23:41:00 +0000

Type: Maintenance

Duration: 1 day and 28 minutes

Affected Components:

Oct 12, 01:13:01 GMT+0
Identified - They have cleaned the ports on the disk adapters and replaced the cables, but yet the error remains. We are consulting with the datacenter on our next move. We anticipate that the next action will take place tomorrow, once we figure out a game plan..

Oct 11, 23:41:01 GMT+0
Identified - Maintenance is now in progress.

Oct 11, 23:41:00 GMT+0
Identified - We have been getting some unusual errors in our logs and after talking to the datacenter, they feel the NVMe adapter or cable may be the culprit. Due to the nature of the errors, we are going to have the datacenter perform an emergency maintenance and replace the adapters and cables. We anticipate around 30 minutes of downtime and expect this to be performed within the hour..

Oct 13, 00:09:11 GMT+0
Completed - The have replaced the hard drive adapter with another model and rebooted the server. Both hard drives are now showing up correctly and we are resyncing the hard drives. The original errors have been resolved with this fix. Once we are done with the raid resync. We will close this incident..

Oct 12, 22:42:43 GMT+0
Identified - The server swap was completed, but we are now facing a hard drive that is not showing up. We have informed the datacenter and they are looking into it. Update: They are taking the server back offline to double check the missing drive..

Oct 13, 00:15:55 GMT+0
Completed - The raid resync has been completed successfully and we are now considering this incident closed. Thanks to all affected customers and their patience during resolving this hardware issue..

Oct 12, 20:49:15 GMT+0
Identified - After talks with the datacenter, we are going to move the drives to a new server (chassis), to rule out any further hardware issues. Please understand that this will involve 20-30 minutes of downtime. We do not have an exact time this will take place, but believe this will be completed within the next couple hours..

Thunder - Motherboard Failure

Wed, 4 Sep 2024 07:45:08 +0000

Type: Incident

Duration: 4 days, 9 hours and 27 minutes

Affected Components:

Sep 4, 07:51:47 GMT+0
Investigating - We have found the server to be offline and have contacted the datacenter as it will not power on. We will update as we have more information..

Sep 4, 07:45:08 GMT+0
Investigating - We are currently investigating an issue with the server Thunder. Our engineers have been alerted and further details will be provided if necessary..

Sep 4, 08:24:48 GMT+0
Investigating - We have contacted the datacenter and they state they will be looking into it shortly. We suspect this is a power supply issue and that is why it is not powering it back on. We will update as the datacenter updates us..

Sep 4, 09:17:09 GMT+0
Identified - The server has been taken completely offline by the datacenter and they are currently working on it. While we are still unsure of the exact cause, we are confident this is hardware related and anticipate the server to back online shortly..

Sep 4, 09:30:34 GMT+0
Resolved - The Thunder server is now back online. It appears the motherboard had a failure and this caused the server to crash and refuse any power options. The motherboard has been replaced and the server is back online..

Investigating issue with Hurricane server

Thu, 29 Aug 2024 17:40:08 +0000

Type: Incident

Duration: 2 days, 16 hours and 33 minutes

Affected Components:

Aug 29, 17:40:08 GMT+0
Investigating - We are currently investigating an issue with the server Hurricane. Our engineers have been alerted and further details will be provided if necessary..

Aug 29, 18:44:40 GMT+0
Resolved - The issue with the Hurricane server has been resolved and all services are running as normal. If you are still facing issues, please submit a ticket to our Support department..

Cpanel Inactive Licenses

Sat, 24 Aug 2024 18:07:27 +0000

Type: Incident

Duration: 1 hour and 30 minutes

Affected Components: , , , , ,

Aug 24, 18:07:27 GMT+0
Identified - It has come to our attention that all our Cpanel Licenses are marked as suspended by mistake. All invoices are paid and up to date, so this seems to be some kind of billing issue. This is the second time this has occurred since they moved to a new billing platform as few months back. We have already contacted Cpanel via support ticket and phone, but are yet to receive a response. Until this is resolved, you most likely will not be able to access cpanel/WHM. We are doing everything in our power to get this resolved ASAP..

Aug 24, 19:37:55 GMT+0
Resolved - We have heard back from the billing department and have confirmed that it is indeed a billing malfunction on their end. While they wait for certain billing staff to be available, they have issued emergency licenses for all servers for temporary reactivation. Once the appropriate staff is available, this should be resolved swiftly and we do not anticipate any further access issues. Cpanel response: "Thank you for contacting cPanel Customer Support! I hope this communication finds you well! This month there was a billing issue causing some customers who had already paid for the month of August to be suspended. I am the only one in the office today, and I have asked my team if anyone can help out. In the meantime, I am working on these concerns in the order that they come in. I do not want you to be further affected by this so, while I am in the process of working through these requests, I have activated emergency cPanel licenses on the following IP(s): ".

Hard Drive Replacement and Emergency Maintenance - Thunder

Sun, 18 Aug 2024 20:33:31 +0000

Type: Incident

Duration: 1 hour and 10 minutes

Affected Components:

Aug 18, 20:33:31 GMT+0
Identified - The server will be going down shortly due to failed hard drive. The datacenter will replace the drive and then we will rebuild the raid. After the raid is done with the resync, we will be updating the firmware on each drive. This will require 2 reboots. So over the next few hours, the server will be rebooted several times and there will be several outages..

Aug 18, 21:43:37 GMT+0
Resolved - The drive and the maintenance has been completed..

U.S. Servers - Datacenter Power Upgrades

Thu, 4 Jul 2024 23:55:00 +0000

Type: Maintenance

Duration: 1 hour and 6 minutes

Affected Components: ,

Jul 4, 23:55:00 GMT+0
Identified - We have been informed by our U.S. datacenter, that on Thursday, 7-4-2024 at 7pm, they will be performing an emergency power component replacement and power upgrade. The downtime is anticipated to be 30 minutes, per the datacenter, but we are going to allocate 1 hour to be on the safe side. During this maintenance window, the servers will be gracefully powered off. Once the power replacement/upgrade is complete, they will then power on the servers. We will be monitoring this closely and will update if their are any delays or issues that arise. .

Jul 5, 01:01:13 GMT+0
Completed - The maintenance is now complete and the servers are back online..

Jul 4, 23:55:01 GMT+0
Identified - Maintenance is now in progress.

Investigating issue with Thunder server

Fri, 21 Jun 2024 07:04:06 +0000

Type: Incident

Duration: 1 day, 13 hours and 47 minutes

Affected Components:

Jun 21, 07:04:06 GMT+0
Investigating - We are currently investigating an issue with the server Thunder. Our engineers have been alerted and further details will be provided if necessary..

Jun 21, 07:39:50 GMT+0
Monitoring - Upon a kernel crash, the server did not come up properly. We initiated a manual reboot and now the server is back online. We will monitor closely and update if there are any further issues..

Jun 21, 07:56:45 GMT+0
Resolved - The server seems to be stable at this time. We will reopen this incident if there are any further issues..

Jun 21, 10:12:35 GMT+0
Monitoring - We are monitoring the server for random reboots to track down the cause. There may be short interruptions though out the day as we attempt to capture logs on the random reboots..

Jun 22, 19:30:12 GMT+0
Identified - There have been a few random reboots, but not quite as frequently as they were. Since this is still ongoing, the datacenter is going to replace the memory and motherboard, that we have replaced all components with new ones. The server is going offline now and is expected to be down for about 30-60 minutes. We will update when the server is back online..

Jun 22, 20:51:08 GMT+0
Resolved - The motherboard and memory have been replaced. The server is now back online. With all new hardware except for the hard drives. This should resolve the reboot issues. So we are gonna consider this closed..

Investigating issue with Thunder server

Wed, 19 Jun 2024 21:00:06 +0000

Type: Incident

Duration: 5 hours and 2 minutes

Affected Components:

Jun 19, 21:00:06 GMT+0
Investigating - We are currently investigating an issue with the server Thunder. Our engineers have been alerted and further details will be provided if necessary..

Jun 19, 21:19:24 GMT+0
Identified - We are working out an issue with repeated reboots on the server. We will update when we have more information..

Jun 19, 21:41:12 GMT+0
Identified - In talks with the datacenter, it appears that the power supply may have become faulty. They are going to take the server down and replace the power supply..

Jun 19, 22:15:30 GMT+0
Monitoring - The Power supply has been replaced and the server is back online. We are going to monitor closely for any further repeated reboots over the next couple hours..

Jun 19, 22:38:00 GMT+0
Resolved - It appears the PSU replacement did the trick and the server is no longer rebooting every few minutes. We will continue to monitor closely over the next few hours and will reopen this incident if the issue arises again..

Jun 20, 00:35:55 GMT+0
Identified - Well the server was running fine for about 1 1/2 hours with no reboots and then the issued returned. We are gonna try to swap out the cpu and hopefully this resolves the problem. If that does not work, the last resort would be to change out the chassis completely. The server will be going down in about 5 minutes to replace the cpu..

Jun 20, 01:18:55 GMT+0
Monitoring - The cpu has been replaced and we will continue to monitor closely to watch for any further reboots..

Jun 20, 02:01:50 GMT+0
Resolved - We are gonna close this incident for now. Meanwhile we will monitor the server closely and if any further issues arrive, we will once again reopen it with additional updates..

Reboot and Failed Raid Drive - Storm

Tue, 28 May 2024 21:59:11 +0000

Type: Incident

Duration: 3 hours and 8 minutes

Affected Components:

May 28, 21:59:11 GMT+0
Identified - We are going to perform an emergency reboot to check a reported failed hard drive in our raid setup. While doing this, we will most likely be doing firmware updates on the motherboard and hard drives to try and resolve these hard drives that keep getting reported as failed. We will update as we determine the course of action..

May 28, 22:26:01 GMT+0
Identified - We have checked the server in recue mode and the drive did not show. We have gotten the server back online with the single drive. So the datacenter will be replacing this drive very shortly. Once the server is back online, we will performing firmware updated so all internal server components to hope this rectifies the hard drives failing or bricking themselves before their life span..

May 28, 22:43:12 GMT+0
Identified - The server has just went down for the hard drive replacement. Please anticipate several shorter downtimes over the next couple hours as we apply these firmware updates in rescue mode. .

May 28, 23:22:45 GMT+0
Identified - The server has been back online for about 30 minutes now. We are waiting for the raid to finish rebuilding and then we will proceed with the firmware updates. .

May 29, 01:06:49 GMT+0
Resolved - The firmware upgrades have been completed and everything is back online. We found there was an issue with earlier versions of the Samsung 990 Pro's that were allowing the disk to fail much earlier than its lifespan. We applied the latest firmware that addresses that issue as well as both of the drives in the server as of now, are the latest production version of the hard drives. So with that said, we do not anticipate any further hard drive failures. .

Hurricane Down

Sun, 12 May 2024 20:07:37 +0000

Type: Incident

Duration: 33 minutes

Affected Components:

May 12, 20:07:37 GMT+0
Investigating - We are currently investigating the outage on the Hurricane server. Please stand by as we gather more information..

May 12, 20:13:59 GMT+0
Identified - The server is having an issue coming up from a reboot. We are currently working on it and will update shortly..

May 12, 20:23:24 GMT+0
Resolved - The server is back online and the grub entry issue has been fixed permanently we feel..

May 12, 20:37:25 GMT+0
Investigating - We are investigating why on reboot the Cloudlinux system did not load..

May 12, 20:41:02 GMT+0
Resolved - This incident has been resolved..

Emergency Downtime - Storm

Wed, 17 Apr 2024 23:00:00 +0000

Type: Maintenance

Duration: 11 hours and 47 minutes

Affected Components:

Apr 18, 10:46:33 GMT+0
Completed - Not sure why our last update did not post, but the hard drive replacement has been completed. We also modified boot records across all drives to fix issues with recent boot records becoming unavailable. This should lead to a more stable environment as compared to the last couple months. .

Apr 17, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 18, 00:48:56 GMT+0
Identified - The server is back online temporarily. We first had to reinstall boot records to the single drive. Now that we know it boots, we are gonna contact the datacenter to replace the failed hard drive. After they replace the drive, the server should boot normally and we will readd the drive to the raid array. Depending on how fast the datacenter can install the new drive will determine when the maintenance window will close. Until they can install the hard drive, the server will be online. The reinstall of the boot records tooo longer than expected due to the server moving the boot folder and modifying the boot entries to the folders new location, which caused havoc on regenerating the boot records due files listing the wrong folder locations. This is not something that usually happens, so it took awhile to track down. We would anticipate the datacenter will replace the drive within the next hour or 2\. .

Apr 17, 23:00:00 GMT+0
Identified - We are scheduling an emergency downtime window of 2 hours, starting at 6pm CST and lasting until 8pm CST. during this emergency downtime, we will be taking the server offline to investigate a failed/offline hard drive in our raid array and also will be doing some maintenance to resolve some of the crashes and reboot issues we have suffered in the past. While we are scheduling a 2 hour window, we believe this will be resolved within 30-40 minutes or so. We understand this is short notice, but we do not want to take the risk of the other drive failing and being forced to reinstall and restore backups. .

Apr 18, 00:17:59 GMT+0
Identified - The maintenance is still in progress, but it is taking longer than we expected. We still hope to have the server back up by the close of the maintenance window. If there will be any further delays, we will update accordingly..

Apr 17, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Investigating issue with Thunder server

Thu, 11 Apr 2024 13:14:06 +0000

Type: Incident

Duration: 6 hours and 33 minutes

Affected Components:

Apr 11, 13:14:06 GMT+0
Investigating - We are currently investigating an issue with the server Thunder. Our engineers have been alerted and further details will be provided if necessary..

Apr 11, 13:25:52 GMT+0
Investigating - We have found that the server crashed and rebooted to the bios screen. The datacenter is currently looking into this to see if it is a hardware issue..

Apr 11, 14:04:16 GMT+0
Monitoring - The server is back online, but we still need to investigate the reason for the crash and boot into bios. We will update this when we have more info..

Apr 11, 16:07:39 GMT+0
Monitoring - We found that the server for some reason just dropped all hard drives from the server. Upon doing a full server reset, the drives then showed back up. We are having the datacenter looking for a resolution so this does not happen again. Hopefully it can be a firmware update of some sort and not actually the motherboard itself. Either way, we will update as we gather more information on a resolution to this..

Apr 11, 19:46:46 GMT+0
Resolved - We are closing this incident for now. We are still working with the datacenter for a fix. This seems to be related to Almalinux and the way it handles the grub and boot partitions when the disk drives are in a raid format. We are exploring options to bypass this can copy the EFI boot partitions across all drives..

Unable to access Cpanels on all servers

Thu, 4 Apr 2024 20:19:05 +0000

Type: Incident

Duration: 3 minutes

Affected Components: , , , , ,

Apr 4, 20:22:14 GMT+0
Resolved - We just got ahold of cpanel and the licenses have been restored..

Apr 4, 20:19:05 GMT+0
Identified - We just came to notice that all servers are not able to access cpanel and WHM. After further investigation we found that all our cpanel licenses are listed as suspended. This appears to be a billing error on Cpanel's side and we are already in contact with them. They recently migrated to a new billing system and it appears to be an error as outlined in this forums post on the cpanel's website. .

Investigating issue with Lightning server

Thu, 4 Apr 2024 04:13:06 +0000

Type: Incident

Duration: 7 hours and 36 minutes

Affected Components:

Apr 4, 11:49:14 GMT+0
Resolved - This incident has been resolved..

Apr 4, 04:13:06 GMT+0
Investigating - We are currently investigating an issue with the server Lightning. Our engineers have been alerted and further details will be provided if necessary..

Apr 4, 04:31:40 GMT+0
Investigating - We have contacted the datacenter as the server is inaccessible. We update as more information comes available..

Apr 4, 05:35:48 GMT+0
Investigating - We are still investigating the issue and awaiting he datacenter to plug in a KVM console for further troubleshooting..

Apr 4, 05:39:06 GMT+0
Monitoring - The server is now back online..

PHP Error/Reboot Issue - Storm Server

Fri, 29 Mar 2024 20:50:23 +0000

Type: Incident

Duration: 3 hours and 5 minutes

Affected Components:

Mar 29, 23:19:02 GMT+0
Monitoring - It appears the last cloudlinux technician that worked on a previous boot issue, did not set the hard drive to boot into the correct partitions. The server is now back online and we will be monitoring..

Mar 29, 23:55:06 GMT+0
Resolved - We are closing this as all issue are resolved..

Mar 29, 20:50:23 GMT+0
Investigating - We are currently investigating an error where PHP versions are not being picked up from PHP Selector. We feel this might be a failed system update and an error with CageFS. So we have reached out to cloudlinux to have a look..

Mar 29, 21:02:36 GMT+0
Identified - It is related to a failed cpanel update earlier in the day. It seems the Cagefs rebuild process during the update got stuck. So Cloudlinux Support is currently working on it..

Mar 29, 21:52:34 GMT+0
Monitoring - After a reboot to try an resolve the issue, the server did not come up from reboot. We are investigating..

Mar 29, 22:37:15 GMT+0
Identified - We are still working on getting the server back online and will update when we have more information..

Investigating issue with Thunder server

Wed, 6 Mar 2024 10:39:06 +0000

Type: Incident

Duration: 2 days, 4 hours and 21 minutes

Affected Components:

Mar 7, 18:01:55 GMT+0
Investigating - Well it appears it has crashed again. We are investigating..

Mar 7, 18:24:15 GMT+0
Investigating - It looks like the server is stuck in a constant reboot. The datacenter is actively working on it and checking the hardware and running some quick hardware tests..

Mar 7, 18:41:07 GMT+0
Identified - The datacenter is moving the hard drives to completely new server setup to rule out any hardware issues. We update as more information comes in..

Mar 6, 10:39:06 GMT+0
Investigating - We are currently investigating an issue with the server Thunder. Our engineers have been alerted and further details will be provided if necessary..

Mar 6, 11:13:30 GMT+0
Identified - We found upon notification of this server being down, that the server crashed and is not botting into the kernel properly. We are working on this and will update as more information comes in..

Mar 6, 12:22:32 GMT+0
Identified - We are continuing to work on the server and track down why it is not booting into the kernel. This may take a few hours to attempt to get the boot records reinstalled and the server booted back up..

Mar 6, 14:41:44 GMT+0
Identified - Just a short update, but we have had the datacenter install a rescue USB and we will continue our troubleshooting to see why it is not properly booting into the kernel. We have also brought aboard the Cloudlinux support team and we are working together to identify and resolve the kernel issue. We have had clients reach out and ask about backups. In a worst case scenario, where we would have to reinstall the server and restore backups, we have backups within 6 hours of the outage. .

Mar 6, 17:05:23 GMT+0
Identified - At this time there is not anything new to report. We are continuing to track down the issue and bring the boot records into place. These types of issues are not quick fixes and we do expect this to be an extended outage. We are doing everything we can to repair the state of the server and avoid a full server reinstall. We post as soon as we have further details..

Mar 6, 19:53:55 GMT+0
Identified - The troubleshooting process is still ongoing. We may have tracked this down to a raid corruption on the server, but we are still not 100% sure on that. We appreciate everyone's patience. We know this is a long outage and we understand everyone's frustrations. We are facing the same frustrations, but some server issues are not always so cut and dry and can take hours and hours of troubleshooting. If at all possible we would rather take a little extra time and try to repair the server, than perform a full rebuild that is not a quick process and "can" come with it's own issues. We will continue to update as things progress..

Mar 6, 21:09:19 GMT+0
Identified - Just a short update. We have pulled in one of the top linux technicians in the country and he has now taken over operations. he is actively working on the server as we speak and hope to have good news soon..

Mar 6, 22:31:22 GMT+0
Monitoring - We are happy to state that the server is back online. We will be monitoring this closely and we are also going to have Cloudlinux examine a kernel dump that was generated on the crash, along with investigating why the one hard drive partially fell out of the raid array. We understand many customers wanted a timeline on when this server was going to be back online, but in situations like this, it is impossible to give a timeline. In the first few hours, we would have thought the server would be back up in an hour or so, but that was not the case. If we give a timeline and miss it, then there is gonna be negative feed back on that. Even further, if we ended up having to reinstall, it could have taken another day. In any outage we would love to give a timeline, but it not always feasible to do that, such in this case. We thank everything for their patience! If anything further pops up, we will update accordingly..

Mar 7, 19:22:42 GMT+0
Monitoring - The server is now back online. It looks to be a hardware issue that is similar we faced with another server in this datacenter. Here is the response from the datacenter: > you are on a new 7950X3D CPU and motherboard. > I'm assuming it is the same issue we've seen with a few other 7950X servers. They would randomly reboot over and over out of no where. We are guessing we got sent a bad batch from the manufacture as we have hundreds of others running fine. We've seen online some others have experienced the same thing with some 7950x's. We started buying 7950X3D, same thing, a little more expensive, but are faster and use less power, seem to be more stable. We strongly believe this should resolve the downtime issue, but we will continue to monitor this closely..

Mar 7, 16:04:43 GMT+0
Resolved - We have now concluded our investigation on the outage of the Thunder server. Looking through the logs and the action taken on the server to bring it back online, we have come to the following conclusions and timeline of events. 1. The server crashed for no apparent reason. This has been an ongoing issue across our whole server fleet. We have setup our servers with kdump that will produce a kernel dump that can be used for analysis. We have submitted the kernel dumps to Cloudlinux from several servers and they stated it will take a few weeks to analyze these. Kernel dumps save the contents of the system memory for analysis and average around 4 GB in size. They require specially trained technicians to read the output of the kernel dump and it is not something we could do. 2. Once we found the server to be down, we discovered the server was not booting back into the kernel and instead was landing on a Grub screen. At that point we began our troubleshooting process and also brought the support staff from Cloudlinux to investigate. We also contacted the datacenter and requested a USB be installed with a rescue system installed. Discovering the cause of a no boot situation is very lengthy process and requires setting up a rescue system to boot into and mounting the original operating system of the server. This all takes extended amounts of time, along with coordinating with the the various staff and CL support team. 3. After server hours of investigation, we found that upon reboot, one the NVMe hard drives that makes up the Raid-1 array on the server, had fallen out of the raid array. To simplify this, a Raid-1 array basically mirrors all the data across 2 drives. If one fails or falls out of the array, the other disk still contains the data on the drive. In this case the one hard drive fell out of one of the arrays, that contains the boot records. With new server technology, there are different boot records stored on each drive and they are not mirrored like the older systems. So when the one drive fell out of the array, it contained part of the boot records for EFI that had to be reinstalled. 4. At this point we went to reinstall the boot grub config and attempted to reboot, but still it went back to booting into the grub screen. At this point we had the staff from Cloudlinux attempt this as well, and still the reboot failed. 5. This is when we made the decision to bring in outside help. We have one of the best Linux technicians in the country on standby, that we have developed a working relationship over the fast few years. While he does come with a premium price per hour, we felt it had reached a point that we had exhausted all our attempts to bring the server back online. At this point we gave him a quick rundown of the state of the server and began his work. After a few attempts to reinstall the boot records himself, he still faced the same issues of a non-booting server. He continued to work the server for another 2 hours and we finally able to get the boot records to stick and the server booted correctly. To verify even further, we performed a second manual reboot and the server booted in the operating system as it should. At this point we will consider the server running stable, but we are still awaiting the analysis of the various kernel dumps that have been provided. There were some changes made since the random reboots/crashes started occurring and this is the first one on 3 weeks, compared to where they were happing several times a week, across several server. So we feel this might have been an isolated case with the hard drive dropping out of the raid array, leading to the kernel panic and crash. We will continue to work with the Cloudlinux team to find the cause of the crashes..

Mar 8, 14:59:51 GMT+0
Resolved - This incident has been resolved..

Investigating issue with Lightning server

Tue, 13 Feb 2024 17:49:07 +0000

Type: Incident

Duration: 4 days, 10 hours and 13 minutes

Affected Components:

Feb 13, 17:49:07 GMT+0
Investigating - We are aware of an issue with the Lightning server. It did several reboots right after another and then has not come back up. We are investigating this and will update when we have more information..

Feb 13, 18:04:35 GMT+0
Investigating - We are still not sure of the issue, but the server is not coming up after a hardware reset. So we have contacted the datacenter to investigate further. We suspect it may be a failed hardware component, but we will wait for them to confirm..

Feb 13, 19:06:53 GMT+0
Investigating - The datacenter has reported that they are investigating the issue. We will update when we get more details..

Feb 13, 19:30:10 GMT+0
Monitoring - The Lightning server is now back online. The datacenter did not detect any hardware issues, so we are going to dig through the server logs and see what is causing this. We will be monitoring the server very closely for any further disruptions. .

Feb 13, 21:04:43 GMT+0
Monitoring - We have brought in Cloudlinux technicians to take a further look at this. There has been similar issues on other servers since installing Cloudlinux and we are doing everything possible to get to the root of the problem..

Feb 14, 01:51:06 GMT+0
Resolved - We are still working with Cloudlinux to troubleshoot these issues, but for now we are going to close this post to unclutter our client portal. We will still post updates as more information comes in or if there are any further issues..

Feb 17, 18:51:04 GMT+0
Identified - The Lightning server seems to have crashed again. We have the datacenter looking into it and investigating the cause of the crash. Updates will follow when more information comes in..

Feb 17, 19:13:50 GMT+0
Identified - We are in talks will replacing the hardware on the server and moving the hard drives to the new server. Once we come to a conclusion to move forward, we will update this incident..

Feb 17, 20:01:45 GMT+0
Identified - We are going to proceed with the hardware replacement to hopefully prevent further outages. You can expect around 2 hours downtime for this to take place. We will update further once the server is back online..

Feb 17, 21:43:01 GMT+0
Identified - We are still waiting on confirmation of the hardware swap completion. We will update as soon as we hear anything..

Feb 17, 22:18:55 GMT+0
Identified - We got notice that the drives were migrated too the new hardware, but now we are facing issues getting the server to boot the os. We are working on this along with the datacenter to get the server back online. In a absolute worst case scenario, we do have up to date backups if it would come to that, but we are not at that stage yet..

Feb 17, 23:46:47 GMT+0
Identified - We have discovered that after the crash of the server, one of the partitions got corrupted. We are working to repair the partition and then go from there. Most likely if we are unable to repair the partition, we will have to do a full reinstall and restore backups. We anticipate that neither process is gonna be quick and you should prepare that this will be a lengthy outage, but do not panic as we have full backups of all accounts from today and they will be restored if we need to reinstall the server..

Feb 18, 04:01:58 GMT+0
Resolved - We are extremely happy to announce that the server is back online in full capacity. With the collaboration of our staff and the Cloudlinux staff, we were able to recover the boot partition and bring the server back online. We found that this corruption of the boot partition happened before the hardware migration took affect. So we are hoping the new hardware will resolve these issue with the random reboots on this server. We also have the Cloudlinux staff reviewing a kernel dump from another server that has a similar issue with the random reboots. Once they analyze the kdump from that server, we should get more information on what is causing the random reboots. Although from what we seen on this server, it might be unrelated and most likely was a failing hardware component. We thank everyone for their patience during all this. This was certainly one of our worst and challenging outages since we have been in business. We really felt it was gonna lead to a server reinstall, but our team and the cloudlinux team really pulled together to pull off a miracle to rebuild and restore the boot partition..

Emergency Hardware Replacement

Mon, 29 Jan 2024 02:00:00 +0000

Type: Maintenance

Duration: 34 minutes

Affected Components:

Jan 29, 02:00:00 GMT+0
Identified - We are reaching out to inform you about an urgent maintenance operation that needs to be conducted on the server hosting your account. Over the past two weeks, we've encountered sporadic reboots on this server, occurring every few days. Initially, we suspected kernel panics as the root cause and took measures to enable extended logging to capture kernel dump logs during the subsequent reboots. However, upon analysis, we found no output from the kernel and no additional logs shedding light on the issue. Given the absence of software-related explanations, we have engaged with our datacenter, and they suspect a faulty CPU may be the culprit. To address this, we will need to replace the CPU, a process that will require the server to be offline for an **estimated period of 30-60 minutes**. Additionally, we've requested the datacenter to swap out the RAM sticks and the power supply concurrently. While the CPU is the primary suspect, we are taking a comprehensive approach to ensure that other hardware components are not contributing to the problem. The replacement of these additional components will only marginally extend the downtime, adding approximately 5-10 minutes. The scheduled maintenance to replace the CPU and other hardware components is set for** 8 PM CST tonight**. Typically, we like to provide more advanced notice for such operations, but given the urgency to mitigate further disruptions caused by the random reboots, we believe swift action is warranted. We sincerely appreciate your understanding and apologize for any inconvenience this may cause. Should you have any concerns or questions, please don't hesitate to reach out to our support team. Thank you for your cooperation. Warm regards, MonsterMegs Team.

Jan 29, 02:33:55 GMT+0
Completed - The hardware components have been replaced and the server is now back online. We will continue to monitor the server closely for any further reboots, but hopefully the replacement of these components will be the solution. .

Jan 29, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Kernel Panics - Thunder

Sat, 20 Jan 2024 12:50:19 +0000

Type: Incident

Duration: 6 days, 6 hours and 58 minutes

Affected Components:

Jan 20, 12:50:19 GMT+0
Investigating - We need to perform an emergency reboot on the server. We have found an issue where the server has been rebooting on its own and we are working with the Cloudlinux team to track down the cause..

Jan 20, 13:54:40 GMT+0
Investigating - The reboot was completed at 6:53am CST. The cloudlinux team are still investigating the issue and we will update as more information comes in. Please be aware that further reboots may be required, but not guaranteed. They also may need to restart services as they troubleshoot, but have been informed to keep any impacts to a minimum..

Jan 20, 14:45:33 GMT+0
Monitoring - We are going to require an additional server reboot to enable additional logging. This is going to happen in the next 5 minutes and should take about 3-5 minutes..

Jan 20, 15:21:16 GMT+0
Resolved - The reboot has been completed and the kdump extended logging has been enabled. It appears there have been several kernel panics that caused these random reboots throughout the week. The addition of kdump will generate a dump file on any kernel panics that Cloudlinux will review to determine the cause of the kernel panics. This know this is directly related to Cloudlinux due to the fact that we had this server running on cpanel for over a month before the migrations, with no reboots or issues of this kind. Since incorporating Cloudlinux into the server, the issue started to present itself a few days later after the migrations were complete. Over the next several days we will be monitoring the server closely for any of these kernel panics and will work closely with the Cloudlinux team to track down the issue and get this resolved ASAP. If any more information comes to light, we will update this post accordingly..

Post Hosting Migration Follow Up - ALL Servers

Sun, 14 Jan 2024 14:42:56 +0000

Type: Incident

Affected Components: , , ,

Jan 14, 14:42:56 GMT+0
Resolved - We are happy to announce that the server migrations of all servers are now complete. We now advise any clients using dns through their domain provider (e.g Godaddy, Enom, etc) or a 3rd party dns providers, such as Cloudflare or DNSMadeEasy, to update your dns zones as follows: - Storm Server: 184.95.50.250 -> 38.46.220.132 - Thunder Server: 108.170.49.202 -> 38.46.220.134 - Lightning Server: 185.56.137.138 -> 46.4.10.227 - Hurricane Server: 185.56.137.130 -> 46.4.10.218 Domains using our nameservers that include ns1.megpanel.com, ns2.megpanel.com...etc, there is no need to make any dns or ip changes. If you have a dedicated ip assigned to your account, you can find this ip listed your cpanel account. It is listed in the right had column under the "General Information" section. We have also added Mailbaby email delivery to the Lightning and Storm servers. This is a email service that we have been using on our Semi-Dedicated hosting plans for the last year and have had great success with email deliverability. Those using a 3rd party dns service will need to add the following to their SPF txt entry. `+include:relay.mailbaby.net` Over the next couple days we will be resolving any lingering issues and ask everyone to be patient as we work through these issues..

Hosting Migrations - US Servers

Sun, 14 Jan 2024 02:00:00 +0000

Type: Maintenance

Duration: 6 days, 4 hours and 28 minutes

Affected Components: ,

Jan 14, 02:00:00 GMT+0
Identified - As mentioned a couple of weeks ago, we will be migrating your server to new updated server and a new datacenter location in Salt Lake City, Utah (Fiberstate). In this email we will address a few details of the server migration process, along with a time line. The time line will be a rough estimate as it is hard to anticipate transfer speeds, but if anything we believe this will go faster than anticipated. We are very excited for this move and we feel our customers are going to love the changes. The server speeds on these new Ryzen 9 7950x servers are nothing short of amazing. These servers are outfitted with top of line components and really shows in the work we have done over the last month to get these servers ready for production. This **Saturday (January 13th, 2024) at 8:00pm CST** we will begin the migration process. We anticipate this transfer to take 4-6 hours, but like stated earlier, we expect it finish quite a bit faster. During this migration there will be no downtime, although there may be slow downs due to the strain on the servers during the migration. We recommend that if possible, to avoid any changes to websites, cpanel settings, etc during this time. These changes could get lost during the transfer process. Once the migration is complete, we have some small back-end tasks to complete before we can consider the migration to complete. Over the following 72 hours we will be tweaking settings and taking care of any small issues that may be detected. So you may notice some small slowdowns or very short periods of failed connections as we restart services. We anticipate the impact will be little to none. With all migrations, the ips that serve your website will be changing. If you are using our nameservers as standard, such as ns1.megpanel.com, ns2.megpanel.com, etc, there will nothing required on your end to point to the new server. As your account migrates to the new server, the new ip will be automatically updated in the dns cluster. Now, if you are using a 3rd party dns provider, such as Cloudflare, you will need to be aware that once the migrations are complete, you will need to update your dns zones directly at the dns provider with the new ip address. The ip changes are as follows: **Storm Server:** 184.95.50.250 -> 38.46.220.132 **Thunder Server:** 108.170.49.202 -> 38.46.220.134 If your account has a dedicated ip and using a 3rd party dns provider, you will need to use that dedicated ip in place of the ips listed above. You will be able to find your assigned dedicated ip in your cpanel interface after the migration. You can find the dedicated ip listed under the "General Information" in the right column of cpanel. We will be posting updates on our Server Status page throughout the entire process. So if you have not yet subscribed, we urge you do so as all updates will be there. Our Server Status page is also directly integrated into our billing portal, so all details will also be found on customer portal homepage. ****All migration updates and post-migration details will be posted on our Server Status page and in our client portal.**** We ask that everyone is patient during the migration process and the days following. Our staff will be incredibly busy with post migration tasks, so we ask that if possible, to hold off on any non-urgent tickets. We will send one final email the day before the migration that will serve as a final reminder..

Jan 14, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 14, 03:35:29 GMT+0
Identified - The server migration of the Thunder server is now complete. The Storm server is at about 74% and should complete within the next 2 hours or so. We will provide another update when the Storm server migration is complete. Once the migration itself is complete, we will have a couple hours of backend task to complete, so please be patient as we update all our backend components..

Jan 14, 04:28:27 GMT+0
Completed - The server migrations are now complete. Over the next few hours we will be resolving any small issues, performing follow up tasks, and doing a post migration audit. If you are using a 3rd party dns provider such as Cloudflare or DNSMadeEASY, you may now update 3rd party dns zone with the new ips: **Storm Server:** 184.95.50.250 -> 38.46.220.132 **Thunder Server:** 108.170.49.202 -> 38.46.220.134 If you have a dedicated ip assigned to your account, you can find this ip listed your cpanel account. It is listed in the right had column under the "General Information" section. We have also added Mailbaby email delivery to the Storm server. This is a email service that we have been using on our Semi-Dedicated hosting plans for the last year and have had great success with email deliverability. Those using a 3rd party dns service will need to add the following to their SPF txt entry. `+include:relay.mailbaby.net` Over the next couple days we will be resolving any lingering issues and ask everyone to be patient as we work through these issues. .

Hosting Migrations - EU Servers

Sat, 6 Jan 2024 22:00:00 +0000

Type: Maintenance

Duration: 15 days

Affected Components: ,

Jan 6, 22:00:00 GMT+0
Identified - As mentioned a couple of weeks ago, we will be migrating your server to new updated servers and a new datacenter location in German (Hetzner). In this email we will address a few details of the server migration process, along with a time line. The time line will be a rough estimate as it is hard to anticipate transfer speeds, but if anything we believe this will go faster than anticipated. We are very excited for this move and we feel our customers are going to love the changes. The server speeds on these new Ryzen 9 7950x servers are nothing short of amazing. These servers are outfitted with top of line components and really shows in the work we have done over the last month to get these servers ready for production. This Saturday (January 6th, 2024) at 6:00pm CST we will begin the migration process. We anticipate this transfer to take 4-6 hours, but like stated earlier, we expect it finish quite a bit faster. During this migration there will be no downtime, although there may be slow downs due to the strain on the servers during the migration. We recommend that if possible, to avoid any changes to websites, cpanel settings, etc during this time. These changes could get lost during the transfer process. Once the migration is complete, we have some small back-end tasks to complete before we can consider the migration to complete. At that time we will send a final email with post migration information and instructions if necessary. Over the following 72 hours we will be tweaking settings and taking care of any small issues that may be detected. So you may notice some small slowdowns or very short periods of failed connections as we restart services. We anticipate the impact will little to none. With all migrations, the ips that serve your website will be changing. If you are using our nameservers as standard, such as ns1.megpanel.com, ns2.megpanel.com, etc, there will nothing required on your end to point to the new server. As your account migrates to the new server, the new ip will be automatically updated in the dns cluster. Now, if you are using a 3rd party dns provider, such as Cloudflare, you will need to be aware that once the migrations are complete, you will need to update your dns zones directly at the dns provider with the new ip address. The ip changes are as follows: - **Lightning Server:** 185.56.137.138 -> 46.4.10.227- - **Hurricane Server:** 185.56.137.130 -> 46.4.10.218 In our final post-migration email, we will announce that it is time for customers using Cloudflare or another 3rd party dns provider, to update the ip addresses. If you have a dedicated ip assigned to your account, you will find the new dedicated ip listed in your cpanel account within the system details box on the right had side. Again if you are using our dns, no changes will be necessary, even if you have a dedicated ip. We ask that everyone is patient during the migration process and the days following. Our staff will be incredibly busy with post migration tasks, so we ask that if possible, to hold off on any non-urgent tickets. .

Jan 6, 22:00:00 GMT+0
Identified - Server migrations have now begun. Updates will be posted as the migrations progress..

Jan 7, 00:56:21 GMT+0
Identified - The server migrations are now complete. Over the next few hours we will be resolving any small issues, performing follow up tasks, and doing a post migration audit. We will leave this notice up for a few days, but do not anticipate any further updates, unless something urgent arises. .

Jan 6, 23:10:47 GMT+0
Identified - The migrations are near complete, but we are having an issue with some databases not being migrated to several errors. We are working on this and once we find the resolution, we will re-import these accounts..

Jan 7, 00:01:11 GMT+0
Identified - We have found the cause of databases larger than 1gb failing to transfer. We have already re-migrated all accounts on the Hurricane server and now we have 4 accounts on the Lightning server that will be migrated. Aside from this issue, the migrations have been going smooth and should be completed within the hour..

Jan 7, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Billing System Upgrades

Sun, 3 Dec 2023 01:00:00 +0000

Type: Maintenance

Duration: 1 hour and 4 minutes

Affected Components: ,

Dec 3, 01:00:00 GMT+0
Identified - We will performing update to our billing system and it may be unavailable to customers for a period of 2 hours. If you have any emergency issues during this time, we ask that you email support@monstermegs.com directly..

Dec 3, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 3, 02:03:56 GMT+0
Completed - Maintenance has completed successfully..

Billing System Maintenance

Wed, 15 Mar 2023 01:00:00 +0000

Type: Maintenance

Duration: 1 hour and 21 minutes

Affected Components: ,

Mar 15, 01:00:00 GMT+0
Identified - In this maintenance window we will be migrating our website and billing system to a server. During this migration, the billing system will be unavailable to customers. We have issued a 2 hours window to complete this transfer, but we anticipate this to completed rather quickly. If any issues arise that will take us outside of this window. We will update our https://status.monstermegs.com page accordingly..

Mar 15, 01:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 15, 02:21:12 GMT+0
Completed - Maintenance has completed successfully. Our website and client portal is back online..

Hosting Server Reboots

Sun, 5 Mar 2023 02:00:00 +0000

Type: Maintenance

Duration: 11 hours and 59 minutes

Affected Components: , , ,

Mar 5, 02:12:12 GMT+0
Completed - Maintenance has completed successfully. All servers are now back online..

Mar 4, 14:13:23 GMT+0
Identified - This is just a reminder that the reboots will begin a 8:00pm tonight. We will post another update when we begin the reboots..

Mar 5, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 5, 02:00:00 GMT+0
Identified - We are planning for a scheduled maintenance period for Saturday, March 4th at 8:00pm. At this time we will be rebooting all web hosting servers to load in updated kernels. Several of our servers have been up for over 2 years without a reboot and while we do use KernelCare to apply kernel level security updates without reboots, it is time to load newer kernels to all hosting servers. We expect the downtime for each server to be around 10-15 minutes. If we run into any further issues, we will update this post with further details..