MonsterMegs - Notice history

We are extremely happy to announce that the server is back online in full capacity. With the collaboration of our staff and the Cloudlinux staff, we were able to recover the boot partition and bring the server back online.

We found that this corruption of the boot partition happened before the hardware migration took affect. So we are hoping the new hardware will resolve these issue with the random reboots on this server. We also have the Cloudlinux staff reviewing a kernel dump from another server that has a similar issue with the random reboots. Once they analyze the kdump from that server, we should get more information on what is causing the random reboots. Although from what we seen on this server, it might be unrelated and most likely was a failing hardware component.

We thank everyone for their patience during all this. This was certainly one of our worst and challenging outages since we have been in business. We really felt it was gonna lead to a server reinstall, but our team and the cloudlinux team really pulled together to pull off a miracle to rebuild and restore the boot partition.

We have discovered that after the crash of the server, one of the partitions got corrupted. We are working to repair the partition and then go from there. Most likely if we are unable to repair the partition, we will have to do a full reinstall and restore backups.

We anticipate that neither process is gonna be quick and you should prepare that this will be a lengthy outage, but do not panic as we have full backups of all accounts from today and they will be restored if we need to reinstall the server.

We got notice that the drives were migrated too the new hardware, but now we are facing issues getting the server to boot the os. We are working on this along with the datacenter to get the server back online.

In a absolute worst case scenario, we do have up to date backups if it would come to that, but we are not at that stage yet.

We are still waiting on confirmation of the hardware swap completion. We will update as soon as we hear anything.

We are going to proceed with the hardware replacement to hopefully prevent further outages. You can expect around 2 hours downtime for this to take place. We will update further once the server is back online.

We are in talks will replacing the hardware on the server and moving the hard drives to the new server. Once we come to a conclusion to move forward, we will update this incident.

The Lightning server seems to have crashed again. We have the datacenter looking into it and investigating the cause of the crash. Updates will follow when more information comes in.

We are still working with Cloudlinux to troubleshoot these issues, but for now we are going to close this post to unclutter our client portal. We will still post updates as more information comes in or if there are any further issues.

We have brought in Cloudlinux technicians to take a further look at this. There has been similar issues on other servers since installing Cloudlinux and we are doing everything possible to get to the root of the problem.

The Lightning server is now back online. The datacenter did not detect any hardware issues, so we are going to dig through the server logs and see what is causing this. We will be monitoring the server very closely for any further disruptions.

The datacenter has reported that they are investigating the issue. We will update when we get more details.

We are still not sure of the issue, but the server is not coming up after a hardware reset. So we have contacted the datacenter to investigate further. We suspect it may be a failed hardware component, but we will wait for them to confirm.

The reboot has been completed and the kdump extended logging has been enabled. It appears there have been several kernel panics that caused these random reboots throughout the week. The addition of kdump will generate a dump file on any kernel panics that Cloudlinux will review to determine the cause of the kernel panics.

This know this is directly related to Cloudlinux due to the fact that we had this server running on cpanel for over a month before the migrations, with no reboots or issues of this kind. Since incorporating Cloudlinux into the server, the issue started to present itself a few days later after the migrations were complete.

Over the next several days we will be monitoring the server closely for any of these kernel panics and will work closely with the Cloudlinux team to track down the issue and get this resolved ASAP.

If any more information comes to light, we will update this post accordingly.

We are going to require an additional server reboot to enable additional logging. This is going to happen in the next 5 minutes and should take about 3-5 minutes.

The reboot was completed at 6:53am CST.

The cloudlinux team are still investigating the issue and we will update as more information comes in. Please be aware that further reboots may be required, but not guaranteed. They also may need to restart services as they troubleshoot, but have been informed to keep any impacts to a minimum.

We have come up with a workaround for the time being. We have come up with a reverse proxy that masks this server as the Enom server and the proxies straight to the Enom ip address.

This workaround will allow customers to access their Enom domains like normal until Enom updates their OpenSSL version.