Fun with Remote Servers

  • user warning: Table 'cache' is marked as crashed and should be repaired query: SELECT data, created, headers, expire FROM cache WHERE cid = 'filter:1:f3e0be56e5c5f0d93995fabe768ae1f6' in /home/dowdle/public_html/montanalinux/includes/database.mysql.inc on line 121.
  • user warning: Table 'cache' is marked as crashed and should be repaired query: UPDATE cache SET data = '<p><span class=\"inline right\"><a href=\"/\" onclick=\"launch_popup(117, 640, 480); return false;\" target=\"_blank\"><img src=\"http://www.montanalinux.org/files/images/Picture 004.thumbnail.jpg\" alt=\"1U Rack Mount\" title=\"1U Rack Mount\" class=\"image thumbnail\" width=\"100\" height=\"75\" /></a><span class=\"caption\" style=\"width: 98px;\">1U Rack Mount</span></span>You may or may not have noticed but this site was down some since late Sunday evening until right before I wrote this. I\'m not exacly sure what happened but I have some sense of it so I thought I\'d jot it down here. I also cover the hardware that runs this system as well as a little about the hosting company we are using. If you are interested, read the rest of the story.</p>\n<p><b>The Physical Server</b></p>\n<p>The physical server that runs this site is actually a 1.8GHz dual-core AMD with 2GB of RAM and three 250GB hard drives (a software RAID 1 with a spare). Two long time Atari buddies of mine (Joe in Connecticut and Rob in Massachusetts) and I went in and bought the parts to build the server with. I only chipped in $200 and Rob and Joe footed the rest of the bill. All of the parts were ordered by Rob and shipped to him... and he put the system together. Rob got a basic Linux install going and got the system on the Internet. I ssh\'ed in. and did a remote, medialess install of <a href=\"http://www.centos.org/\" target=\"_centsite\">CentOS</a> 4.3 with the GUI installer piped to me via VNC. Rebooted, did all of the updates... and then installed <a href=\"http://www.openvz.org/\" target=\"_ovzsite\">OpenVZ</a>. Shortly after that CentOS 4.4 came out and I upgraded it along with all of the VPSes. Then Rob shipped it to the hosting company.</p>\n<p><b>The Hosting Company</b></p>\n<p><span class=\"inline left\"><a href=\"/\" onclick=\"launch_popup(118, 640, 480); return false;\" target=\"_blank\"><img src=\"http://www.montanalinux.org/files/images/Picture 002.thumbnail.jpg\" alt=\"CPU\" title=\"CPU\" class=\"image thumbnail\" width=\"100\" height=\"75\" /></a><span class=\"caption\" style=\"width: 98px;\">CPU</span></span>The hosting company that provides the physical server with its connection to the Internet is <a href=\"http://www.colopronto.com/\" target=\"_cpsite\">ColoPronto</a> which is a division of <a href=\"http://www.serverpronto.com/\" target=\"_spsite\">ServerPronto</a>... which I believe is owned by <a href=\"http://www.infolink.com/\" target=\"_ilsite\">InfoLink</a>. We actually have a small block of IP addresses and are currently running the Host OS and seven VPSes each with their own IP address.</p>\n<p>I believe the physical complex where the server is located is in Florida somewhere. While I wish I could have the server within 10 miles of where I live so I could actually have physical access to it... since I don\'t pay for the connection... which would certainly be several times what Rob is paying... I can\'t really complain. I believe with the high speed connection and the extra IP addresses, the bill is about $20.93 a month... which is barely higher than dialup here in Montana.</p>\n<p><b>The Problem Begins</b></p>\n<p><span class=\"inline right\"><a href=\"/\" onclick=\"launch_popup(113, 640, 480); return false;\" target=\"_blank\"><img src=\"http://www.montanalinux.org/files/images/Picture 008.thumbnail.jpg\" alt=\"Motherboard\" title=\"Motherboard\" class=\"image thumbnail\" width=\"100\" height=\"75\" /></a><span class=\"caption\" style=\"width: 98px;\">Motherboard</span></span>The system and connection have been fairly stable until about 10 minutes before New Years Day (Sunday night). I noticed the server was down (Warren sent me an IM message about it too) and did a traceroute to it. The traceroute timed out at the hop just above the machine... so I submitted a trouble ticket via the hosting provider\'s online customer support site. About 7:30 AM (Monday) I noticed the server was back up. Yeah! I checked the trouble ticket system and didn\'t see a response.</p>\n<p>Then on Monday night I noticed the system had gotten rebooted... and a trouble ticket response. They said they had rebooted the system (even though it had come back up several hours before) and that during the boot Kudzu had come up with a screen saying it had detected a new network card. They took it upon themselves to configure this new NIC and gave it the IP address assigned to the Host OS.</p>\n<p>I could still access the server but something was weird. I did an <code>ifconfig</code> and it showed two network interface cards (eth0 and eth1) and both of them were configured to have the same IP address. ARGH!. Rob thought he had disabled the second NIC in the BIOS of the machine... but perhaps that got reset somehow... OR perhaps a newer Linux kernel could see the card despite it being turned off in the BIOS. Whatever the case, the hosting provider tech support person ended up with both interfaces configured although only one had a network cable plugged into it.</p>\n<p>It is not a good thing when two NICs have the same address... and we started running into connectivity issues. I submitted another trouble ticket explaining that we did NOT want eth1 setup... and to check what port actually had the network cable plugged in. Seems the technician who had configured eth1 had also moved the network cable too. </p>\n<p>The VPSes were trying to go through eth0 so they had lost connectivity. Since I could get to the Host OS still, I manually restarted each VPS (<code>vzctl restart VPID</code>) and they started working again as that caused them to start routing through eth1.</p>\n<p><b>The Fix?</b></p>\n<p><span class=\"inline right\"><a href=\"/\" onclick=\"launch_popup(115, 640, 480); return false;\" target=\"_blank\"><img src=\"http://www.montanalinux.org/files/images/Picture 006.thumbnail.jpg\" alt=\"Hard drive\" title=\"Hard drive\" class=\"image thumbnail\" width=\"100\" height=\"75\" /></a><span class=\"caption\" style=\"width: 98px;\">Hard drive</span></span>I submitted yet another response to the trouble ticket as follows:<br />\n<blockquote><i> Please review the following steps and see if they make sense to you. If they do, please follow them and report any success. :)</p>\n<ol>\n<li>Login to the console as root</li>\n<li>Move the network cable from current port (eth1) to other port (eth0)</li>\n<li>restart networking<br />\n&nbsp;&nbsp;&nbsp;service network restart</li>\n<li>Manually take down eth1<br />\n&nbsp;&nbsp;&nbsp;ifconfig eth1 down</li>\n<li>Remove the config for eth1<br />\n&nbsp;&nbsp;&nbsp;rm /etc/sysconfig/network-scripts/ifcfg-eth1</li>\n<li>restart networking again<br />\n&nbsp;&nbsp;&nbsp;service network restart</li>\n<li>Verify that eth0 is configured and up and that eth1 does not show up<br />\n&nbsp;&nbsp;&nbsp;ifconfig</li>\n<li>Verify connectivity with the outside world<br />\n&nbsp;&nbsp;ping something outside</li>\n</ol>\n<p></i></p></blockquote>\n<p><b>Working for Good?</b></p>\n<p>I got a response back from them that said they had followed my instructions and that it appeared to be working again.</p>\n<p>I ssh\'ed in again, saw that indeed eth0 was setup and working, and that eth1 was no longer configured. I had to restart the VPSes again so they would switch from eth1 back to eth0... but I was expecting that. I guess I could reboot the machine and make sure it comes up like expected, but I\'m just not in the mood for it quite yet.</p>\n<p><b>Let\'s hope the system stays up for a long time to come... knock on wood.</b></p>\n<br class=\"clear\" />', created = 1411091435, expire = 1411177835, headers = '' WHERE cid = 'filter:1:f3e0be56e5c5f0d93995fabe768ae1f6' in /home/dowdle/public_html/montanalinux/includes/database.mysql.inc on line 121.
  • user warning: Table 'cache' is marked as crashed and should be repaired query: SELECT data, created, headers, expire FROM cache WHERE cid = 'filter:1:481f46b7c227328a9938b55e3d18bc61' in /home/dowdle/public_html/montanalinux/includes/database.mysql.inc on line 121.
  • user warning: Table 'cache' is marked as crashed and should be repaired query: UPDATE cache SET data = '<p>I have only had a few down times with them... and in almost every case it was something going on with my server rather than their hosting. Those times I have had to file a trouble ticket they have been reasonably responsive and gotten my problem resolved. </p>\n<p>I do maintain my own backups and don\'t really rely on them for much more than an occasional reboot... but they did fix a broken boot loader on a software RAID system for me once. They also rebooted someone else\'s server by mistake when I asked them for a reboot... but other than that, it is more than reasonable service for the price.</p>\n<p>I\'d really like to have a second machine with them so I could do faster backups and avoid downtime as the result of hardware failures.</p>\n<br class=\"clear\" />', created = 1411091435, expire = 1411177835, headers = '' WHERE cid = 'filter:1:481f46b7c227328a9938b55e3d18bc61' in /home/dowdle/public_html/montanalinux/includes/database.mysql.inc on line 121.
  • user warning: Table 'cache' is marked as crashed and should be repaired query: SELECT data, created, headers, expire FROM cache WHERE cid = 'filter:1:5afda500d3031c0b75dc56f43524b552' in /home/dowdle/public_html/montanalinux/includes/database.mysql.inc on line 121.
  • user warning: Table 'cache' is marked as crashed and should be repaired query: UPDATE cache SET data = '<p>Interesting, I\'m deciding if I want to use Colopronto. I don\'t want to send my server to a place I\'m unsure about because it was expensive. Would you say you are happy with the service they are giving you?</p>\n<br class=\"clear\" />', created = 1411091435, expire = 1411177835, headers = '' WHERE cid = 'filter:1:5afda500d3031c0b75dc56f43524b552' in /home/dowdle/public_html/montanalinux/includes/database.mysql.inc on line 121.

1U Rack Mount1U Rack MountYou may or may not have noticed but this site was down some since late Sunday evening until right before I wrote this. I'm not exacly sure what happened but I have some sense of it so I thought I'd jot it down here. I also cover the hardware that runs this system as well as a little about the hosting company we are using. If you are interested, read the rest of the story.

The Physical Server

The physical server that runs this site is actually a 1.8GHz dual-core AMD with 2GB of RAM and three 250GB hard drives (a software RAID 1 with a spare). Two long time Atari buddies of mine (Joe in Connecticut and Rob in Massachusetts) and I went in and bought the parts to build the server with. I only chipped in $200 and Rob and Joe footed the rest of the bill. All of the parts were ordered by Rob and shipped to him... and he put the system together. Rob got a basic Linux install going and got the system on the Internet. I ssh'ed in. and did a remote, medialess install of CentOS 4.3 with the GUI installer piped to me via VNC. Rebooted, did all of the updates... and then installed OpenVZ. Shortly after that CentOS 4.4 came out and I upgraded it along with all of the VPSes. Then Rob shipped it to the hosting company.

The Hosting Company

CPUCPUThe hosting company that provides the physical server with its connection to the Internet is ColoPronto which is a division of ServerPronto... which I believe is owned by InfoLink. We actually have a small block of IP addresses and are currently running the Host OS and seven VPSes each with their own IP address.

I believe the physical complex where the server is located is in Florida somewhere. While I wish I could have the server within 10 miles of where I live so I could actually have physical access to it... since I don't pay for the connection... which would certainly be several times what Rob is paying... I can't really complain. I believe with the high speed connection and the extra IP addresses, the bill is about $20.93 a month... which is barely higher than dialup here in Montana.

The Problem Begins

MotherboardMotherboardThe system and connection have been fairly stable until about 10 minutes before New Years Day (Sunday night). I noticed the server was down (Warren sent me an IM message about it too) and did a traceroute to it. The traceroute timed out at the hop just above the machine... so I submitted a trouble ticket via the hosting provider's online customer support site. About 7:30 AM (Monday) I noticed the server was back up. Yeah! I checked the trouble ticket system and didn't see a response.

Then on Monday night I noticed the system had gotten rebooted... and a trouble ticket response. They said they had rebooted the system (even though it had come back up several hours before) and that during the boot Kudzu had come up with a screen saying it had detected a new network card. They took it upon themselves to configure this new NIC and gave it the IP address assigned to the Host OS.

I could still access the server but something was weird. I did an ifconfig and it showed two network interface cards (eth0 and eth1) and both of them were configured to have the same IP address. ARGH!. Rob thought he had disabled the second NIC in the BIOS of the machine... but perhaps that got reset somehow... OR perhaps a newer Linux kernel could see the card despite it being turned off in the BIOS. Whatever the case, the hosting provider tech support person ended up with both interfaces configured although only one had a network cable plugged into it.

It is not a good thing when two NICs have the same address... and we started running into connectivity issues. I submitted another trouble ticket explaining that we did NOT want eth1 setup... and to check what port actually had the network cable plugged in. Seems the technician who had configured eth1 had also moved the network cable too.

The VPSes were trying to go through eth0 so they had lost connectivity. Since I could get to the Host OS still, I manually restarted each VPS (vzctl restart VPID) and they started working again as that caused them to start routing through eth1.

The Fix?

Hard driveHard driveI submitted yet another response to the trouble ticket as follows:

Please review the following steps and see if they make sense to you. If they do, please follow them and report any success. :)

  1. Login to the console as root
  2. Move the network cable from current port (eth1) to other port (eth0)
  3. restart networking
       service network restart
  4. Manually take down eth1
       ifconfig eth1 down
  5. Remove the config for eth1
       rm /etc/sysconfig/network-scripts/ifcfg-eth1
  6. restart networking again
       service network restart
  7. Verify that eth0 is configured and up and that eth1 does not show up
       ifconfig
  8. Verify connectivity with the outside world
      ping something outside

Working for Good?

I got a response back from them that said they had followed my instructions and that it appeared to be working again.

I ssh'ed in again, saw that indeed eth0 was setup and working, and that eth1 was no longer configured. I had to restart the VPSes again so they would switch from eth1 back to eth0... but I was expecting that. I guess I could reboot the machine and make sure it comes up like expected, but I'm just not in the mood for it quite yet.

Let's hope the system stays up for a long time to come... knock on wood.


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Scott Dowdle's picture

Been there a while now

I have only had a few down times with them... and in almost every case it was something going on with my server rather than their hosting. Those times I have had to file a trouble ticket they have been reasonably responsive and gotten my problem resolved.

I do maintain my own backups and don't really rely on them for much more than an occasional reboot... but they did fix a broken boot loader on a software RAID system for me once. They also rebooted someone else's server by mistake when I asked them for a reboot... but other than that, it is more than reasonable service for the price.

I'd really like to have a second machine with them so I could do faster backups and avoid downtime as the result of hardware failures.


Interesting, I'm deciding if

Interesting, I'm deciding if I want to use Colopronto. I don't want to send my server to a place I'm unsure about because it was expensive. Would you say you are happy with the service they are giving you?


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.