VDS (Chicago) Limiting Outbound Connections?

[Paronity]

Is there anything in the network architecture that would prevent a high number of MySQL requests from occurring in quick succession? We wrote a plugin for ProCon which has been running successfully on a different server for months without a single issue. Now that we have it on our vds server, the ODBC connection(s) keeps getting dropped. We can fire up the other server (which we are keeping around as a backup for the RCON tools) and everything works just fine again. The issue only occurs on this VDS. Can you think of anything that might cause this issue?

http://img.paronity.com/EjfR

I own the MySQL server as well and I can guarantee that it has not gone offline or many of my web customers would be very upset with me. Anyone have any idea what might cause this issue?

[Edge100x]

There's nothing on our end that would prevent or limit MySQL connections. The MySQL server may be set up by default to limit the connection count from external IPs, however -- its logs might provide more information on whether it's restricting the VDS IP.

[Paronity]

There's nothing on our end that would prevent or limit MySQL connections. The MySQL server may be set up by default to limit the connection count from external IPs, however -- its logs might provide more information on whether it's restricting the VDS IP.

It's not. The MySQL is configured to allow infinite connections from this host as it will be getting them continuously. I don't believe the MySQL server is at fault since I can run the exact same configuration of ProCon from the other server, my personal computer, and the founder of my clan without any issues.

Edit - I'm not trying to sound rude....as I re-read my post it could have come off that way. Apologies if that is how you took it. Just trying to figure out what's going on. Thanks for your help already.

[Edge100x]

With that error message, it's unfortunately very difficult to say what's going on. But, it's safe to say that it's not a problem with the hypervisor, as we use MySQL connections extensively from our VDSes.

Have you tested for packet loss between the VDS and your MySQL server?

Is the connection rate exactly the same before and now? If you are waiting longer between connections now, they might time out, and if there are far more, then extra threads might step on each others' toes if there are locking problems.

[Paronity]

With that error message, it's unfortunately very difficult to say what's going on. But, it's safe to say that it's not a problem with the hypervisor, as we use MySQL connections extensively from our VDSes.

I agree. That is why I posted here to see if anyone else could try and help me figure this out. We love the responsiveness we get from NFO servers and their support so we are trying to run everything we have on NFO servers now.

Have you tested for packet loss between the VDS and your MySQL server?

Pinged from my server to the VDS for 25 minutes straight. Not one packets was dropped. I also checked my error log on the MySQL server. Nothing. Lastly, I did a tcpdump on port 3306 to see if any errors were being sent that were not being reported any way. Ran strings on the resulting output and there are no errors coming from the MySQL server at all.

Is the connection rate exactly the same before and now? If you are waiting longer between connections now, they might time out, and if there are far more, then extra threads might step on each others' toes if there are locking problems.

Yes. Server count has actually decreased from when we had it on the other box. Was 4 64 man and 2 32 man servers. Now it is just 2 and 2. Sometimes they work as expected. We get stats about 85% of the time. I would hate for it to miss someone that was hacking and now we don't have their GUID to ban them later on.

Can you think of any other troubleshooting steps that I could do. I'm really open to anything at this point because I'm not sure what else to try. Thanks in advance.

[Edge100x]

Updating MySQL might be worth a try, to be safe.

Does a MySQL connection ever go a number of hours without activity? If so, it could be timing out.

[Paronity]

Updating MySQL might be worth a try, to be safe.

Running the latest version already.

Does a MySQL connection ever go a number of hours without activity? If so, it could be timing out.

No. The connections that have issues are having issues during connection initiation. If the server is connected, there is never a problem. It's initiating the connection when the MySQL server "goes away". Also to note, the servers NEVER throw the error at the same time. It is sporadic across all servers. One will throw the error while the others keep on chugging without any issues.

I did increase some settings that I don't think I'm hitting. Maybe one of those is responsible after all (which would be weird that the server isn't logging them).

max_user_connection
max_connect_errors

Thanks again for your input. It has been appreciated. Sometimes, you just need to bounce stuff off people to figure it out.

[Edge100x]

Running a tcpdump/windump on both sides as this connection attempt is made, is everything being sent on each side coming through to the other (the SYN/ACK, etc)?

[Paronity]

Well, it must be my MySQL server and something that just recently happened. For fun, I tried again on my desktop and other RCON server and they are now doing it as well. It didn't before, so I must be hitting some config limit that I wasn't hitting before. I am looking into what I might be hitting but no luck yet. Thanks for all your assistance thus far. It is much appreciated, and one of the many reason we will never go anywhere else for our server needs.

I don't think there is anything else that you need to assist with since we have now incontrovertibly proved it's not just the VDS. Thanks again.

[.=QUACK=.Major.Pain]

I had the same issue with Hostgator for several of my websites.
After it happened several times, they reinstalled a lot of the software like apache, mysql, etc, and made some adjustments.

They felt it was overload on the sites, and they told me that it was alot to do with spiders and crawlers totally overloading everything.

Made some changes to .htacess and robots.txt, and things have been great since.