What you need to know about the recent compromise here

This is used for general discussion that is not necessarily server-related.
User avatar
Edge100x
Founder
Founder
Posts: 12424
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

What you need to know about the recent compromise here

Post by Edge100x » Fri Jul 31, 2015 11:46 pm

On July 29, 2015, NFOservers suffered a compromise of some of its systems. In this document, we detail what we know was accessed, and we issue guidance for how you should respond (most importantly, by changing all your passwords). We also discuss how we've responded to the breach, and some of what we know about how it was done.

Our guidance here is based on worst-case assumptions about what the attacker might have obtained and accessed. In reality, it is likely that the attacker did a subset of what he could have, and unlikely that he has used many stolen credentials (yet).

The damage

On July 29, 2015, an attacker gained system-level access to our main webserver, including its files and databases. The attacker then used the webserver's credentials to log in to our mail server, and briefly logged in to several hosted website machines (hosted, hosted2, hosted3, hosted11, hosted30, and hosted32).

The databases on the webserver store information on most services here, including any passwords shown in the control panel (such as webhosting system, MySQL, and stats passwords; VNC passwords for VDSes; default VDS system passwords; and default dedicated server passwords); customer names; customer email addresses; the prices and configurations of services; and communications made between us and customers in our support system. The mail server contains all stored email messages (those which customers have not deleted), as well as email login and password combinations. Hosted website machines contain customers' files and databases for shared webhosting accounts.

Control panel login passwords are stored using a one-way salted hash, which means that the attacker does not have the passwords themselves, but he or she would be able to locally brute-force simple passwords using culled database information. Passwords that are displayed in the control panel are stored as clear text in the database (we could have two-way encrypted them, but in a case like this, the attacker would have the tools to decrypt them, so it would not have helped). Email passwords are also stored as clear text in their database.

We know that the attacker exported control panel login and password combinations, and that the attacker exported VDS passwords (both VNC and system default), at approximately 10:30am PDT on July 29. We assume that the attacker then retrieved these exports. It is likely that the attacker also read other files and databases, but we do not know the extent to which this was done; it could be that nothing else was read, or that essentially everything stored on the webserver was read. The attacker had access to the webserver for most of the day, so he or she had a significant window of opportunity.

The attacker was logged in to the mail server for a shorter period of time than the webserver -- about one hour. This would have been enough time to export passwords and potentially read some emails directly from disk, though not long enough to retrieve a significant quantity of them. We did not detect IMAP or POP3 activity suggesting that the attacker tried to remotely retrieve emails using those mechanisms.

The hosted website machines were only logged into briefly (a few minutes apiece). It is possible that the attacker directly read a limited number of customer files on just those machines during that time.

Our payment system, which stores credit card information, has many extra layers of encryption and security, in accordance with (and exceeding) PCI requirements. It is on a separate machine that can only be partially accessed from the webserver using a custom API. Credit card numbers were not obtained by the attacker in this breach.

Steps you should take in response to the breach

Change all passwords associated with NFO services. These include:
  • Your control panel password.
  • If you have an unmanaged VDS with us, your VNC password (and then shut down the VDS and start it back up through the control panel, to make sure that the new password takes effect).
  • If you have an unmanaged VDS or machine with us and were using the default root/administrator password, that root/administrator password. The new password that you choose does not need to match what you might see in the control panel; it is actually more secure if it doesn't.
  • If you have webhosting with us, all three webhosting passwords listed in the control panel (system, MySQL, and stats). If you have important files on the webhosting, consider that there is a slim chance that they could have been read, and respond appropriately (such as by changing other passwords that you use internally on that hosting).
  • If you have an account on our forums, its password.
  • Any passwords on third-party websites that might match those you use here (you should use a different, well-constructed password on every website, whenever practical).
Check your NFO account and services for signs of unauthorized access or tampering. For instance, check the "Access log" pages of your control panel and look for any IP addresses that you do not recognize or actions that you don't remember taking. If you have a game server, check the FTP log for the same sort of things. If you have a VDS or machine, check its system logs for signs that someone might have tried to log in.

If you use an email account with us, have the administrator of the domain for that email address set a new password for you. Because there is a chance that your emails themselves might have been accessed, if you had credentials from external websites in stored emails, make sure to also change those passwords.

Be wary of phishing attempts. Since the attacker may have obtained account email addresses, he or she may send emails to you that pretend to be from NFO, from banking or other websites, or that include malware as attachments. Be very careful about clicking URLs included in emails, and always hover your mouse over them to make sure that they lead where you expect (and that the part between http:// or https:// and the next slash is a legitimate domain that you recognize). Do not open an attachment unless it is from a trusted sender and you expected it to be sent.

The attacker was fully locked out of the webserver at 2:30am PDT on July 30. If you already changed passwords before then, to be safe, we recommend changing them again.

Steps we have taken in response to the breach

Upon discovering a compromise, and after researching it to determine its scope (we initially thought that much less data was exposed), we notified all customers to change passwords in the system here, through a control panel event.

When we later determined that the mail server had been accessed, we posted another event, to customers with webhosting accounts. At this time, we also invalidated all email passwords, because we knew that the attacker could potentially use them immediately.

We removed the original source of the breach. We invalidated the method that the attacker was using to access the webserver and other machines and added firewall rules to prevent any sort of further external command-line access to the mail server and webserver. We changed all internal passwords, including to all machines that we run (even though these weren't stored and therefore should be unknown to the attacker), to our own user accounts on those machines, to our databases, to our internal file-transferring mechanisms, to our switches and routers, to our remote-controlled power outlets, to our email accounts, and to our payment processor access accounts.

We moved our forums to a separate machine and updated them to the next major release.

We updated the kernel and other applications on our main webserver, and we are in the process of updating hosted website machines.

We invalidated some control panel user account passwords -- ones which were last changed in perhaps the last five years or so were stored using a different, less-secure type of hashing algorithm, and were more vulnerable to being brute-forced.

We are posting this document here and sending an email linking to it to all customers.

We are contacting law enforcement.

We are continuing to monitor closely for further signs of unauthorized access to any of our machines.

We are reviewing how we can improve our internal security further, both to prevent future breaches and to better limit damage if another one were to occur. We are prioritizing these next steps and will begin their implementation. (We did many things right when it came to securing our systems, but there is always room for improvement.)

How the attacker obtained access

The attacker initially got in through our customer forum. He or she appears to have used an unannounced/unknown phpBB vulnerability targeting our version of the software to log in as a forums administrator without knowing the administrator's password. The attacker then used facilities in phpBB to install scripts allowing access to the system as its user (which we intentionally kept separate from other users on the system and gave very little access).

Using an unannounced/unknown method of privilege escalation, the attacker obtained access to another user on the system -- the user for our main website and control panel.

The attacker performed a further privilege escalation (likely using the same unannounced/unknown method, potentially through the kernel itself) to obtain system-level privileges (which we are extremely careful to prevent from happening using any known methods).

The main webserver intentionally has significant access to many other systems on our network. Using an authentication key retrieved from the webserver, the attacker logged in to the mail server and several hosted website machines.

User avatar
Markie
A regular
A regular
Posts: 56
Joined: Mon Oct 01, 2012 11:55 am
Location: AS14586

Re: What you need to know about the recent compromise here

Post by Markie » Fri Jul 31, 2015 11:53 pm

Yeah, it sucks that it happened, however, I'm extremely pleased with the level of transparency shown here.

User avatar
ZacharyS
A semi-regular
A semi-regular
Posts: 26
Joined: Fri Oct 10, 2014 6:32 pm
Location: Somewhere off the South Coast - UK
Contact:

Re: What you need to know about the recent compromise here

Post by ZacharyS » Sat Aug 01, 2015 1:27 am

I'd like to thank you for the information you've provided us here. It does mean some extra work for our Community's administrators, but that's a small price to pay for security.

This stuff happens to anyone and everyone and it's just a shame that people feel the need to do this kind of stuff (hacking I mean). As usual the support and information you've supplied us with is exemplary (is that spelt right? :lol: )

As usual you and your Company have reinforced the reasons that we have as a Gaming Community for using your excellent services and we'll continue to do so for a long time yet, finances permitting.

Now I need to leave and check that everything that needed doing has been done :wink:

Thanks again.

Zac. :D

sewi
New to forums
New to forums
Posts: 6
Joined: Sat Jun 08, 2013 4:50 am

Re: What you need to know about the recent compromise here

Post by sewi » Sat Aug 01, 2015 2:57 am

Edge100x wrote:Be wary of phishing attempts. Since the attacker may have obtained account email addresses, he or she may send emails to you that pretend to be from NFO, from banking or other websites, or that include malware as attachments. Be very careful about clicking URLs included in emails, and always hover your mouse over them to make sure that they lead where you expect (and that the part between http:// or https:// and the next slash is a legitimate domain that you recognize). Do not open an attachment unless it is from a trusted sender and you expected it to be sent.
This will be the main backlash - be prepared for very official sounding phishing mails, addressing you correctly, using your correct server names, and so on.
A good advise is also to not click any links in mails that look like they're coming from nfoservers.com, and rather navigate to the page directly, and then check your tickets there.


Apart from that, I agree that the level of transparency you provide is exceptional; it allows your customers to paint their own pictures of the scope, and adjust accordingly.

When it comes to security considerations, the worst case is not a bad place to start.

EvilJellyDonut
New to forums
New to forums
Posts: 1
Joined: Sat Mar 09, 2013 3:04 pm

Re: What you need to know about the recent compromise here

Post by EvilJellyDonut » Sat Aug 01, 2015 4:14 am

Weren't your passwords encrypted?

Bubka3
Compulsive poster
Compulsive poster
Posts: 68
Joined: Fri Feb 21, 2014 10:24 am

Re: What you need to know about the recent compromise here

Post by Bubka3 » Sat Aug 01, 2015 5:24 am

Welp, I took this opportunity to rotate most of my passwords.

noname
A semi-regular
A semi-regular
Posts: 16
Joined: Thu Dec 18, 2014 7:55 am

Re: What you need to know about the recent compromise here

Post by noname » Sat Aug 01, 2015 6:02 am

I see that "customer names" is mentioned among the leaked data.

Was this the name on the account itself ? The name that you are greeded by when logging in to your control panel "Welcome Name" and the one that displays when you create a support ticket ? or was it the name used for CC payments ?

Are you 101% sure no home address, phone number, country, CC expiery date were leaked ?

It is VERY BAD this compromise has taken place...this is why i mostly avoid using companies that don't take anon forms of payment such as bitcoin, paysafecard or similar, so i don't have to worry as much when stuff like this happens.....very bad guys, + keeping information in plain text...wtf.

mohhk
New to forums
New to forums
Posts: 3
Joined: Mon Feb 23, 2015 7:04 pm

Re: What you need to know about the recent compromise here

Post by mohhk » Sat Aug 01, 2015 6:14 am

I am sorry you guys had to go through this stressful incident. Thanks for the very fast reaction and communication.

Net security is really a subtle art sometimes, no hard feelings. Actually, this is yet another reason for me to recommend your company. Really shows how hard working and professional you are. :)

Thank you, keep up the good work, and keep healthy!

kendsie
New to forums
New to forums
Posts: 7
Joined: Thu Mar 06, 2014 8:47 am

Re: What you need to know about the recent compromise here

Post by kendsie » Sat Aug 01, 2015 6:28 am

This is what I really love about NFO. Complete transparency. Being a long timer with NFO, I can attest to the fact that they just "ROCK"! John and his boys, and girls, are the best! Yea, it was a pain in the buttox to change all of my passwords, and the passwords of the other game servers that I regulate, but this is a small price to pay for piece of mind.

Thanks NFO, for all your hard work! I am thinking, that maybe we should change our passwords more often than every 2 years... :lol: !

Great job people!

«RG»BMF
http://WWW.RGCLAN.NET

jacobe
New to forums
New to forums
Posts: 1
Joined: Sat Aug 01, 2015 7:06 am

Re: What you need to know about the recent compromise here

Post by jacobe » Sat Aug 01, 2015 7:09 am

Thank you for the transparency!

eckospider
New to forums
New to forums
Posts: 3
Joined: Sat Aug 01, 2015 7:08 am

Re: What you need to know about the recent compromise here

Post by eckospider » Sat Aug 01, 2015 7:14 am

[*]Any passwords on third-party websites that might match those you use here (you should use a different, well-constructed password on every website, whenever practical).[/list]

Bubka3
Compulsive poster
Compulsive poster
Posts: 68
Joined: Fri Feb 21, 2014 10:24 am

Re: What you need to know about the recent compromise here

Post by Bubka3 » Sat Aug 01, 2015 8:19 am

noname wrote:Was this the name on the account itself?
Probably. It seems all the payment source data is not on the web server and really locked down.
noname wrote: keeping information in plain text...wtf.
It is not standard practice to encrypt any names or email addresses. The control panel passwords were properly hashed and salted. As John said, the default passwords for other services would gain minimal benefits from being encrypted because it would be a two-way function (so that it can be displayed, later).

Overall, it seems John handled this appropriately, and getting a full report within 3 days of the hack is unheard of. Very good work.

theRadAleks
This is my homepage
This is my homepage
Posts: 199
Joined: Wed Feb 19, 2014 6:07 pm
Location: Dallas, TX

Re: What you need to know about the recent compromise here

Post by theRadAleks » Sat Aug 01, 2015 10:36 am

noname wrote:I see that "customer names" is mentioned among the leaked data.

Was this the name on the account itself ? The name that you are greeded by when logging in to your control panel "Welcome Name" and the one that displays when you create a support ticket ? or was it the name used for CC payments ?

Are you 101% sure no home address, phone number, country, CC expiery date were leaked ?

It is VERY BAD this compromise has taken place...this is why i mostly avoid using companies that don't take anon forms of payment such as bitcoin, paysafecard or similar, so i don't have to worry as much when stuff like this happens.....very bad guys, + keeping information in plain text...wtf.
No credit card information was leaked because as John stated it's hosted on a separate machine "Our payment system, which stores credit card information, has many extra layers of encryption and security, in accordance with (and exceeding) PCI requirements. It is on a separate machine that can only be partially accessed from the webserver using a custom API. Credit card numbers were not obtained by the attacker in this breach." Please read everything before just commenting.

User avatar
Edge100x
Founder
Founder
Posts: 12424
Joined: Thu Apr 18, 2002 11:04 pm
Location: Seattle
Contact:

Re: What you need to know about the recent compromise here

Post by Edge100x » Sat Aug 01, 2015 11:40 am

Was this the name on the account itself ? The name that you are greeded by when logging in to your control panel "Welcome Name" and the one that displays when you create a support ticket ? or was it the name used for CC payments ?

Are you 101% sure no home address, phone number, country, CC expiery date were leaked ?
The API exposed to the webserver from the payments system allows for the information that you can see on the "Payments" page of the control panel to be retrieved by the webserver only (not by any external system). Credit card numbers simply can't be read through this system, and there is no other way for the webserver to access the payment machine, which is why we know that they were not obtained by the attacker. The other information could have been read in theory; however, our logs do not suggest that it was.

User avatar
ZacharyS
A semi-regular
A semi-regular
Posts: 26
Joined: Fri Oct 10, 2014 6:32 pm
Location: Somewhere off the South Coast - UK
Contact:

Re: What you need to know about the recent compromise here

Post by ZacharyS » Sat Aug 01, 2015 1:19 pm

Seems to me that they (nfo) did everything they could and as has been said in other posts - it's refreshing to be kept informed, not something that the majority of other companies would do I'm sure :wink:

Keep up the good work!

Zac. :D

Locked