Our guidance here is based on worst-case assumptions about what the attacker might have obtained and accessed. In reality, it is likely that the attacker did a subset of what he could have, and unlikely that he has used many stolen credentials (yet).
The damage
On July 29, 2015, an attacker gained system-level access to our main webserver, including its files and databases. The attacker then used the webserver's credentials to log in to our mail server, and briefly logged in to several hosted website machines (hosted, hosted2, hosted3, hosted11, hosted30, and hosted32).
The databases on the webserver store information on most services here, including any passwords shown in the control panel (such as webhosting system, MySQL, and stats passwords; VNC passwords for VDSes; default VDS system passwords; and default dedicated server passwords); customer names; customer email addresses; the prices and configurations of services; and communications made between us and customers in our support system. The mail server contains all stored email messages (those which customers have not deleted), as well as email login and password combinations. Hosted website machines contain customers' files and databases for shared webhosting accounts.
Control panel login passwords are stored using a one-way salted hash, which means that the attacker does not have the passwords themselves, but he or she would be able to locally brute-force simple passwords using culled database information. Passwords that are displayed in the control panel are stored as clear text in the database (we could have two-way encrypted them, but in a case like this, the attacker would have the tools to decrypt them, so it would not have helped). Email passwords are also stored as clear text in their database.
We know that the attacker exported control panel login and password combinations, and that the attacker exported VDS passwords (both VNC and system default), at approximately 10:30am PDT on July 29. We assume that the attacker then retrieved these exports. It is likely that the attacker also read other files and databases, but we do not know the extent to which this was done; it could be that nothing else was read, or that essentially everything stored on the webserver was read. The attacker had access to the webserver for most of the day, so he or she had a significant window of opportunity.
The attacker was logged in to the mail server for a shorter period of time than the webserver -- about one hour. This would have been enough time to export passwords and potentially read some emails directly from disk, though not long enough to retrieve a significant quantity of them. We did not detect IMAP or POP3 activity suggesting that the attacker tried to remotely retrieve emails using those mechanisms.
The hosted website machines were only logged into briefly (a few minutes apiece). It is possible that the attacker directly read a limited number of customer files on just those machines during that time.
Our payment system, which stores credit card information, has many extra layers of encryption and security, in accordance with (and exceeding) PCI requirements. It is on a separate machine that can only be partially accessed from the webserver using a custom API. Credit card numbers were not obtained by the attacker in this breach.
Steps you should take in response to the breach
Change all passwords associated with NFO services. These include:
- Your control panel password.
- If you have an unmanaged VDS with us, your VNC password (and then shut down the VDS and start it back up through the control panel, to make sure that the new password takes effect).
- If you have an unmanaged VDS or machine with us and were using the default root/administrator password, that root/administrator password. The new password that you choose does not need to match what you might see in the control panel; it is actually more secure if it doesn't.
- If you have webhosting with us, all three webhosting passwords listed in the control panel (system, MySQL, and stats). If you have important files on the webhosting, consider that there is a slim chance that they could have been read, and respond appropriately (such as by changing other passwords that you use internally on that hosting).
- If you have an account on our forums, its password.
- Any passwords on third-party websites that might match those you use here (you should use a different, well-constructed password on every website, whenever practical).
If you use an email account with us, have the administrator of the domain for that email address set a new password for you. Because there is a chance that your emails themselves might have been accessed, if you had credentials from external websites in stored emails, make sure to also change those passwords.
Be wary of phishing attempts. Since the attacker may have obtained account email addresses, he or she may send emails to you that pretend to be from NFO, from banking or other websites, or that include malware as attachments. Be very careful about clicking URLs included in emails, and always hover your mouse over them to make sure that they lead where you expect (and that the part between http:// or https:// and the next slash is a legitimate domain that you recognize). Do not open an attachment unless it is from a trusted sender and you expected it to be sent.
The attacker was fully locked out of the webserver at 2:30am PDT on July 30. If you already changed passwords before then, to be safe, we recommend changing them again.
Steps we have taken in response to the breach
Upon discovering a compromise, and after researching it to determine its scope (we initially thought that much less data was exposed), we notified all customers to change passwords in the system here, through a control panel event.
When we later determined that the mail server had been accessed, we posted another event, to customers with webhosting accounts. At this time, we also invalidated all email passwords, because we knew that the attacker could potentially use them immediately.
We removed the original source of the breach. We invalidated the method that the attacker was using to access the webserver and other machines and added firewall rules to prevent any sort of further external command-line access to the mail server and webserver. We changed all internal passwords, including to all machines that we run (even though these weren't stored and therefore should be unknown to the attacker), to our own user accounts on those machines, to our databases, to our internal file-transferring mechanisms, to our switches and routers, to our remote-controlled power outlets, to our email accounts, and to our payment processor access accounts.
We moved our forums to a separate machine and updated them to the next major release.
We updated the kernel and other applications on our main webserver, and we are in the process of updating hosted website machines.
We invalidated some control panel user account passwords -- ones which were last changed in perhaps the last five years or so were stored using a different, less-secure type of hashing algorithm, and were more vulnerable to being brute-forced.
We are posting this document here and sending an email linking to it to all customers.
We are contacting law enforcement.
We are continuing to monitor closely for further signs of unauthorized access to any of our machines.
We are reviewing how we can improve our internal security further, both to prevent future breaches and to better limit damage if another one were to occur. We are prioritizing these next steps and will begin their implementation. (We did many things right when it came to securing our systems, but there is always room for improvement.)
How the attacker obtained access
The attacker initially got in through our customer forum. He or she appears to have used an unannounced/unknown phpBB vulnerability targeting our version of the software to log in as a forums administrator without knowing the administrator's password. The attacker then used facilities in phpBB to install scripts allowing access to the system as its user (which we intentionally kept separate from other users on the system and gave very little access).
Using an unannounced/unknown method of privilege escalation, the attacker obtained access to another user on the system -- the user for our main website and control panel.
The attacker performed a further privilege escalation (likely using the same unannounced/unknown method, potentially through the kernel itself) to obtain system-level privileges (which we are extremely careful to prevent from happening using any known methods).
The main webserver intentionally has significant access to many other systems on our network. Using an authentication key retrieved from the webserver, the attacker logged in to the mail server and several hosted website machines.