Update: Website Hacked

Hi all,

As you probably know, the website was hacked earlier this month. I’ve been working on fixing this up over the past couple of weeks, and can now report back with an update.

What Happened
A zero-day exploit for vBulletin was recently discovered. This affected any forums where the /install/upgrade.php file was left in place after an installation or upgrade. The exploit enables a remote attacker to silently create a new administrator account, with full access to the forums control panel. Pretty nasty stuff.

When last upgrading this website, I made the mistake of deleting /install/install.php, but not doing a full clean-up of the directory. This opened up a vulnerability which was exploited to create three new admin accounts over the course of a few days. One of these accounts then installed a PHP backdoor, and used this remote shell to manipulate files on the server, and deface the website.

How It Was Fixed
As soon as I discovered the site had been compromised, I took it offline and changed the passwords for important logins. Then I restored the site from backups to a new server (this is where daily backups really come in handy).

It took quite a while to get rid of the infected files, clean up the malicious administrator accounts, and check for any further vulnerabilities. The same attack was actually repeated while I was in the process of applying security patches! But I finally got it all cleaned up, and the site went live again roughly 24 hours after the homepage was defaced.

Since then I have been working with the security team at CodeGuard and analysing the logs to confirm further details of the incident, and find what can be done to prevent similar attacks in the future.

How It Affects You
There’s no way to know for sure what the attacker(s) did while the site was compromised. The particular vulnerability which was exploited has affected many other vBulletin forums recently. So far I have not found any indication that this was a targeted attack against MMWA in particular.

The safest way to approach this is to assume a worst-case scenario. The attacker had direct access to the filesystem, and full admin privileges on the forums. This means they could potentially access everything on the site, including:
[ul]
[li]Private messages[/li][li]Order history on the Gigs site[/li][li]Email addresses[/li][li]Hashed passwords[/li][/ul]

All passwords were salted and hashed, so there is a low risk of plaintext passwords being recovered. But this is still possible, especially if you used a weak password which might be vulnerable to a dictionary attack.
If you used this password on any other websites besides MMWA Forums, I recommend you change it immediately!

What Happens Next
Later today I will be resetting the password on all accounts as a routine precaution. You should receive an email with your username, and a link to set a new password.

I’ve implemented much more rigorous logging and backup procedures since this attack. Notifications have been set up so that any suspicious activity will be noticed immediately. While I already had regular backups in place, the recovery process took longer than expected. This is something I’ll be much more prepared for after the experience of last week.

I’m also looking into more secure hosting solutions. Up until recently I was using shared hosting, with a number of domains on the same filesystem. As the site continues to grow and gains a higher profile, I think it’s worth investing in infrastructure which will be both secure and scalable moving forward in the medium term. Most likely I will be moving the blog and forums to separate hosting providers, with a specialty in WordPress and vBulletin respectively.

Conclusion
The purpose of this post is to be transparent about what’s happened, and exactly what information may have been compromised. While websites are hacked all the time, I realise you’ve trusted me with certain information by participating in this community, and I do feel a personal obligation to protect that information. I’m sorry that your trust has been let down on this occasion. It’s the first major security breach I’ve had to handle in my time as a webmaster, so it’s been very much an education for me in best practices, backups, and security in general. I can assure you I’m doing my best to prevent anything like this happening again.

Please let me know if you have any questions or comments about what’s happened. And thank you for your continue support of this community!

Cheers,
David

May I ask, how did you find out they hacked the site?

@Martin - from what I remember they’ve changed the front page and everything.

One morning I woke up and checked the site, only to find the homepage had been replaced with a “Hacked” message. That’s when I first realised that something was wrong.

Unfortunately two admin accounts had already been created a couple of days earlier. Due to the nature of the exploit used, I never got any notification of this activity (whereas I normally get notifications about new registrations, possible spam, etc.) If I had been manually checking for new admin accounts on a regular basis, it’s possible that some of this could have been avoided. Which is why I’ve now ramped up the logging and notifications, and will be doing manual reviews to make sure anything like this is caught much earlier in the future.

That means people who registered or even logged in AFTER the day of the hack might have their passwords recorded - not only hashes! Have you checked if they modified the forum scripts for that? Unless they’ve only got access to the forum admin panel and not the filesystem?

Would love to see a screen shot of that message. I did see the unavailable message that you had put up. Cloudflare was still serving the older cached version of the forum.

Good point. They did install a modification to the forum scripts, which provided a PHP-based backdoor with filesystem & shell access. A similar backdoor was also copied to a few different locations on the filesystem.

Apart from this, I’ve found no indication of anything else being modified in the forum software. vBulletin does log admin actions, which provides an extra level of checking. And I’ve compared the active plugins to a clean vBulletin install to remove the compromised component there.

However the web host does not log shell commands or database queries directly. So there is no way to tell exactly what information (if any) was accessed using these methods. The Apache logs aren’t really much to go off.

Ah, good old Cloudflare. They’re great for keeping the site running - but actually made it harder to sift through the logs on this occasion, thanks to the lack of mod_cloudflare on my host’s Apache installation

Here’s a screenshot of what the site looked like before it was taken down:

And it seems this site wasn’t the only target, according to HACK DB | Hacker.

I Had one of my sites hacked a few years back. Also a deface, I don’t think they ever really got on the server it just turned out I had a sql injection leak in one of my pages. It does not necessary mean they have downloaded the data, but it could.

Just registered to say it takes decency to be upfront with your users. Respect.

now i can’t reach the forum via tapatalk or even via my mobile browsers. what should i do?

Thank God is fixed now, but what about dDos protection? i must wait about 5 seconds to visit site?

It’s a tool which helps to detected the ddos guy. A small help.

The site was being hammered with traffic this week, which overloaded the database and slowed down or broke lots of things. It looks like a bot was causing most of the hits, so I turned on CloudFlare’s DDoS protection which temporarily mitigated the issue. It should be back to normal again now.

Thanks a lot for sharing such a useful informations.this article is very useful for me.i got a lot of useful informations to this article.i would like to know more about this topic.i hope you more useful post.thank you.Insurance Brokers Erp

Great job, David!