Roon services outage

Hi Everyone,

As of approximately 6:45PM EDT one of our service providers is experiencing an outage which is causing intermittent connectivity between user cores and our back-end services. This will result in some functions being degraded or unavailable.

We don’t yet have an ETA for resolution, but the team is working to get everything back online.

We’re very sorry for the inconvenience.

1 Like

I believe the outage affects logging into Roon?

It was a denial of service attack :frowning:

It’s been mitigated and we are keeping a watch on it to stay on top of the ongoing attack.

2 Likes

Who has Roon gone and upset to have one of those?

We are constantly being attacked. Everyone is. Mostly these attackers are pretty good about rate-limiting and hiding their actions in the noise of normal traffic. They aren’t looking to take us down, they are looking to steal your Roon accounts and resell them.

They have massive email/username + password lists from other sites and just test all the time. Once in a while, someone leaves a script running or forgets to rate limit, and when our servers start bcrypting 10s of thousands of passwords a second, machines start falling over.

4 Likes

It’s a sad world, but thank you for all your efforts.

1 Like

So when will you implement TOTP seed based MFA to help combat against phished credentials or password stuffing?

(Please don’t do SMS-based MFA. SMS is easily compromised these days.)

1 Like

First we’ll switch to an OAuth based login that happens in the browser where we can deploy more countermeasures, then add MFA on top of that.

It’s a project already underway.

That said, we’ve already done some stuff that makes it harder to steal accounts. For example, you can’t change passwords or emails on accounts without access to the original email account. It sometimes poses some inconvenience, but theft of accounts plummeted when we did that. It’s not fool proof though, since many users get their email accounts compromised too. MFA is not fool proof either, as most users refuse to use it.

6 Likes

This is good to hear. Thanks.

1 Like

Whew, I had no idea.

If even 10% of these hackers put their talents to useful endeavors the world would be in a much better place🤔

2 Likes

Danny 100% Agreed on optional MFA.
After 18 months being turned on in one of the fairly large networks I manage, less than 5% of users had opted in.

When I gave a deadline for mandatory MFA, with no remote access allowed without it being enabled, with a month to go we hit 25% opt in

2 weeks after we had switched off access we still had about 15% people who could not get on the VPN because in their words “I would not change the policy”. Eventually at about 99% active and the other 1% only work physically in the office.

It’s hard to force people, but for those that refuse TOTP based solutions, you can always send a text or email. They then get the level of security they deserve

Good luck with this work Danny, it is important work

5 Likes

We had no options but use MFA for any of our logins at work. Film industry is very panicky about security.

Well after the Sony hack that’s to be expected and as it should be.

We launched ours just before lock down. That should say it all really. Couldn’t track anyone down and 90% working remotely

It’s mainly Marvel and Disney, they make you jump through hoops. Infact one of the measures they asked us to do on a recent projects our server storage system, brought it to its knees, we had to revert the changes they insisted on it after it was finished and it’s been fine.

Oh we are a licensor for both so trust me I understand some of their absolutely crazy requests.
Star wars reboot was a new high in crazy Disney demands :roll_eyes:

1 Like

And yet for their own teams the security is way more lax. Which grates.

1 Like

That’s because in their mind, all suppliers are the weakest link.

1 Like