Roon services outage

Hi Everyone,

As of approximately 6:45PM EDT one of our service providers is experiencing an outage which is causing intermittent connectivity between user cores and our back-end services. This will result in some functions being degraded or unavailable.

We donā€™t yet have an ETA for resolution, but the team is working to get everything back online.

Weā€™re very sorry for the inconvenience.

1 Like

I believe the outage affects logging into Roon?

It was a denial of service attack :frowning:

Itā€™s been mitigated and we are keeping a watch on it to stay on top of the ongoing attack.

2 Likes

Who has Roon gone and upset to have one of those?

We are constantly being attacked. Everyone is. Mostly these attackers are pretty good about rate-limiting and hiding their actions in the noise of normal traffic. They arenā€™t looking to take us down, they are looking to steal your Roon accounts and resell them.

They have massive email/username + password lists from other sites and just test all the time. Once in a while, someone leaves a script running or forgets to rate limit, and when our servers start bcrypting 10s of thousands of passwords a second, machines start falling over.

4 Likes

Itā€™s a sad world, but thank you for all your efforts.

1 Like

So when will you implement TOTP seed based MFA to help combat against phished credentials or password stuffing?

(Please donā€™t do SMS-based MFA. SMS is easily compromised these days.)

1 Like

First weā€™ll switch to an OAuth based login that happens in the browser where we can deploy more countermeasures, then add MFA on top of that.

Itā€™s a project already underway.

That said, weā€™ve already done some stuff that makes it harder to steal accounts. For example, you canā€™t change passwords or emails on accounts without access to the original email account. It sometimes poses some inconvenience, but theft of accounts plummeted when we did that. Itā€™s not fool proof though, since many users get their email accounts compromised too. MFA is not fool proof either, as most users refuse to use it.

6 Likes

This is good to hear. Thanks.

1 Like

Whew, I had no idea.

If even 10% of these hackers put their talents to useful endeavors the world would be in a much better placešŸ¤”

2 Likes

Danny 100% Agreed on optional MFA.
After 18 months being turned on in one of the fairly large networks I manage, less than 5% of users had opted in.

When I gave a deadline for mandatory MFA, with no remote access allowed without it being enabled, with a month to go we hit 25% opt in

2 weeks after we had switched off access we still had about 15% people who could not get on the VPN because in their words ā€œI would not change the policyā€. Eventually at about 99% active and the other 1% only work physically in the office.

Itā€™s hard to force people, but for those that refuse TOTP based solutions, you can always send a text or email. They then get the level of security they deserve

Good luck with this work Danny, it is important work

5 Likes

We had no options but use MFA for any of our logins at work. Film industry is very panicky about security.

Well after the Sony hack thatā€™s to be expected and as it should be.

We launched ours just before lock down. That should say it all really. Couldnā€™t track anyone down and 90% working remotely

Itā€™s mainly Marvel and Disney, they make you jump through hoops. Infact one of the measures they asked us to do on a recent projects our server storage system, brought it to its knees, we had to revert the changes they insisted on it after it was finished and itā€™s been fine.

Oh we are a licensor for both so trust me I understand some of their absolutely crazy requests.
Star wars reboot was a new high in crazy Disney demands :roll_eyes:

1 Like

And yet for their own teams the security is way more lax. Which grates.

1 Like

Thatā€™s because in their mind, all suppliers are the weakest link.

1 Like