Need for speed (v1.2 b165, 64bit, Windows)

[mannp]

@ben what do you class as a particularly large library? Mines not huge, but small/med i guess.

[Carl]

The OP has 500k tracks, in my book that’s very large… I’d say large is > 200K tracks… but it’s all subjective.

[ben (Ben Coburn)]

I’d be interested in any library that feels “slow” subjectively. That said, I’ve just gotten a 400k track export, which is a lot more than the one I’ve been using and is probably sufficient for my purposes.

[mzbe]

@Ben: thanks for taking time to share here. I hugely appreciate you giving the community a heads up on how things are going.

Just for fun, to explore “the art of the possible”, I dropped the XLSX library export into Microsoft PowerBI (desktop version is a free download, no strings attached).

Observations:

• The resulting .pbix file is 60.9MB (raw XML export from roon = 386 MB).
• PowerBI is using in-memory row-level compression.
• Any and all full text (“contains”) queries (that I constructed) on the full library take a split second to execute.

Takeaways:

• Even with this “large” library, with the most inefficient data representation (.XML export), the total track list only amounts to 386MB. Of course not counting indices (increasing size) or tabular compression (decreasing size).
• The effective size seems to indicate a possibility of always keeping the db in memory (is this already done in v1.2 today?).
• Perhaps tweaking the current engine will already produce great results. Maybe some 3rd party component could help, e.g. https://inmemory.net/ (disclaimer: I haven’t used this one, don’t know if it’s applicable).

[evand]

I’d be interested in any library that feels “slow” subjectively. That said, I’ve just gotten a 400k track export, which is a lot more than the one I’ve been using and is probably sufficient for my purposes.

If you want another around the same size that is slow browsing/ navigating at times and always when searching I’ll install Roon on Windows and export the db.

[Boris_Morris]

Have you considered a search index library? Something like Apache SOLR? That thing has pretty incredible performance for absolutely ridiculous index sizes - I think the Alfresco guys say they have more or less instantaneous results on content libraries with billions of items. And Alfresco has full text search on its content - which you obviously don’t need to music.

Although; can one search on lyrics in Roon…?

[Rugby (aka Daniel or Big Dan)]

Although; can one search on lyrics in Roon…?

No, but that is a great idea!

[mannp]

Ok, fair enough. Mine is more like 40k, so very very small in comparison; and is fast for me on a Xeon rig. So not a good comparison then.

[Nyquist]

Can’t help but think that a lot of the lag problems haven’t got anything to do with processor power, memory or database size but wih the status of your home network. I have had tons of issues with other software, file sharing, stable connections etc at home untill I bought a decent router. Solved almost aal of my problems instantly. Most ISP’s provide low-end crappy modem/routers wich are fine for one internet connection but not for a complicated home network with many devices talking and streaming to each other. My Roon server works smoothly on my music computer wich is allready 8 years old. It’s a core2duo e8200 with ‘only’ 2GB ram and win10 from ssd. Music on internal spinning disc.With the ISP router quite terrible, lots of connection losses and lag. Withn new Asus router smoothly. O.k.database is relatively small compared to lot’s of you, ‘only’ 2GB in size, +/- 3000 albums, but still.

[evand]

Enterprise gigabit switches deployed here, router only provides DNS.

[mzbe]

Same here - dedicated gigabit connection to NAS. Also, performance indicators during search indicate that performance is not dependent on network speed. Let’s wait for Ben to do his magic with v1.3

[Nyquist]

A switch is nothing more then a connection divider for trafic to and from your router. There is no direct traffic between devices connected to the same switch. All trafic goes trough the router. Disconnect the router and all your devices won’t see each other anymore. That’s what a router does, controlling traffic between devices, provide ip numbers etc. A switch does not do that. Your router quality still makes a difference nomatter how high quality switch you are using. Almost all ISP router are of the cheapest lowest quality unless you have a serious buisiness ISP with idem dito serious monthly fees. Only if you connect a router to the WAN port (Not a LAN port) of your ISP router it will only provide DNS and nothing else. With a switch that’s impossible.

[Boris_Morris]

You are completely incorrect.

Whilst this was true a very, very long time ago. It is no longer true now.

What you are taking about is a hub, a switch is an intelligent network device that switches traffic based on MAC address; as opposed to a hub which repeats all traffic to all ports.

In order to send a packet from computer A to computer B, computer A looks at the target IP address of the packet (lets ignore DNS for now). Computer A looks at its routing table to determine whether this address is on any subnet of its network interfaces. If it is not, then it determines the correct gateway for the traffic and sends it there - this is your router.

If computer A determines that computer B is on the same subnet, the it will broadcast on that network interface to ask who has that IP address, receiving a MAC address is response from the owner of the IP address. This mapping of IP to MAC is stored in the ARP table.

Once an ARP table entry is established, traffic is simply sent from computer A to computer B via the switch.

A router plays no role in local network traffic. It routes traffic across subnets.

[anon55914447]

Look, we know that dB performance is comparatively poor when measured against other popular library managers, and it’s easily reproducible by doing searches on a large library directly on the Core. This eliminates networking as a potential bottleneck.

O.k.database is relatively small compared to lot’s of you, ‘only’ 2GB in size, +/- 3000 albums, but still.

But still what?

[ben (Ben Coburn)]

• Even with this “large” library, with the most inefficient data representation (.XML export), the total track list only amounts to 386MB. Of course not counting indices (increasing size) or tabular compression (decreasing size).
• The effective size seems to indicate a possibility of always keeping the db in memory (is this already done in v1.2 today?).
• Perhaps tweaking the current engine will already produce great results. Maybe some 3rd party component could help, e.g. https://inmemory.net/ (disclaimer: I haven’t used this one, don’t know if it’s applicable).

We do keep all this data in memory in Roon 1.2 right now. Our performance problem isn’t that we’re going out to the disk for anything, it’s that the algorithm we use to actually search the data in memory is simple instead of fast. I’m working on replacing it with one that trades that simplicity for speed.

If you want another around the same size that is slow browsing/ navigating at times and always when searching I’ll install Roon on Windows and export the db.

The export to Excel I need actually doesn’t require Windows specifically, it should work from both Mac and Windows desktop versions. I’ll send a PM with upload instructions in a few minutes.

Have you considered a search index library? Something like Apache SOLR? That thing has pretty incredible performance for absolutely ridiculous index sizes - I think the Alfresco guys say they have more or less instantaneous results on content libraries with billions of items. And Alfresco has full text search on its content - which you obviously don’t need to music.

Although; can one search on lyrics in Roon…?

We did consider looking for an outside library for this problem, but we’d generally prefer to write code that does only and exactly what Roon needs it to. I’m going to be responsible for all the properties of this search thing, so I’d rather have direct control over the code.

[Nyquist]

But still what?

database performance and connection stability improved a lot even though i have a relatively small database.

[Boris_Morris]

We did consider looking for an outside library for this problem, but we’d generally prefer to write code that does only and exactly what Roon needs it to. I’m going to be responsible for all the properties of this search thing, so I’d rather have direct control over the code.

That’s perfectly understandable - and in my experience a very difficult decision to make. So many tradeoffs between maintainability and wheel-reinvention.

I can certainly see why you wouldn’t want to bolt on the goliath that is SOLR onto the relatively slim Roon platform.

Sounds like a very interesting, algorithmically, problem to solve in any case…

[evand]

can’t recall the user but there was an example of striping 2xSSD on an i7 and getting good performance on a very large uber-library. Worth reading that user’s experiences.

I’ve just done the same using 2x 60GB Kingston mS200 SSD’s using mdadm raid0, formatted f2fs. I’ll soon see whether it delivers any improvement.

[andybob (Andrew Cox)]

Hi @ben,

I’ve got a small library (20-30k Tracks or so) and am using an i7 BRIX as a Core with an internal SSD. The search response using the magnifying glass tool is quick, but the Focus Text search tool feels slow and laggy. Not sure if your current work extends to the Focus tool, but it is worth a look.

[RBM (René)]

Really? Mine is a bit smaller (16K tracks), but text entry & filtering is lightning quick (as in almost realtime, split second response). I can imagine differences with the 100K+ guys, but from 16 to 30K?