Talk me out of building a 10GB (really 2GB) switch plane

I echo the comment above is the jump from 900Mbit/s to 1.2Gbit/s worth it?

Yes, I understand your backplane with be 2.5G but again to utilise this is some data shifting.

I have taken my internet and network to a 1Gbit/s service (1G down, 50M up), with a Cisco RV340 providing a NATing speed of 942Mbit/s over a 1Gbit backplane through a main 8-port switch for 3 NAS units (2 with bonded 1Gbit LAN connection), Roon Core, WAP (WiFi 5 with 1.3Gbit/s on 5GHz, 450Mbit/s on 2.4GHz) and a supplementary 5-port switch for IOT & some 100Mbit/s connections.

This makes general internet snappy over WiFi, instantaneous over wired machines, plenty of bandwidth for video calls, streamed video/music and any downloads faster than the server connection providing the content.

I have also, through a secondary service, now have resilience WAN connection into the RV340, with a 5G Modem as backup ISP.

I think this 1G/50M is as fast as the coaxial cable into the property can take, so unless there is some major infrastructure shift to optical FTTH in Dublin, this is it.

Step 1 complete: 1/2.5/5/10GB SFP+ in port 10, changed wan1 to port 10 (took some finding in the interface, no lie).

Now awaiting LTE modem and new cable modem.

From there I have to decide if the shenanigans will continue. As of right now all further L3 switch shenanigans are on pause, mostly because I donā€™t fancy rebuilding my rack. Plus I didnā€™t get my lowball offer accepted for a 48 Pro. And, to everyoneā€™s points above, I have no need. But I promise to keep everyone informed if I succumb.

Iā€™m not going to talk you out of it, if you find it interesting and rewarding. I have way more network capacity and infrastructure than I need. My stereo system, including Roon, is way better than I need. I have way more books than I need. My native plant garden/habitat is way more complex and bigger than I need. (N.B.: I said need, not want.)

Guess what my hobbies are.

I say go for it, and have fun going for it.

Whatā€™s the part you used for that? That seems like a simple and smart upgrade.

That is this (there are many brands, this seems to work fine). Make sure you get the slightly more expensive 1/2.5/5/10G and not the 10 if you want it to speak to the 1/2.5 Motorola modem. Hot swappable so stupidly easy at the hardware level. And donā€™t (like me initially) forget a cat 6 patch cable, youā€™ve probably never actually wanted to even theoretically bother with 5 vs 6 before.

And remember, all this gets you is the full capability of the modem/plan at your UDM.

https://a.co/d/8jyjhYw

If someday youā€™re connecting two SFP+ ports and want 10G between them, and not going far enough to bother with fiber, thereā€™s this ridiculously comparatively cheap ā€œtwo modules and a permanently connected cableā€ unit.

https://a.co/d/8EANJhy

Though if youā€™re getting an L3 switch just so you can have one, maybe youā€™ll go with fiber just to say you have. Hypothetically speaking of course.

I also just learned that Comcast is building out its DOCSIS 4.0 network, and has announced plans to get symmetrical 2Gb connections to most US households in the next couple years. They just sent me an email to let me know that my internet will be out for a full day in a couple weeks as they complete network upgrades.

Not that any of this matters to me now, but boy they are making a substantial bet that the metaverse or something like that is coming and we are all gonna be saturating the links we have. Qobuz & Tidal ainā€™t seen nothing yet. Looked back at an old bill - wasnā€™t very long ago that I had the 25Mb plan and was psyched about it and laughing at my friends who overpaid for a 100Mb plan and were buying dongles because they were trying to solve latency & poor Wi-Fi coverage inside their homes, whereas I had decent wired connections around my home (pulled pre-cut cables through the basement and up through A/C registers!) and so Netflix worked without glitches. I think I only got 100Mb in 2012 or something.

Thanks. I wondered if that was the part you used. According to Amazon, I ordered one in 2020. I now remember doing this and realizing, at the time, that the modem on the other end was 1Gb, so I recycled it. Have another inbound now.

I use the UniFi branded version of this to connect my UDM Pro and my switch. You can see it in my rack photo in an earlier post. After getting it working at 1Gb, but not faster, I looked at the specs of my switch. You already know where that led. But it works, so I left it alone.

Iā€™m cat6 throughout including those little patch cables. :slight_smile:

My AT&T U-Verse internet is $30 per month for 50Mbps down and 12Mbps up. Works great for Roon, Audirvana, and Apple Music. Problem is, they keep jacking the price of AT&T U-Verse TV and my only alternatives are Charter Spectrum or Direct TV.

Yo, @Johnny_Ooooops ā€¦ do you actually have that SFP+ module running at 2.5Gbps with the Motorola MB8611? I just tried it and couldnā€™t get it to work.

Looking at UDM Pro specs, and reading various forum posts, I get the impression that the UDM Pro WAN SFP+ port supports only 1 or 10Gb but not 2.5. What are you seeing on your end and if you do have it running at 2.5Gbps, can you post a picture of configured or running at that speed so I know what I should be looking for?

Thanks!

Yo @gTunes so hereā€™s the dealā€¦ I donā€™t have the Motorola yetā€¦ itā€™s coming tomorrow and Iā€™ll let you know. Right now I have the SFP+ / Port 10 hooked up to my Netgear CM600 which has a gigabit Ethernet port. I have Port 10 configured as WAN1 and to force 10Gb (see images below). The SFP+ module supposedly presents as 10Gb to the UDM and then autonegotiates internally. Currently it must be doing that at 1Gb to have a happy connection with the CM600, though I have no sign on the Unifi (which just thinks itā€™s 10Gb). I presume itā€™ll just work out when I plug in the Motorola and automagically figure out that 2.5Gb is where it should be, and Iā€™ll see the famous blue light on the Motorola and see the speed when I run a Speed Test directly from the UDM console. But I got no proof, just prior reports from the Tong Family that it works thusly. Itā€™s worth a double check that you have one with a sticker on it that says 1/2.5/5/10, because but for the sticker itā€™s indistinguishable from the 1 or the 10 model (which Iā€™ve read about in my travels).

Will report back once I have the new modem in place.

In other random stuff, just realized my (non-rack-mount) UPS battery is long dead, so I have to replace that, and thatā€™ll involve some amount of rack fussery. Pandoraā€™s box and all that.


Interesting. I get where youā€™re going with this now. I was expecting the UDM to understand 2.5Gbps but I see now, from what you wrote and write-ups elsewhere, that the UDM will see it as 10Gbps. Canā€™t play more with it now but Iā€™ll try again later. Thanks.

All working.

image

Iā€™m consistently seeing at-the-router tests of a bit north of gigabit. Thatā€™s about a 15% increase.

Thanks for the pointers, @Johnny_Ooooops!

1 Like

Funnily enough, since I embarked on this whole journey, Iā€™ve upgraded a few things, and the best thing that Iā€™ve seen is totally outside of my control and just coincidentally happened at the same time.

So what did I do?

  • got 2*1Gb LACP link between my 2 24-port unifi switches
  • got a Netgear LB1120 modem on Google Fi for when Comcast falls over, set it up as a failover WAN2 on my UDM Pro
  • got a Motorola DOCSIS 3.1, and figured out how to get it connected to my UDM Pro via a 2.5Gb connection so I can see whatever it gives me at my router, even if I canā€™t get all that speed out to a single device because Iā€™m bottlenecked by the 1Gb connection from my UDM Pro to my existing switches

Amidst all this, I realized my office ethernet was down a pair, so was getting 100Mb, so that was hilarious. It took a few hours sleuthing to figure out which punch down was the culprit and fix it. Frustrating, but ironic to be chasing all this when the one connection where Iā€™m likely to use the speed couldnā€™t use a fraction of it.

But finally, what was the awesome thing that happened you may ask? Right in the middle of all this, Comcast rolls through my neighborhood to install their fancy new ā€œ4.0ā€ (scare quotes because I think itā€™s a meaningless term) gear. And what does that result in? 1.2Gb measured at my router when Iā€™m paying for 900Mb, but more importantly to me, consistently low latency. I used to have multiple spikes to 30-100ms each day. Now itā€™s pretty solid at 10-18ms consistently for a week or more, Max of 27 in last 24 hours vs 150ms typical previously. Donā€™t know if that will continue, but thatā€™s probably worth way more to me than more bandwidth at this point. Even when I add a bunch of albums adding up to 15Gb, my automated house-to-house sync across my OpenVPN tunnel completes in less than an hour without appreciably getting close to saturating the link so no one notices. But low latency is gold.