Talk me out of building a 10GB (really 2GB) switch plane

Tinkering
1

So, the product of me getting an LTE modem to act as a failover internet connection for Comcast is that I spent a bunch of time looking at unifi networking kit and the Xfinity site.

  • turns out that Xfinity is giving me 900GB for what I had been paying for 600GB, and my old Netgear CM700 is throttling my connection speed
  • I could get 1.2gbps for only $5/month more
  • in order to take advantage of that I would need a CM2000 or a Motorola MB8611; let’s pick the Motorola because it’s cheaper and Wirecutter likes it and it gets decent reviews
  • then I need to get an SFP+ 10G/2.5/1.25 copper module for my Unifi UDM Pro, and I’ll need to swap my WAN1 and WAN2 ports (the port 9 which is default for WAN1 is “only” 1GB and will be used for the new LTE modem, while the SFP+ Port 10 will be swapped and will handle the Motorola at 2.5GB)
  • but then I could also get a Unifi Aggregation switch with its 8 SFP+ ports, and connect it to my UDM Pro and my 2 different Switch 24 POEs via link aggregation of 2*1GB, or just via SFP 1GB (all the switch 24’s have). Note this gets harder because the Switch 24’s only have SFP, not SFP+. And the UDM-Pro doesn’t support link aggregation. So it’s all puzzle pieces.

But what am I achieving? I’ve never had local network congestion. I’ve never had an internet connection this fast nor missed it. Am I chasing stupidity? A solution in search of a problem?

My “semi-pragmatic” answer is to “just” get the modem, the SFP+ 2.5GB module, the LTE backup and wait until I saturate my local network. But… is there a reason to go further sooner?

1 Like
2

I have the xfinity 1.2 connection, I even pay a $5/month penalty fee for using my own gear. I won’t put their box on my network because they won’t give me the root password. That would NEVER stay in that state so why even pretend. Otherwise it is pretty solid.

Why would you need to put fibre into the Unifi? I got confused here.

I’m not sure you gain much for the costs involved. If you aren’t locally oversubscribed I would hold off for faster/cheaper networking.

EG I just bought a workstation with built in 2.5g/10g rj45 ports . . . that I hope work at 1g.

1 Like
3

I would get whatever you need to comfortably support your needs and give you the cellular failback you want. Don’t waste money for no reason.

1 Like
4

I just went through this in January. We have similar equipment - UDM Pro, Switch 24 PoE.

TL;DR: I got pulled into the cable modem upgrade you’re considering. I didn’t go as deep as you’re considering going. I think it’s a waste of time and money.

I had a Netgear CM1150 which Xfinity started telling me was throttling my speed and no longer supported. I’m paying for the 1.2Gbps tier and while the CM1150 is functionally capable of supporting it up to 2Gbps, it Xfinity only “certifies” it for 800Mbps. They’ve done similar things with other self-owned cable modems. I’m not sure if the they are actually throttling speeds to “certified” limits or if that’s just part of their strategy to get people that own their own gear to move to the Xfinity “gateway”.

I don’t think we need to discuss any more about why @MamaTried, @Johnny_Ooooops and I don’t want to use the Xfinity gear (e.g., Home Hotspot).

I actually fell for the whole thing about 1.Gbps without thinking it through the way you are.

We moved last year from an area serviced by Ziply Fiber (derivative of FIOS) to an Xfinity-only area. Ziply gave us bi-directional 1Gbps. That’s an amazing service tier - I miss it. Had Xfinity over there, too, in parallel. Every once in a while, I re-seed my offsite Azure backup and it’s amazing to watch it fly at a legitimate 1Gbps upstream. I still haven’t quit wrapped my head around being back in the world of asymmetric speeds and so when Xfinity started telling me that I was being throttled, my thinking got a little muddled and I bought a new cable modem without thinking it all through.

I did the same research you did and bought the Motorola MB6811.

I’ve owned my own Xfinity gear for many, many years and am no stranger to cable modem upgrades. My experience has been that they are mostly self-service now. The MB6811 update wasn’t - was one of those hours on the phone experiences. They just couldn’t get it provisioned. They finally did.

And that, my friend, is where my story ends. Because I realized, like you, that the next step would be to try to connect my UDM Pro to the Motorola using the WAN SFP+ port and an adapter. If I were to get that working, I would have essentially raised my downstream bandwidth capacity between the UDM Pro and my UDM Switch (a USW-24 PoE) from 1Gbps to 1.2Gbps. And that would accomplish absolutely nothing. Because I don’t have a switch congestion problem. Neither do you. If I did, then the Xfinity upgrade would be an orthogonal problem. On top of that, my UDM Pro has never once measured my Xfinity downstream speed at greater than about 900Mbs. I suppose that could change if I had a faster interconnect between the UDM Pro and the cable modem, but I seriously doubt it.

You’re considering going deep. I wouldn’t. It’s just running in place until you decide you want to solve for higher bandwidth wired connections to your client machines, which you might never decide.

Maybe upgrade your cable modem if it’s throttled to 600Mbps but consider stopping there.

I’m curious what you’re talking about with port swapping, though. You can see how I’ve got my switch cabled to the UDM Pro. Are you already doing this or are you doing something different?

Also…look how nicely that RS1221+ pairs with the UniFi gear :slight_smile:

EA7E0F68-B73C-4B95-9E99-571974749853
EA7E0F68-B73C-4B95-9E99-5719747498534032×3024 3.72 MB

3 Likes
5

Ok jealous. I know, I know. You are def pro grade and I am thoroughly jealous.

I always know I can count on you to give me the reality check. Thank you.

So, if you want to get >1GB into the UDM Pro, you need to use one of these nifty 2.5/5/10GB SFP+ Cat 6a adapters in Port 10 of the UDM Pro. Then you need to specify Port 10 as WAN1 and force it to 10GB (the device will present as 10GB to the UDM and auto negotiate 2.5 with the modem). That in turn means first disabling Port 9 (the default WAN port) in the Unifi console. Then your Motorola will negotiate a 2.5GB link to your UDM Pro. Phew. You know all this - you said it above.

So now you can get the full 1.2 into your UDM. But big whoop, you can’t get >1GB out to a single device unless you use the SFP+ in LAN Port 11. But I don’t own any switches which have SFP+, just two 24’s which have SFP. And the UDM Pro doesn’t do LACP, so I couldn’t even get to 2GB without adding or swapping a device.

I finally got to “well, I could get a USW-Pro-48-PoE for only $1100 and replace my dual 24 port switches” and then I woke up from a bad dream.

I guess if I have devices on both switches that in total each want to consume 600MB, I could saturate the 1.2. That would be fun. And pointless. Until my wife and I are on simultaneous holographic VR zoom meetings or something.

The whole point of this was to get an LTE modem up and running so internet outages are more redundant and transparently so. I’ll get there eventually. Here’s the part you were asking about… it’ll be On Port 9, which will now be LAN2 with failover. But I’m going to try to do all the above first and get the full speed ™ onto the UDM Pro.

Boy, you’ve made me excited for the provisioning :slight_smile:

2 Likes
6

I didn’t not know all of this. As tempting as it is to fool around with this, I’m going to pass. But I’m going to file this info away in case I decide to play with it some other time. Thanks for explaining.

Good luck with the modem. I hope it’s an easier transition for you than it was for me.

2 Likes
7

And, ugh, baby, this rathole is deep.

The UDM-Pro has a 1GB backplane, so anything crossing VLANs gets bottlenecked there. This is WAY above my pay grade, so not really concerned too much about it, but it seems like the only way to get the full speed is if I have an L3 switch connected via SFP+. Not saying this is all correct below on Reddit, but if it is. I’m stuck with basically 1GB cap in any case.

1 Like
8

What’s wrong with 1GB and what’s the incremental cost and effort to get 1.2GB? Is it worth it and will it actually make a difference?

My internet is 50Mbps down and 12Mbps up and works perfectly with Roon. Just saying…

9

Sorry, this is no longer about Roon. Roon is great with my connection as is, for sure. This is just the only networking-savvy site I know where people will talk to me and I won’t be treated like an idiot because I’m figuring this all out in real time. Roon set me off on this tack of internet connection redundancy, now I’m just learning out loud. Apologies for the disruption! :slight_smile:

3 Likes
10

This isn’t going to help answer your question but you seem to be getting bytes and bits mixed up. When I first read your post I thought you were talking about monthly cable modem caps. 10GBps (or GB/sec, gigaBYTES per sec) is not the same as 10gbps (or Gbps, gigaBITS per second), not even close. I would love to have a 900GBps connection. :slight_smile:

11

Many thanks. It’s actually hilarious. I often get the harder stuff / complicated details right, and whiff on writing out the basics. Weird behavior I know. It’s how you can tell I really don’t know what I’m talking about even if I fake it pretty good :slight_smile:

2 Likes
12

Only your notation seemed to be off and you are asking the right questions.

Speaking of which, to try and answer your question. You currently have a 900mpbs connection to the Internet and thinking of upgrading to 1200mpbs (1.2gbps), right? Can you even saturate your current connection?

I find that most sites, even from big tech giants, max out at about 2-300 mbps sustained so unless you have a lot of people in your house hold downloading the latest iPhone or Windows/macOS update at the same time I doubt you could saturate your current connection (and even then would it be worth the cost just to get the latest update that much faster?). Certainly unlikely to do so with just one device.

I have 1gbps symmetrical fiber and I have yet to come close to saturating it and I am a heavy user (probably on order of double digit terabytes every month). With your hardware costs to make it happen I would say stay where you are.

One other thing to note. A 5 port switch with all 1gb ports should have a backplane that is double that (so 10bps). Why? Switched networks are full duplex unlike hubs or WiFi.

13

I haven’t tracked this stuff closely because there’s nothing about my connection to the internet or my in-home use cases that push me to care. I think the “1 gigabit” total switching capacity assertion has been debunked. So if the issue you’re concerned about is routing between the ports on the UDM Pro, then I don’t think that’s an issue.

But…I think there is more than one revision of the UDM Pro and that the switch → router bandwidth was 1Gbps in the original revision but has been upgraded to 2.5Gbps in later revision(s). I’m not motivated enough to dig into this more deeply. If you do, can you share what you learn?

See:

14

Actually, it looks more confusing than that. It looks like the first rev had a 2.5Gbps uplink to the CPU (switch → router) but they decreased that to 1Gbps in subsequent revs. So slower. But they decoupled WAN+LAN.

Good luck making your way through this, if you care enough to try :frowning:

image

https://ubntwiki.com/products/unifi/unifi_dream_machine_pro

16

No. You don’t need it.

It’s more consumption and more environmental waste.

:innocent:

1 Like
17

It’s all true. 100%.

And yet, earlier today I put a bid on a 48 Pro PoE switch with SFP+ for $400 on eBay, as I know I can sell my two 24 Port switches for more than that. Doubt I’ll get it, as that’s ~1/3 of retail. This is probably the dumbest project I have ever taken on. But not gonna let myself lose $$ or buy anything new, for roughly reasons you cite, so maybe it won’t happen.

2 Likes
18

Sometimes we do things, just because we want to. Everything doesn’t always need to make perfect sense. What fun is that?

1 Like
19

Keep us posted please!

20

Repeat after me:
“I do not need an L3 switch. I do not need an L3 switch. I do not need an L3 switch.”

21

I believe it is a violation of community guidelines to imply that this forum is intended for the discussion of necessary things. I’m tempted to report your post. Please consider this a warning. :slight_smile:

2 Likes