JUNIPER JNCIP-SP: USING INDEPENDENT-DOMAIN TO TUNNEL CUSTOMER BGP VPN PREFIXES

Let’s imagine we’re an internet provider, with an autonomous system number of 69. Nice! Nice.

Now let’s further imagine we have an MPLS L3VPN customer, who will talk BGP with us. Boy oh boy, this keeps getting better and better.

Usually the customer would talk eBGP to the service provider. However, there is one downside to this: if Site A wanted to set the Local Preference on their advertisements, this wouldn’t be seen by the ISP: the two organisations are in a different autonomous system, after all. There’s ways around it, for example Site A could could add a community that the customer asks (pays) the ISP to not remove, so that Site B can see the community and set the local pref accordingly. Still, it would be nice if there was an easier way, like just talking internal BGP.

Another scenario: the customer might simply turn around to us and say this: “I don’t want to see 69 in the AS-PATH of the prefixes I receive. The number 69 is not, as you claim, “nice”: it’s rude, and I’ll have no rudeness in my network, thank you very much.” Of course, legally this is enough to make any customer contract null and void. That’s definitely true, in all countries. Don’t look it up.

But let’s suppose we want to honour the customer’s request, no matter how wrong they are. What if the customer doesn’t want to see the ISP’s AS number in the ASPATH at all?

To achieve either or both of these things, we can use a command called independent-domain.

 

WHAT DOES INDEPENDENT-DOMAIN DO?

This command does something interesting. As far as the customer’s on-site routers (CPEs) are concerned, they now have an internal BGP peering to the ISP. The CPEs see the ISP as being in the same AS as themselves.

But in reality, the ISP is doing something cheeky: it’s taking the BGP prefixes it learns from the CPE, and tunneling the NLRI (Network Layer Reachability Information – the various BGP attributes of the prefix) across the service provider’s MPLS network, unchanged. As a result, the Provider Edge (PE) routers can pass on the BGP advertisement to the customer, unchanged. Now, if Site A sets a localpref, Site B receives that localpref. Cool!

If you’re running a full-mesh topology, configuring it is super easy: all the service provider has to do is add independent-domain on the relevant VRFs. However, if you’re running hub-and-spoke, there’s something extra to consider. Let’s take a look at both.

 

OUR TOPOLOGY TODAY

This is actually a topology I borrowed from an earlier post about hub and spoke VPNs. Today we’re just interested in the three sites in yellow, belonging to Customer A.

Notice that Site 1 has a LAN prefix of 192.168.10.0/24. In this post we’re going to be following this prefix on the journey of a lifetime.

In the first full-mesh scenario we’ll talk about, we’ll just be watching this prefix go from CPE1 to CPE2. In this scenario, there’s just one single logical link between CPE2 and PE2.

The second scenario will be hub-and-spoke, so later on we’ll watch how the advertisement goes from CPE1 to CPE2, and then from CPE2 to CPE3. In this scenario, there’s one physical link between CPE2 and PE2, split into two logical sub-interfaces. If you’re unsure how or why we’d do that, give that hub-and-spoke VPN post a read. But don’t worry, you won’t need to know about that to understand this post.

 

INDEPENDENT-DOMAIN IN A FULL-MESH TOPOLOGY

Let’s see what the config looks like for a full-mesh L3VPN, starting with the PE routers. The config below is from PE1, but it’ll be broadly the same on any of the PE routers in this topology.

What do we see in this config? It’s mostly the usual stuff:

• We’ve got a VRF called CUSTOMER_A
• There’s a route distinguisher
• An interface
• A policy for the route targets
• And some BGP to the customer

But in red you’ll notice one extra bit: the independent-domain command! Notice also that we’re setting the VRF itself to be using AS64512, as opposed to normal where the ISP’s own AS would be used by default:

root@Router_1_PE> show configuration routing-instances CUSTOMER_A
instance-type vrf;
interface ge-0/0/0.0;
route-distinguisher 1.1.1.1:1;
vrf-import CUSTOMER_A_IMPORT;
vrf-export CUSTOMER_A_EXPORT;
routing-options {
    autonomous-system 64512 independent-domain;
}
protocols {
    bgp {
        group CUSTOMER_A_CPE_1 {
            type internal;
            peer-as 64512;
            neighbor 10.10.250.1;
}}}

Now let’s look at CPE1. We set the BGP peering to a type of “internal”, and we set the peer-as like this:

root@CUSTOMER_A_CPE_1> show configuration protocols bgp
group TO_ISP {
    type internal;
    export EXPORT_CONNECTED;
    peer-as 64512;
    neighbor 10.10.250.0;
}

To see the effect of independent-domain, let’s change the local-preference that CPE 1 advertises. Let’s set it to 666. CPE1’s export policy looks like this:

root@CUSTOMER_A_CPE_1> show configuration policy-options
policy-statement EXPORT_CONNECTED {
    from protocol direct;
    then {
        community add ORIGIN_COMMUNITY_CUSTOMER_A_CPE_1;
        local-preference 666;
        accept;
}}

Let’s hope back to the service provider side. We see that PE learns the prefix as normal. Note the AS path: it’s just internal. And look at our local pref: 666! Delightfully devilish, Seymour!

root@Router_1_PE> show route receive-protocol bgp 10.10.250.1 table CUSTOMER_A.inet.0

CUSTOMER_A.inet.0: 6 destinations, 7 routes (6 active, 0 holddown, 0 hidden)
  Prefix   Nexthop       MED     Lclpref    AS path
  10.10.250.0/31          10.10.250.1                  666        I
* 192.168.10.0/24         10.10.250.1                  666        I

Let’s follow the 192.168.10.0/24 advertisement. In our lab, the prefix gets sent to a route reflector called PE 4, who reflects it to PE 2. How does PE2 see this prefix?

root@Router_2_PE> show route receive-protocol bgp 4.4.4.4 table VRF_CUSTOMER_A_SPOKES_ADVERTISING_TO_HUB.inet.0 192.168.10.0/24

VRF_CUSTOMER_A_SPOKES_ADVERTISING_TO_HUB.inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)

  Prefix   Nexthop       MED     Lclpref    AS path
* 192.168.10.0/24         1.1.1.1                      100        I

Wait: why is the local pref 100? Didn’t we set it to 666?

Remember, the attributes are being tunneled through the ISP’s network. So yes: in AS69 (nice), the localpref is indeed 100. But when it gets advertised to the CPE at site 2, the localpref is restored:

root@CUSTOMER_A_CPE_2> show route receive-protocol bgp 10.10.250.100 192.168.10.0/24

inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)

  Prefix   Nexthop       MED     Lclpref    AS path
* 192.168.10.0/24         10.10.250.100                666        I

There it is: 666. The devil is in the detail!

 

INDEPENDENT-DOMAIN IN A HUB-AND-SPOKE TOPOLOGY

Now, if this were full-mesh, that would be the end of the story. Hunky dory! But what if this were hub-and-spoke? What if Site 1 and Site 3 were spokes, and Site 2 were the hub? Well, Site 1 would advertise the prefix to the PE as normal, and the PE would send it on to the hub, Site 2.

Now at this stage, we’d want Site 2 to somehow then re-send it on to Site 3. This is hub-and-spoke, after all.

But remember the rule of iBGP: if you learn a prefix by iBGP, don’t re-advertise it to other iBGP neighbours. And there’s the problem: as far as the hub is concerned, it’s now learning prefixes by iBGP – and therefore, it won’t re-advertise it back to the spokes, because the hub sees all is peerings as iBGP now!

If we reconfigure the network to be hub and spoke, CPE2’s advertisements to PE2 now look like this: the only things it’s advertising are its local prefixes. Our lovely 192.168.10.0/24 prefix isn’t being advertised at all. CPE2 learned this by iBGP, so it doesn’t get re-advertised.

root@CUSTOMER_A_CPE_2> show route advertising-protocol bgp 10.10.250.102

inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
  Prefix   Nexthop       MED     Lclpref    AS path
* 10.10.250.100/31        Self                         100        I
* 10.10.250.102/31        Self                         100        I
* 192.168.20.0/24         Self                         100        I

So, how do we go about fixing this. Perhaps we can think of one special situation where the usual rules of iBGP are broken. Hmm… is the special situation perhaps… the Queen’s Birthday? No, silly: I’m talking about route reflectors!

Have you ever configured a customer CPE to act as a route reflector? No? Well today’s your lucky day, Susan.

Our new BGP config on the hub CPE looks like this. Remember that in this scenario, the link between PE2 and CPE2 is made up of two sub-interfaces: one for BGP advertisements coming in, and one for BGP advertisements going back out again.

In the configuration below I have one BGP group called “TO_ISP”. Both neighbors are in this group, but notice that they’ve got different export policies. The post dedicated to hub-and-spoke explains why that is. I gave each neighbour a description too, to make it a bit easier to read.

But crucially, we add a cluster ID to the group as a whole. And as such, prefixes that CPE2 receives from one iBGP peer will now get re-advertised to other iBGP peers, with the cluster ID attached.

root@CUSTOMER_A_CPE_2> show configuration protocols bgp
group TO_ISP {
    type internal;
    cluster 69.69.69.69;
    peer-as 64512;
    neighbor 10.10.250.100 {
        description "TO ISP, SPOKE PREFIXES IN";
        export POLICY_NOTHING_EXPORTED;
    }
    neighbor 10.10.250.102 {
        description "TO ISP, HUB AND SPOKE PREFIXES OUT";
        import POLICY_NOTHING_IMPORTED;
        export EXPORT_CONNECTED;
}}

Did it work? Is the hub CPE 2 now advertising CPE 1’s LAN range back to PE 2? Let’s find out by going straight to CPE 3, and see if it sees anything for CPE 1’s LAN.

root@CUSTOMER_A_CPE_3> show route 192.168.10.0/24

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:01:53, localpref 666
                      AS path: I
                    > to 10.10.250.6 via ge-0/0/0.0

Well look at that – indeed it does! And, here’s the magic: the AS-PATH is still fully internal! Our localpref of 666 is even still there. A job well done! Please email my boss and tell him to give me a massive pay rise. I mean MASSIVE.

 

FUN STORY

I once worked at an ISP where every single customer was configured with independent-domain. I queried why this was, and a senior engineer explained to me that “if you turn it on for just one customer, it breaks every customer.”

Hmm… that didn’t sound right. So, I tested this in my lab. Initially, VPN Customer A and VPN Customer B were not running independent-domain. I ran a constant rapid ping on our other customer from Customer B’s CPE 1 to Customer B’s CPE 3. While that ping was going, I reconfigured all of Customer A to be independent-domain. I checked back on Customer B’s ping… and not a single packet was lost.

So, either that engineer had made a mistake somewhere, or there was a bug in Junos. Which to be fair, isn’t the most outlandish idea in the world. In any case, you can definitely run some customers as independent-domain, and some without. Which makes sense: if you could only run all customers this way or no customers this way, then that would be a very strange and unhelpful restriction!

 

THAT’S IT!

Do you use independent-domain in your network? Do you have any stories of using it in production? Be sure to comment below if you do!

If you enjoyed this post, I’d love you to share it on your favourite social media. The more readers I get, the more I’m inspired to write even more posts to help you on your studies.

If you’re on Mastodon, follow me to find out when I make new posts. I’ve got all sorts of bad opinions, and you can be among the very first people to hear them. (2024 edit: I’m also on BlueSky nowadays too. I was once on Twitter, but I’ve given up on it, on account of… well, I don’t need to finish that sentence, do I.)

And if you fancy some more learning, take a look through my other posts. I’ve got plenty of cool new networking knowledge for you on this website, especially covering Juniper tech and service provider goodness.

It’s all free for you, although I’ll never say no to a donation. This website is 100% a non-profit endeavour, in fact it costs me money to run. I don’t mind that one bit, but it would be cool if I could break even on the web hosting, and the licenses I buy to bring you this sweet sweet content.

Thank you very much for reading!