A QUIRK OF TRACEROUTE THAT YOU’LL WANT TO KNOW ABOUT

Most network folks know how traceroute works: by manipulating a feature in the IPv4 header called the Time To Live – the most threateningly-named portion of the TCP/IP stack! The only way it could be worse is if they’d called it the “Time To Die”.

The TTL value in the IPv4 header (or the Hop Limit value in the IPv6 header) goes down by 1 as it passes from one router to the next. If the TTL ever reaches zero then the device that sets the TTL to zero will (hopefully!) send an ICMP “Time Exceeded: TTL expired in transit” message back to the source.

Traceroute takes advantage of this: it starts by sending a packet with a TTL of 1, and as such our immediate next-hop is forced to reply, because it will set the TTL to zero. Voila: by forcing it to reply, we’ve just discovered our first hop in the chain! We then send another packet, this time with a TTL of 2. Our immediate next-hop receives the packet, sets the TTL to 1, passes it to the second router in the chain, who sets the TTL to zero, and then sends it own TTL expired message. Voila: we now know the second hop in the chain! Gosh, it’s almost like magic!

It’s important to be aware that the routers in the path don’t *have* to respond. To quote RFC 792, which is the RFC for ICMP: “If the gateway processing a datagram finds the time to live field is zero it must discard the datagram. The gateway may also notify the source host via the time exceeded message.” Notice the words “must” and “may” in that quote. Don’t be fooled if you don’t get a response from a hop in the chain of a traceroute: it could just be that the device has been set up to not respond.

You could argue it’s a good security practice to not respond, because it stops people working out what devices are on a network. An alternative view is that if you turn this stuff off, you make it harder to troubleshoot. You will of course have your own opinions on this, and whatever they are, I’m sure that you’re correct, and everyone else is wrong. Well done to you for being so clever!

There’s a chance you knew everything I’ve said so far already. Here’s something you might not have known though. We’ve talked so far about how devices send out a “packet” when performing a traceroute. But what does that mean? What’s actually in that packet? Many people assume it’s just a ping. And on Windows this is indeed the case: traceroutes from Windows machines send out an ICMP Echo Request.

But did you know that on Linux and Unix machines (like MacOS), the default is to actually send a UDP packet on ports 33434 to 33534? Cisco also use UDP 33434, but the port goes up with each hop. As for Juniper, it’s FreeBSD under the hood, so it too uses UDP.

Why does this matter? Because it can have an impact on the results! If a device is in front of a firewall, or if the device itself has any kind of firewall filtering functionality, then it could feasibly be allowing ICMP but denying whatever UDP port the traceroute traffic comes in on. And this is exactly the scenario a customer of ours reported recently.

Below I’m doing a traceroute to a real-world IP address. However, I’ve taken the real-world IPs out of the traceroute output, for obvious reasons, and I’ve replaced the final three hops with the obviously fake and not-legitimate IPs address of 444.444.444.444, 555.555.555.555, and 666.666.666.666. Three addresses guaranteed to not exist in the real world!

Here’s what a traceroute to our alleged 666.666.666.666 address looks like by default from my Macbook. The traceroute is going out as a UDP packet, and the very final hop doesn’t reply.

Chris-Parkers-MacBook-2:~ chrisparker$ traceroute 666.666.666.666
traceroute to 666.666.666.666 (666.666.666.666), 64 hops max, 52 byte packets
 1 10.8.159.254 (10.8.159.254)  1.487 ms  1.100 ms  1.126 ms
 2 213.86.87.226 (213.86.87.226)  1.502 ms  1.520 ms  1.368 ms
 3 80.169.31.29 (80.169.31.29)  2.176 ms  2.083 ms  2.080 ms
 4 212.36.135.119 (212.36.135.119)  3.725 ms  3.752 ms  3.705 ms
 5 212.36.135.119 (212.36.135.119)  3.516 ms  3.581 ms  5.666 ms
 6 ge-2-1-0.mpr1.lhr2.uk.above.net (195.66.224.76)  8.533 ms  5.862 ms  5.485 ms
 7 ae13.mpr3.lhr3.uk.zip.zayo.com (64.125.30.54)  3.944 ms  4.163 ms  4.050 ms
 8 4444.4444.4444.uk.zip.zayo.com (444.444.444.444)  4.635 ms  4.481 ms  4.311 ms
 9 555.555.555.555.ipyx-xxxxxx-zyo.above.net (555.555.555.555)  5.074 ms  5.224 ms  4.614 ms
 10 * * *
 11 * * *
 12 * * *
 13 *^C
Chris-Parkers-MacBook-2:~ chrisparker$

You wouldn’t know it from the traceroute above, but I can tell you for fact that the final hop is up, and live – but isn’t accepting UDP messages. As such, the end device never even gets to a stage where it can process the packet and set the TTL from 1 to 0. And so, no ICMP TTL exceeded message is sent back to the source.

Notice how the traceroute carries on past line 10, which is where the end destination should live. Why is this? Our source machine doesn’t realise it’s already reached the end destination, so it sends out yet another set of three packets, this time with the TTL set one higher than before. By the time it reaches our end destination, the TTL will actually be 2 – but it doesn’t matter, because this packet is still being dropped. It’s UDP, and just like the last packet it isn’t allowed.

Similarly, the packet at line 12 would have had a TTL of 3. Our plucky source machine just assumes that if a hop in the chain doesn’t respond, there could well be other stuff beyond it. Our source device doesn’t know that it is in fact at the end destination. Bless!

Now let’s see what happens when I use the -I (capital i, not lowercase L!) switch, to force my Mac to use ICMP. Previously, line 10 was where things ended. Will we get a different result this time? (Spoiler alert: yes!)

Chris-Parkers-MacBook-2:~ chrisparker$ traceroute -I 666.666.666.666
traceroute to 666.666.666.666 (666.666.666.666), 64 hops max, 72 byte packets
 1 10.8.159.254 (10.8.159.254)  2.273 ms  1.233 ms  3.960 ms
 2 213.86.87.226 (213.86.87.226)  1.570 ms  1.516 ms  1.315 ms
 3 80.169.31.29 (80.169.31.29)  2.111 ms  1.938 ms  1.911 ms
 4 212.36.135.119 (212.36.135.119)  3.765 ms  3.903 ms  3.689 ms
 5 212.36.135.119 (212.36.135.119)  3.616 ms  3.493 ms  3.319 ms
 6 ge-2-1-0.mpr1.lhr2.uk.above.net (195.66.224.76)  3.565 ms  3.470 ms  7.246 ms
 7 ae13.mpr3.lhr3.uk.zip.zayo.com (64.125.30.54)  3.865 ms  3.724 ms  3.667 ms
 8 4444.4444.4444.uk.zip.zayo.com (444.444.444.444)  4.348 ms  4.440 ms  4.219 ms
 9 555.555.555.555.ipyx-xxxxxx-zyo.above.net (555.555.555.555)  4.818 ms  5.503 ms  4.368 ms
 10 666.666.666.666 (666.666.666.666)  6.876 ms  5.495 ms  7.177 ms
Chris-Parkers-MacBook-2:~ chrisparker$

The traceroute actually completed! Our box accepted ICMP traffic, and as such it replied.

The problem our customer mentioned was that they thought there was a routing problem, because “sometimes the last hop responded, and sometimes it didn’t”, so they said. Turns out that it “wasn’t working” from a Linux box, but it was from a Windows box.

Ultimately they could always ping the end destination, so I was able to reassure them that everything was working as expected. But when the ping didn’t convince them, showing them these two traceroutes did the trick. A bit of extra knowledge about the inner-workings of the protocol helped me to reassure them just a little bit more. I hope you find this info helpful too!

While we’re here, let’s end with another bit of useful traceroute info. Heed these words: slow response times from a device don’t necessarily indicate slow speeds on the line. It could very likely indicate that the control plane of the box is doing a lot of hard work, and so isn’t prioritising the generation of TTL exceeded messages. Transit traffic (ie traffic passing through the router, in the data plane) could be absolutely fine! If one single hop has high response times, but all the hops after it have speeds that you’d expect, then this is almost definitely the case.

Be careful with traceroute: it doesn’t always tell you what you think it’s telling you. If you want to know more, there’s an hour-long video on YouTube by an engineer called Richard Steenbergen, who’ll tell you all about it. It’s well worth your time if you want to know more!

Hey there: thanks for reading this! If you’re on Mastodon, follow me to find out when I make new posts. (2024 edit: I’m also on BlueSky nowadays too. I was once on Twitter, but I’ve given up on it, on account of… well, I don’t need to finish that sentence, do I.)

If you enjoyed this post, of course I’d love you to share it on your favourite social media of choice. Go on: be the hero you wish existed in the world!

And if you fancy some more learning, take a look through my other posts. I’ve got plenty of cool new networking knowledge for you on this website, especially covering Juniper tech and service provider goodness.

It’s all free for you, although I’ll never say no to a donation. This website is 100% a non-profit endeavour, in fact it costs me money to run. I don’t mind that one bit, but it would be cool if I could break even on the web hosting, and the licenses I buy to bring you this sweet sweet content.