Friday, September 6, 2019

The sad tale of the ISP that didn’t deploy IPv6

Once upon a time in the not so distant past, a large ISP dominated a country’s 
telecommunications market and felt powerful and without competition. Whenever someone 
needed to log on to the 
Internet they would use their services. Everyone envied their market penetration.

This large ISP, however, had never wanted to deploy IPv6 because they thought their stock
 of IP addresses was enough and saw no indicator telling them that they needed the new 
protocol.
During the course of those years, another smaller ISP began implementing IPv6 and slowly 
began to grow, as they realized that the protocol did indeed make a difference in the eyes of their 
clients and that it was helping them win over new users.

The small ISP’s market penetration continued to grow, as did their earnings and general respect
 for their services. As they grew, it became easier for them to obtain better equipment, traffic 
and interconnection prices. Everything was going very well. The small ISP couldn’t believe that
something as simple as deploying IPv6 could be paying off so spectacularly. Their customers
 told them their needs included running VPNs and holding conference calls with partners in 
other parts of the world, and that their subsidiaries, customers and business partners in Europe
 and Asia had already adopted IPv6.

Despite being so powerful, the large ISP began experiencing internal problems that were 
neither billing nor money related. Sales staff complained that they were having trouble closing 
many deals because customers had started asking for IPv6 and, although their ISP was so
 large and important, they simply did not have IPv6 to offer. Both corporate customers and 
residential users were asking for IPv6; even major state tenders were requiring IPv6.

When this started happening, the Sales Manager complained to the Products, Engineering and 
Operations departments. The latter were left speechless and some employees were let go by 
the company. In the end, Sales did not care where the fault lay – they were simply unable to 
gain new customers. Realizing that they were losing customers, some of the salespeople 
accepted job offers at the small ISP who was looking to grow their staff as they could now 
afford the best sales force. 

Then the same thing happened with the larger ISP’s network manager, an expert who knew 
a lot about IPv6 but who had been unable to overcome the company’s bureaucracy and bring
 the new protocol into production. Logically, the network manager was followed by his trusted 
server administrator and head of security. The large ISP couldn’t believe what was happening
 right before their very eyes. The sales force hired by the smaller ISP (those who used to 
work for the large ISP) brought with them their huge customer base, all of them potential prospects.

A stampede of the large ISP’s clients was on the way. The months went by and the smaller
 ISP was no longer simply offering Internet access – its Data Center had grown, major 
companies brought in new cache servers and much more. They were now offering co-location, 
hosting, virtual hosting, voice and video, among many other services.

When the large provider decided to deploy IPv6, it had to do so very quickly. Things went 
wrong; many errors were made. In addition, certain consultants and companies took 
advantage of their problems and charged higher rush fees. Network downtime increased,
 as did the number of calls to the call center. The large ISP’s reputation started to crumble.


As expected, in the end, everyone who was part of this story – clients and providers alike – 
ended up deploying IPv6. Some ended up happier than others, but everyone adopted IPv6
 on their networks.

By Alejandro Acosta

Monday, July 8, 2019

BGP: To filter or not to filter by prefix size. That is the question


Introduction


In order to write these post the R+D team and the WARP team joint together after analyzing some security incidents related with BGP, network accessibility, network hijacks and network visibilities.


As you probably know, in the BGP world, there are dozen of ways to filter prefixes. The goal of this post is to show some recommendations in order to have a more stable network, keep the visibility of your prefixes as much as possible and of course reduce the calls to the NOC


Scenario

Many ISPs around the world can not (or do not wish) to receive the full routing table (DFZ) which by the time of writing this text is about 750.000 prefixes (IPv4)


  • The above description could be due to some -not limited- of the following causes:
  • The routers does not have enough RAM to learn all the prefixes (please also note that there could be also several BGP session at the same time)
  • The network admin wants to save CPU cycles in their devices
  • The upstream providers is not announcing the full routing table
  • The network admin wishes to keep his network simple and easy

Anyhow, in the end of the story, the router is not learning the full routing table.



Problem

Not learning the full routing table can bring many partial inconveniences that in the end brings connectivity problems, users complaints, issues accessing some web sites and more.



Why?

Please try to imagine the following case



  1. I have a router (property of EXAMPLE) in Internet that is ONLY learning a partial DFZ
  2. The routers mentioned in “1” is only learning “big network”, over /20. I mean, the router learns /20, /19, /18, etc. (of course, we are talking about IPv4)
  3. Based in the configuration indicated in number “2”, the router is not going to learn prefixes such as /21, /22, /23 nor /24
  4. So far so good. However, somewhere in Internet, some people hijacked an /21 to the company ACME (hijack, bad configuration, whatever)
  5. ACME decides to perform more specific prefix advertisements, so, he takes his /21 and announces 8 /24 prefixes to the DFZ.
  6. Because of the filters configured by EXAMPLE, he will never learn the legitimate /24 prefixes advertised by ACME
  7. EXAMPLE will keep learning the hijacked /21 which obviously will bring connectivity problems aim to the legitimate owner of the network




Topology

The following diagram represents the hypothesis presented in the previous point in a graphic manner to facilitate its understanding.








Recommendation

The following recommendations only for networks that CAN NOT LEARN the full DFZ. One more time, they were found after studying many cases of connectivity problem and network hijacks.:


  • Do not filter more specific network,. We mean, it’s better to learn more specific networks such as: /24, /23, /22 (IPv4 world)
  • Filter using AS_PATH (like, 2,3 or 4 deep ASs)
  • Please create your ROS and use RPKI



Some examples (Cisco like)1. Learning only /22, /23 or /24:


router bgp 65002
  neighbor 10.0.0.1 remote-as 65001
  neighbor 10.0.0.1 route-map FILTRO-IN in
!
ip prefix-list SMALLNETWORKS seq 5 permit 0.0.0.0/0 ge 22 le 24
!
route-map FILTRO-IN permit 10
  match ip address prefix-list SMALLNETWORKS
!



2. Only learning two AS beyond us:

router bgp 65001
  neighbor 10.0.0.2 remote-as 65002
  neighbor 10.0.0.2 route-map ASFILTER-IN in


!
ip as-path access-list 5 permit ^[0-9]+_$
ip as-path access-list 5 permit ^[0-9]+ [0-9]+_$
!
route-map ASFILTER-IN permit 10
  match as-path 5
!



More information

Prefix hijack demonstration 1 / 2 (in Spanish):
https://www.youtube.com/watch?v=X5RNSs8y8Ao&t=39s


Prefix hijack demonstration 2/2 (in Spanish):
https://www.youtube.com/watch?v=m51WtuEZOKI


BGP Prefix-Based Outbound Route Filtering
http://www.cisco.com/c/en/us/td/docs/ios/12_2s/feature/guide/fsbgporf.html


Ejemplo de configuración de BGP con dos prestadores de servicio diferentes (conexiones múltiples)
http://www.cisco.com/cisco/web/support/LA/7/75/75930_27.html


Certificación de Recursos (RPKI)
http://www.lacnic.net/web/lacnic/certificacion-de-recursos-rpki


Información General sobre Certificación de Recursos (RPKI)
http://www.lacnic.net/web/lacnic/informacion-general-rpki




Authors:

Dario Gomez (https://twitter.com/daro_ua)

Alejandro Acosta (https://twitter.com/ITandNetworking)