Your VPS Server!

Friday, September 6, 2019

The sad tale of the ISP that didn’t deploy IPv6

Once upon a time in the not so distant past, a large ISP dominated a country’s telecommunications market and felt powerful and without competition. Whenever someone needed to log on to the Internet they would use their services. Everyone envied their market penetration.


This large ISP, however, had never wanted to deploy IPv6 because they thought their stock of IP addresses was enough and saw no indicator telling them that they needed the new protocol.
During the course of those years, another smaller ISP began implementing IPv6 and slowly began to grow, as they realized that the protocol did indeed make a difference in the eyes of their clients and that it was helping them win over new users.

The small ISP’s market penetration continued to grow, as did their earnings and general respect for their services. As they grew, it became easier for them to obtain better equipment, traffic and interconnection prices. Everything was going very well. The small ISP couldn’t believe that something as simple as deploying IPv6 could be paying off so spectacularly. Their customers told them their needs included running VPNs and holding conference calls with partners in other parts of the world, and that their subsidiaries, customers and business partners in Europe and Asia had already adopted IPv6.

Despite being so powerful, the large ISP began experiencing internal problems that were neither billing nor money related. Sales staff complained that they were having trouble closing many deals because customers had started asking for IPv6 and, although their ISP was so large and important, they simply did not have IPv6 to offer. Both corporate customers and residential users were asking for IPv6; even major state tenders were requiring IPv6.

When this started happening, the Sales Manager complained to the Products, Engineering and Operations departments. The latter were left speechless and some employees were let go by the company. In the end, Sales did not care where the fault lay – they were simply unable to gain new customers. Realizing that they were losing customers, some of the salespeople accepted job offers at the small ISP who was looking to grow their staff as they could now afford the best sales force.

Then the same thing happened with the larger ISP’s network manager, an expert who knew a lot about IPv6 but who had been unable to overcome the company’s bureaucracy and bring the new protocol into production. Logically, the network manager was followed by his trusted server administrator and head of security. The large ISP couldn’t believe what was happening right before their very eyes. The sales force hired by the smaller ISP (those who used to work for the large ISP) brought with them their huge customer base, all of them potential prospects.

A stampede of the large ISP’s clients was on the way. The months went by and the smaller ISP was no longer simply offering Internet access – its Data Center had grown, major companies brought in new cache servers and much more. They were now offering co-location, hosting, virtual hosting, voice and video, among many other services.

When the large provider decided to deploy IPv6, it had to do so very quickly. Things went wrong; many errors were made. In addition, certain consultants and companies took advantage of their problems and charged higher rush fees. Network downtime increased, as did the number of calls to the call center. The large ISP’s reputation started to crumble.


As expected, in the end, everyone who was part of this story – clients and providers alike – ended up deploying IPv6. Some ended up happier than others, but everyone adopted IPv6 on their networks.

By Alejandro Acosta

Monday, July 8, 2019

BGP: To filter or not to filter by prefix size. That is the question


Introduction


In order to write these post the R+D team and the WARP team joint together after analyzing some security incidents related with BGP, network accessibility, network hijacks and network visibilities.


As you probably know, in the BGP world, there are dozen of ways to filter prefixes. The goal of this post is to show some recommendations in order to have a more stable network, keep the visibility of your prefixes as much as possible and of course reduce the calls to the NOC


Scenario

Many ISPs around the world can not (or do not wish) to receive the full routing table (DFZ) which by the time of writing this text is about 750.000 prefixes (IPv4)


  • The above description could be due to some -not limited- of the following causes:
  • The routers does not have enough RAM to learn all the prefixes (please also note that there could be also several BGP session at the same time)
  • The network admin wants to save CPU cycles in their devices
  • The upstream providers is not announcing the full routing table
  • The network admin wishes to keep his network simple and easy

Anyhow, in the end of the story, the router is not learning the full routing table.



Problem

Not learning the full routing table can bring many partial inconveniences that in the end brings connectivity problems, users complaints, issues accessing some web sites and more.



Why?

Please try to imagine the following case



  1. I have a router (property of EXAMPLE) in Internet that is ONLY learning a partial DFZ
  2. The routers mentioned in “1” is only learning “big network”, over /20. I mean, the router learns /20, /19, /18, etc. (of course, we are talking about IPv4)
  3. Based in the configuration indicated in number “2”, the router is not going to learn prefixes such as /21, /22, /23 nor /24
  4. So far so good. However, somewhere in Internet, some people hijacked an /21 to the company ACME (hijack, bad configuration, whatever)
  5. ACME decides to perform more specific prefix advertisements, so, he takes his /21 and announces 8 /24 prefixes to the DFZ.
  6. Because of the filters configured by EXAMPLE, he will never learn the legitimate /24 prefixes advertised by ACME
  7. EXAMPLE will keep learning the hijacked /21 which obviously will bring connectivity problems aim to the legitimate owner of the network




Topology

The following diagram represents the hypothesis presented in the previous point in a graphic manner to facilitate its understanding.








Recommendation

The following recommendations only for networks that CAN NOT LEARN the full DFZ. One more time, they were found after studying many cases of connectivity problem and network hijacks.:


  • Do not filter more specific network,. We mean, it’s better to learn more specific networks such as: /24, /23, /22 (IPv4 world)
  • Filter using AS_PATH (like, 2,3 or 4 deep ASs)
  • Please create your ROS and use RPKI



Some examples (Cisco like)1. Learning only /22, /23 or /24:


router bgp 65002
  neighbor 10.0.0.1 remote-as 65001
  neighbor 10.0.0.1 route-map FILTRO-IN in
!
ip prefix-list SMALLNETWORKS seq 5 permit 0.0.0.0/0 ge 22 le 24
!
route-map FILTRO-IN permit 10
  match ip address prefix-list SMALLNETWORKS
!



2. Only learning two AS beyond us:

router bgp 65001
  neighbor 10.0.0.2 remote-as 65002
  neighbor 10.0.0.2 route-map ASFILTER-IN in


!
ip as-path access-list 5 permit ^[0-9]+_$
ip as-path access-list 5 permit ^[0-9]+ [0-9]+_$
!
route-map ASFILTER-IN permit 10
  match as-path 5
!



More information

Prefix hijack demonstration 1 / 2 (in Spanish):
https://www.youtube.com/watch?v=X5RNSs8y8Ao&t=39s


Prefix hijack demonstration 2/2 (in Spanish):
https://www.youtube.com/watch?v=m51WtuEZOKI


BGP Prefix-Based Outbound Route Filtering
http://www.cisco.com/c/en/us/td/docs/ios/12_2s/feature/guide/fsbgporf.html


Ejemplo de configuración de BGP con dos prestadores de servicio diferentes (conexiones múltiples)
http://www.cisco.com/cisco/web/support/LA/7/75/75930_27.html


Certificación de Recursos (RPKI)
http://www.lacnic.net/web/lacnic/certificacion-de-recursos-rpki


Información General sobre Certificación de Recursos (RPKI)
http://www.lacnic.net/web/lacnic/informacion-general-rpki




Authors:

Dario Gomez (https://twitter.com/daro_ua)

Alejandro Acosta (https://twitter.com/ITandNetworking)