Friday, September 6, 2019

The sad tale of the ISP that didn’t deploy IPv6

Once upon a time in the not so distant past, a large ISP dominated a country’s 
telecommunications market and felt powerful and without competition. Whenever someone 
needed to log on to the 
Internet they would use their services. Everyone envied their market penetration.

This large ISP, however, had never wanted to deploy IPv6 because they thought their stock
 of IP addresses was enough and saw no indicator telling them that they needed the new 
During the course of those years, another smaller ISP began implementing IPv6 and slowly 
began to grow, as they realized that the protocol did indeed make a difference in the eyes of their 
clients and that it was helping them win over new users.

The small ISP’s market penetration continued to grow, as did their earnings and general respect
 for their services. As they grew, it became easier for them to obtain better equipment, traffic 
and interconnection prices. Everything was going very well. The small ISP couldn’t believe that
something as simple as deploying IPv6 could be paying off so spectacularly. Their customers
 told them their needs included running VPNs and holding conference calls with partners in 
other parts of the world, and that their subsidiaries, customers and business partners in Europe
 and Asia had already adopted IPv6.

Despite being so powerful, the large ISP began experiencing internal problems that were 
neither billing nor money related. Sales staff complained that they were having trouble closing 
many deals because customers had started asking for IPv6 and, although their ISP was so
 large and important, they simply did not have IPv6 to offer. Both corporate customers and 
residential users were asking for IPv6; even major state tenders were requiring IPv6.

When this started happening, the Sales Manager complained to the Products, Engineering and 
Operations departments. The latter were left speechless and some employees were let go by 
the company. In the end, Sales did not care where the fault lay – they were simply unable to 
gain new customers. Realizing that they were losing customers, some of the salespeople 
accepted job offers at the small ISP who was looking to grow their staff as they could now 
afford the best sales force. 

Then the same thing happened with the larger ISP’s network manager, an expert who knew 
a lot about IPv6 but who had been unable to overcome the company’s bureaucracy and bring
 the new protocol into production. Logically, the network manager was followed by his trusted 
server administrator and head of security. The large ISP couldn’t believe what was happening
 right before their very eyes. The sales force hired by the smaller ISP (those who used to 
work for the large ISP) brought with them their huge customer base, all of them potential prospects.

A stampede of the large ISP’s clients was on the way. The months went by and the smaller
 ISP was no longer simply offering Internet access – its Data Center had grown, major 
companies brought in new cache servers and much more. They were now offering co-location, 
hosting, virtual hosting, voice and video, among many other services.

When the large provider decided to deploy IPv6, it had to do so very quickly. Things went 
wrong; many errors were made. In addition, certain consultants and companies took 
advantage of their problems and charged higher rush fees. Network downtime increased,
 as did the number of calls to the call center. The large ISP’s reputation started to crumble.

As expected, in the end, everyone who was part of this story – clients and providers alike – 
ended up deploying IPv6. Some ended up happier than others, but everyone adopted IPv6
 on their networks.

By Alejandro Acosta

Monday, July 8, 2019

BGP: To filter or not to filter by prefix size. That is the question


In order to write these post the R+D team and the WARP team joint together after analyzing some security incidents related with BGP, network accessibility, network hijacks and network visibilities.

As you probably know, in the BGP world, there are dozen of ways to filter prefixes. The goal of this post is to show some recommendations in order to have a more stable network, keep the visibility of your prefixes as much as possible and of course reduce the calls to the NOC


Many ISPs around the world can not (or do not wish) to receive the full routing table (DFZ) which by the time of writing this text is about 750.000 prefixes (IPv4)

  • The above description could be due to some -not limited- of the following causes:
  • The routers does not have enough RAM to learn all the prefixes (please also note that there could be also several BGP session at the same time)
  • The network admin wants to save CPU cycles in their devices
  • The upstream providers is not announcing the full routing table
  • The network admin wishes to keep his network simple and easy

Anyhow, in the end of the story, the router is not learning the full routing table.


Not learning the full routing table can bring many partial inconveniences that in the end brings connectivity problems, users complaints, issues accessing some web sites and more.


Please try to imagine the following case

  1. I have a router (property of EXAMPLE) in Internet that is ONLY learning a partial DFZ
  2. The routers mentioned in “1” is only learning “big network”, over /20. I mean, the router learns /20, /19, /18, etc. (of course, we are talking about IPv4)
  3. Based in the configuration indicated in number “2”, the router is not going to learn prefixes such as /21, /22, /23 nor /24
  4. So far so good. However, somewhere in Internet, some people hijacked an /21 to the company ACME (hijack, bad configuration, whatever)
  5. ACME decides to perform more specific prefix advertisements, so, he takes his /21 and announces 8 /24 prefixes to the DFZ.
  6. Because of the filters configured by EXAMPLE, he will never learn the legitimate /24 prefixes advertised by ACME
  7. EXAMPLE will keep learning the hijacked /21 which obviously will bring connectivity problems aim to the legitimate owner of the network


The following diagram represents the hypothesis presented in the previous point in a graphic manner to facilitate its understanding.


The following recommendations only for networks that CAN NOT LEARN the full DFZ. One more time, they were found after studying many cases of connectivity problem and network hijacks.:

  • Do not filter more specific network,. We mean, it’s better to learn more specific networks such as: /24, /23, /22 (IPv4 world)
  • Filter using AS_PATH (like, 2,3 or 4 deep ASs)
  • Please create your ROS and use RPKI

Some examples (Cisco like)1. Learning only /22, /23 or /24:

router bgp 65002
  neighbor remote-as 65001
  neighbor route-map FILTRO-IN in
ip prefix-list SMALLNETWORKS seq 5 permit ge 22 le 24
route-map FILTRO-IN permit 10
  match ip address prefix-list SMALLNETWORKS

2. Only learning two AS beyond us:

router bgp 65001
  neighbor remote-as 65002
  neighbor route-map ASFILTER-IN in

ip as-path access-list 5 permit ^[0-9]+_$
ip as-path access-list 5 permit ^[0-9]+ [0-9]+_$
route-map ASFILTER-IN permit 10
  match as-path 5

More information

Prefix hijack demonstration 1 / 2 (in Spanish):

Prefix hijack demonstration 2/2 (in Spanish):

BGP Prefix-Based Outbound Route Filtering

Ejemplo de configuración de BGP con dos prestadores de servicio diferentes (conexiones múltiples)

Certificación de Recursos (RPKI)

Información General sobre Certificación de Recursos (RPKI)


Dario Gomez (

Alejandro Acosta (

Friday, July 13, 2018

How to run Flask framework to listen in both IPv4 and IPv6 (DualStack)

  How to run Flask framework to listen in both IPv4 and IPv6 (DualStack)


Start your machine learning or AI journey with Runpod. If you need affordable GPU rental , RunPod has the lowest prices. Rent a NVIDIA RTX A6000 with 48GB VRAM, or other models like NVIDIA 3090.

Thursday, August 24, 2017

Google DNS --- Figuring out which DNS Cluster you are using

(this is -almost- a copy / paste of an email sent by Erik Sundberg to nanog mailing list on 
August 23).

This post is being posted with his explicit permission.
I sent this out on the outage list, with a lots of good feedback sent to me. So I figured it 
would be useful to share the information on nanog as well. A couple months ago had to 
troubleshoot a google DNS issue with Google’s NOC. Below is some helpful information 
on how to determine which DNS Cluster you are going to. Let’s remember that Google runs 
DNS Anycast for DNS queries to and Anycast routes your DNS queries to 
the closes DNS cluster based on the best route / lowest metric to Google 
has deployed multiple DNS clusters across the world and each DNS Cluster has multiple 
servers. So a DNS query in Chicago will go to a different DNS clusters than queries from 
a device in Atlanta or New York.

How to get a list of google DNS Cluster’s.
dig -t TXT +short @

How to print this list in a table format. 
-- Script from: -- 

for LOC in $(dig -t TXT +short @
  case $LOC in
    '') : ;;
    *.*|*:*) printf '%s ' ${LOC} ;;
    *) printf '%s\n' ${LOC} ;;

Which will give you a list like below. This is all of the IP network’s that 
google uses for their DNS Clusters and their associated locations. iad iad syd lhr mrn tpe atl mrn tul lpp bru cbf bru lpp chs cbf chs lpp dls dub mrn cbf lpp cbf tul mrn atl cbf bru cbf cbf chs dls dls sin tul lhr lhr sin syd syd fra fra fra bom bom gru atl gru cbf scl tpe cbf tul chs lpp tul mrn tul atl cbf nrt nrt nrt iad grq grq tpe
2404:6800:4000::/48 bom
2404:6800:4003::/48 sin
2404:6800:4006::/48 syd
2404:6800:4008::/48 tpe
2404:6800:400b::/48 nrt
2607:f8b0:4001::/48 cbf
2607:f8b0:4002::/48 atl
2607:f8b0:4003::/48 tul
2607:f8b0:4004::/48 iad
2607:f8b0:400c::/48 chs
2607:f8b0:400d::/48 mrn
2607:f8b0:400e::/48 dls
2800:3f0:4001::/48 gru
2800:3f0:4003::/48 scl
2a00:1450:4001::/48 fra
2a00:1450:4009::/48 lhr
2a00:1450:400b::/48 dub
2a00:1450:400c::/48 bru
2a00:1450:4010::/48 lpp
2a00:1450:4013::/48 grq

There are
IPv4 Networks: 68
IPv6 Networks: 20
DNS Cluster’s Identified by POP Code’s: 20
DNS Clusters identified by POP Code to City, State, or Country. Not all 
of these are 
Google’s Core Datacenters, some of them are Edge Points of Presences (POPs). and 

Most of these are airport codes, it did my best to get the location correct.
iad          Washington, DC
syd         Sydney, Australia
lhr          London, UK
mrn        Lenoir, NC
tpe         Taiwan
atl          Altanta, GA
tul          Tulsa, OK
lpp          Findland
bru         Brussels, Belgium
cbf         Council Bluffs, IA
chs         Charleston, SC
dls          The Dalles, Oregon
dub        Dublin, Ireland
sin          Singapore
fra          Frankfort, Germany
bom       Mumbai, India
gru         Sao Paulo, Brazil
scl          Santiago, Chile
nrt          Tokyo, Japan
grq         Groningen, Netherlans

Which Google DNS Server Cluster am I using. I am testing this from Chicago, 
# dig -t txt +short @ "" 
<<<<<<DNS Server IP, reference the list above to get the cluster, Council 
Bluffs, IA 
"edns0-client-subnet" <<<< Your Source IP Block Side note, 
the google 
dns servers will not respond to DNS queries to the Cluster’s Member’s IP, 
they will 
only respond to dns queries to and So the following will 
not work.

dig @ Now to see the DNS Cluster load balancing 
in action. 
I am doing a dig query from our Telx\Digital Realty POP in Atlanta, GA. 
We do peer 
with google at this location. I dig a dig query about 10 times and received 
following unique dns cluster member ip’s as responses. 

dig -t txt +short @

Which all are Google DNS Networks in Atlanta.







Just thought it would be helpful when troubleshooting google DNS 

(this is -almost- a copy / paste of an email sent by Erik Sundberg to nanog mailing list on 
August 23 2017).
 This post is being posted with his explicit permission.

Monday, August 21, 2017

My humble results. Testing ping to loopback using v4 and v6

Hello there,
  Regarding RTT v4 vs v6 I did something "interesting" recently, would like to know your thoughs.
  If you ping6 your loopback (let´s say 1000 packets) interface with Windows or Linux, v6 is faster.
  Now try the same on MAC (El capitan for example).., v6 is 20-25% slower.
  I did the above with many devices (and asked some friends) and the behavior was pretty much the same.

--- ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.037/0.098/1.062/0.112 ms

--- ::1 ping6 statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.058/0.120/0.194/0.027 ms

--- ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 98999ms
rtt min/avg/max/mdev = 0.015/0.021/0.049/0.007 ms

--- ::1 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 99013ms
rtt min/avg/max/mdev = 0.019/0.031/0.040/0.004 ms

Windows 10:

Ping statistics for ::1:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Ping statistics for
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 4ms, Average = 0ms

Where to order pain Relief without prescription
vivo v15 pro

Monday, February 29, 2016

Read a BGP live stream from CAIDA

  Read a BGP live stream from CAIDA and insert them into a BGP session

What do we need
  bgpreader from the bgpstream core package provided by Caida obtained in github

  We will read the BGP live stream feed using bgpreader, then the standard output of it will be redirected to a pipe file (mkfifo) where a perl script called bgpsimple will be reading this file. This very same script will established the BGP session against a BGP speaker and announce the prefixes received in the stream.

LAB Topology
  The configuration was already tested in Cisco & Quagga
  The BGP Speaker (Cisco/Quagga) has the IPv4 address
  The BGP Simple Linux box has the IP

How does it works?
  bgpreader has the ability to write his output in the -m format used by libbgpdump (by RIPENCC), this is the very same format bgpsimple uses as stdin. That's why myroutes is a PIPE file (created with mkfifo).



First install general some packages:
apt-get install apt-file libsqlite3-dev libsqlite3 libmysqlclient-dev libmysqlclient
apt-get install libcurl-dev libcurl  autoconf git libssl-dev
apt-get install build-essential zlib1g-dev libbz2-dev
apt-get install libtool git
apt-get install zlib1g-dev

Also intall wandio
git clone


cd wandio
./configure && ./make && ./make install

to test wandio:

Download bgp reader tarball from:

#ldconfig (before testing)

#mkfifo myroutes

to test bgpreader:
./bgpreader -p caida-bmp -w 1453912260 -m
(wait some seconds and then you will see something)

# git clone

Finally run everything
In two separate terminals (or any other way you would like to do it):

./bgpreader -p caida-bmp -w 1453912260 -m > /usr/src/bgpsimple/myroutes
./ -myas 65000 -myip -peerip -peeras 65000 -p myroutes

One more time, what will happen behind this?
bgpreader will read an online feed from a project called caida-bmp with starting timestamp 1453912260 (Jan 27 2016, 16:31) in "-m" format, It means a libbgpdump format (see references). The stardard output of all this will be send to the file /usr/src/bgpsimple/myroutes which is a "pipe file". At the same time, will create an iBGP session againts peer (a bgp speaker such as Quagga or Cisco). will read myroutes files and send what it seems in this file thru the iBGP Session.

Important information
- The BGP Session won't be established until there is something in the file myroutes
- eBGP multi-hop session are allowed
- You have to wait short time (few seconds) until bgpreaders start to actually see something and starts to announce to the BGP peer

References / More information:
-Part of the work was based on:

- Caida BGP Stream:

- bgpreader info:

- RIPE NCC libbgpdump:

- Introduction of "Named Pipes" (pipe files in Linux):