Bar Chart Race IPv6 Deployment in LAC Region - May 2014 to June 2023

Strange ssh behavior on MAC - Copy/Paste Issues

 Situation:

   Strange behavior of SSH on MAC, problems with copy/paste in terminal during ssh. Does the clipboard work in other applications?

Solution:

   At least in "vi" the solution is very simple. Edit the file: ~/.vimrc and paste the following content:

if !has("gui_running")

   set mouse=

endif

Luck!

A Look at LACNIC’s IPv6-Only Members

Introduction

This paper seeks to analyze the announcement and status of the prefixes held by organizations known as IPv6-only LACNIC members. These are organizations that have received IPv6 prefixes, that may or may not have received an autonomous system (ASN), and that have not been assigned an IPv4 prefix by LACNIC. The results of this analysis will help us improve our understanding of the uses and needs of our members across the region.

Information sources used in this analysis

The source of information used in this analysis was LACNIC whois, with data obtained throughout the month of January 2023. This information only includes those members who have been assigned IPv6 but not IPv4 resources by LACNIC. They may or may not have been assigned an autonomous system number.

Data processing

Python3 on a Jupyter notebook. We used the public APIs of LACNIC and RIPE NCC.

LACNIC WHOIS

RIPE NCC API

APNIC IPv6 Penetration by ASN

LACNIC’s delegated-extended File:

Findings

We found 483 organizations that are considered IPv6-only LACNIC members, which we divided into:

            IPv6-only members with ASN: 343 members

            IPv6-only members without ASN: 140 members

Results for the 483 IPv6-only members (with and without ASN):

An analysis of the 483 members allowed us to obtain the following information:

– 261 announce the entire or a portion of the prefix

– 208 announce the entire prefix they received from LACNIC

– 53 announce a portion of the prefix they received from LACNIC

– 222 do not announce the prefix

Results for the 343 IPv6-only members with at least one ASN:

A total of 343 of the 483 IPv6-only members have at least one ASN (71%).

Out of the 343 members that have an ASN:

                        163 announce the prefix (47.52%)

120 announce the entire IPv6 prefix (73.61%)

43 announce a portion of the IPv6 prefix (26.38%)

180 do not announce the IPv6 prefix (52.48%)

Results for the 140 IPv6-only members without an ASN:

A total of 140 of the 483 IPv6-only members do not have an ASN (29%).

Out of these 140 IPv6-only members with no ASN:

98 announce the prefix (70%)

82 announce the entire prefix (83.67%)

16 announce a portion of the prefix (16.32%)

42 do not announce the prefix (30%)

What can we conclude from the charts above?

The first thing that stands out is that IPv6-only members without an ASN have 23% more announcements than LACNIC members who do have an ASN. In other words, these members have asked another organization to announce their prefix.

In both cases, the number of unannounced prefixes is very high, and it might be interesting to find out whether there is a reason for this. 

It is particularly striking that the percentage of partial announcements is almost identical (11% of members with an ASN and 12% of members without an ASN).

Trying to identify whether the ASNs have IPv6 traffic

Knowing that there are 343 ASNs with assigned IPv6 prefixes and considering that these are also IPv6-only members, one can presume that they must have “medium high” IPv6 traffic.  Therefore, we analyzed each autonomous system to determine its IPv6 traffic on 23 January 2023.

How do we find out if an ASN has IPv6 traffic?

As many of you know, APNIC has been measuring IPv6 traffic for each ASN for many years. You can learn more about these studies here: https://stats.labs.apnic.net/ipv6

Based on the above, we wanted to find out if the known ASNs of IPv6-only LACNIC members actually had IPv6 traffic (beyond announcing their prefix). 

Unfortunately, we were unable to obtain information for 274 (79.88%) of the 343 autonomous systems that were analyzed. However, it is important to note that this does not necessarily mean that they have not deployed IPv6, but that the traffic generated is very low and does not appear in the measurements by APNIC. A total of 32 ASNs were reported with 0% IPv6 traffic and 1 ASN with 88% IPv6 traffic.

Are our IPv6-only members actually IPv6-only?

Finally, we wanted to find out if our members (IPv6-only with an ASN) are truly as IPv6-only as their name implies.

For this particular case, we relied on the RIPE NCC API to obtain the information presented below.

Out of the 343 IPv6-only members with at least one ASN:

– A total of 540 prefixes were being announced (including v4 and v6)

                        IPv4 announcements: 220

                        IPv6 announcements: 320

                        In IPv4 announcements, the average prefix length is: 23.56

                        In IPv6 announcements, the average prefix length is: 38.09

– 66 members “became” DualStack, i.e., the ASN announces both IPv4 and IPv6

– 90 members announce IPv6 only

– 27 members announce IPv4 only

– 160 do not announce any prefix

And to which RIR do the IPv4 prefixes announced by IPv6-only LACNIC members belong?

LACNIC ARIN RIPE NCC AFRINIC APNIC

28 166 17 9 0

Sankey Diagram – A Look at LACNIC’s IPv6-Only Members

Conclusions

Results suggest that, while a significant number of LACNIC members consider themselves to be IPv6-only, we observed that more than 50% of these members have not started announcing their IPv6 prefix. At the same time, we noticed that many of those who have deployed IPv6 continue to use IPv4. This means that, although in recent years IPv6 adoption has grown in the region, there is still a long way to go to achieve widespread IPv6 deployment. In any case, it is important to continue to monitor the adoption of the new protocol.

Finally, being an IPv6-only LACNIC member also allows an organization to participate in the Internet ecosystem.

The Game of Dominoes and TCP/IP

As many of us do, I have more than one passion: family, work, sports, a category in which I include the beautiful game of dominoes. I reached my best level in this game about 25 years ago, when I participated in several tournaments (a few of which I won) and the icing on the cake was that I took the sixth place in a national tournament. I should also mention that I come from a family with a certain tradition of domino fans, including two uncles, my father, and my brother.

Playing dominoes is one of the most beautiful things that family and close and not so close friends can share. But how is the game of dominoes similar to the TCP/IP protocol? Some of you must probably be thinking “Alejandro has gone crazy.” Perhaps I haven’t gone crazy but likely already was.

But I digress… I will show you that TCP/IP and dominoes do have a lot in common.

IBM [1] defines the TCP/IP protocols as follows:

“Protocols are sets of rules for message formats and procedures that allow machines and application programs to exchange information. These rules must be followed by each machine involved in the communication in order for the receiving host to be able to understand the message.”

Sounds interesting… but you probably still don’t see what this has to do with dominoes and think it’s crazy. Do not despair, we’ll get there in a moment!

This is what ChatGPT has to say about the game of dominoes:

“The game of dominoes is a board game in which players place tiles with numbers on both ends on a board. The object of the game is to place all the tiles before the other players.In the game of dominoes, communication between players is based on strategy and planning. Players must communicate which tiles they have and which tiles they can play, and they must work together to find the best way to place the tiles on the board. Players must also pay attention to the other players’ moves and adapt their strategy accordingly. In short, communication during a game of dominoes is essential to a team’s success and to winning the game.”

Now, let’s think at the macro level. At this point, we can see that both involve elements that must be sent/played, and which must maintain an order to achieve successful communication. Likewise, both require strategy and planning to achieve the objective: one connects tiles, the other connects devices. Am I starting to convince you?

Now let’s talk about communication and partnership dominoes, a game where players are paired into teams. This is at the heart of the topic that I want to get to.

Regardless of the style of dominoes (Cuban, Latino, Mexican, Chilean, etc.), partnership dominoes is a communication game and is not different from a data network. A player needs to communicate with his partner (A → B, B → A) to specify which tiles he has or doesn’t have.

But how do partners communicate if one of the rules is that you are not allowed to talk? Therein lies the greatness of any good player: just like TCP/IP —and any other communication protocol— there are certain rules that must be followed.

After three decades of experience in both worlds, I will share with you what I believe are the main moves in the partnership dominoes ecosystem and their counterpart in TCP/IP:

The pose

In a game of dominoes, the “pose” —the first tile to be placed on the board— is identical to a TCP SYN Packet, more specifically, a TCP Fast Open SYN with payload. This is a beautiful comparison, as in partnership dominoes the pose *always* transmits information, usually related to the suit (number) one most desires. TCP Fast Open —a thing of beauty from the world of TCP— is defined in RFC 7413 and its main objective is to be able to send information in the first packet with which all TCP communication begins (SYN).

The pauses (double ACK)

Payload is considered unnecessary in some networks, yet it may be worth the trouble. A player’s “pause” explicitly reveals that the player has more than one tile of the suit (number) they have played. With this, the player is efficiently communicating information to their partner, who should be aware of these pauses, much like those servers and networks where devices are configured to send more than one ACK in TCP (acknowledgment).

Passing (packet dropped)

In TCP, when a packet is dropped, the “Congestion Avoidance” phase begins. There, the TCP window decreases by 50% and so does transmission speed. Nothing more similar to passing in a game of dominoes. Of course, here a player goes into a panic, especially when they have a lot to transmit.

The version

In the world of IP, we are used to IPv4 and IPv6; in dominoes, the only difference is that versions range from 0 (blank) to 6. (Yes, I know, double-9 domino sets have the numbers 0 to 9, every rule has its exception ;-) 

False thinking

TCP Half-Open. Remember? This is when a three-way handshake (SYN, SYN+ACK, ACK) cannot be completed (but careful! this is the modern concept and, to be honest, it does not follow RFC 793). It is also commonly used to launch DoS attacks.

Load size (total length)

As we draw our tiles, how many points do we add?

Source and destination address

Here we are specifically in the Layer-3 world. This is a very interesting case where, just as in a communication network, one host communicates with another. The same thing happens in dominoes: while some “packets” are aimed at your partner, some may be aimed at our opponents, as needed (for example, if we are trying to find a specific suit).

Thinking about each move

Bufferbloat is a very particular situation, yet it occurs quite frequently. Basically, these are situations where hosts (mainly middleware, routers, firewalls, or switches) add delay (excess buffering) when switching packets. This creates unnecessary latency and jitters. If you manage a network, please be sure to check if you are suffering from bufferbloat.

Block the game/hand

In dominoes, this is the moment when it is no longer possible to make a play. In TCP/IP, it means that the network is down, so packets cannot be sent.

Consequences of good and poor communication

Just as in any network protocol, in partnership dominoes both proper and poor communication have their consequences. In a game of dominoes, if communication is good, it will most likely lead to victory. If communication is deficient, the team will lose the game. In TCP/IP, if communication is good, the connection will be properly established and the data will be delivered. If the communication is poor, the data will be corrupted and/or the connection will not be established.

Head and tail

In dominoes, a “head and tail” (capicúa) is when your last tile has two different numbers. What can this be compared to? Well, it can be compared to TCP Multipath (MPTCP), defined in RFC 8684, which allows operating connections through different paths. This is why MPTCP provides redundancy and efficiency in terms of bandwidth consumption.

Have I not convinced you yet? OK, I have one more chance to see if I can do it.

The table below shows a layer-to-layer comparison of TCP/IP and dominoes.

TCP/IP model (+ user layer)	Domino model
User	Player
Application	Set up the play
Transport	Select the tile
Network	Pick up the tiles
Link	Tiles
Physical	Place the tile on the table

In the TCP/IP model, a packet is built from the upper to the lower layers (it is then injected into the network, etc.), it is received by the destination host, and processed “in reverse”, from the physical to the application layer. The exact same thing happens in the domino model: the player sets up their move, selects their tile, picks it up, and then injects it into the game.

Conclusion

Comparing the game of partnership dominoes and the TCP/IP communication protocol may seem strange at first. However, a closer look reveals the similarities. In dominoes, there are two players who act as senders and receivers of information, as they each have their own set of tiles and must communicate with each other to decide which tile to play each turn. In the same way, in TCP/IP there are devices that act as senders and receivers of information.

These contrasts illustrate how similarities can often exist between seemingly dissimilar systems, and that understanding these similarities can help us understand complex concepts and see things from a different perspective.

Python: reading a text file - character

Situation:

   Reading a text file in python3 (csv or txt) there is a character that can be appreciated using "more" in terminal but in python3 the situation is more complicated.

Example:

  $ more epa.csv

<U+FEFF>the text

Problem:

   Python3 reads the file well, it doesn't throw an error, but that invisible "character" remains in the variables, the texts, etc. and can cause some inconvenience.

Solution:

   The solution is to read the file and specify the encoding, something as simple as:

FILENAME="epa.csv"

with open(FILENAME, encoding='utf-8-sig') as file:

     for line in file:

         print(line)

Explanation (taken from: https://stackoverflow.com/questions/17912307/u-ufeff-in-python-string):

The Unicode character U+FEFF is the byte order mark, or BOM, and is used to tell the difference between big- and little-endian UTF-16 encoding.

Good luck,

An Interesting Change Is Coming to BGP

A route leak is defined as the propagation of routing announcement(s) beyond their intended scope (RFC 7908). But why do route leaks occur? The reasons are varied and include errors (typos when entering a number), ignorance, lack of filters, social engineering, and others.

Although there are several ways to prevent route leaks and, in fact, their number has decreased over the past three years (thanks to RPKI, IRR, and other mechanisms), I will try to explain what I believe BGP configurations will look like in the future. To do so, I will talk about RFC 9234, Route Leak Prevention and Detection Using Roles in UPDATE and OPEN Messages. And the part I would like to highlight is “role detection” as, after this RFC, in the future, we will assign roles in our BGP configurations.

To understand what we want to achieve, let’s recall some typical situations for an ISP: 

a new customer comes along with whom we will speak BGP,

a connection to an IXP,

the ISP buys capacity from a new upstream provider,

a new private peering agreement.

In all these cases decisions need to be made. There are multiple ways to configure BGP, including route maps, AS filters, prefix lists, communities, ACLS, and others. We may even be using more than one of these options.  

This is where RFC 9324 enters the picture: the document establishes the roles in the BGP OPEN message, i.e., it establishes an agreement of the relationship on each BGP session between autonomous systems. For example, let’s say that I am a router and I speak to another router and tell them that I am a “customer”; in turn, the other router’s BGP session can say “I am your provider.” Based on this exchange, all configurations (i.e., filters) will be automatic, which should help reduce route leaks.

These capabilities are then negotiated in the BGP OPEN message.

The RFC defines five roles:

Provider – sender is a transit provider to neighbor;

Customer – sender is a transit customer of neighbor;

RS – sender is a Route Server, usually at an Internet exchange point (IX);

RS-client – sender is client of an RS;

Peer – sender and neighbor are peers.

How are these roles configured?

If, for example, on a relationship in a BGP session between ASes, the local AS role is performed by the Provider, the remote AS role must be performed by the Customer and vice versa. Likewise, if the local AS role is performed by a Route Server (RS), the remote AS role must be performed by an RS-Client and vice versa. Local and remote AS roles can also be performed by two Peers (see table).

An example is included below.

BGP Capabilities

BGP capabilities are what the router advertises to its BGP peers to tell them which features it can support and, if possible, it will try to negotiate that capability with its neighbors. A BGP router determines the capabilities supported by its peer by examining the list of capabilities in the OPEN message. This is similar to a meeting between two multilingual individuals, one of whom speaks English, Spanish and Portuguese, while the other speaks French, Chinese and English. The common language between them is English, so they will communicate in that language. But they will not do so in French, as only one of them speaks this language. This is basically what has allowed BGP to grow so much with only a minor impact on our networks, as it incorporates these backward compatibility notions that work seamlessly.

This RFC has added a new capability.

Does this code work? Absolutely. Here’s an example in FRR:

Strict Mode

Capabilities are generally negotiated between the BGP speakers, and only the capabilities supported by both speakers are used. If the Strict Mode option is configured, the two routers must support this capability.

In conclusion, I believe the way described in RFC 9234 will be the future of BGP configuration worldwide, replacing and greatly improving route leaks and improper Internet advertisements. It will make BGP configuration easier and serve as a complement to RPKI and IRR for reducing route leaks and allowing for cleaner routing tables.

Click here to watch the full presentation offered during LACNIC 38 LACNOG 2022.

https://news.lacnic.net/en/events/an-interesting-change-is-coming-to-bgp

DNSSEC Deployment in the Region – Statistics and Measurements

By Hugo Salgado, Research and Development at NIC Chile; Dario Gomez Technology Consultant; Alejandro Acosta R+D Coordinator at LACNIC

Introduction

In this article we would like to talk about some recent studies we have conducted on a topic we are very passionate about: DNSSEC. Note that we are using the plural term “studies”, as there are two studies on DNSSEC that we began at the same time… please continue reading to find out what these two studies are about!

About DNSSEC

DNSSEC incorporates additional security to the DNS protocol, as it allows checking the integrity and authenticity of the data, preventing spoofing and tampering attacks through the use of asymmetric cryptography, better known as public/private key cryptography. By using these keys and digital signatures based on public key cryptography, it is possible to determine whether a query has been altered, which allows guaranteeing the integrity and authenticity of the message. If these signatures are checked and they do not match, it means that the chain of trust has been broken and the query cannot be validated as legitimate.

Having DNSSEC depends on your ISP or Internet service provider, who is the one responsible for configuring the protocol. There are several different tools to find out if you have DNSSEC, such as the following:

https://dnssec-analyzer.verisignlabs.com/

https://dnsviz.net/

As many already know, the DNSSEC protocol has been growing a lot in recent years. Four aspects have marked the growth of DNSSEC deployment:

    DNSSEC is enabled by default on some recursive servers (BIND).

    Great progress has been achieved in making it easier for major registrars to enable DNSSEC on the different domains.

    All major recursive DNS servers perform DNSSEC validation (Google Public DNS, Cloudflare, Quad9, etc.).

    Apple very recently informed that iOS16 and macOS Ventura will allow DNSSEC validation at the stub resolver.

DNSSEC has always been very important for LACNIC, and we have organized many events and activities around this topic. However, to date we had never conducted our own study on the matter.

What is the study about?

LACNIC’s R&D department wanted to carry out a study to understand the status and progress of DNSSEC deployment in the region.

Data sources

We have two very reliable data sources:

    We used RIPE ATLAS probes https://atlas.ripe.net/

    We performed traffic captures (tcpdump) that were then anonymized on authoritative servers

Dates:

We began gathering data in November 2021. Currently, data is gathered automatically, and weekly and monthly reports are produced. [1]

How can one identify if a server is performing DNSSEC?

We need to look at two different things:

Atlas probes:

DNS resolution requests for a domain name that has intentional errors in its signatures and therefore cannot be validated with DNSSEC are sent to all available probes in Latin America and the Caribbean. An error response (SERVFAIL) means that the resolver used by that probe is using DNSSEC correctly. If, on the other hand, a response is obtained (NOERROR), the resolver is not performing any validation. Note that, interestingly, the goal is to obtain a negative response from the DNS server – this is the key to knowing whether the recursive server validates DNSSEC. Now, an example. If you visit dnssec-failed.org and can open the page, it means that your recursive DNS is not performing DNSSEC validation – you shouldn’t be able to open the page! :-)

Traffic captures (tcpdump):

Before sharing what we do with traffic captures, we will first expand a bit on the concept of DNSSEC. Just as DNS has traditional records (A, AAAA, MX, etc.), new records were added for DNSSEC: DS, RRSIG, NSEC, NSEC3, and DNSKEY. In other words, a recursive DNS server can query the AAAA record to learn the IPv6 address of a name and also the Delegation Signer (DS) record to check the authenticity of the child zones. The key to this study is that servers that don’t perform DNSSEC validation don’t query DNSSEC records!

Based on the above, when making the capture, tcpdump is asked to take the entire packet (flag -s 0). Thus, we have all its content, from Layer 3 to Layer 7. While processing the packet, we look for specific DNSSEC records (again: DS, RRSIG, NSEC, NSEC3, and DNSKEY). If we can obtain any of these records, then the recursive server is indeed performing DNSSEC validation, otherwise it is not.

Where is this capture performed?

The capture is performed specifically in one of the instances of the reverse DNS D server (D.IP6-SERVERS.ARPA). The following command is used: /usr/sbin/tcpdump -i $INTERFAZ -c $CANTIDAD -w findingdnssecresolvers-$TODAY.pcap -s 0 dst port $PORT and ( dst host $IP1 or dst host $IP2 )

Processing the data

First, processing the data comprises several steps, all of which are performed entirely using open-source software, specifically, Bash, Perl, and Python3 on Linux.

Second, let’s not forget that there are two sources of information: traffic captures (PCAPs) and Atlas probes. Below is the methodology followed in each case.

    Processing of PCAPs: After obtaining the PCAPs, a series of steps are performed, including the following:

        Processing of PCAP files in Python3 using the pyshark library.

        Cleaning of unprocessable data (incorrectly formed, damaged, non-processable packets, etc.)

        Elimination of duplicate addresses

        Anonymization of the data

        Generation of result.

        Generation of charts and open data

    Processing of the information obtained in RIPE Atlas:

Traffic data captures are performed with monthly measurements on the RIPE Atlas platform, using its command line API. They are then collected and processed using a series of Perl scripts and, finally, they are plotted using the Google Charts API. In addition, we always make the data available in open data format.

Let’s keep in mind that, in order to determine if a probe is using a validating resolver, it uses a domain name with intentionally incorrect signatures. Because the name is not valid according to DNSSEC, a validating probe should return an error when attempting to resolve that name. On the contrary, a positive response is obtained for the name, the resolver is not validating, as it has ignored the incorrect signature.

Results

The chart below shows the number of servers that were studied, specifying those that use DNSSEC and those that do not. The blue and red lines represent servers for which DNSSEC is and is not enabled, respectively.

Nro de Servidores DNSs estudiados

Chart No. 1

As the image shows, on 2 June 2022 there were more recursive servers that did not perform DNSSEC validation than servers that did. Between 33,000 and 55,000 IP addresses were analyzed each week. In general, an average of approximately 55% of servers that do not use the protocol and 45% of positive samples is maintained.

Nro de Servidores IPv6 DNSs estudiados

Chart No. 2

Chart No. 2 shows the history of DNSSEC queries over IPv6. It is worth noticing that for various sampling periods there were more DNSSEC queries over IPv6 than over IPv4. Undoubtedly, the intention is for the red line to gradually decrease and for the blue line to gradually increase.

Ranking of countries with the highest DNSSEC validation rates

Using the RIPE Atlas measurement platform, it is possible to measure each country’s ability to validate DNSSEC. Each measurement can be grouped by country to create a ranking:

Ranking DNSSEC por pais

Chart No. 3

Ranking based on average DNSSEC validation rates from networks in the countries of Latin America and the Caribbean corresponding to May 2022.

The numbers inside the bars show the number of participating ASs for each country. Countries where we measured only one AS were not included.

Summary

Based on the study of traffic captures with data obtained over a period of eight months, the graph suggests a slow decrease in the number of NO-DNSSEC servers. There also appears to be greater DNSSEC deployment in IPv6 than in IPv4 servers.

It is to be expected that an analysis of the data obtained using Atlas probes suggests greater deployment of DNSSEC validation than other, more generic data sources, as these probes are usually hosted on more advanced networks or by users who would deliberately enable DNSSEC. However, it somehow represents the “upper limit” of DNSSEC penetration and is also an important indicator of its evolution over time.

Open data

As usual, we at LACNIC want to make our information available so that anyone who wishes to do so can use it in their work:

https://stats.labs.lacnic.net/DNSSEC/opendata/

https://mvuy27.labs.lacnic.net/datos/

This data is being provided in the spirit of “Time Series Data.” In other words, we are making available data collected over time, which will make it very easy for our statistics to fluctuate and to identify increases and/or drops in DNSSEC deployment by country, region, etc.

As always when we work on this type of project, we welcome suggestions for the improvement of both the implementation and the visualization of the information obtained.

References:

[1] https://stats.labs.lacnic.net/DNSSEC/dnssecstats.html

https://mvuy27.labs.lacnic.net/datos/dnssec-ranking-latest.html