Python: reading a text file - character

Situation:

   Reading a text file in python3 (csv or txt) there is a character that can be appreciated using "more" in terminal but in python3 the situation is more complicated.

Example:

  $ more epa.csv

<U+FEFF>the text

Problem:

   Python3 reads the file well, it doesn't throw an error, but that invisible "character" remains in the variables, the texts, etc. and can cause some inconvenience.

Solution:

   The solution is to read the file and specify the encoding, something as simple as:

FILENAME="epa.csv"

with open(FILENAME, encoding='utf-8-sig') as file:

     for line in file:

         print(line)

Explanation (taken from: https://stackoverflow.com/questions/17912307/u-ufeff-in-python-string):

The Unicode character U+FEFF is the byte order mark, or BOM, and is used to tell the difference between big- and little-endian UTF-16 encoding.

Good luck,

Tiny script: how to get the RRSIG DNS records with python3

 (I guess there are many other ways to do this, even more elegant)

import dns.resolver
domain='lacnic.net'
domain = dns.name.from_text(domain)
request = dns.message.make_query(domain, dns.rdatatype.ANY)
response = dns.query.tcp(request,'8.8.8.8')
#print(response)

for item in str(response).splitlines( ):
  if 'RRSIG' in item: print (item)

Read a BGP live stream from CAIDA

Objective
  Read a BGP live stream from CAIDA and insert them into a BGP session

What do we need
  bgpreader from the bgpstream core package provided by Caida
  bgp_simple.pl obtained in github

Overview
  We will read the BGP live stream feed using bgpreader, then the standard output of it will be redirected to a pipe file (mkfifo) where a perl script called bgpsimple will be reading this file. This very same script will established the BGP session against a BGP speaker and announce the prefixes received in the stream.

LAB Topology
  The configuration was already tested in Cisco & Quagga
  The BGP Speaker (Cisco/Quagga) has the IPv4 address 192.168.1.1
  The BGP Simple Linux box has the IP 192.168.1.2

How does it works?
  bgpreader has the ability to write his output in the -m format used by libbgpdump (by RIPENCC), this is the very same format bgpsimple uses as stdin. That's why myroutes is a PIPE file (created with mkfifo).

Steps:  

INSTALL BGP READER - UBUNTU 15.04

First install general some packages:
apt-get install apt-file libsqlite3-dev libsqlite3 libmysqlclient-dev libmysqlclient
apt-get install libcurl-dev libcurl  autoconf git libssl-dev
apt-get install build-essential zlib1g-dev libbz2-dev
apt-get install libtool git
apt-get install zlib1g-dev

Also intall wandio
wandio-1.0.3
git clone https://github.com/alistairking/wandio

./configure

cd wandio
./bootstrap.sh
./configure && ./make && ./make install
wandiocat http://www.apple.com/library/test/success.html

to test wandio:
wandiocat http://www.apple.com/library/test/success.html

Download bgp reader tarball from:
https://bgpstream.caida.org/download

#ldconfig (before testing)

#mkfifo myroutes

to test bgpreader:
./bgpreader -p caida-bmp -w 1453912260 -m
(wait some seconds and then you will see something)

# git clone https://github.com/xdel/bgpsimple


Finally run everything
In two separate terminals (or any other way you would like to do it):

./bgpreader -p caida-bmp -w 1453912260 -m > /usr/src/bgpsimple/myroutes
./bgp_simple.pl -myas 65000 -myip 192.168.1.2 -peerip 192.168.1.1 -peeras 65000 -p myroutes

One more time, what will happen behind this?
bgpreader will read an online feed from a project called caida-bmp with starting timestamp 1453912260 (Jan 27 2016, 16:31) in "-m" format, It means a libbgpdump format (see references). The stardard output of all this will be send to the file /usr/src/bgpsimple/myroutes which is a "pipe file". At the same time, bgp_simple.pl will create an iBGP session againts peer 192.168.1.1/AS65000 (a bgp speaker such as Quagga or Cisco). bgp_simple.pl will read myroutes files and send what it seems in this file thru the iBGP Session.

Important information
- The BGP Session won't be established until there is something in the file myroutes
- eBGP multi-hop session are allowed
- You have to wait short time (few seconds) until bgpreaders start to actually see something and bgp_simple.pl starts to announce to the BGP peer

References / More information:
-Part of the work was based on:
http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/

- Caida BGP Stream:
https://bgpstream.caida.org/

- bgpreader info:
https://bgpstream.caida.org/docs/tools/bgpreader

- RIPE NCC libbgpdump:
http://www.ris.ripe.net/source/bgpdump/

- Introduction of "Named Pipes" (pipe files in Linux):
http://www.linuxjournal.com/article/2156

Error 1017 in Cacti. MySQL. Blank Graphics

Error:
4/7/2012 5:37:49 PM - CMDPHP: Poller [0] ERROR: SQL Cell Failed!, Error: '1017 ', SQL: "SELECT count (*) FROM polle
WHERE r_time poller_id = 0 AND end_time> '0000-00-00 00:00:00 '"

4/7/2012 5:38:34 PM - CMDPHP: Poller [0] ERROR: SQL Cell Failed!, Error: '1017 ', SQL: "SELECT count (*) FROM polle
r_time WHERE end_time = '0000-00-00 00:00:00 '"

Procedure:
If the file cacti.log you see the above error tt indicates a failure with poller_output mysql table.
In this respect, I have seem three different solutions, order from the less risky to the highest.

1) Repair mysql table with a php script that brings cacti

# php $PATH-TO-CACTI/cli/repair_database.php

2) Repair the table with mysql command:

# mysql> REPAIR TABLE poller_output;

If you do not know the user information for the mysql database, it can be found here:

$PATH-TO-CACTI/include/config.php


3) Remove the table and do it again (this solution despite being aggressive works perfectly and you do not lose the historic Cacti information). Just enters the mysql CLI and copy/paste the following:

-
- Table structure for table `poller_output`
-
DROP TABLE IF EXISTS `poller_output`;
CREATE TABLE `poller_output` (
MEDIUMINT local_data_id `` (8) unsigned NOT NULL default '0 ',
Rrd_name `` varchar (19) NOT NULL default'',
`Time` datetime NOT NULL default '0000-00-00 00:00:00 ',
`Output` text NOT NULL,
PRIMARY KEY (`local_data_id`, `rrd_name`, `time`)
) TYPE = MyISAM;
-
- Dumping data for table `poller_output`
-
Poller_output `LOCK TABLES` WRITE;
/ *! 40000 ALTER TABLE `poller_output` DISABLE KEYS * /;
/ *! 40000 ALTER TABLE `poller_output` ENABLE KEYS * /;
UNLOCK TABLES;


Ready!, Then you can wait for the poller runs 15 minutes and you will have something in your graphs. If you want to force the poller command is as follows:

# /usr/bin/php-q /var /www/miServer-cacti/poller.php  -force

I hope it will be useful,

Disable / shutdown iptables on Linux

Introduction:

Sometimes it is necessary to "shutdown" or disable our Linux iptables, the procedure depends on the Linux distribution you're using.  

1) Procedure if you are using Redhat, Fedora, Mandriva / Mandrake or Centos, you just have to run the following:
# service iptables save
# service iptables stop
# chkconfig iptables off
or
# / etc /init.d/iptables stop


2) How to disable iptables on Debian or Ubuntu

a) Create a script called fw.stop with the following contents:

# /bin/sh
echo "Stopping firewall and Allowing everyone ..."
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

b) Give execute permission to the script:

# chmod + x / root / fw.stop
or
# chmod 755 fw.stop
c) You can run the scritp with the following command:
# ./fw.stop

More info at:
http://sources68.com/linux-disable-remove-the-iptables-firewall-1fa67761.html

http://blog.acostasite.com/2012/09/deshabilitar-iptables-en-linux.html