BGPGREP

NAME

bgpgrep — filter and print BGP data within MRT dumps

SYNOPSIS

bgpgrep [OPTIONS]... [FILES]... [EXPRESSION]

DESCRIPTION

The bgpgrep utility reads each possibly compressed Multithreaded Routing Toolkit (MRT) dump file specified by FILES and formats its contents to standard output (or any custom output file as specified by OPTIONS). bgpgrep may optionally evaluate a predicate defined by EXPRESSION over every MRT record containing a BGP message. Whenever such predicate evaluates as false the relevant BGP message is discarded from output (see the EXAMPLES section below).

EXPRESSION predicates only affect BGP messages output, other kind of information, such as state changes, is always printed by bgpgrep regardless of any filtering rule.

bgpgrep prints diagnostics to standard error, it detects and tolerates data corruption as much as possible. Corruption within a BGP message causes only the affected message to be dropped. Though unrecoverable errors affecting the entire MRT dump file may require it to be dropped as a whole, bgpgrep will then move to the next file in FILES, if any.

Such events are always reported with reasonable diagnostic errors. Parsed data up to the corruption point may still be printed to regular output.

OPTIONS

bgpgrep expects its options before the FILES list. This is due to the fact that bgpgrep must distinguish between options and its expression predicate (see the OPERANDS section below, for details on how bgpgrep makes such distinction).

The following options are supported:
−−dump−bytecode

Debug option, causes bgpgrep to dump its filtering engine bytecode to standard error before starting MRT dump files processing.

−−no−color

bgpgrep may ease visualization by surrounding some output with color escape sequences, on terminals that support this feature. This option forces colored text output off.

−h or −−help

Prints a short help message, summarizing bgpgrep functionality.

OPERANDS

bgpgrep interprets all its operands up to but not including the first operand that starts with a ‘−’ , or is a ‘(’ or a ‘!’, as the FILES operand. Each of these operands are pathnames to a MRT dump file to be processed. An actual ‘−’ is a placemarker to read uncompressed MRT data from standard input at that point during processing (see STDIN section below).

The first operand that starts with a ‘−’ , or is a ‘(’ or a ‘!’, and any subsequent arguments, are interpreted as an EXPRESSION predicate. Legal predicates are composed of the following terms:
−type msg−type

Evaluates as true if the BGP message type matches ‘msg−type’. Message types may be provided by a human readable type name, as defined by RFC 4271, Section 4 (e.g. OPEN, UPDATE), or any other relevant RFC defining the message type (e.g. ROUTE_REFRESH). An explicit numeric code may also be provided (e.g. 1 as an equivalent to OPEN).

−attr attribute−type

Evaluates as true if the BGP message is an UPDATE containing a path attribute of type ‘attribute−type’. The attribute type may be specified by a human readable name, as defined by RFC 4271, Section 5.1 (e.g. AS_PATH, ATOMIC_AGGREGATE), or any other relevant RFCs defining the interesting path attribute (e.g. COMMUNITY). An explicit numeric code may also be provided (e.g. 8 as an equivalent to COMMUNITY), which is especially useful to specify non-standard path attributes.

−aspath pattern

Evaluates as true if the BGP message is an UPDATE with an AS_PATH that matches the regular expression specified by ‘pattern’. See the AS_PATH REGULAR EXPRESSIONS section below for details.

−peer peer−expression

Evaluates as true if ‘peer−expression’ matches the peer that provided the BGP data. This term matches the PEER field as displayed by the LINE ORIENTED OUTPUT format (see section below for details). Supported peer matching expressions are documented in the PEER MATCHING EXPRESSIONS section below.

−bogon−asn

Evaluates as true if BGP message is an UPDATE whose AS_PATH contains bogon ASN. Any of the following is classified as a bogon ASN:

−exact prefix−expression

Evaluates as true if BGP message is an UPDATE and contains at least one of the relevant networks of interest specified in ‘prefix−expression’. See PREFIX MATCHING EXPRESSIONS section for details.

−supernet prefix−expression

Similar to −exact, but evaluates as true if BGP message contains supernets of the relevant networks of interest, or the actual networks themselves.

−subnet prefix−expression

Similar to −exact, but evaluates as true if BGP message contains subnets of the relevant networks of interest.

−related prefix−expression

Similar to −exact but evaluates as true if BGP message contains prefixes related to the relevant networks of interest. A related network is defined to be either a subnet or a supernet of the specified prefix.

−timestamp time−expression

Evaluates as true if the timestamp at which the BGP data was originated or collected matches the specified ’time−expression’. Accepted expressions are described in the TIMESTAMP MATCHING EXPRESSIONS section.

−communities communities−expression

Evaluates as true if BGP message is an UPDATE whose path attributes contains at least one community specified within ’communities−expression’, see the COMMUNITY MATCHING EXPRESSIONS section below for details.

−all−communities communities−expression

Similar to −communities, but requires all communities inside ‘communities−expression’ to be present inside UPDATE message path attributes.

Terms can be combined with the following operators (in order of decreasing precedence):
( expression )

Evaluates as true if expression is true, may be used to explicitly control precedence.

! expression or -not expression

Negation of expression; the unary NOT operator.

expression [−a] expression or expression [−and] expression

Conjunction of expressions; the AND operator is implicit if no other operator is provided inbetween two consecutive expressions, but can be made explicit by explicitly inserting the −a or −and operators. The second expression is not evaluated if the first one is false.

expression −o expression or expression −or expression

Alternation of expressions; the OR operator. The second expression is not evaluated if the first one is true.

AS_PATH REGULAR EXPRESSIONS

bgpgrep uses a specialized regular expression (regexp) style pattern matching approach for AS_PATH.

AS_PATH regular expressions support most features found in string pattern matching, except backreferences, classes and counted repetition.

ASN are specified by a numeric value, for example 19819 represents AS19819. In their simplest form, AS_PATH expressions match an ASN sequence against the merged BGP data AS_PATH (as specified by RFC 4893), indipendently by its starting position. In the same way a regexp would match a string of characters. For example ‘19819 172’ matches AS_PATH ‘AS121 AS19819 AS172 AS1111’.

The following features, commonly found in regular expressions, are supported by bgpgrep:
Complements

The prefix ‘!’ operator can be used to match any but the specified ASN, for example ‘!871’ matches any ASN but AS871.

Grouping and alternation

Groups can be defined inside regexp by enclosing them inside parentheses, for example ‘( 202 397 )’ defines a group with the single ASN sequence ‘AS202 AS397’. The alternation operator ‘|’ provides additional flexibility, allowing multiple sequences inside groups, like ‘( 202 397 | 1111 5439 )’, which would match both ‘AS1921 AS202 AS397’ and ‘AS2431 AS1111 AS5439 AS79’. Alternation can be used even outside groups and alternatives may very well be more than two. Groups may be nested.

Metacharacters

The ‘.’ metacharacter can be used to match any ASN in its position. The metacharacters ‘*’, ‘?’ and ‘+’ are repetition operators, they can be used to match the preceding ASN or group multiple times in different ways. ‘191*’ matches AS191 zero or more times, ‘191?’ matches AS191 zero or one time, while ‘191+’ matches AS191 one or more times.

These features may be combined at will to provide powerful expressions, for example ‘ˆ !432’ matches any AS_PATH that does not start with AS432.

Extensive usage examples can be found in the EXAMPLES section.

PEER MATCHING EXPRESSIONS

Peer matching expressions specify a set of relevant peers, either by providing their IP address, their ASN, or both. The constructed set is then matched against the peer providing the BGP data inside the MRT input files.

Allowed constructs are:

peer−address

Only peer IP address is matched for equality.

"peer−address peer−asn"

Both peer IP address and ASN are tested for equality.

When both IP address and ASN are provided, the match should be quoted so that it is understood to be a single match as opposed to one match by peer address followed by another match by peer ASN.

Multiple peer matches can be provided at the same time by enclosing them in parentheses, for example ‘( '199036' '173.2.2.1 7566' )’ matches both peer AS199036 and peer AS7566 with IP address 173.2.2.1.

Whenever a peer matching expression is expected, a filepath to a text file may be specified in its place. In this case bgpgrep will read the peer matches directly from file. Matches inside file may be separated by either spaces or newlines. No parentheses are necessary, though quoting may still be necessary for matches specifying both peer address and ASN. Typical C and C++ style comments are supported within the file.

See the EXAMPLES section for usage examples.

COMMUNITY MATCHING EXPRESSIONS

COMMUNITY matching expressions define a set of interesting communities. Communities may be specified in any of the following ways:

Multiple communities may be listed by enclosing them in parentheses, for example ‘( '65535:*' '0:*' )’ matches any reserved COMMUNITY as per RFC 1997.

Whenever a community matching expression is expected, a filepath to a text file may be provided in its place. In this case bgpgrep will parse the communities from the file itself. Each COMMUNITY inside file may be separated by either spaces or newlines. No parentheses are required. Typical C and C++ style comments are supported within the file.

See the EXAMPLES section for usage examples.

PREFIX MATCHING EXPRESSIONS

Prefix matching expressions define a set of interesting networks. Networks are specified as prefixes in their CIDR notation, for example 193.0.0.0/16 or 2001:67c::/32. If prefix length is not defined explicitly, it is taken to be the full IP address length, that is 32 for IPv4 addresses and 64 for IPv6 addresses.

Prefix matching can be restricted to announcements or withdrawals. Syntax is:

If none of ‘+’ or ‘-’ is prepended, then matching takes place on both announcements and withdrawals.

Multiple prefixes can be specified at the same time by enclosing them in parentheses, for example: ‘( '+193.0.0.0/16' '2001:67c::/32' )’.

Whenever a prefix matching expression is expected, a filepath to a text file may be specified in its place. In this case bgpgrep will parse the relevant prefixes from the file itself. Inside file, prefixes may be separated by either spaces or newlines. No parentheses are required. Typical C and C++ style comments are supported within the file.

See the EXAMPLES section for usage examples.

TIMESTAMP MATCHING EXPRESSIONS

Timestamp matching expressions define a time point and a matching direction. Expressions are matched either to the MRT header timestamp, in case of BGP4MP and ZEBRA records (commonly referred to as updates), or to the ORIGINATED field in case of TABLE_DUMPV2 or TABLE_DUMP snapshots (commonly referred to as RIB snapshots). Timestamps may be specified in either of the two following formats:

Matching direction may be any of the following:
>=timestamp

Matches if BGP data was originated after or exactly at the relevant timestamp.

>timestamp

Matches if BGP data was originated after the relevant timestamp.

=timestamp

Matches if BGP data was originated exactly at the relevant timestamp.

<timestamp

Matches if BGP data was originated before the relevant timestamp.

<=timestamp

Matches if BGP data was originated before or exactly at the relevant timestamp.

If no matching direction is provided, ‘=’ is implicitly assumed. See the EXAMPLES section for usage examples.

LINE ORIENTED OUTPUT

bgpgrep prints each MRT record into multiple lines, each one representing either ROUTE INFORMATION or BGP SESSION STATUS.

ROUTE INFORMATION can be either an announcement, a route withdrawn or a routing table (RIB) snapshot.

Each ROUTE INFORMATION line is a sequence of the following ‘|’ separated fields:

TYPE|PREFIXES|PATH ATTRIBUTES|PEER|TIMESTAMP|ASN32BIT

Fields have the following meaning:
TYPE

Single character describing the route information type, may be ‘=’ (RIB snapshot entry), ‘+’ (announcement), or ‘-’ (withdrawn).

PREFIXES

The list of prefixes carried into the message. If the information is an announcement, then this enumerates the prefixes within NLRI and MP_REACH_NLRI fields. If the information is a withdrawn, then this enumerates the prefixes within WITHDRAWN_ROUTES and MP_UNREACH_NLRI fields. If the information is a RIB snapshot entry, then this is the prefix related to the current RIB entry. Multiple prefixes are separated by a single space.

PATH ATTRIBUTES

This is a ‘|’ separated list of the most common BGP path attributes characterizing a route. Each field is left empty if the corresponding path attribute is not present in the collected BGP data (e.g. route announcements without optional attributes, or route withdrawals). The currently displayed path attributes are:

AS_PATH|NEXT_HOP|ORIGIN|ATOMIC_AGGREGATE|AGGREGATOR|COMMUNITIES

If the BGP peer does not support ASN32BIT capability, then the AS_PATH field contains the result of the merging procedure between AS_PATH and AS4_PATH attributes, according to RFC 4893, and the AGGREGATOR field contains the AS4_AGGREGATOR attribute (if present). Otherwise, AS_PATH and AGGREGATOR fields contain the respective attribute.

NEXT_HOP field contains either the NEXT_HOP attribute (IPv4) or the next hop address(es) listed into the MP_REACH_NLRI attribute (IPv6), as described in RFC 4760.

ORIGIN contains the corresponding attribute encoded as a single character, ‘i’ (IGP), ‘e’ (EGP), ‘?’ (INCOMPLETE).

ATOMIC_AGGREGATE field contains AT if the attribute is set, it is left empty otherwise.

COMMUNITIES field contains both COMMUNITY ( RFC 1997 ) and LARGE_COMMUNITY ( RFC 8092 ) displayed in their canonical representation. Well−known communities are displayed according to their IANA assigned names (e.g. NO_EXPORT instead of ‘65535:65281’).

PEER

The BGP peer that provided the BGP message. If the peer uses the ADD−PATH extension ( RFC 7911 ) to announce BGP data, then it is displayed as ‘peer−address peer−asn path−id’, otherwise as ‘peer−address peer−asn’.

TIMESTAMP

Displays the Unix epoch time at which the information was collected. If extended timestamp information is available, the Unix Epoch time is followed by a ‘.’ and the microsecond precision is appended right after it. Timestamp is displayed as a raw numerical value.

ASN32BIT

May be either 1, if BGP data has ASN32BIT capability, or 0.

The BGP SESSION STATUS is encoded as a BGP session state change according to RFC 6936, Section 4.4.1. The format of a line representing a state change is the following:

#|OLD_STATE-NEW_STATE|||||||PEER|TIMESTAMP|ASN32BIT

Each field has the following format:
OLD_STATE−NEW_STATE

Represents the old and new state of the BGP session respectively, according to the BGP Finite State Machine states numerical codes.

PEER, TIMESTAMP, ASN32BIT

Have identical format and meaning with regards to the ROUTING INFORMATION case.

Each line produced always has the same ‘|’ character count, in both ROUTING INFORMATION’s and BGP SESSION STATUS’s case. This facilitates the task of writing simple scripts that manipulate bgpgrep’s output text.

EXIT STATUS

The following exit values are returned:

STDIN

The standard input is used only if no FILES arguments are provided, or when any of the specified FILES arguments is ‘−’ , in which case MRT data is read from standard input at that point, up to an <end−of−file>.

Whenever bgpgrep reads from standard input, MRT data is assumed to be uncompressed.

INPUT FILES

bgpgrep supports most MRT dump formats as written by the majority of Route Collecting projects (see the STANDARDS section below for additional references). MRT dumps may be provided either in their plain uncompressed form, or as files compressed by gzip(1), bzip2(1), or xz(1). bgpgrep performs appropriate decompression on the fly. File extension is used, in a case insensitive way, to discriminate among supported compression formats. If the file extension is not recognized, or there is no extension, then it is assumed to be uncompressed.

STDOUT

Unless redirected explicitly via OPTIONS, the standard output is used to print a human readable text representation of BGP message data, nothing else shall be written to the standard output. bgpgrep may detect and treat as error whenever the standard output is a regular file, and is the same file as any of the FILES arguments. The default output format used by bgpgrep is documented in the LINE ORIENTED OUTPUT section.

STDERR

The standard error is used only for diagnostic messages and error reporting. Any BGP message output is exclusive to standard output or any file explicitly specified by OPTIONS.

EXAMPLES

This section contains some useful examples, starting from trivial ones, demonstrating basic bgpgrep usage, to more complex ones employing sophisticated filtering predicates. Examples in this section use paranoid quoting, since this a worthwhile habit that eliminates potential pitfalls introduced by shell expansion.

bgpgrep

It formats and prints all the BGP data found inside the uncompressed MRT input data available from standard input.

bgpgrep -- -peer '199036'

finds all BGP data announced by peer AS199036, taking MRT input data implicitly from standard input. Notice how an explicit ‘−−’ is necessary for bgpgrep to interpret −peer as an actual EXPRESSION operand, rather than incorrectly mistaking it for OPTIONS.

bgpgrep ./rib.1.bz2 -peer '199036'

but takes MRT input data from a bzip(1), compressed file. The file argument removes the necessity of an explicit ‘−−’ to separate FILES and EXPRESSION operands.

bgpgrep ./updates.*.gz -aspath 'ˆ199036'

finds every message whose first ASN in AS_PATH is AS199036, inside all gzip(1) compressed files resulting from the glob expansion.

bgpgrep ./updates.*.gz -aspath '3333$'

is similar to the previous example, but uses AS_PATH REGULAR EXPRESSIONS to find every BGP message whose last ASN in AS_PATH is AS3333.

bgpgrep ./updates.*.gz -aspath '174 3356'

demonstrates yet another basic use of AS_PATH REGULAR EXPRESSIONS to find every BGP message whose AS_PATH crosses link AS174 AS3356. Notice how the argument of −aspath needs quoting.

bgpgrep ./updates.*.gz -aspath '174 (2603+|202+|303+) 3356'

It finds every BGP message whose AS_PATH crosses AS174 and AS3356, through one intermediate ASN among AS2603, AS202, or AS303. It also takes care of possible prepending for the inbetween ASN.

bgpgrep ./updates.*.gz -aspath 'ˆ7854 .* 5032$' -or -aspath '109 .* 9081$'
bgpgrep ./updates.*.gz -aspath 'ˆ7854 .* 5032$ | ˆ109 .* 9081$'

They both find every BGP message whose AS_PATH starts at AS7854 and terminates at AS5032, or starts at AS109 and terminates at AS9081. The second being the most efficient. This example illustrates the use of alternation inside AS_PATH REGULAR EXPRESSIONS to test multiple patterns at the same time.

bgpgrep ./rib.*.xz -subnet '192.65.0.0/16' -aspath '174 137'

finds all subnets of 192.65.0.0/16 crossing link AS174 AS137. It combines two EXPRESSION terms with an implicit AND operator, since no explicit −and and no −or was provided, as detailed by the OPERANDS section.

bgpgrep ./rib.*.gz \( -subnet '193.0.0.0/16' -or -subnet '2001:67c::/32' \) -aspath '3333$'
bgpgrep ./rib.*.gz -subnet \( '193.0.0.0/16' '2001:67c::/32' \) -aspath '3333$'

They both print every message containing subnets of 193.0.0.0/16 or 2001:67c::/32 destinated to AS3333, the second being a more efficient alternative. In the latter, notice the use of ‘(’ and ‘)’ inside −subnet to provide multiple arguments. This behavior is further explained in the PREFIX MATCHING EXPRESSIONS section, and is common to most matching expressions provided by bgpgrep.

bgpgrep ./rib.*.bz2 -peer '202' -timestamp '>=2012-10-01' -timestamp '<2012-11-01' -loops

is another example combining multiple EXPRESSION terms to achieve complex filtering. It scans all bzip2(1) compressed MRT input files resulting from glob expansion, and prints every BGP message provided by peer AS202 during the month of October, 2012 containing loops in its AS_PATH. Notice how multiple −timestamp terms can be combined to effectively define bounded time ranges.

bgpgrep ./rib.*.bz2 -communities \( '0:*' '65535:*' \) -peer '185.169.236.135 201'

prints all the BGP messages containing reserved communities, provided by peer AS201 with IP address 185.169.236.135.

bgpgrep ./rib.*.bz2 -all-communities ./commlist.tpl -subnet ./netlist.tpl

demonstrates the use of filepath arguments for −all−communities and −subnet EXPRESSION terms. bgpgrep will parse the two text files and use their contents as arguments. This is especially useful to create templates containing relevant networks, communities, or peers and reuse them across various runs.

APPLICATION USAGE

The bgpgrep utility and its filtering engine have been designed for performance. Providing predicates has the double role of cleaning the output of irrelevant data, without resorting to complex scripting, and avoid wasting time to decode useless data. Therefore bgpgrep can gain a lot performance−wise when provided with restrictive predicates, that cut away significant amounts of BGP data from its input files.

bgpgrep deliberately mimics the find(1) utility operands convention, in an attempt to feel familiar to the experienced shell user and provide a powerful EXPRESSION syntax that feels both expressive and readable. Though, many of find(1)’s subtleties also apply to bgpgrep. When writing EXPRESSION, it should be noted that ‘!’, ‘(’, ‘<’ and ‘>’ have special meaning to the shell, and should be quoted accordingly. bgpgrep provides the alternative −not syntax for the unary NOT ! operator that avoids the problem. Still, care should be used with other EXPRESSION terms arguments. When in doubt use explicit quotes, as demonstrated in the EXAMPLES section.

bgpgrep attempts to provide descriptive output for syntax errors that should help with most of these problems.

Another common source of errors is the distinction between FILES and EXPRESSION. bgpgrep treats any operand starting with ‘−’ and followed by at least one character as the beginning of an EXPRESSION, and an actual ‘−’ as a placeholder for standard input (see STDIN and OPERANDS sections for details). In the unlikely event of having to deal with files that may generate ambiguity (e.g. a file named ‘−’), make the file reference explicit by prepending ‘./’ (e.g. ‘./−’ to reference a file named ‘−’ in the current directory). If the FILES list should be left empty, but an EXPRESSION should still be applied, then provide an explicit ‘−−’ to mark the empty file list, as shown in the EXAMPLES section.

SEE ALSO

awk(1), grep(1)

STANDARDS

The bgpgrep utility conforms to:

AUTHOR

bgpgrep was written by Lorenzo Cogotti. bgpgrep is an evolution over bgpscanner originally developed by the same author at the Institute of Informatics and Telematics of the Italian National Research Council (IIT−CNR), with significant contributions by the Isolario project development team at the time.