next up previous
Next: Research issuesTesting Up: Routing Information Service Previous: Prototype Setup

Final Setup

 

Figure 3 shows the proposed setup for the RIS in more detail.

Data-collection machines (the ``Remote Route Collector'' (RRC) in figure 3) will be installed at a few major topologically interesting points on the Internet, such as major exchange points and the like. These Remote Route Collectors will collect routing information either directly over the exchange LAN of the site or using the BGP-multi-hop mode to nearby ISP's. All peerings will be set up by hand and only data from ISP's interested in this project will be collected.

The optimal number of Remote Route Collectors and their location still has to be investigated. In first order, it is expected that around 10 such RRC's will be installed in the RIPE service area.

In principle, all peerings via BGP-multi-hop could be done by the central machine at the RIPE NCC. However, that would involve the transfer of a large amount of data over the networks involved. One central collector will not scale to several hundred peers. It is therefore more efficient to do some of the BGP-multi-hop peerings with the RRC's, reduce and compress the data there and only transfer the results to the RIPE NCC.

The RRC's are discussed in more detail in section 3.2.1.

Data is then transferred to a central machine at the NCC and stored into a database. Design considerations for this database are discussed in section 3.1.2.

Users of the RIS can access the database through various interfaces discussed in section 3.1.3. It is a goal of the project to transfer the data as often as practically possible without overloading the networks or databases, in order to minimize the time between data-taking and availability of the data to the users.

 figure140
Figure:   Detailed overview of the final set up.

Software for the RIS

An overview of the various software modules used in the final setup is shown in figure 4.

Software for the Remote Route Collectors

Remote Route collectors will run BGP listener which logs all updates received from its peers and also maintains a routing table. In the first instance, it is planned to run the MRT [9] and GNU-Zebra [8] software under a supported UNIX platform on each RRC. These products were described in section 2.

Depending on the development paths of MRT and Zebra, the RIPE NCC might decide to re-write parts of those software packages in order to add missing features or to optimize performance, or to use parts of other software packages. The parts of any external software essential for the RIS will be reviewed and documented according to appropriate RIPE NCC guidelines in order to remain maintainable in the future.

The BGP listener software will collect time stamped BGP4 announcements, including:

As both Zebra and MRT supports IPv4 and IPv6, it will be easy to extend the RIS to include IPv6 routing information when native IPv6 networks start to be deployed on a large scale.

 figure152

Receiving speed:

Figure 5 shows the number of updates received by an RRC as a function of time. When the RRC is started, it receives some 65,000 initial prefix announcements in 90 seconds from a single peer. After the initial setup, the rate, averaged over a 1 week period, will drop to 2 updates/second/peergif.

Assuming 60 to 100 peerings at a single collection point, the typical number seen with the prototype will translates to a peek of 50,000 to 80,000 announcements per second to be record by a RRC. Collector software will be resource intensive in terms of memory, to keep several views of routing table and I/O intensive to record the updates and snapshots.

During major instabilities, the number of updates might even be higher. In order not to interfere with operation of the routers, it will be considered to limit the data-flow from the routers to the RRC's.

Database software

 

The main requirements for the database are:

Insertion speed:

Assuming all RRC's are restarted once on a same day total number of updates collected from 10 RRC's during one day will be about 110 M prefix updates. To summerize and insert this data into database in about 3 hours, required processing speed will be 10,000 prefixes/sec. The database will have to buffer incoming data and the situation where not all data has been inserted in the database will be flagged. The acceptable time elapsed between data-taking and availability in the data-base still has to be investigated.

Query speed:

This should be such that an individual user will receive an answer to a simple query (e.g. what was the route from A to B at a given time) in O(10)&thicksp;s. If this takes much longer, extracting data will be too cumbersome and the RIS will not be used.

Merging data:

The software should be able to merge data from remote points into database if they have been disconnected from the RIPE NCC while still collecting data without ``falling behind'' with the regular updates. The amount of data to be merged at once here may be of the order of the data collected during a week. (For example, an RRC that got disconnected from the Internet at the start of a long weekend, followed by a few days necessary to solve the problem.)

RRC NCC Central
(GByte) (GByte)
Peer/Day Daily Monthly Monthly Yearly
BGP announcements
Uncompressed 0 . 007 0 . 490 14 . 3 143 . 0 - . -
Compressed 0 . 005 0 . 350 - . - - . - 1002 . 0
Routing table dumps
Uncompressed 0 . 021 1 . 430 43 . 0 430 . 0 - . -
Compressed 0 . 002 0 . 143 - . - - . - 430 . 0
Storage needed
Hard disk 0 . 028 0 . 633 57 . 3 573 . 0 - . -
Tape - . - - . - - . - - . - 1700 . 0
Data to be transferred
Compressed - . - 0 . 500 - . - - . - - . -


[]   Storage requirements for the RIS. The first 3 columns refer to the individual RRC collecting BGP-announcements and routing table dumps, the last 2 columns to the central collection point at the NCC. The top 3 rows show the size of the BGP-announcements, routing table dumps and total storage space needed at that stage. The last row shows the amount of data to be transferred. Data will be stored either compressed or uncompressed, ``-.-'' indicates that data will not be stored in the (un-)compressed format at that stage. A total of 10 collection points has been assumed. The numbers in this table are based on the prototype and will be further refined as we gain more experience. The numbers also do not include index-files and other overhead introduced by the data-base program.

Data volumes:

The prototype shows that the data at a typical collection point consists of: The maximum number of collection points is of the order of 40, approximately 30 core locations in the RIPE region and about 10 core locations elsewhere in the world. However, there is a considerable overlap between the data that would be collected at all 40 points, so we assume that data collection at 10 points will be sufficient. Data-collection might also be done by exchanging data with similar projects (see 6), though this does not change the total data-volume.

An estimate of the upper limit of the total data volume based on these results can be found in table 1. The numbers in these table are based on our studies with the prototype and will be further refined as we gain more experience. Assuming that data will be available on disk for 1 month and is then moved to tape, the RIS will require some 600&thicksp;Gbyte of disk-space as well as a tape storage device capable of handling some 2&thicksp;Tbyte.

Handling such an amount of data is by no means trivial. The database should be able to handle this amount of data and be able to cope with future expansion.

Experience with the prototype has shown that the popular MySQL database cannot cope with these requirements. Other products are being investigated.

User Interface

 

Regardless of the method chosen to store the data, it is planned to implement the following ways to access the data (in this order):

  1. All (or a selected subset of) the raw data (both the full routing tables and the individual updates) will be made available to the sites participating in the project via FTP or HTTP.
  2. Command line queries, a user enters a command and the resulting output will be shown on the screen, similar to the well known RIPE NCC whois server. Typical queries that one can think of are, for example: These queries will be refined and expanded based on the input from the users.
  3. Queries through a cgi-script. An interface to command-line queries via a web-interface will be provided. This interface will also allow us to provide graphical output, for example, a map of ASN's and their connections.
  4. Reports and statistics. The data will be collected and reduced to reports showing statistics of Internet routing. For example: This will again be expanded based on user input.
In the first instance, it is foreseen that all queries will be done to the RIPE NCC RIS server. At a later stage, the possibility to do local queries to the individual RRC's might be added. The advantages of this are two-fold, it will reduce the load on the central server and it will allow the users to use the RIS data when a major network problem causes the central server to be unreachable.

Data collection at the Remote points

The RRC's will need careful monitoring and security since they are interacting with critical network elements like border routers, are located at important network points, and problems may trigger alarms at peer NOC's. For example, if a box goes down NOC monitors may generate alarms equivalent to peer going down even though an RRC is not announcing any routes. This should be avoided.

For security reasons, the RIS will get a separate AS-number. The RRC's will also not announce any routes though peers may wish to set up filters to block any announcements from the RRC's.

Although it is planned to control the RRC's from a central point with no operators or service required at the local sites, it is expected that each site that hosts a RRC appoints a local contact. This contact should take care of things that cannot be done remotely, such as rebooting the machine or copy information from the console in case of network or hardware problems, keep the RIPE NCC informed about planned maintenance of network and such. The local contact, obviously, has to be reachable by phone or email.

Hardware

Hardware for the RRC's.

 

The prototype shows that route collection is a memory and I/O intensive process, CPU power is less important. Approximately 256&thicksp;Mbyte of memory is needed to run all software without significant swapping.

Although data will be pulled to the RIPE NCC at regular intervals, one should consider the situation where connectivity to the RIPE NCC is lost for some time and that data will have to be buffered at the RRC's. In order to cope with this situation, the RRC's will have to be equipped with at least 20&thicksp;Gbyte of disk space.

Over the last year's the RIPE NCC has gained experience with the operation of remote machines for data-collection purposes [7]. It is planned to use that experience to run the RRC's, no particular problems are foreseen in this respect.

There is also considerable interest in the installation of Test-boxes at the same location as the RRC's and it has been suggested to use the same hardware for both projects. This idea will be investigated further.

Hardware for the central collection point

The central machine will collect data in the same way as the RRC's. Besides that, it will run the database where all collected data is stored as well as handle external queries. The machine will be equipped with sufficient disk-space to keep approximately one month of data online. Older data will be stored on tape.

Peering with the route collector

The RIS project will use AS Number 12654 for peering at all collection points. This ASN 12654 will not announce any routesgif. Ideally, the RIS should peer with all members of the site for their entire default free routing table. These peerings will be set up using BGP4. The peering policy will be added to auto-num objects in the RIPE whois database.

The RIPE NCC central collector will also setup a few multi-hop BGP4 peerings to ISP/NAP's border routers.

Bandwidth requirements for the RRC's.

Approximately 500&thicksp;Mbyte (see table 1) will be transferred from each RRC to the RIPE NCC daily. This translates into a constant flow of about 6&thicksp;kbyte/s. The central collection point at the NCC will see an incoming flow of about 60&thicksp;kbyte/s. However, the data volume that has to be transfered can reduced only by sending the differences between the periodic dumps, rather than the full dumps. We will ensure that these data-transfers will not affect normal operations.


next up previous
Next: Research issuesTesting Up: Routing Information Service Previous: Prototype Setup

Henk Uijterwaal
Thu Oct 14 13:53:27 MEST 1999