From wilhelm@ripe.net Tue Sep 14 14:30:55 1999
Date: Fri, 10 Sep 1999 12:31:38 +0200
From: Rene Wilhelm 
To: tt-ops@ripe.net
Subject: failing RCDP file rotation [Re: output from Cron] [ripe-ttraffic #8757]


Data collection cronjob on ginkgo reported:

> collect_data: RCDP.tt25.ripe.net.19990907-170448 size differences?
>   remote: 13265831, local: 7590628 
>   local file renamed, getting new copy ...

This is a sign of failing RCDP file rotation: the file of July7 afternoon
keeps receiving new data, whereas yesterday's data should have gone to files
with 19990909 in the date part of the filename.

Recipe to fix this:

1. STOP processes from appending to the file

  sshtt tt25
  [TT@tt25:117] ps x |grep received_data
  24327  p0- I  0:08.11 /usr/local/bin/perl5 -w /home/ttraffic/bin/receive_data 
  [TT@tt25:118] kill 24327
  [TT@tt25:119] ps x |grep Control
   3109  ??  S      0:40.69 PacketCatcher emon  (462) n  (462) (ControlDaemon)
   3110  ??  S      4:27.50 UDPProcces port 6000 t 6000 000 (ControlDaemon)
   3111  ??  S      7:17.77 DataHandler emon  (462) n  (462) (ControlDaemon)
  [TT@tt25:121] kill 3109 3110 3111

check that processes are indeed gone, then restart:

  [TT@tt25:123] ~/bin/start_processes.pl

2. Split the large RCDP file into 'per-day' components:

  (this works thanks to the briliant idea of using the UNIX time,
   seconds since 1970, as the serial number in packets send by hosts)

  [TT@tt25:124] cd ~/data
  [TT@tt25:134] ~/bin/split_rcdp.pl  RCDP.tt25.ripe.net.19990907-170448
  [TT@tt25:135] ls -ltr |grep RCDP |tail 
  ...
  -rw-r--r-- 15280948 Sep 10 09:50 RCDP.tt25.ripe.net.19990907-170448
  -rw-r--r--  1613218 Sep 10 09:59 new-RCDP.tt25.ripe.net.19990907-170448
  -rw-r--r--  5583842 Sep 10 09:59 new-RCDP.tt25.ripe.net.19990908-000000
  -rw-r--r--  5663967 Sep 10 10:00 new-RCDP.tt25.ripe.net.19990909-000000
  -rw-r--r--  2419921 Sep 10 10:00 new-RCDP.tt25.ripe.net.19990910-000000
  -rw-r--r--    66598 Sep 10 10:10 RCDP.tt25.ripe.net.19990910-095255

3. double check we're not missing anything:

   [TT@tt25:136]  wc RCDP.tt25.ripe.net.19990907-170448
     161180 1934160 15280948 RCDP.tt25.ripe.net.19990907-170448

   [TT@tt25:137] wc new-*
      17020  204240 1613218 new-RCDP.tt25.ripe.net.19990907-170448
      58896  706752 5583842 new-RCDP.tt25.ripe.net.19990908-000000
      59757  717084 5663967 new-RCDP.tt25.ripe.net.19990909-000000
      25507  306084 2419921 new-RCDP.tt25.ripe.net.19990910-000000
     161180 1934160 15280948 total

   (total of new-* equals original)

4. rename files

mv -i RCDP.tt25.ripe.net.19990907-170448 old.RCDP.tt25.ripe.net.19990907-170448
mv -i new-RCDP.tt25.ripe.net.19990907-170448 RCDP.tt25.ripe.net.19990907-170448
mv -i new-RCDP.tt25.ripe.net.19990908-000000 RCDP.tt25.ripe.net.19990908-000000
mv -i new-RCDP.tt25.ripe.net.19990909-000000 RCDP.tt25.ripe.net.19990909-000000
mv -i new-RCDP.tt25.ripe.net.19990910-000000 RCDP.tt25.ripe.net.19990910-000000

5. final check

  [TT@tt25:145] ls -ltr |grep RCDP |tail 

  should not report any files starting with "new-RCDP" string


Datataking on ginkgo will pick up the newly created RCDP files, recognizes
the different date strings in the filenames, and will redo the merging
and root_data creation for these days.


-- Rene