Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-453

Syslog filling up cluster VM's - one VM at 12MB

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Won't Do
    • Icon: Medium Medium
    • Frankfurt Release
    • None
    • None

      As a watch for periodic saturation of the FS - see the 40G issue in
      https://lists.onap.org/pipermail/onap-discuss/2018-June/010050.html
      https://lists.onap.org/pipermail/onap-discuss/2018-June/010056.html

      on cluster.onap.info I only see 12Mb

      ubuntu@cluster1.onap.info
      ubuntu@ip-172-31-28-156:~$ df
      Filesystem                                       1K-blocks     Used        Available Use% Mounted on
      udev                                              15691140        0         15691140   0% /dev
      tmpfs                                              3139820   351036          2788784  12% /run
      /dev/xvda1                                        81254044 65222160         16015500  81% /
      tmpfs                                             15699096     6892         15692204   1% /dev/shm
      tmpfs                                                 5120        0             5120   0% /run/lock
      tmpfs                                             15699096        0         15699096   0% /sys/fs/cgroup
      fs-023adc1b.efs.us-west-1.amazonaws.com:/ 9007199254739968  3729408 9007199251010560   1% /dockerdata-nfs
      tmpfs                                              3139820        0          3139820   0% /run/user/1000
      
      -rw-r-----  1 syslog adm     8292567 Jun  4 21:07 syslog
      -rw-r-----  1 syslog adm    12395747 Jun  4 06:25 syslog.1
      -rw-r-----  1 syslog adm      710414 Jun  3 06:25 syslog.2.gz
      -rw-r-----  1 syslog adm      700347 Jun  2 06:25 syslog.3.gz
      -rw-r-----  1 syslog adm      721147 Jun  1 06:25 syslog.4.gz
      -rw-r-----  1 syslog adm      636081 May 31 06:25 syslog.5.gz
      -rw-r-----  1 syslog adm      373696 May 30 06:25 syslog.6.gz
      -rw-r-----  1 syslog adm      109797 May 29 06:25 syslog.7.gz
      
      cluster0.onap.info
      ubuntu@ip-172-31-17-88:~$ df
      Filesystem                                       1K-blocks     Used        Available Use% Mounted on
      udev                                              15691140        0         15691140   0% /dev
      tmpfs                                              3139820   327024          2812796  11% /run
      /dev/xvda1                                        81254044 66859720         14377940  83% /
      tmpfs                                             15699096    14160         15684936   1% /dev/shm
      tmpfs                                                 5120        0             5120   0% /run/lock
      tmpfs                                             15699096        0         15699096   0% /sys/fs/cgroup
      fs-023adc1b.efs.us-west-1.amazonaws.com:/ 9007199254739968  3729408 9007199251010560   1% /dockerdata-nfs
      tmpfs                                              3139820        0          3139820   0% /run/user/1000
      
      
      -rw-r-----  1 syslog adm    4497347 Jun  4 21:11 syslog
      -rw-r-----  1 syslog adm    8147958 Jun  4 06:25 syslog.1
      -rw-r-----  1 syslog adm     527451 Jun  3 06:25 syslog.2.gz
      -rw-r-----  1 syslog adm     528870 Jun  2 06:25 syslog.3.gz
      -rw-r-----  1 syslog adm     558364 Jun  1 06:25 syslog.4.gz
      -rw-r-----  1 syslog adm     386747 May 31 06:25 syslog.5.gz
      -rw-r-----  1 syslog adm       2876 May 30 06:25 syslog.6.gz
      -rw-r-----  1 syslog adm       2493 May 29 06:25 syslog.7.gz
      
      cluster2.onap.info
      ubuntu@ip-172-31-28-244:~$ df
      Filesystem                                       1K-blocks     Used        Available Use% Mounted on
      udev                                              15691140        0         15691140   0% /dev
      tmpfs                                              3139820   180076          2959744   6% /run
      /dev/xvda1                                        81254044 66635832         14601828  83% /
      tmpfs                                             15699096    14472         15684624   1% /dev/shm
      tmpfs                                                 5120        0             5120   0% /run/lock
      tmpfs                                             15699096        0         15699096   0% /sys/fs/cgroup
      fs-023adc1b.efs.us-west-1.amazonaws.com:/ 9007199254739968  3729408 9007199251010560   1% /dockerdata-nfs
      tmpfs                                              3139820        0          3139820   0% /run/user/1000
      ubuntu@ip-172-31-28-244:~$ sudo ls /var/log/sys* -la
      -rw-r----- 1 syslog adm 2468538 Jun  4 21:12 /var/log/syslog
      -rw-r----- 1 syslog adm 2473441 Jun  4 06:25 /var/log/syslog.1
      -rw-r----- 1 syslog adm  158894 Jun  3 06:25 /var/log/syslog.2.gz
      -rw-r----- 1 syslog adm  156342 Jun  2 06:25 /var/log/syslog.3.gz
      -rw-r----- 1 syslog adm  160135 Jun  1 06:25 /var/log/syslog.4.gz
      -rw-r----- 1 syslog adm  202650 May 31 06:25 /var/log/syslog.5.gz
      -rw-r----- 1 syslog adm  109941 May 30 06:25 /var/log/syslog.6.gz
      -rw-r----- 1 syslog adm  109698 May 29 06:25 /var/log/syslog.7.gz
      
      master
      cluster.onap.info
      ubuntu@ip-172-31-24-168:~$ df
      Filesystem                                       1K-blocks    Used        Available Use% Mounted on
      udev                                              15691140       0         15691140   0% /dev
      tmpfs                                              3139820    8996          3130824   1% /run
      /dev/xvda1                                        81254044 8288004         72949656  11% /
      tmpfs                                             15699096     380         15698716   1% /dev/shm
      tmpfs                                                 5120       0             5120   0% /run/lock
      tmpfs                                             15699096       0         15699096   0% /sys/fs/cgroup
      tmpfs                                              3139820       0          3139820   0% /run/user/1000
      fs-023adc1b.efs.us-west-1.amazonaws.com:/ 9007199254739968 3729408 9007199251010560   1% /dockerdata-nfs
      -rw-r-----  1 syslog adm     15585 Jun  4 21:11 syslog
      -rw-r-----  1 syslog adm     21077 Jun  4 06:25 syslog.1
      -rw-r-----  1 syslog adm      1798 Jun  3 06:25 syslog.2.gz
      -rw-r-----  1 syslog adm      1973 Jun  2 06:25 syslog.3.gz
      -rw-r-----  1 syslog adm      1711 Jun  1 06:25 syslog.4.gz
      -rw-r-----  1 syslog adm      1796 May 31 06:25 syslog.5.gz
      -rw-r-----  1 syslog adm      1710 May 30 06:25 syslog.6.gz
      -rw-r-----  1 syslog adm      1732 May 29 06:25 syslog.7.gz
      
      
      and the long lived CD system under jenkins.onap.info
      
      master0.onap.info
      ubuntu@ip-172-31-18-85:~$ df
      Filesystem                                       1K-blocks      Used        Available Use% Mounted on
      udev                                              62905708         0         62905708   0% /dev
      tmpfs                                             12582736    127836         12454900   2% /run
      /dev/xvda1                                       162574680 132935868         29622428  82% /
      tmpfs                                             62913664     12700         62900964   1% /dev/shm
      tmpfs                                                 5120         0             5120   0% /run/lock
      tmpfs                                             62913664         0         62913664   0% /sys/fs/cgroup
      fs-5fdd1b26.efs.us-east-2.amazonaws.com:/ 9007199254739968   5152768 9007199249587200   1% /dockerdata-nfs
      tmpfs                                             12582736         0         12582736   0% /run/user/1000
      ubuntu@ip-172-31-18-85:~$ sudo ls /var/log/sys* -la
      -rw-r----- 1 syslog adm 1847140 Jun  4 21:13 /var/log/syslog
      -rw-r----- 1 syslog adm 2655658 Jun  4 06:25 /var/log/syslog.1
      -rw-r----- 1 syslog adm  337236 Jun  3 06:25 /var/log/syslog.2.gz
      -rw-r----- 1 syslog adm  307204 Jun  2 06:25 /var/log/syslog.3.gz
      -rw-r----- 1 syslog adm  290348 Jun  1 06:25 /var/log/syslog.4.gz
      -rw-r----- 1 syslog adm  929035 May 31 06:25 /var/log/syslog.5.gz
      

            pau2882 pau2882
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: