Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-71

Some of the AAI ELK logs are missing from 20171004 that were in around 20170928

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Medium Medium
    • Beijing Release
    • None
    • None
    • LOG Sprint 8

      On an older master we had a lot more logs after some gui and robot actions, currently in master 20171004 with/without the AAI filebeat patch - we are missing some.

      It is looking like the logs in AAI that were split into 2 will need their appender changed (would make sense since the war writes to different locations in K8S) - in rackspace we see logs on PUTS ok - in OOM they are missing - looking into it

       Live

      <file>${logDirectory}/rest/audit.log</file>

      defined in aai/resources/aai-resources/ajsc-shared-config/etc/logback.xml

      <property name="logDirectory" value="${AJSC_HOME}/logs" />

       Proposed

      logging-analytics/onap/aai/aai-resources/conf/logback.xml

       

       

      7 day old

      source.keyword: Descending  Count 
      /var/log/onap/mso/asdcController/debug.log 59,831
      /var/log/onap/policy/drools/error.log 39,983
      /var/log/onap/sdc/sdc-be/error.log 33,604
      /var/log/onap/appc/karaf.log 12,102
      /var/log/onap/sdnc/karaf.log 11,467
      /var/log/onap/appc/debug.log 2,720
      /var/log/onap/vid/application.log 262
      /var/log/onap/policy/ep_sdk_app/debug.log 196
      /var/log/onap/mso/asdcController/error.log 164
      /var/log/onap/policy/xacml-pdp-rest/debug.log 84
      /var/log/onap/mso/asdcController/metrics.log 82
      /var/log/onap/EELF/application.log 36
      /var/log/onap/mso/networkAdapter/debug.log 9
      /var/log/onap/mso/tenantAdapter/debug.log 9
      /var/log/onap/mso/vnfAdapter/debug.log 9
      /var/log/onap/sdc/sdc-be/audit.log 8
      /var/log/onap/policy/xacml-pap-rest/debug.log 7
      /var/log/onap/mso/apiHandlerInfra/debug.log 5
      /var/log/onap/mso/bpmn/debug.log 5
      /var/log/onap/mso/sdncAdapter/debug.log 5
      /var/log/onap/vid/error.log 5
      /var/log/onap/EELF/error.log 4
      /var/log/onap/policy/ep_sdk_app/error.log 4
      /var/log/onap/mso/requestDB/debug.log 3
      /var/log/onap/vid/debug.log 2
      /var/log/onap/mso/bpmn/audit.log 1
      /var/log/onap/policy/ep_sdk_app/audit.log 1
      /var/log/onap/policy/ep_sdk_app/metrics.log 1
      /var/log/onap/vid/audit.log 1
      /var/log/onap/vid/metrics.log 1

       

      current system 20171004 + oom-112 patch

      source.keyword: Descending  Count 
      /var/log/onap/sdc/sdc-be/error.log 337,829
      /var/log/onap/mso/asdcController/debug.log 262,176
      /var/log/onap/policy/drools/error.log 178,300
      /var/log/onap/sdnc/karaf.log 38,334
      /var/log/onap/appc/karaf.log 21,043
      /var/log/onap/aai/aai-ml/error.log 1,809
      /var/log/onap/appc/debug.log 1,360
      /var/log/onap/aai/aai-sdb/error.log 4
      /var/log/onap/sdc/sdc-be/audit.log 4

       

      In the screencap - the left is from a 7 day old view, the right is yesterdays master + the --OOM-112-- aai filebeat patch

      root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it aai-resources-338473047-9mz6t bash
      Defaulting container name to aai-resources.
      Use 'kubectl describe pod/aai-resources-338473047-9mz6t' to see all of the containers in this pod.
      root@aai-resources-338473047-9mz6t:/opt/app/aai-resources# ls
      bin data etc init-chef.sh log services
      bundleconfig docker-entrypoint.sh extApps jetty logs
      conf docs extJars lib scripts
      root@aai-resources-338473047-9mz6t:/opt/app/aai-resources# cd /var/log/
      alternatives.log btmp faillog onap/
      apt/ dmesg fsck/ wtmp
      bootstrap.log dpkg.log lastlog
      root@aai-resources-338473047-9mz6t:/opt/app/aai-resources# cd /var/log/onap/
      root@aai-resources-338473047-9mz6t:/var/log/onap# ls
      aai
      root@aai-resources-338473047-9mz6t:/var/log/onap# cd aai/aai-resources/
      dmaapAAIEventConsumer/ perf-audit/ rest/
      root@aai-resources-338473047-9mz6t:/var/log/onap# cd aai/aai-resources/rest
      root@aai-resources-338473047-9mz6t:/var/log/onap/aai/aai-resources/rest# ls
      audit.log debug.log error.log metrics.log sane.log translog.log
      <7-9mz6t:/var/log/onap/aai/aai-resources/rest# ls -la
      total 8
      drwxr-xr-x 2 root aaiadmin 4096 Oct 4 18:11 .
      drwxr-xr-x 5 root aaiadmin 4096 Oct 4 18:11 ..
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 audit.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 debug.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 error.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 metrics.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 sane.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 translog.log
      root@aai-resources-338473047-9mz6t:/var/log/onap/aai/aai-resources/rest# exit
      exit
      root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it aai-traversal-2033243133-4vhwt bash
      Defaulting container name to aai-traversal.
      Use 'kubectl describe pod/aai-traversal-2033243133-4vhwt' to see all of the containers in this pod.
      <4vhwt:/opt/app/aai-traversal# cd /var/log/onap/aai/
      dmaapAAIEventConsumer/ perf-audit/ rest/
      <4vhwt:/opt/app/aai-traversal# cd /var/log/onap/aai/rest/
      root@aai-traversal-2033243133-4vhwt:/var/log/onap/aai/rest# ls -la
      total 8
      drwxr-xr-x 2 root aaiadmin 4096 Oct 4 18:11 .
      drwxr-xr-x 5 root aaiadmin 4096 Oct 4 18:11 ..
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 audit.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 debug.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 error.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 metrics.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 sane.log
      -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 translog.log
      root@aai-traversal-2033243133-4vhwt:/var/log/onap/aai/rest# exit
      exit
      root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it model-loader-service-3356570452-0dsp6 bash
      Defaulting container name to model-loader-service.
      Use 'kubectl describe pod/model-loader-service-3356570452-0dsp6' to see all of the containers in this pod.
      root@model-loader-service-3356570452-0dsp6:/# ls -la /var/log/
      alternatives.log dpkg.log onap/
      apt/ faillog unattended-upgrades/
      bootstrap.log fontconfig.log upstart/
      btmp fsck/ wtmp
      dmesg lastlog
      <356570452-0dsp6:/# ls -la /var/log/onap/aai/aai-ml/
      audit.log error.2017-10-04.0.log metrics.log
      debug.log error.log
      <356570452-0dsp6:/# ls -la /var/log/onap/aai/aai-ml/
      total 888
      drwxr-xr-x 2 root root 4096 Oct 4 20:00 .
      drwxr-xr-x 3 root root 4096 Oct 4 18:08 ..
      -rw-r--r-- 1 root root 0 Oct 4 18:08 audit.log
      -rw-r--r-- 1 root root 0 Oct 4 18:08 debug.log
      -rw-r--r-- 1 root root 111123 Oct 4 19:59 error.2017-10-04.0.log
      -rw-r--r-- 1 root root 777882 Oct 5 15:29 error.log
      -rw-r--r-- 1 root root 0 Oct 4 18:08 metrics.log
      root@model-loader-service-3356570452-0dsp6:/# exit
      exit
      root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it search-data-service-2366687049-1hnml bash
      Defaulting container name to search-data-service.
      Use 'kubectl describe pod/search-data-service-2366687049-1hnml' to see all of the containers in this pod.
      <66687049-1hnml:/# ls -la /var/log/onap/aai/aai-sdb/
      audit.log debug.log error.log metrics.log
      <66687049-1hnml:/# ls -la /var/log/onap/aai/aai-sdb/
      audit.log debug.log error.log metrics.log
      <66687049-1hnml:/# ls -la /var/log/onap/aai/aai-sdb/
      total 36
      drwxr-xr-x 2 root root 4096 Oct 4 18:08 .
      drwxr-xr-x 3 root root 4096 Oct 4 18:08 ..
      -rw-r--r-- 1 root root 0 Oct 4 18:08 audit.log
      -rw-r--r-- 1 root root 0 Oct 4 18:08 debug.log
      -rw-r--r-- 1 root root 28450 Oct 4 18:08 error.log
      -rw-r--r-- 1 root root 0 Oct 4 18:08 metrics.log
      <66687049-1hnml:/# cat /var/log/onap/aai/aai-sdb/error.log
      2017-10-04T22:08:30.764Z [main] ERROR ajsc.VandelayService NMBS-VANDELAYSVC-0565: For user [search-data-service service], [{}] failed to import due to non-existent Context name

      missing

      root@mso-3911927766-bmww7:/var/log/onap/mso/bpmn# ls -la
      total 36
      drwxr-xr-x 2 root root 4096 Oct 4 10:09 .
      drwxr-xr-x 10 root root 4096 Oct 4 10:09 ..
      -rw-r--r-- 1 root root 496 Oct 4 10:09 audit.log
      -rw-r--r-- 1 root root 23268 Oct 4 10:09 debug.log
      -rw-r--r-- 1 root root 0 Oct 4 10:09 error.log
      -rw-r--r-- 1 root root 0 Oct 4 10:09 metrics.log
      root@mso-3911927766-bmww7:/var/log/onap/mso/bpmn# cat debug.log
      2017-10-04 14:09:20,623 [ServerService Thread Pool -- 88] INFO com.att.eelf.debug ServiceName=contextInitialized AlertSeverity=0 ErrorDesc= TargetServiceName= ServiceNameIsMethodName=true ErrorCode=null TargetEntity= MSO-GENERAL-5414I Prepare to initialize loggers from /etc/mso/config.d/logback.bpmn.xml
      
      
      
      

            michaelobrien michaelobrien
            michaelobrien michaelobrien
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: