-
Bug
-
Resolution: Cannot Reproduce
-
Medium
-
None
-
None
-
LOG Sprint 8
On an older master we had a lot more logs after some gui and robot actions, currently in master 20171004 with/without the AAI filebeat patch - we are missing some.
It is looking like the logs in AAI that were split into 2 will need their appender changed (would make sense since the war writes to different locations in K8S) - in rackspace we see logs on PUTS ok - in OOM they are missing - looking into it
Live
<file>${logDirectory}/rest/audit.log</file>
defined in aai/resources/aai-resources/ajsc-shared-config/etc/logback.xml
<property name="logDirectory" value="${AJSC_HOME}/logs" />
Proposed
logging-analytics/onap/aai/aai-resources/conf/logback.xml
7 day old
source.keyword: Descending | Count |
---|---|
/var/log/onap/mso/asdcController/debug.log | 59,831 |
/var/log/onap/policy/drools/error.log | 39,983 |
/var/log/onap/sdc/sdc-be/error.log | 33,604 |
/var/log/onap/appc/karaf.log | 12,102 |
/var/log/onap/sdnc/karaf.log | 11,467 |
/var/log/onap/appc/debug.log | 2,720 |
/var/log/onap/vid/application.log | 262 |
/var/log/onap/policy/ep_sdk_app/debug.log | 196 |
/var/log/onap/mso/asdcController/error.log | 164 |
/var/log/onap/policy/xacml-pdp-rest/debug.log | 84 |
/var/log/onap/mso/asdcController/metrics.log | 82 |
/var/log/onap/EELF/application.log | 36 |
/var/log/onap/mso/networkAdapter/debug.log | 9 |
/var/log/onap/mso/tenantAdapter/debug.log | 9 |
/var/log/onap/mso/vnfAdapter/debug.log | 9 |
/var/log/onap/sdc/sdc-be/audit.log | 8 |
/var/log/onap/policy/xacml-pap-rest/debug.log | 7 |
/var/log/onap/mso/apiHandlerInfra/debug.log | 5 |
/var/log/onap/mso/bpmn/debug.log | 5 |
/var/log/onap/mso/sdncAdapter/debug.log | 5 |
/var/log/onap/vid/error.log | 5 |
/var/log/onap/EELF/error.log | 4 |
/var/log/onap/policy/ep_sdk_app/error.log | 4 |
/var/log/onap/mso/requestDB/debug.log | 3 |
/var/log/onap/vid/debug.log | 2 |
/var/log/onap/mso/bpmn/audit.log | 1 |
/var/log/onap/policy/ep_sdk_app/audit.log | 1 |
/var/log/onap/policy/ep_sdk_app/metrics.log | 1 |
/var/log/onap/vid/audit.log | 1 |
/var/log/onap/vid/metrics.log | 1 |
current system 20171004 + oom-112 patch
source.keyword: Descending | Count |
---|---|
/var/log/onap/sdc/sdc-be/error.log | 337,829 |
/var/log/onap/mso/asdcController/debug.log | 262,176 |
/var/log/onap/policy/drools/error.log | 178,300 |
/var/log/onap/sdnc/karaf.log | 38,334 |
/var/log/onap/appc/karaf.log | 21,043 |
/var/log/onap/aai/aai-ml/error.log | 1,809 |
/var/log/onap/appc/debug.log | 1,360 |
/var/log/onap/aai/aai-sdb/error.log | 4 |
/var/log/onap/sdc/sdc-be/audit.log | 4 |
In the screencap - the left is from a 7 day old view, the right is yesterdays master + the ---- aai filebeat patchOOM-112
root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it aai-resources-338473047-9mz6t bash Defaulting container name to aai-resources. Use 'kubectl describe pod/aai-resources-338473047-9mz6t' to see all of the containers in this pod. root@aai-resources-338473047-9mz6t:/opt/app/aai-resources# ls bin data etc init-chef.sh log services bundleconfig docker-entrypoint.sh extApps jetty logs conf docs extJars lib scripts root@aai-resources-338473047-9mz6t:/opt/app/aai-resources# cd /var/log/ alternatives.log btmp faillog onap/ apt/ dmesg fsck/ wtmp bootstrap.log dpkg.log lastlog root@aai-resources-338473047-9mz6t:/opt/app/aai-resources# cd /var/log/onap/ root@aai-resources-338473047-9mz6t:/var/log/onap# ls aai root@aai-resources-338473047-9mz6t:/var/log/onap# cd aai/aai-resources/ dmaapAAIEventConsumer/ perf-audit/ rest/ root@aai-resources-338473047-9mz6t:/var/log/onap# cd aai/aai-resources/rest root@aai-resources-338473047-9mz6t:/var/log/onap/aai/aai-resources/rest# ls audit.log debug.log error.log metrics.log sane.log translog.log <7-9mz6t:/var/log/onap/aai/aai-resources/rest# ls -la total 8 drwxr-xr-x 2 root aaiadmin 4096 Oct 4 18:11 . drwxr-xr-x 5 root aaiadmin 4096 Oct 4 18:11 .. -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 audit.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 debug.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 error.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 metrics.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 sane.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 translog.log root@aai-resources-338473047-9mz6t:/var/log/onap/aai/aai-resources/rest# exit exit root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it aai-traversal-2033243133-4vhwt bash Defaulting container name to aai-traversal. Use 'kubectl describe pod/aai-traversal-2033243133-4vhwt' to see all of the containers in this pod. <4vhwt:/opt/app/aai-traversal# cd /var/log/onap/aai/ dmaapAAIEventConsumer/ perf-audit/ rest/ <4vhwt:/opt/app/aai-traversal# cd /var/log/onap/aai/rest/ root@aai-traversal-2033243133-4vhwt:/var/log/onap/aai/rest# ls -la total 8 drwxr-xr-x 2 root aaiadmin 4096 Oct 4 18:11 . drwxr-xr-x 5 root aaiadmin 4096 Oct 4 18:11 .. -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 audit.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 debug.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 error.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 metrics.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 sane.log -rw-r--r-- 1 root aaiadmin 0 Oct 4 18:11 translog.log root@aai-traversal-2033243133-4vhwt:/var/log/onap/aai/rest# exit exit root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it model-loader-service-3356570452-0dsp6 bash Defaulting container name to model-loader-service. Use 'kubectl describe pod/model-loader-service-3356570452-0dsp6' to see all of the containers in this pod. root@model-loader-service-3356570452-0dsp6:/# ls -la /var/log/ alternatives.log dpkg.log onap/ apt/ faillog unattended-upgrades/ bootstrap.log fontconfig.log upstart/ btmp fsck/ wtmp dmesg lastlog <356570452-0dsp6:/# ls -la /var/log/onap/aai/aai-ml/ audit.log error.2017-10-04.0.log metrics.log debug.log error.log <356570452-0dsp6:/# ls -la /var/log/onap/aai/aai-ml/ total 888 drwxr-xr-x 2 root root 4096 Oct 4 20:00 . drwxr-xr-x 3 root root 4096 Oct 4 18:08 .. -rw-r--r-- 1 root root 0 Oct 4 18:08 audit.log -rw-r--r-- 1 root root 0 Oct 4 18:08 debug.log -rw-r--r-- 1 root root 111123 Oct 4 19:59 error.2017-10-04.0.log -rw-r--r-- 1 root root 777882 Oct 5 15:29 error.log -rw-r--r-- 1 root root 0 Oct 4 18:08 metrics.log root@model-loader-service-3356570452-0dsp6:/# exit exit root@kos1001:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it search-data-service-2366687049-1hnml bash Defaulting container name to search-data-service. Use 'kubectl describe pod/search-data-service-2366687049-1hnml' to see all of the containers in this pod. <66687049-1hnml:/# ls -la /var/log/onap/aai/aai-sdb/ audit.log debug.log error.log metrics.log <66687049-1hnml:/# ls -la /var/log/onap/aai/aai-sdb/ audit.log debug.log error.log metrics.log <66687049-1hnml:/# ls -la /var/log/onap/aai/aai-sdb/ total 36 drwxr-xr-x 2 root root 4096 Oct 4 18:08 . drwxr-xr-x 3 root root 4096 Oct 4 18:08 .. -rw-r--r-- 1 root root 0 Oct 4 18:08 audit.log -rw-r--r-- 1 root root 0 Oct 4 18:08 debug.log -rw-r--r-- 1 root root 28450 Oct 4 18:08 error.log -rw-r--r-- 1 root root 0 Oct 4 18:08 metrics.log <66687049-1hnml:/# cat /var/log/onap/aai/aai-sdb/error.log 2017-10-04T22:08:30.764Z [main] ERROR ajsc.VandelayService NMBS-VANDELAYSVC-0565: For user [search-data-service service], [{}] failed to import due to non-existent Context name
missing
root@mso-3911927766-bmww7:/var/log/onap/mso/bpmn# ls -la total 36 drwxr-xr-x 2 root root 4096 Oct 4 10:09 . drwxr-xr-x 10 root root 4096 Oct 4 10:09 .. -rw-r--r-- 1 root root 496 Oct 4 10:09 audit.log -rw-r--r-- 1 root root 23268 Oct 4 10:09 debug.log -rw-r--r-- 1 root root 0 Oct 4 10:09 error.log -rw-r--r-- 1 root root 0 Oct 4 10:09 metrics.log root@mso-3911927766-bmww7:/var/log/onap/mso/bpmn# cat debug.log 2017-10-04 14:09:20,623 [ServerService Thread Pool -- 88] INFO com.att.eelf.debug ServiceName=contextInitialized AlertSeverity=0 ErrorDesc= TargetServiceName= ServiceNameIsMethodName=true ErrorCode=null TargetEntity= MSO-GENERAL-5414I Prepare to initialize loggers from /etc/mso/config.d/logback.bpmn.xml
- blocks
-
LOG-80 Kibana does not seem to show all the logs from application pods
- Closed
- is blocked by
-
OOM-332 Add AAI filebeat container - blocked by LOG-67
- Closed
-
OOM-362 CNFE: AAI-traversal/aai-resources failing to find EcompEncoder during logging bootstrap
- Closed
-
OOM-112 OOM ONAP Logging - Configuration of all ONAP components to emit canonical logs
- Closed
- relates to
-
LOG-67 Add AAI filebeat
- Closed
-
LOG-64 Logger field has a length restriction of 36 which needs a fix
- Closed
-
OOM-377 update logback.xml for centralized logging
- Closed
- mentioned in
-
Page Loading...