Currently log collection occurs via the filebeat sidecar container. There is 1 container per pod that pushes logs from the emptydir PV volume to logstash.

Design Issues:
DI1: Logstash pulls directly from /var/lib/kubelet/pods***empty~dir directories
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
DI2: handle differences in where the kubelet dir is places in k8s deployments (rancher, kubeadm)
DI3: handle KaaS deployments (secondary) - like https://console.aws.amazon.com/eks/home?region=us-east-1#/home

Notes

There is an optimization that can be prototyped where we instead push logs directly from each cluster VM right from the kubelet logs containing the PV's for each container on that VM - via a DaemonSet filebeat container (1 per VM instead of 1 per microservice)

The logstash container was recently changed to a DaemonSet deployment - use this as a chart reference
Logstash is already a DaemonSet – you can use that as the deployment part of the example (small change) – the larger piece will be to point filebeat to a subset of the emptyDir PV directory or target individual pod PVs on that VM for log shipping.
Example of where logs are put for the emptyDir PV

/var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/onap-*l/*/*/*.log
ubuntu@ip-172-31-55-235:~/_dev/20180712-log-137/oom/kubernetes$ 
sudo ls /var/lib/kubelet/pods
/b3f9e7bc-8d79-11e8-969d-02a1d5a57c25/
volumes/
kubernetes.io~empty-dir/
onap-logdemonode-logs/logdemonode/node -la
-rw-r--r-- 1 root root    0 Jul 22 06:37 audit.log
-rw-r--r-- 1 root root  548 Jul 22 07:01 debug.log
-rw-r--r-- 1 root root    0 Jul 22 06:37 error.log
-rw-r--r-- 1 root root    0 Jul 22 06:37 metrics.log

for example you will need to correlate the PV UUID on the VM with the kubernetes deployment pod id to map the correct pod to it’s emptyDir pv – or just push everything appearing in the kubelet dir.
I am hoping that a switch of the service to a DaemonSet – a one line change and blind pushing of all or a subset (minus the kubernetes/rancher/Grafana pods) of the volumes/kubernetes.io~empty-dir would be sufficient.

Architectural discussion with Roger today 20180802 - the solution above is not fully viable because the filebeat DS will push to the logstash service (not the DS) - which will fanout the requests to possibly other VM's and cause unnecessary network traffic.

requirements
R1: reduce network traffic outside individual VM
R2: reduce # of sidecar containers
R3: Handle log load spike - (filebeat (has a cache) or logstash) must queue/cache traffic to es - when es is n/a (temp) - until es is clustered/(its own daemonset)
If we switch to logstash and there is no cache - add redis? inbetween ls and es

Better solution is to either
0) currently processing logs in logstash DS, by shipping filebeat sidecar logs pre/uService
1) process logs in the logstash DS, ship the logs to FB DS which has a buffer where FB sends to ES.
issues:
vm0 could send logs to vm1 (because of the load-balances ls (DS) service) - too much network traffic
2) process Logs in the logstash DS ship to ES as usual (no FB required)
3) Process logs in the logstash DS ship to ES with a redis cache in between

#2 is likely the way we should go

Work items
WI1: verify logstash file plugin can read logs in multiple dirs from /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/onap-*l///.log on a particular VM

• turn off filebeat - by commenting the "name" section in charts/pdp/templates/statefullset.yaml for filebeat
• restart the service look for only 1/1
• follow https://wiki.onap.org/pages/viewpage.action?pageId=28378955#ONAPApplicationLoggingSpecificationv1.2(Casablanca)-Deployingdemopod (--set policy.enabled=false , do a make all, make policy then flip the enabled flag back to true)
  sudo helm upgrade -i onap local/onap --namespace onap --set policy.enabled=true
• pick some logs already appearing in kibana and backtrack to the vm emiting these - ie onap-pdp - you are using policy so no work here
• use kubectl get pods --all-namespaces -o wide
• to get the host
• goto /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/onap-*l///.log
• actual work is....
  2 phase - turn off filebeat, then turn on logstash file plugin
  change logstash pipeline to also use files not just the port
  https://git.onap.org/oom/tree/kubernetes/log/charts/log-logstash/resources/config/onap-pipeline.conf
  turn off filebeat
  recomplile/restart policy
  check kibana for logs from ls (no fb should be coming through)

emails

From: Michael O'Brien
Sent: Thursday, August 2, 2018 9:03 AM
To: GENTILE, ALEXANDER <ag943j@att.com>
Cc: THAKORE, SHISHIR B <st2373@att.com>; onap-discuss@lists.onap.org
Subject: RE: Understanding Daemonset in ONAP

Alexander,
Hi, sounds good. I have some time later today (pomba at 8, tsc at 10, acumos at 1130 – free at 1400 GMT-4 today, more free friday.
Shishir and his team(Sanjay, Spondon, Lorraine, Jun, Dave) at AT&T have been instrumental over the last 6 months in finalizing the logging spec, getting the EPICs up to date/defined and pushing the overall project – along with our collaboration in Acumos around logging.

Expanding to the list as usual for the rest of the team.
We can use my zoom http://zoom.us/j/7939937123 at 2 pm

The refactoring of the filebeat sidecar container approach to use a daemonSet per cluster VM needs to be evaluated as part of the Casablanca scope – it is #9 on the list
https://wiki.onap.org/display/DW/Logging+Casablanca+Scope
There is a story off the S3P epic
https://jira.onap.org/browse/LOG-380
for a per VM Filebeat DaemonSet – that you can use for your work.
https://jira.onap.org/browse/LOG-587
Keep documentation/experimentation notes on the jira or the DI section of the wiki
https://wiki.onap.org/pages/viewpage.action?pageId=28378955#ONAPApplicationLoggingSpecificationv1.2(Casablanca)-DI8:LOG-587:RefactoringfilebeattorunasaperVMDaemonSetdeployment

Logstash is already a DaemonSet – you can use that as the deployment part of the example (small change) – the larger piece will be to point filebeat to a subset of the emptyDir PV directory or target individual pod PVs on that VM for log shipping.
for example you will need to correlate the PV UUID on the VM with the kubernetes deployment pod id to map the correct pod to it’s emptyDir pv – or just push everything appearing in the kubelet dir.
I am hoping that a switch of the service to a DaemonSet – a one line change and blind pushing of all or a subset (minus the kubernetes/rancher/Grafana pods) of the volumes/kubernetes.io~empty-dir would be sufficient.

Example of where logs are put for the emptyDir PV
ubuntu@ip-172-31-55-235:~/_dev/20180712-log-137/oom/kubernetes$ sudo ls /var/lib/kubelet/pods/b3f9e7bc-8d79-11e8-969d-02a1d5a57c25/volumes/kubernetes.io~empty-dir/onap-logdemonode-logs/logdemonode/node -la
rw-rr- 1 root root 0 Jul 22 06:37 audit.log
rw-rr- 1 root root 548 Jul 22 07:01 debug.log
rw-rr- 1 root root 0 Jul 22 06:37 error.log
rw-rr- 1 root root 0 Jul 22 06:37 metrics.log

You can use the RI logdemo pod 30258 port – to exercise the ELK+filebeat stack without running the rest of onap – as long as you deploy on a cluster with at least 2 nodes and modify the logdemo deployment to also be a daemonset (currently a single line change)
https://git.onap.org/logging-analytics/tree/reference/logging-kubernetes
documented on
https://wiki.onap.org/pages/viewpage.action?pageId=28378955#ONAPApplicationLoggingSpecificationv1.2(Casablanca)-DeveloperGuide

If you need help with setting up a cluster you can use either the procedures/scripts in readthedocs http://onap.readthedocs.io/en/latest/submodules/oom.git/docs/oom_setup_kubernetes_rancher.html for openstack, I use an AWS cluster so if you need a cloud native cluster like on AWS use the wiki and code below
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-ClusteredDeployment
https://git.onap.org/logging-analytics/tree/deploy/aws
undercloud agnostic setup
https://git.onap.org/logging-analytics/tree/deploy/rancher/oom_rancher_setup.sh

thank you
/michael

From: GENTILE, ALEXANDER <ag943j@att.com>
Sent: Wednesday, August 1, 2018 6:34 PM
To: Michael O'Brien <Frank.Obrien@amdocs.com>
Cc: THAKORE, SHISHIR B <st2373@att.com>
Subject: Understanding Daemonset in ONAP

Hello Michael,

My name is Alex Gentile and I am an intern at AT&T working with Shishir on logging in ONAP. My project is about deploying Filebeat as a Daemonset to collect logs from containers rather than having Filebeat containers wrapped up with pods. Due to time constraints, I will be testing my architecture using existing Logstash pods deployed as a Daemonset as part of ONAP. I was hoping I could speak to you sometime tomorrow, 8/2, to briefly discuss the implementation of these Daemonsets so I might have a better understanding of how they work.

The last day of my internship is next Friday 8/10 so I have very limited time left to conduct my analysis. I am available any time tomorrow, so we can talk any time that is best for you.

Thank you,
Alex Gentile