Uploaded image for project: 'Logging analytics'
  1. Logging analytics
  2. LOG-930

POMBA/LOG Integration with Datalake as a Service platform - TSC 20190117 - via DCAE

    Details

    • Epic Name:
      POMBA/LOG Integration with Datalake as a Service platform - TSC 20190117 - via DCAE

      Description

      discuss how for example we provide or consume
      ElasticSearch as a Service
      Kibana as a Service
      update:
      https://wiki.onap.org/display/DW/PTL+2019-02-04

      aka firehose(twitter) for dmaap
      https://wiki.onap.org/display/DW/TSC+2019-01-17

      From Kenny Paul (LFN) to Everyone: (09:56)
      
#topic datalake
Guobiao Mo reviedes his slides

      From Me to Everyone: (10:03)
      
Q: we could provide hooks into the 3 existing ELK stacks (LOG, POMBA, CLAMP) - most already have external facing nodeports for their elasticsearch containers - btw there are 5 elasticsearch, 5 cassandara, 14+ mariadb/myssql 1 mongodb, 3 postgres, 2 redis and one graphdb in onap
there is a mongodb container in nbi/extapi

      From Alexander Vul to Everyone: (10:07)
      
Is this infrastructure meant to be deployed as part of the ONAP footprint?

      From Keong Lim k00759777 to Everyone: (10:08)
      
since DMaaP is based on Kafka, couldn't this also be done by simply retaining all the messages in Kafka?

      From Alexander Vul to Everyone: (10:08)
      
Kafka is not meant to handle the volume of data we are talking about here...

      From Keong Lim k00759777 to Everyone: (10:09)
      
there are orgs that do exactly that already with kafka

      From Alexander Vul to Everyone: (10:09)
      
I am sure there are...

      From Catherine Lefevre to Everyone: (10:10)
      
@Denghui, Andy - agenda updated but have you planned a TSC Readout about NS domain / Internal NS Descriptor today?

      From Michael O'Brien to Everyone: (10:11)
      
So datalake will be Elasticsearch as a service - we should discuss this

      From Catherine Lefevre to Everyone: (10:11)
      
if not then I can remove this readout for this week

      From Michael O'Brien to Everyone: (10:12)
      
datalake pods ideally will be native kubernetes charts not cloudify blueprints for el-alto
      
      From Michael O'Brien to Everyone: (10:12)
      
datalake pods ideally will be native kubernetes charts not cloudify blueprints for el-alto

      From Ramki Krishnan to Everyone: (10:12)
      
Fair point raised by Keong Lim on Kafka - Kafka has programmable buffering.

      From Keong Lim k00759777 to Everyone: (10:12)
      
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/

      From Michael O'Brien to Everyone: (10:13)
      
woudn't we need DCAEGEN2 deployed plus secondary dep-* pods to use datalake

      From Srini Addepalli (Intel) to Everyone: (10:13)
      
Is this activity intend to be done as part of POMBA or DCAE?

      
      s this activity intend to be done as part of POMBA or DCAE?

      From Jason Hunt to Everyone: (10:14)
      
Wouldn’t we expect that operators already have data lakes?  Should we focus on how to enable them to link ONAP to their data lakes?

      From Brian to Everyone: (10:14)
      
+1 Jason's comment

      From Michael O'Brien to Everyone: (10:14)
      
+1 Jason - pluggable data lake - so this datalake is just an RI

      From Eric Debeau to Everyone: (10:14)
      
Most of operators have deployed their own data lakes. I agree that we need to explain how to link ONAP with existing data lakes

      From Fernando (Fred) Oliveira to Everyone: (10:15)
      
I agree, ONAP should enable the feeds to an external data lake

      From Andreas Geissler (Deutsche Telekom) to Everyone: (10:16)
      
agree with Eric and Jason, but as ONAP Addon the component would be useful.

      From Lingli to Everyone: (10:16)
      
+1

      From Michael O'Brien to Everyone: (10:18)
      
More very interested on the Datalake as a Platform (pluggable) aspect of this project - the actual implementation is more of an example

      From Brian to Everyone: (10:18)
      
can we do it like netbox for IPAM as a contrib ?

      From Michael O'Brien to Everyone: (10:18)
      
yes, contrib is a good idea

      From Mike Elliott to Everyone: (10:19)
      
+1 - contrib (optional)
      
      +1 - contrib (optional)

      From Me to Everyone: (10:23)
      
onap          onap-contrib-netbox-app-84f877fb76-qhc9t                      1/1       Running            0          9h
onap          onap-contrib-netbox-app-provisioning-8f2vk                    0/1       Error              0          9h
onap          onap-contrib-netbox-app-provisioning-gwtqt                    0/1       Completed          0          9h
onap          onap-contrib-netbox-app-provisioning-j6hn9                    0/1       Error              0          9h
onap          onap-contrib-netbox-nginx-5c85578697-lp2bd                    1/1       Running            0          9h
onap          onap-contrib-netbox-postgres-65f4bb6459-rc5k8                 1/1       Running            0          9h
the error is normal - the completed is the 2nd run
above https://git.onap.org/oom/tree/kubernetes/contrib

      From Eric Debeau to Everyone: (10:28)
      
+1 for contrilb

      From Ramki Krishnan to Everyone: (10:28)
      
+1 on contrib

      From Eric Debeau to Everyone: (10:28)
      
I think we should share about contrib at TSC level

      From Srini Addepalli (Intel) to Everyone: (10:29)
      
+1 on contrib. ONAP to external datalake bus as Kafka is good enough in my view.

      From Pamela Dragosh to Everyone: (10:30)
      
At the very least, trial it with DCAE and see if it fits. If not, go from there.

      From Srini Addepalli (Intel) to Everyone: (10:30)
      
It is very common practice in Industry to use Kafka as a bus.

      From Arash Hekmat (Amdocs) to Everyone: (10:30)
      
A contrib is a module that is not part of OOM master Helm chart for ONAP install. A contrib module has a separate chart in OOM and has to be installed separately.

      From Catherine Lefevre to Everyone: (10:31)
      
TSC - are you ok to vote +1 POC only for Dublin, part of DCAE; 0 no opinion, -1 need to more time to reflect and descope

      From Me to Everyone: (10:32)
      
arash is right - DCAEGEN2 not compatable yet with native kubernetes charts via contrib

      From Andreas Geissler (Deutsche Telekom) to Everyone: (10:32)
      
+1

      From Lingli to Everyone: (10:32)
      
+1

      From Chaker Al-Hakim to Everyone: (10:32)
      
#vote +1

      From Stephen Terrill to Everyone: (10:32)
      
#vote: +1

      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

              People

              • Assignee:
                michaelobrien Michael O'Brien
                Reporter:
                michaelobrien Michael O'Brien
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: