-
Epic
-
Resolution: Done
-
Medium
-
None
-
POMBA/LOG Integration with Datalake as a Service platform - TSC 20190117 - via DCAE
discuss how for example we provide or consume
ElasticSearch as a Service
Kibana as a Service
update:
https://wiki.onap.org/display/DW/PTL+2019-02-04
aka firehose(twitter) for dmaap
https://wiki.onap.org/display/DW/TSC+2019-01-17
From Kenny Paul (LFN) to Everyone: (09:56) #topic datalake Guobiao Mo reviedes his slides From Me to Everyone: (10:03) Q: we could provide hooks into the 3 existing ELK stacks (LOG, POMBA, CLAMP) - most already have external facing nodeports for their elasticsearch containers - btw there are 5 elasticsearch, 5 cassandara, 14+ mariadb/myssql 1 mongodb, 3 postgres, 2 redis and one graphdb in onap there is a mongodb container in nbi/extapi From Alexander Vul to Everyone: (10:07) Is this infrastructure meant to be deployed as part of the ONAP footprint? From Keong Lim k00759777 to Everyone: (10:08) since DMaaP is based on Kafka, couldn't this also be done by simply retaining all the messages in Kafka? From Alexander Vul to Everyone: (10:08) Kafka is not meant to handle the volume of data we are talking about here... From Keong Lim k00759777 to Everyone: (10:09) there are orgs that do exactly that already with kafka From Alexander Vul to Everyone: (10:09) I am sure there are... From Catherine Lefevre to Everyone: (10:10) @Denghui, Andy - agenda updated but have you planned a TSC Readout about NS domain / Internal NS Descriptor today? From Michael O'Brien to Everyone: (10:11) So datalake will be Elasticsearch as a service - we should discuss this From Catherine Lefevre to Everyone: (10:11) if not then I can remove this readout for this week From Michael O'Brien to Everyone: (10:12) datalake pods ideally will be native kubernetes charts not cloudify blueprints for el-alto From Michael O'Brien to Everyone: (10:12) datalake pods ideally will be native kubernetes charts not cloudify blueprints for el-alto From Ramki Krishnan to Everyone: (10:12) Fair point raised by Keong Lim on Kafka - Kafka has programmable buffering. From Keong Lim k00759777 to Everyone: (10:12) https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ From Michael O'Brien to Everyone: (10:13) woudn't we need DCAEGEN2 deployed plus secondary dep-* pods to use datalake From Srini Addepalli (Intel) to Everyone: (10:13) Is this activity intend to be done as part of POMBA or DCAE? s this activity intend to be done as part of POMBA or DCAE? From Jason Hunt to Everyone: (10:14) Wouldn’t we expect that operators already have data lakes? Should we focus on how to enable them to link ONAP to their data lakes? From Brian to Everyone: (10:14) +1 Jason's comment From Michael O'Brien to Everyone: (10:14) +1 Jason - pluggable data lake - so this datalake is just an RI From Eric Debeau to Everyone: (10:14) Most of operators have deployed their own data lakes. I agree that we need to explain how to link ONAP with existing data lakes From Fernando (Fred) Oliveira to Everyone: (10:15) I agree, ONAP should enable the feeds to an external data lake From Andreas Geissler (Deutsche Telekom) to Everyone: (10:16) agree with Eric and Jason, but as ONAP Addon the component would be useful. From Lingli to Everyone: (10:16) +1 From Michael O'Brien to Everyone: (10:18) More very interested on the Datalake as a Platform (pluggable) aspect of this project - the actual implementation is more of an example From Brian to Everyone: (10:18) can we do it like netbox for IPAM as a contrib ? From Michael O'Brien to Everyone: (10:18) yes, contrib is a good idea From Mike Elliott to Everyone: (10:19) +1 - contrib (optional) +1 - contrib (optional) From Me to Everyone: (10:23) onap onap-contrib-netbox-app-84f877fb76-qhc9t 1/1 Running 0 9h onap onap-contrib-netbox-app-provisioning-8f2vk 0/1 Error 0 9h onap onap-contrib-netbox-app-provisioning-gwtqt 0/1 Completed 0 9h onap onap-contrib-netbox-app-provisioning-j6hn9 0/1 Error 0 9h onap onap-contrib-netbox-nginx-5c85578697-lp2bd 1/1 Running 0 9h onap onap-contrib-netbox-postgres-65f4bb6459-rc5k8 1/1 Running 0 9h the error is normal - the completed is the 2nd run above https://git.onap.org/oom/tree/kubernetes/contrib From Eric Debeau to Everyone: (10:28) +1 for contrilb From Ramki Krishnan to Everyone: (10:28) +1 on contrib From Eric Debeau to Everyone: (10:28) I think we should share about contrib at TSC level From Srini Addepalli (Intel) to Everyone: (10:29) +1 on contrib. ONAP to external datalake bus as Kafka is good enough in my view. From Pamela Dragosh to Everyone: (10:30) At the very least, trial it with DCAE and see if it fits. If not, go from there. From Srini Addepalli (Intel) to Everyone: (10:30) It is very common practice in Industry to use Kafka as a bus. From Arash Hekmat (Amdocs) to Everyone: (10:30) A contrib is a module that is not part of OOM master Helm chart for ONAP install. A contrib module has a separate chart in OOM and has to be installed separately. From Catherine Lefevre to Everyone: (10:31) TSC - are you ok to vote +1 POC only for Dublin, part of DCAE; 0 no opinion, -1 need to more time to reflect and descope From Me to Everyone: (10:32) arash is right - DCAEGEN2 not compatable yet with native kubernetes charts via contrib From Andreas Geissler (Deutsche Telekom) to Everyone: (10:32) +1 From Lingli to Everyone: (10:32) +1 From Chaker Al-Hakim to Everyone: (10:32) #vote +1 From Stephen Terrill to Everyone: (10:32) #vote: +1
- blocks
-
ONAPARC-264 ONAP DataLake
- Closed
- relates to
-
LOG-707 Logging El-Alto (moved from) Dublin Scope
- Closed