Ensure that Helm charts are parameterized to ensure that one can instantiate the service multiple times.
Prove with some sample application to ensure that one can write the data in OpenTSDB and read the data from OpenTSDB.
Since OpenTSDB requires HDFS and HBASE, you may want to look at these as starting point:
Since we are not using YARN, but K8S based scheduling, Not sure whether Yarn deployment is required.
This is breakdown of activities I see :
- Setup POC : To understand how HDFS, HBASE, Spark and OpenTSDB work together
- Use existing helm charts as done by others. I did not find OpenTSDB charts, but I find K8S yaml file for OpenTSDB here: https://github.com/harryge00/hbase-kubernetes. You may like to convert this to Helm charts. Seems simple as it does not use too many Kubernetes resources.
- Wherever operators are available, use them. But ensure that operators are installed using Helm Charts. And also, any CRDs and configuration information is also represented as Helm Charts.
- For Spark on K8S, use operator as mentioned here: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
- For Zookeeper on K8S, check the applicability of https://github.com/pravega/zookeeper-operator
- For HDFS and HBASE, I did not find any operator and hence use helm charts to deploy Hadoop and HBASE. And we can think of starting something to provide operators for HDFS and HBASE in future.
- Ensure that they work fine by running some tests on OpenTSDB.
- Now identify the items that are to be parameterized to ensure that same Helm charts are used to deploy multiple deployments. Some in the same cluster in different namespace and some across clusters in various cloud-regions.
- Update the helm charts
- Ensure that they all work fine by deploying some test application in various deployments (this test would require its own Helm chart)
- Test above using K8S plugin API of ONAP.
- Test above using SDC and SO of ONAP
- Integrate these helm charts with rest of analytics framework
- Integrate with Prometheus helm charts.
- Integrate with Kafka broker helm charts
- Create one sample Analytics application that leverage data in M3DB/OpenTSDB and generate alerts over Kafka bus.