-
Story
-
Resolution: Unresolved
-
Highest
-
None
-
None
Today PNDA deployment manager can't leverage K8S based scheduling of Spark driver & Executors. Also, deployment manager seems to be calling spark-submit directly. To make it K8S friendly, K8S based mechanism are expected to be used. Also, for workflows, currently oozie plugn is used in deployment manager. it is highly recommended to support anothyer plugin using K8S argo framework.
Since Dublin scope is to instantiate the analytics applications in the K8S based Cloud regions, K8S facilities using Argo & spark-k8s-operator can be used to submit the applications to the spark at the cloud-region. Hence, deployment manager may not be used for running the application.
Study to see whether PNDA deployment manager can be leveraged for synchronizing spark application images and ML/DL models between ONAP-Central & cloud regions.
Some background material:
- Use spark-on-k8s-operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) for deploying analytics app workloads on analytics framework.
- Use https://github.com/argoproj/argo workflow framework when workflows are required.