In Dublin release, it is decided to use Consul as configuration store for analytics application components. That is, expectation is that the all analytics application components have Consul agent that gets notified when the configuration is written in Consul in appropriate keys and that reads the configuration from consul and apply it in its application portion.
On framework configuration: On behalf of analytics applications, yet times, some configuration needs to be applied in framework components. For example, creating kafka topic etc.. Since we can't expect that framework components have consul agent as they are opensource frameworks, it is necessary to work with configuration mechanisms exposed by these components.
Keeping above in mind, it is important that config-sync component (that would be in the sites) have plugin approach to talk to various configuration mechanisms.
Activity summary:
- Create a plugin (analytics-config-plugin) in Multi-Cloud K8S config service.
- Create a sync micro service (analytics-config-sync) that normally is deployed as part of analytics framework in cloud-region, which does following:
- Gets notified by analytics-config-plugin with new configuration(s).
- Use various adapters to synchronize the configuration
- Support for consul adapter to synchronize relevant configuration in ConsulDB.
- Support for talking Kafka-broker to create/delete config (such as topics)
- Identify any gaps in Multi-Cloud/K8S-config-service