Uploaded image for project: 'Data Movement as a Platform'
  1. Data Movement as a Platform
  2. DMAAP-518

Extend DMaapClient to use Apache Kafka API Client artifact

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Medium Medium
    • Casablanca Release
    • None

      Due to expected high data volume that is going to be sent to DMaaP, it has been decided to extend DMaaPClient to use Apache Kafka API.
      It means that DMaaPClient is going to connect directly to Apache Kafka for publishing/reading messages to/from topics.
      DMaapClient using/wrapping REST API will be left for backward compatibility.

      Tasks (draft):
      1 Check version of Apache Kafka/Zookeper used in DMaaP currently, upgrade if needed
      2 Add dependency in DMaaP Client for Apache Kafka API Client correcpoding to Kafka version
      3 Add Apache Kafka Producer using native API
      4 Add Apache Kafka Consumer using native API
      5 Check security possibility for TLS in Apache Kafka
      6 Check security possibility for authorization in Apache Kafka
      7 Check security possibility for authentication in Apache Kafka
      8 Check how to use AAF in DMaaP
      9 Data partitioning  design and usage
      10 Usage of several brokers - how to
      11

            mmigdal mmigdal
            mmigdal mmigdal
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: