kafka.tools

MirrorMaker

object MirrorMaker extends Logging with KafkaMetricsGroup

The mirror maker has the following architecture: - There are N mirror maker thread shares one ZookeeperConsumerConnector and each owns a Kafka stream. - All the mirror maker threads share one producer. - Each mirror maker thread periodically flushes the producer and then commits all offsets.

Note

For mirror maker, the following settings are set by default to make sure there is no data loss:

  1. use new producer with following settings acks=all retries=max integer block.on.buffer.full=true max.in.flight.requests.per.connection=1 2. Consumer Settings auto.commit.enable=false 3. Mirror Maker Setting: abort.on.send.failure=true
Linear Supertypes
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MirrorMaker
  2. KafkaMetricsGroup
  3. Logging
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. trait MirrorMakerMessageHandler extends AnyRef

    If message.

  2. class MirrorMakerThread extends Thread with Logging with KafkaMetricsGroup

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def cleanShutdown(): Unit

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def commitOffsets(mirrorMakerConsumer: MirrorMakerBaseConsumer): Unit

  10. def createNewConsumers(numStreams: Int, consumerConfigPath: String, customRebalanceListener: Option[ConsumerRebalanceListener], whitelist: Option[String]): Seq[MirrorMakerBaseConsumer]

  11. def debug(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  12. def debug(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  13. def debug(msg: ⇒ String): Unit

    Definition Classes
    Logging
  14. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  15. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  16. def error(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  17. def error(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  18. def error(msg: ⇒ String): Unit

    Definition Classes
    Logging
  19. def fatal(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  20. def fatal(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  21. def fatal(msg: ⇒ String): Unit

    Definition Classes
    Logging
  22. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  23. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  24. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  25. def info(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  26. def info(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  27. def info(msg: ⇒ String): Unit

    Definition Classes
    Logging
  28. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  29. var logIdent: String

    Attributes
    protected
    Definition Classes
    Logging
  30. lazy val logger: Logger

    Definition Classes
    Logging
  31. val loggerName: String

    Definition Classes
    Logging
  32. def main(args: Array[String]): Unit

  33. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  34. def newGauge[T](name: String, metric: Gauge[T], tags: Map[String, String] = Map.empty): Gauge[T]

    Definition Classes
    KafkaMetricsGroup
  35. def newHistogram(name: String, biased: Boolean = true, tags: Map[String, String] = Map.empty): Histogram

    Definition Classes
    KafkaMetricsGroup
  36. def newMeter(name: String, eventType: String, timeUnit: TimeUnit, tags: Map[String, String] = Map.empty): Meter

    Definition Classes
    KafkaMetricsGroup
  37. def newTimer(name: String, durationUnit: TimeUnit, rateUnit: TimeUnit, tags: Map[String, String] = Map.empty): Timer

    Definition Classes
    KafkaMetricsGroup
  38. final def notify(): Unit

    Definition Classes
    AnyRef
  39. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  40. def removeMetric(name: String, tags: Map[String, String] = Map.empty): Unit

    Definition Classes
    KafkaMetricsGroup
  41. def swallow(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  42. def swallowDebug(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  43. def swallowError(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  44. def swallowInfo(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  45. def swallowTrace(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  46. def swallowWarn(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  47. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  48. def toString(): String

    Definition Classes
    AnyRef → Any
  49. def trace(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  50. def trace(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  51. def trace(msg: ⇒ String): Unit

    Definition Classes
    Logging
  52. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  53. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  54. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  55. def warn(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  56. def warn(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  57. def warn(msg: ⇒ String): Unit

    Definition Classes
    Logging

Inherited from KafkaMetricsGroup

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped