Skip to content

MySQL -> ES, 自定义的 ES output,执行时 Execute 函数接受到 Msgs 数量为1 #254

Description

@Stanley3

MySQL Input 的配置

[input]
    type = "mysql"
    mode = "batch"

[input.config]
    nr-scanner = 10
    table-scan-batch = 10000
    batch-per-second-limit = 10
    max-full-dump-count = 10000

ES Output 的配置

[output]
    type="custom-elasticsearch"

[output.config]
    ignore-bad-request = true

Scheduler 的配置

[scheduler]
    type = "batch-table-scheduler"

[scheduler.config]
    nr-worker = 10
    batch-size = 1000
    queue-size = 1024
    sliding-window-size = 10240
    nr-retries = 3
    retry-sleep = "2s"

按照这个配置,在执行到 func (output *customESOutput) Execute(msgs []*core.Msg) error 函数时,msgs 数量总是为1,导致 ES 的 bulk request 形同虚设。
请问,应该如何调整配置,解决这个问题。

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions