网站自动化采集,成都网站设计费用,巴中做网站,贵阳app下载场景#xff1a;kafka配置文件log.dirs增加了几个目录#xff0c;但是新目录没有分区数据写入#xff0c;所以打算进行重分区一下。 1.生成迁移计划
进入kafka/bin目录
新建 topic-reassign.json,把要重分区的topic按下面格式写。 { topics: [{ …场景kafka配置文件log.dirs增加了几个目录但是新目录没有分区数据写入所以打算进行重分区一下。 1.生成迁移计划
进入kafka/bin目录
新建 topic-reassign.json,把要重分区的topic按下面格式写。 { topics: [{ topic: VOLTE_GM },{ topic: VOLTE_GM_MEDIA },{ topic: VOLTE_MW },{ topic: VOLTE_RX },{ topic: VOLTE_SH },{ topic: VOLTE_SV }], version: 1 } 执行命令 ./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --generate --topics-to-move-json-file topic-reassign.json --broker-list 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 broker-list 是kafka配置文件中的 broker.id
输出
Current partition replica assignment当前分区策略建议保存回退的时候会用到。
Proposed partition reassignment configuration建议分区策略 Current partition replica assignment {version:1,partitions:[{topic:VOLTE_GM,partition:1,replicas:[35,18],log_dirs:[any,any]},{topic:VOLTE_GM_MEDIA,partition:2,replicas:[0,19],log_dirs:[any,any]},{topic:VOLTE_RX,partition:0,replicas:[33,2],log_dirs:[any,any]},{topic:VOLTE_SV,partition:1,replicas:[10,2],log_dirs:[any,any]},{topic:VOLTE_SH,partition:0,replicas:[14,13],log_dirs:[any,any]},{topic:VOLTE_GM,partition:2,replicas:[36,19],log_dirs:[any,any]},{topic:VOLTE_SV,partition:2,replicas:[11,3],log_dirs:[any,any]},{topic:VOLTE_GM_MEDIA,partition:1,replicas:[39,18],log_dirs:[any,any]},{topic:VOLTE_SH,partition:2,replicas:[16,15],log_dirs:[any,any]},{topic:VOLTE_SH,partition:1,replicas:[15,14],log_dirs:[any,any]},{topic:VOLTE_MW,partition:1,replicas:[38,23],log_dirs:[any,any]},{topic:VOLTE_MW,partition:2,replicas:[39,24],log_dirs:[any,any]},{topic:VOLTE_GM_MEDIA,partition:0,replicas:[38,17],log_dirs:[any,any]},{topic:VOLTE_RX,partition:2,replicas:[35,4],log_dirs:[any,any]},{topic:VOLTE_MW,partition:0,replicas:[37,22],log_dirs:[any,any]},{topic:VOLTE_GM,partition:0,replicas:[34,17],log_dirs:[any,any]},{topic:VOLTE_RX,partition:1,replicas:[34,3],log_dirs:[any,any]},{topic:VOLTE_SV,partition:0,replicas:[9,1],log_dirs:[any,any]}]} Proposed partition reassignment configuration{version:1,partitions:[{topic:VOLTE_RX,partition:0,replicas:[28,21],log_dirs:[any,any]},{topic:VOLTE_SH,partition:0,replicas:[1,25],log_dirs:[any,any]},{topic:VOLTE_GM_MEDIA,partition:0,replicas:[36,19],log_dirs:[any,any]},{topic:VOLTE_SH,partition:2,replicas:[3,27],log_dirs:[any,any]},{topic:VOLTE_MW,partition:0,replicas:[15,13],log_dirs:[any,any]},{topic:VOLTE_RX,partition:2,replicas:[30,23],log_dirs:[any,any]},{topic:VOLTE_GM,partition:1,replicas:[12,32],log_dirs:[any,any]},{topic:VOLTE_SV,partition:0,replicas:[35,25],log_dirs:[any,any]},{topic:VOLTE_MW,partition:2,replicas:[17,15],log_dirs:[any,any]},{topic:VOLTE_GM_MEDIA,partition:2,replicas:[38,21],log_dirs:[any,any]},{topic:VOLTE_SV,partition:2,replicas:[37,27],log_dirs:[any,any]},{topic:VOLTE_SH,partition:1,replicas:[2,26],log_dirs:[any,any]},{topic:VOLTE_RX,partition:1,replicas:[29,22],log_dirs:[any,any]},{topic:VOLTE_GM,partition:0,replicas:[11,31],log_dirs:[any,any]},{topic:VOLTE_MW,partition:1,replicas:[16,14],log_dirs:[any,any]},{topic:VOLTE_GM_MEDIA,partition:1,replicas:[37,20],log_dirs:[any,any]},{topic:VOLTE_SV,partition:1,replicas:[36,26],log_dirs:[any,any]},{topic:VOLTE_GM,partition:2,replicas:[13,33],log_dirs:[any,any]}]} 将标红的json保存到topic-todo.json
2.执行迁移命令 ./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --reassignment-json-file topic-todo.json --execute --throttle 1048576000 throttle带宽单位是B上面表示100M/s带宽。
3.验证迁移结果 ./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --reassignment-json-file topic-todo.json --verify 分区多数据量大的话用时多一些半个小时一个小时也是可能的。
