排查logstash3.4升级到5.0版本后kafka不兼容问题

参考文档：

/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/CHANGELOG.md/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/DEVELOPER.md/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/README.md.md/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.0.4/lib/logstash/outputs/kafka.rb

缘由：

之前对ELKB环境从2.4版本升级到最新的5.0稳定版本，主要升级步骤可以参考http://jerrymin.blog.51cto.com/3002256/1870205，后来发现kafka集群运行报错，现在把排查过程记录如下，仅供参考

之前环境：

logstash3.4

logstash-input-kafka-2.0.9

logstash-output-kafka-2.0.5

kafka_2.10-0.8.2.2.tgz

升级后环境：

logstash6.0

logstash-input-kafka-2.0.9

logstash-output-kafka-2.0.5

报错信息：

[2016-11-16T14:35:44,739][ERROR][logstash.inputs.kafka]Unknownsetting'zk_connect'forkafka[2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka]Unknownsetting'topic_id'forkafka[2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka]Unknownsetting'reset_beginning'forkafka

实施步骤：

1，根据错误查看程序哪里报错

grep"Unknownsetting"/usr/share/logstash/-R/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:self.logger.error("Unknownsetting'#{name}'for#{@plugin_name}")

2，查看程序相关代码，发现需要查看plugins的config定义文件等

defvalidate_check_invalid_parameter_names(params)invalid_params=params.keys#Filteroutparametersthatmatchregexpkeys.#Thesearedefinedinpluginslikethis:#config/foo.*/=>...@config.each_keydo|config_key|ifconfig_key.is_a?(Regexp)invalid_params.reject!{|k|k=~config_key}elsifconfig_key.is_a?(String)invalid_params.reject!{|k|k==config_key}endendifinvalid_params.size>0invalid_params.eachdo|name|self.logger.error("Unknownsetting'#{name}'for#{@plugin_name}")endreturnfalseend#ifinvalid_params.size>0returntrueend#defvalidate_check_invalid_parameter_names

3，进入插件总目录查看具体信息

cd /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5

发现重点查看如下文件

grepconfig./*-R|awk'{print$1}'|uniq./CHANGELOG.md:./DEVELOPER.md:See./lib/logstash/inputs/kafka.rb:#./lib/logstash/inputs/kafka.rb:./README.md:-Binary

1）首先看CHANGELOG.md，就有发现logstash-input-3.0.0.beta1开始就不在向后兼容，且剔除了jruby-kafka，注意这里有个坑2）会讲到，4.0.0版本说开始支持kafka 0.9，5.0.0又说开始

支持0.10切不向后兼容，这破坏性更新也是够了。看来问题找到了我的kafka版本是kafka_2.10-0.8.2.2.tgz，kafka版本不兼容导致的。

CHANGELOG.md部分文档如下：

##5.0.4-UpdatetoKafkaversion0.10.0.1forbugfixes##5.0.0-SupportforKafka0.10whichisnotbackwardcompatiblewith0.9broker.##4.0.0-Republishallthegemsunderjruby.-Updatetheplugintotheversion2.0ofthepluginapi,thischangeisrequiredforLogstash5.0compatibility.Seehttps://github.com/elastic/logstash/issues/5141-SupportforKafka0.9forLS5.x##3.0.0.beta1-RefactortousenewJavabasedconsumer,bypassingjruby-kafka-Breaking:ChangeconfigurationtomatchKafka'sconfiguration.Thisversionisnotbackwardcompatible

2）之前我看DEVELOPER.md文档时,看配置语法都正确，还以为是却少依赖关系jruby-kafka library呢，这个再logstash3.x是在用的(另外对比logstash6.x发现5版本少了不少插件。另外

kafka版本写的是0.8.1.1，看来这个DEVELOPER.md没有及时更新（与后面kafka.rb文件不一致），谁要是看到了麻烦及时更新啊，虽是小问题但是也可能误导我等屁民。当然也有可能是我没

有全面看文档导致的。

DEVELOPER.md文档结尾如下：

Dependencies====================*ApacheKafkaversion0.8.1.1*jruby-kafkalibrary

3）开始看README.md文档，特意看了下kafka的兼容性 看来logstas-input-kafka5.0.5和logstash-output-kafka5.0.4只能用kafka0.10了。如果你想用Kafka0.9还想用Logstash6.0,你的

logstash-input-kafka和logstash-output-kafka只能降级版本到4.0.0了，这里都说他是中间过渡版本了，所以还是随大流吧。

##KafkaCompatibilityHere'satablethatdescribesthecompatibilitymatrixforKafkaBrokersupport.Pleaserememberthatitisgoodadvicetoupgradebrokersbeforeconsumers/producerssincebrokerstargetbackwardscompatibility.The0.9brokerwillworkwithboththe0.8consumerand0.9consumerAPIsbutnottheotherwayaround.|KafkaBrokerVersion|LogstashVersion|InputPlugin|OutputPlugin|Why?||:---------------:|:------------------:|:--------------:|:---------------:|:------||0.8|2.0-2.x|<3.0.0|<3.0.0|Legacy,0.8isstillpopular||0.9|2.0-2.3.x|3.0.0|3.0.0|Intermediatereleasebefore0.10thatworkswitholdRubyEventAPI`[]`||0.9|2.4,5.0|4.0.0|4.0.0|Intermediatereleasebefore0.10withnewget/setAPI||0.10|2.4,5.0|5.0.0|5.0.0|TracklatestKafkarelease.Notcompatiblewith0.9broker|

4）现在看来只能升级kafka版本了。最后我看了下jar-dependencies发现了kafka-clients-0.10.0.1.jar

ls/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/vendor/jar-dependencies/runtime-jars/kafka-clients-0.10.0.1.jarlog4j-1.2.17.jarlz4-1.3.0.jarslf4j-api-1.7.21.jarslf4j-log4j12-1.7.21.jarsnappy-java-1.1.2.6.jar

5）还有一个文件没有看，怀着好奇心我看了一眼，发现之前都白费力气了，这里才是最有参考价值的的主参考文档啊，是捷径啊，隐藏的够深的，差点错过了，汗！

/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb

kafka.rb部分文档如下：

#ThisinputwillreadeventsfromaKafkatopic.Itusesthethenewlydesigned#0.10versionofconsumerAPIprovidedbyKafkatoreadmessagesfromthebroker.##Here'sacompatibilitymatrixthatshowstheKafkaclientversionsthatarecompatiblewitheachcombination#ofLogstashandtheKafkainputplugin:##[options="header"]#|==========================================================#|KafkaClientVersion|LogstashVersion|PluginVersion|SecurityFeatures|Why?#|0.8|2.0.0-2.x.x|<3.0.0||Legacy,0.8isstillpopular#|0.9|2.0.0-2.3.x|3.x.x|BasicAuth,SSL|WorkswiththeoldRubyEventAPI(`event['product']['price']=10`)#|0.9|2.4.0-5.0.x|4.x.x|BasicAuth,SSL|Workswiththenewgetter/setterAPIs(`event.set('[product][price]',10)`)#|0.10|2.4.0-5.0.x|5.x.x|BasicAuth,SSL|Notcompatiblewiththe0.9broker#|==========================================================##NOTE:WerecommendedthatyouusematchingKafkaclientandbrokerversions.Duringupgrades,youshould#upgradebrokersbeforeclientsbecausebrokerstargetbackwardscompatibility.Forexample,the0.9broker#iscompatiblewithboththe0.8consumerand0.9consumerAPIs,butnottheotherwayaround.

6）升级kafka_2.10-0.8.2.2.tgz为kafka_2.11-0.10.0.1.tgz （我看kafka-clients-0.10.0.1.jar，所以没有用最新的kafka_2.11-0.10.1.0.tgz）

大概步骤

关闭老kafka

/usr/local/kafka/bin/kafka-server-stop.sh /usr/local/kafka/config/server.properties

备份老配置文件

server.properties和zookeeper.properties

删除kafka

rm -rf /usr/local/kafka/

rm -rf /data/kafkalogs/*

安装配置新kafka

wget http://mirrors.hust.edu.cn/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz

tar zxvf kafka_2.11-0.10.0.1.tgz -C /usr/local/

ln -s /usr/local/kafka_2.11-0.10.0.1 /usr/local/kafka

diff了下server.properties和zookeeper.properties变动不大可以直接使用

启动新kafka

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &

7)注意几个关键配置需要修改

config :bootstrap_servers, :validate => :string, :default => "localhost:9092"

config :group_id, :validate => :string, :default => "logstash"

config :topics, :validate => :array, :default => ["logstash"]

config :consumer_threads, :validate => :number, :default => 1

除了上面的几个关键配置外，kafka的topic分片信息需要重新create一份，否则KafkaMonitor监控不出Active Topic Consumer图形，但实际是在工作中。