升级Zabbix proxy到3.0.2后无法启动报错

2025-04-15 技术教程

一问题描述

将一台zabbix proxy由2.4.5升级到3.0.2后启动了就直接崩溃了。

错误信息如下：

2367:20160508:153246.830Onechildprocessdied(PID:42385,exitcode/signal:11).Exiting...

42367:20160508:153248.904ZabbixProxystopped.Zabbix3.0.2(revision59540).

将日志级别调高点，设置DebugLevel=4，查看报错

42629:20160508:153529.004Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42628:20160508:153529.004Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42634:20160508:153529.004Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42651:20160508:153529.004Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42659:20160508:153529.004Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42661:20160508:153529.005Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42655:20160508:153529.005Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42663:20160508:153529.005Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42664:20160508:153529.005Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42666:20160508:153529.005Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42665:20160508:153529.006Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42667:20160508:153529.006Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42630:20160508:153529.006Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42668:20160508:153529.006Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42671:20160508:153529.006Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42670:20160508:153529.007Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42647:20160508:153529.007Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42674:20160508:153529.007Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42680:20160508:153529.007Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...42679:20160508:153529.008Gotsignal[signal:15(SIGTERM),sender_pid:42623,sender_uid:498,reason:0].Exiting...

9102:20160508:170455.283Endofzbx_strpool_destroy()9102:20160508:170455.283Endoffree_configuration_cache()9102:20160508:170455.283Infree_ipmi_handler()9102:20160508:170455.283Endoffree_ipmi_handler()9102:20160508:170455.284Infree_selfmon_collector()collector:0x7fa793e230009102:20160508:170455.284Endoffree_selfmon_collector()9102:20160508:170455.284Inunload_modules()9102:20160508:170455.284Endofunload_modules()9102:20160508:170455.284ZabbixProxystopped.Zabbix3.0.2(revision59540).zabbix_proxy[9152]:[file:'selfmon.c',line:375]lockfailed:[22]Invalidargument

总共有4台proxy，每台proxy的操作系统版本都是CentOS6。就单独这一台zabbix proxy启动有问题，刚启动就有进程死掉。

二问题解决

查看这台proxy服务器的内核版本是3.9.8-1.el6.elrepo.x86_64，而其他几台proxy的内核版本是2.6.32-358.el6.x86_64 刚开始也没有朝着内核版本的问题方向思考。以为还是proxy的版本或者系统参数设置有问题。

1.排查Linux下的信号量

查看Zabbix的官方BUG追踪平台Invalid argument可能是信号量达到了限制

可以尝试增大信号量的值

echo256400003232000>/proc/sys/kernel/sem

这个四个参数分别是

SEMMSL 每个信号量集的含有的信号量的最大值

SEMMNS 系统层面的信号量数量最大限制

SEMOPM semop(2)调用可以指定的最大操作数量

SEMMNI 系统层面的信号数量最小限制

通过echo方式更改重启后会失效，可以更改/etc/sysctl.conf

kernel.sem=2503200010010000

然后执行sysctl -p

更改参数后proxy启动仍然会崩溃

2.排查内核版本原因

Updating to latest kernel (3.10.0-327.10.1.el7.x86_64) + reboot solved it for me
The error reported originally may be something else than what affected me.

查看帖子怀疑是内核版本造成的，这台proxy以前有人升级过内核，更改linux启动项设置默认启动内核为2.6.32.重启后问题得到解决，proxy不再崩溃。但是之前proxy 2.4.5在3.9.8的内核上可以正常运行。

遇到这个BUG的人挺多的，有好几个帖子都是反馈这个问题。

三补充知识

这个BUG涉及到Linux下的信号量相关的知识。信号量是为了解决在多进程编程下资源共享的问题。著名的用信号量来解决的就是哲学家就餐问题。

Linux下使用ipcs命令来查看共享内存，信号量和消息队列

ipcs 默认是-a参数，显示所有信息

#ipcs------SharedMemorySegments--------keyshmidownerpermsbytesnattchstatus0x6c6c65360root600409600x680316181835009zabbix600838858300x6c030ad711632648zabbixagen600365056130x680316722392073zabbix600838858300x780316722424842zabbix6001677721600x740316722457611zabbix600419430400x670316722490380zabbix600133693440000x730316722523149zabbix6002359296000------SemaphoreArrays--------keysemidownerpermsnsems0x000000000root60010x0000000032769root60010x000000004325378apache60010x7a031618622595zabbix600120x000000004358148apache60010x7a030ad74292613zabbixagen600130x7a031672851974zabbix600120x7a0316381015815zabbix600120x7a0316201441800zabbix60012------MessageQueues--------keymsqidownerpermsused-bytesmessages

ipcs -m 单独显示共享内存片段信息

ipcs -s 单独显示信号量

ipcs -q 单独显示消息队列

#ipcs-u------SharedMemoryStatus--------segmentsallocated71pagesallocated1795473pagesresident68130pagesswapped57636Swapperformance:0attempts0successes------SemaphoreStatus--------usedarrays=16allocatedsemaphores=152------Messages:Status--------allocatedqueues=0usedheaders=0usedspace=0bytes

参考资料：

https://support.zabbix.com/browse/ZBX-10657

http://www.cnblogs.com/forilen/p/4316358.html

https://support.zabbix.com/browse/ZBX-3974

http://blog.zabbix.com/mysterious-zabbix-problems-how-we-debug-them/1023/

https://en.wikipedia.org/wiki/Semaphore_(programming)