OS操作系统：SUSE Linux 11

数据库版 本：MYSQL 5.6

客户反映MYSQL数据库因为内存不足被重启。

2016-10-24 12:28:49 29226 [Note] Plugin'FEDERATED' is disabled.

2016-10-24 12:28:49 29226 [Note] InnoDB:Using atomics to ref count buffer pool pages

2016-10-24 12:28:49 29226 [Note] InnoDB: TheInnoDB memory heap is disabled

2016-10-24 12:28:49 29226 [Note] InnoDB:Mutexes and rw_locks use GCC atomic builtins

2016-10-24 12:28:49 29226 [Note] InnoDB:Compressed tables use zlib 1.2.3

2016-10-24 12:28:49 29226 [Note] InnoDB:Using Linux native AIO

2016-10-24 12:28:49 29226 [Note] InnoDB:Using CPU crc32 instructions

2016-10-24 12:28:49 29226 [Note] InnoDB:Initializing buffer pool, size = 2.0G

2016-10-24 12:28:50 29226 [Note] InnoDB:Completed initialization of buffer pool

2016-10-24 12:28:50 29226 [Note] InnoDB:Highest supported file format is Barracuda.

2016-10-24 12:28:50 29226 [Note] InnoDB: Logscan progressed past the checkpoint lsn 27852365485

2016-10-24 12:28:50 29226 [Note] InnoDB:Database was not shutdown normally!

2016-10-24 12:28:50 29226 [Note] InnoDB:Starting crash recovery.

2016-10-24 12:28:50 29226 [Note] InnoDB:Reading tablespace information from the .ibd files...

2016-10-24 12:28:50 29226 [Note] InnoDB:Restoring possible half-written data pages

2016-10-24 12:28:50 29226 [Note] InnoDB:from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to logsequence number 27852370215

2016-10-24 12:28:51 29226 [Note] InnoDB:Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 13 14 15 16 17 1819 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 4445 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 7071 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 9697 98 99

InnoDB: Apply batch completed

InnoDB: Last MySQL binlog file position 0401322767, file name TSPTONLDB01-it_mgmad-bin.000023

2016-10-24 12:28:52 29226 [Note] InnoDB: 128rollback segment(s) are active.

2016-10-24 12:28:52 29226 [Note] InnoDB:5.6.19 started; log sequence number 27852370215

2016-10-24 12:28:52 29226 [Note] Recoveringafter a crash using /dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin

2016-10-24 12:29:00 29226 [Note] Startingcrash recovery...

2016-10-24 12:29:00 29226 [Note] Crashrecovery finished.

2016-10-24 12:29:06 29226 [Note] Serverhostname (bind-address): '*'; port: 60002

2016-10-24 12:29:06 29226 [Note] IPv6 isavailable.

2016-10-24 12:29:06 29226 [Note] - '::' resolves to '::';

2016-10-24 12:29:06 29226 [Note] Serversocket created on IP: '::'.

2016-10-24 12:29:06 29226 [Warning]'proxies_priv' entry '@ root@tsptonldb01' ignored in --skip-name-resolve mode.

2016-10-24 12:29:06 29226 [Note] EventScheduler: Loaded 0 events

2016-10-24 12:29:06 29226 [Note]/usr/local/mysql/bin/mysqld: ready for connections.

Version: '5.6.19-upsql-0.1.1-log' socket: '/etc/mysock/it_mgmad.sock' port: 60002 UpSQL Server

2016-10-24 12:30:21 29226 [ERROR]/usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)

2016-10-24 12:30:21 29226 [ERROR]Out of memory; check if mysqld or some other process uses all available memory;if not, you may have to use 'ulimit' to allow mysqld to use more memory or youcan add more swap space

161024 12:31:35 mysqld_safe Number ofprocesses running now: 0

161024 12:31:35 mysqld_safe mysqld restarted

2016-10-24 12:31:37 29659[Warning] Buffered warning: Could not increase number of max_open_files to morethan 10240 (request: 25000)

从上面的信息，我们看到， MSYQL报内存不足的错误无法启动。

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482]Mem-Info:

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996483] Node 0 DMA per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996485] CPU 0: hi: 0, btch: 1 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996486] CPU 1: hi: 0, btch: 1 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996487] Node 0 DMA32 per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996488] CPU 0: hi: 186, btch: 31 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996489] CPU 1: hi: 186, btch: 31 usd: 5

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996490] Node 0 Normal per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996491] CPU 0: hi: 186, btch: 31 usd: 30

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996492] CPU 1: hi: 186, btch: 31 usd: 144

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996495] active_anon:658762 inactive_anon:260562 isolated_anon:0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996496] active_file:353inactive_file:1716 isolated_file:0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996496] unevictable:0 dirty:6writeback:5 unstable:0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996497] free:21800slab_reclaimable:2214 slab_unreclaimable:10742

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996498] mapped:765 shmem:456 pagetables:6664 bounce:0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996499] Node 0 DMA free:15880kB min:256kB low:320kB high:384kBactive_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kBunevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kBmlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kBslab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kBwriteback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996505] lowmem_reserve[]: 0 3000 4010 4010

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996507] Node 0 DMA32 free:54356kB min:50368kB low:62960kBhigh:75552kB active_anon:2275764kB inactive_anon:569032kB active_file:12kBinactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kBpresent:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:12kBslab_reclaimable:1896kB slab_unreclaimable:7060kB kernel_stack:2904kBpagetables:6588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:611all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996513] lowmem_reserve[]: 0 0 1010 1010

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996515] Node 0 Normal free:16964kB min:16956kB low:21192kBhigh:25432kB active_anon:359284kB inactive_anon:473216kB active_file:1400kBinactive_file:6776kB unevictable:0kB isolated(anon):0kB isolated(file):0kBpresent:1034240kB mlocked:0kB dirty:24kB writeback:20kB mapped:3028kBshmem:1812kB slab_reclaimable:6960kB slab_unreclaimable:35908kBkernel_stack:3624kB pagetables:20068kB unstable:0kB bounce:0kBwriteback_tmp:0kB pages_scanned:12878 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996520] lowmem_reserve[]: 0 0 0 0

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996522] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996544] Node 0 DMA32: 1308*4kB 973*8kB 448*16kB 181*32kB 91*64kB43*128kB 11*256kB 8*512kB 8*1024kB 1*2048kB 0*4096kB = 54456kB

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996549] Node 0 Normal: 439*4kB 303*8kB 169*16kB 83*32kB 18*64kB5*128kB 4*256kB 3*512kB 3*1024kB 0*2048kB 0*4096kB = 16964kB

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996553] 38343 total pagecache pages

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996554] 35726 pages in swap cache

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996556] Swap cache stats: add 5506519, delete 5470793, find22920240/23085658

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996557] Free swap = 0kB

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317509.996557] Total swap = 2097148kB

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006955] 1048560 pages RAM

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006957] 67010 pages reserved

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006958] 1978 pages shared

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006959] 953768 pages non-shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006960][ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006974] [ 571] 0 571 4223 1 1 -17 -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006981] [ 1689] 0 1689 998 1 0 0 0 acpid

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006983] [ 1704] 100 1704 5444 117 0 0 0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006986] [ 1722] 0 1722 4824 62 0 0 0 syslog-ng

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006988] [ 1725] 0 1725 1065 0 0 0 0 klogd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006991] [10038] 010038 31835 168 1 0 0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006994] [10096] 010096 15193 1 0 0 0 VGAuthService

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006996] [10185] 10110185 8608 146 0 0 0 hald

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.006999] [10188] 010188 27278 0 0 0 0 console-kit-dae

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007001] [10189] 010189 4505 2 0 0 0 hald-runner

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007004] [10284] 010284 8690 0 0 0 0 hald-addon-inpu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007007][10308] 0 10308 8690 17 0 0 0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007009] [10312] 10110312 9129 0 1 0 0 hald-addon-acpi

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007012] [10313] 010313 8690 16 0 0 0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007014] [12046] 012046 7409 8 0 -17 -1000 auditd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007017] [12048] 012048 3606 0 0 0 0 audispd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007020] [12081] 012081 4723 16 0 0 0 rpcbind

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007022] [12083] 012083 2088 17 0 0 0 haveged

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007025] [12140] 012140 4216 1 1 -17 -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007027] [12141] 012141 4216 1 0 -17 -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007029] [12224] 012224 6759 1 1 0 0 mcelog

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007031][12701] 0 12701 17296 28 0 -17 -1000 sshd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007034] [12715] 012715 17933 0 1 0 0 cupsd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007036] [12730] 012730 59557 71 0 0 0 nscd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007038] [12798] 012798 9552 28 0 0 0 master

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007041] [12825] 5112825 10052 31 1 0 0 qmgr

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007043] [12864] 012864 16711 47 1 0 0 gdm

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007046] [12878] 0 12878 27377 121 0 0 0 X

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007048] [12885] 012885 3697 15 0 0 0 cron

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007051] [12924] 012924 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007055] [12925] 012925 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007058] [12926] 012926 1121 2 0 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007061] [12927] 012927 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007064] [12928] 012928 1121 2 1 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007067] [12929] 012929 1121 2 0 0 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007071] [13064] 10713064 5548 0 1 0 0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007074] [13065] 10713065 3362 0 1 0 0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007077] [13066] 10713066 39551 82 0 0 0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007080] [13079] 10713079 9863 67 0 0 0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007083] [13083] 10713083 31861 0 1 0 0 at-spi-registry

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007086] [13084] 10713084 70831 2912 0 0 0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007090] [13086] 10713086 20207 0 1 0 0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007092] [13094] 10713094 57536 135 0 0 0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007095] [13096] 10713096 10854 0 0 0 0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007098] [13097] 10713097 54246 0 0 0 0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007101] [21349] 101821349 2857 2 1 0 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007105] [12777] 101612777 2857 2 0 0 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007109] [18450] 018450 13384 49 1 0 0 gdm-simple-slav

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007112] [18452] 018452 66809 13462 1 0 0 X

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007115] [18489] 10718489 5548 0 1 0 0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007118] [18518] 018518 24613 0 1 0 0 gdm-session-wor

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007121] [18540] 018540 39584 89 1 0 0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007124] [18613] 018613 4178 19 0 0 0 gpg-agent

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007126][18619] 0 18619 5548 0 1 0 0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007129] [18620] 018620 3429 0 0 0 0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007132] [18623] 018623 9887 123 1 0 0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007135] [18627] 018627 91696 332 0 0 0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007138] [18631] 018631 23814 1 1 0 0 gnome-keyring-d

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007141] [18635] 018635 10908 0 1 0 0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007144] [18639] 018639 16822 19 0 0 0 gvfs-fuse-daemo

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007147] [18645] 018645 51782 400 1 0 0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007149] [18650] 018650 93659 621 1 0 0 gnome-panel

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007152] [18651] 018651 135905 13563 1 0 0 nautilus

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007155] [18653] 018653 36624 0 0 0 0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007160] [18659] 018659 82105 455 1 0 0 main-menu

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007171] [18662] 018662 12083 0 1 0 0 gvfsd-trash

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007174] [18666] 018666 13609 1 0 0 0 gvfs-hal-volume

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007178] [18668] 018668 15774 1 1 0 0 gvfs-gphoto2-vo

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007181] [18677] 018677 75456 1 1 0 0 gnome-volume-co

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007184] [18681] 018681 56947 1 0 0 0 python

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007187] [18686] 018686 39298 113 0 0 0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007189] [18688] 018688 57564 125 0 0 0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007192] [18692] 018692 38020 114 1 0 0 gnome-screensav

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007195] [18693] 018693 48186 0 1 0 0 pulseaudio

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007198] [18718] 018718 10962 0 0 0 0 gvfsd-burn

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007201] [21417] 021417 59168 142 1 0 0 application-bro

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007204] [21420] 021420 57129 199 0 0 0 baobab

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007208] [21463] 021463 35883 1 0 0 0 gnomesu

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007211] [21464] 021464 13445 4 0 0 0 gnomesu-pam-bac

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007214] [21471] 021471 3221 1 0 0 0 yast2

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007217] [21501] 021501 118060 114 0 0 0 y2base

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007221] [21621] 021621 55974 123 0 0 0 gnome-terminal

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007224] [21622] 021622 1616 0 0 0 0 gnome-pty-helpe

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007227] [21623] 021623 3505 1 0 0 0 bash

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007230] [21649] 021649 9244 1 0 0 0 gvfsd-metadata

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007233] [31142] 101631142 1100222 219744 1 0 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007236] [31181] 101831181 1039509 163992 1 0 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007239] [ 699] 1017 699 2856 98 0 0 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007242] [ 1625] 1017 1625 1076677 414298 1 0 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007246] [ 1861] 1019 1861 641372 48223 1 0 0 java

Oct 25 13:41:07 TSPTONLDB01 kernel:[14317510.007284] [14971] 5114971 10014 128 0 0 0 pickup

Oct 25 13:41:07 TSPTONLDB01kernel: [14317510.007287] Out of memory: Kill process 31142 (mysqld) score 283or sacrifice child

Oct 25 13:41:07 TSPTONLDB01kernel: [14317510.007358] Killed process 31142 (mysqld) total-vm:4400888kB,anon-rss:878976kB, file-rss:0kB

Oct 25 13:45:45 TSPTONLDB01kernel: [14317788.351765] vmtoolsd invoked oom-killer: gfp_mask=0x201da,order=0, oom_adj=0, oom_score_adj=0

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351772] vmtoolsd cpuset=/ mems_allowed=0

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351777] Pid: 10038, comm: vmtoolsd Tainted: G X 3.0.13-0.27-default #1

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351780] Call Trace:

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351803] [<ffffffff810048b5>] dump_trace+0x75/0x300

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351814] [<ffffffff8143ea0f>] dump_stack+0x69/0x6f

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351825] [<ffffffff810f2b6f>] dump_header+0x8f/0x1f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351834] [<ffffffff810f3235>]oom_kill_process+0x95/0x180

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351841] [<ffffffff810f346c>] out_of_memory+0x14c/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351848] [<ffffffff810f8efb>] __alloc_pages_slowpath+0x7ab/0x7f0

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351854] [<ffffffff810f90f1>] __alloc_pages_nodemask+0x1b1/0x1c0

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351863] [<ffffffff811312b5>] alloc_pages_current+0xa5/0x120

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351871] [<ffffffff810f08f2>] filemap_fault+0x172/0x3d0

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351878] [<ffffffff8111916a>] __do_fault+0x7a/0x5d0

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351884] [<ffffffff8111975d>]handle_pte_fault+0x9d/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351893] [<ffffffff81445327>] do_page_fault+0x207/0x4c0

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.351900] [<ffffffff81442065>] page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.353040] DWARF2 unwinder stuck at page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel:[14317788.353042]

通过日志发现发现在10月24日到11月份出现多次内存不足，操作系统 KILL 掉MYSQLD进程的情况。

在该服务器上有3个MYSQL实例，每个MYSQL实例的 INNODB 缓冲池设置为2GB，但操作系统的物理内存

只有4GB，另外在服务器上还有一个数据同步的 JAVA进程。结合日志分析，MYSQL 内存分配不合理，建议把服务器内存扩展到10GB。

建议MYSQL INNODB池计算方法： INNODB 缓冲池内存= 操作系统物理内存 - （预留给操作系统的内存1-2GB）