[svlug] Sync command slowdown
James Sparenberg
james at linuxrebel.org
Tue Nov 25 15:55:48 PST 2008
On Monday 24 November 2008 17:32:40 James Sparenberg wrote:
> All,
>
> I've got 4 new DB servers exhibiting a rather problematic set of
> actions with disk sync. Running the sync command (as is done when
logs
> are rotated.) takes 4-6 seconds (not micro, full seconds) Other than
> this these boxes are running 10x faster than the servers they replaced
> and making our customers much happier.
>
> Further it's only happening on my 4 new servers not on the older
> systems (I have them in QA, etc). CPU load is at best 15% per core,
> normally down in the dirt, as the primary CPU function is to manage
disk
> I/0 (most queries are the same so they are cached, and writes are
> limited.) RAM is fine and the box never touches it's swap partition.
> I'm personally suspecting that it is Kernel module vs LSI raid
> controller, on these boxes.
>
> Additionally our DBA has discovered that it seems to work faster on
> smaller partitions than the larger one we have for our DBs. The only
> tweak I'm currently using beyond defaults in fstab are noatime (RHEL 5
> mount command doesn't recognize relatime) however changing out the
> noatime has no affect on the sync command.
>
> Has anyone out there seen anything at all similar? I'm hoping to
> avoid having to go to software raid only because it will mean a fairly
> extensive rebuild of the system, and from now until mid February I'm
not
> going to be able to take that time. For now the mid level solution is
> going with smaller binlogs and rotating a lot more often.
>
> Details on the systems.
>
> Manf
> Penguin Computing (And yes they are providing some assistance on
> this, they really are proactive.)
>
> OS:
> CentOS 5.2 x86_64 (fully updated) running 2.6.18-92
>
> CPU:
> Single Quad Core Intel Xeon
>
> RAM:
> 16GB
>
> HDD
> LSI Raid 1 controller
>
> lspci:
>
> 00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller
> Hub (rev b1)
> 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express
x8
> Port 2-3 (rev b1)
> 00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express
x4
> Port 3 (rev b1)
> 00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express
x8
> Port 4-5 (rev b1)
> 00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express
x4
> Port 5 (rev b1)
> 00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express
x8
> Port 6-7 (rev b1)
> 00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express
x4
> Port 7 (rev b1)
> 00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA
> Engine (rev b1)
> 00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB
Registers
> (rev b1)
> 00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB
Registers
> (rev b1)
> 00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB
Registers
> (rev b1)
> 00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
> Registers (rev b1)
> 00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved
> Registers (rev b1)
> 00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD
Registers
> (rev b1)
> 00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD
Registers
> (rev b1)
> 00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI
> Express Root Port 1 (rev 09)
> 00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #1 (rev 09)
> 00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #2 (rev 09)
> 00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #3 (rev 09)
> 00:1d.3 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> UHCI USB Controller #4 (rev 09)
> 00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset
> EHCI USB2 Controller (rev 09)
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
> 00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC
> Interface Controller (rev 09)
> 00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE
Controller
> (rev 09)
> 00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset
> SATA IDE Controller (rev 09)
> 00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus
> Controller (rev 09)
> 01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Upstream Port (rev 01)
> 01:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to
> PCI-X Bridge (rev 01)
> 02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Downstream Port E1 (rev 01)
> 02:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Downstream Port E3 (rev 01)
> 03:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1064ET
> PCI-Express Fusion-MPT SAS (rev 04)
> 04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
> Ethernet Controller (Copper) (rev 01)
> 04:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
> Ethernet Controller (Copper) (rev 01)
> 07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit
Ethernet
> Controller (rev 06)
> 07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit
Ethernet
> Controller (rev 06)
> 0b:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200e
> [Pilot] ServerEngines (SEP1) (rev 02)
>
> Stack Trace
>
> execve("/bin/sync", ["sync"], [/* 16 vars */]) = 0
> brk(0) = 0x1ce91000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0)
> = 0x2aaaaaaab000
> uname({sys="Linux", node="my.server.name", ...}) = 0
> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
> directory)
> open("/etc/ld.so.cache", O_RDONLY) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=23796, ...}) = 0
> mmap(NULL, 23796, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aaaaaaac000
> close(3) = 0
> open("/lib64/libc.so.6", O_RDONLY) = 3
> read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\331!
> \2773\0\0\0"..., 832) = 832
> fstat(3, {st_mode=S_IFREG|0755, st_size=1699880, ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0)
> = 0x2aaaaaab2000
> mmap(0x33bf200000, 3481848, PROT_READ|PROT_EXEC, MAP_PRIVATE|
> MAP_DENYWRITE, 3, 0) = 0x33bf200000
> mprotect(0x33bf34a000, 2093056, PROT_NONE) = 0
> mmap(0x33bf549000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|
> MAP_DENYWRITE, 3, 0x149000) = 0x33bf549000
> mmap(0x33bf54e000, 16632, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|
> MAP_ANONYMOUS, -1, 0) = 0x33bf54e000
> close(3) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0)
> = 0x2aaaaaab3000
> arch_prctl(ARCH_SET_FS, 0x2aaaaaab3250) = 0
> mprotect(0x33bf549000, 16384, PROT_READ) = 0
> mprotect(0x33bf01a000, 4096, PROT_READ) = 0
> munmap(0x2aaaaaaac000, 23796) = 0
> brk(0) = 0x1ce91000
> brk(0x1ceb2000) = 0x1ceb2000
> open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=56470832, ...}) = 0
> mmap(NULL, 56470832, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aaaaaab4000
> close(3) = 0
> sync() = 0
> ###NOTE: This is where it pauses ###
> close(1) = 0
> exit_group(0) = ?
>
>
> _______________________________________________
> svlug mailing list
> svlug at lists.svlug.org
> http://lists.svlug.org/lists/listinfo/svlug
>
Well 24 hours later (so that we could monitor) and the conclusion seem
to be that yes it does appear to be related somehow to the ext3 FS and
no changing to writeback doesn't improve the speed. It seems that the
problem has to do with the size of the ext3 FS, in that over some (as
yet) unknow size there is a problem with the speed of the sync command.
James
More information about the svlug
mailing list