PHP 5.3.6 SIGSEGV (with Zend_Locale?)

We have a 64-bit web server with 7GB memory, PHP crashed in the weekend running our site that is powered by Codeigniter with selective Zend classes. Nearly all PHP requests going to our site led to SIGSEGV. Our other servers on the same spec running the site has far less of these. Not sure that is related or not, we are running on APC (the version shipped with PHP 5.3.6, I think it's 3.1.3p1) and we found we had lots of "Unable to allocate memory for pool." in the error log. Running the APC stats I found the cache was full.

So my first question, I think having a full APC cache should not have led to SIGSEGV?

It's PHP 5.3.6 as on title and gdb it I came up with the following backtrace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f9d259007e0 (LWP 6020)]
strlen () at ../sysdeps/x86_64/strlen.S:31
31              pcmpeqb (%rdi), %xmm2
(gdb) backtrace
#0  strlen () at ../sysdeps/x86_64/strlen.S:31
#1  0x00007f9d1e14e274 in xbuf_format_converter (xbuf=0x7fffa8560f10, fmt=0x7f9d1e26d0df "s, %s%s given, called in %s on line %d and defined", ap=0x7fffa8560fb0)
    at /usr/src/debug/php-5.3.6/main/spprintf.c:574
#2  0x00007f9d1e14f164 in vspprintf (pbuf=0x7f9d270327d8, max_len=0, format=, ap=) at /usr/src/debug/php-5.3.6/main/spprintf.c:797
#3  0x00007f9d1e19b56f in zend_error (type=4096, format=0x7f9d1e26d0b8 "Argument %d passed to %s%s%s() must %s%s, %s%s given, called in %s on line %d and defined")
    at /usr/src/debug/php-5.3.6/Zend/zend.c:1051
#4  0x00007f9d1e1eac56 in zend_verify_arg_error (execute_data=0x7f9d257e88f8) at /usr/src/debug/php-5.3.6/Zend/zend_execute.c:471
#5  zend_verify_arg_type (execute_data=0x7f9d257e88f8) at /usr/src/debug/php-5.3.6/Zend/zend_execute.c:505
#6  ZEND_RECV_SPEC_HANDLER (execute_data=0x7f9d257e88f8) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:449
#7  0x00007f9d1e1c0e30 in execute (op_array=0x7f9d26ffe998) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:107
#8  0x00007f9d1e19b0fd in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/debug/php-5.3.6/Zend/zend.c:1194
#9  0x00007f9d1e1487c8 in php_execute_script (primary_file=0x7fffa8563640) at /usr/src/debug/php-5.3.6/main/main.c:2268
#10 0x00007f9d1e2246a5 in php_handle开发者_运维知识库r (r=0x7f9d26dbc8c0) at /usr/src/debug/php-5.3.6/sapi/apache2handler/sapi_apache2.c:669
#11 0x00007f9d2593d8b0 in ap_run_handler (r=0x7f9d26dbc8c0) at /usr/src/debug/httpd-2.2.16/server/config.c:158
#12 0x00007f9d2594116e in ap_invoke_handler (r=0x7f9d26dbc8c0) at /usr/src/debug/httpd-2.2.16/server/config.c:376
#13 0x00007f9d2594c5fc in ap_internal_redirect (new_uri=, r=) at /usr/src/debug/httpd-2.2.16/modules/http/http_request.c:502
#14 0x00007f9d1f3637a5 in handler_redirect (r=0x7f9d26dcad38) at /usr/src/debug/httpd-2.2.16/modules/mappers/mod_rewrite.c:4831
#15 0x00007f9d2593d8b0 in ap_run_handler (r=0x7f9d26dcad38) at /usr/src/debug/httpd-2.2.16/server/config.c:158
#16 0x00007f9d2594116e in ap_invoke_handler (r=0x7f9d26dcad38) at /usr/src/debug/httpd-2.2.16/server/config.c:376
#17 0x00007f9d2594c7c0 in ap_process_request (r=0x7f9d26dcad38) at /usr/src/debug/httpd-2.2.16/modules/http/http_request.c:282
#18 0x00007f9d25949698 in ap_process_http_connection (c=0x7f9d26dabf78) at /usr/src/debug/httpd-2.2.16/modules/http/http_core.c:190
#19 0x00007f9d259453c8 in ap_run_process_connection (c=0x7f9d26dabf78) at /usr/src/debug/httpd-2.2.16/server/connection.c:43
#20 0x00007f9d259510b7 in child_main (child_num_arg=) at /usr/src/debug/httpd-2.2.16/server/mpm/prefork/prefork.c:662
#21 0x00007f9d259513ca in make_child (s=0x7f9d268a8860, slot=17) at /usr/src/debug/httpd-2.2.16/server/mpm/prefork/prefork.c:758
#22 0x00007f9d2595204c in perform_idle_server_maintenance (_pconf=, plog=, s=) at /usr/src/debug/httpd-2.2.16/server/mpm/prefork/prefork.c:893
#23 ap_mpm_run (_pconf=, plog=, s=) at /usr/src/debug/httpd-2.2.16/server/mpm/prefork/prefork.c:1097
#24 0x00007f9d25929840 in main (argc=1, argv=0x7fffa8563c88) at /usr/src/debug/httpd-2.2.16/server/main.c:740

With some simple gdb I found execution seems to have reached Zend_Locale::_prepareLocale() and apparently while PHP tried to output an error message some bad pointers led to the SIGSEGV:

(gdb) frame 9
#9  0x00007f9d1e1487c8 in php_execute_script (primary_file=0x7fffa8563640) at /usr/src/debug/php-5.3.6/main/main.c:2268
2268                    retval = (zend_execute_scripts(ZEND_REQUIRE TSRMLS_CC, NULL, 3, prepend_file_p, primary_file, append_file_p) == SUCCESS);
(gdb) p *primary_file
$1 = {type = ZEND_HANDLE_FILENAME, filename = 0x7f9d26dcd2d0 "/mnt/goserver/goanimate.com/index.php", opened_path = 0x0, handle = {fd = 646922536, fp = 0x7f9d268f4128, stream = {handle = 0x7f9d268f4128,
      isatty = 0, mmap = {len = 140312938585584, pos = 140312938463096, map = 0x7f9d26dca068, buf = 0x7f9d26dabff8 "\310\307\332&\235\177", old_handle = 0x7f9d25948971, old_closer = 0}, reader = 0xc,
      fsizer = 0x7f9d26dc3840, closer = 0x7f9d26dca078}}, free_filename = 0 '\000'}
(gdb) frame 7
#7  0x00007f9d1e1c0e30 in execute (op_array=0x7f9d27000c40) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:107
107                     if ((ret = EX(opline)->handler(execute_data TSRMLS_CC)) > 0) {
(gdb) p *op_array
$2 = {type = 2 '\002', function_name = 0x7f9d0ef9aec0 "_prepareLocale", scope = 0x7f9d26ffd228, fn_flags = 1025, prototype = 0x0, num_args = 2, required_num_args = 1, arg_info = 0x7f9d0ef9ae68,
  pass_rest_by_reference = 0 '\000', return_reference = 0 '\000', done_pass_two = 1 '\001', refcount = 0x7f9d27000d38, opcodes = 0x7f9d0efad760, last = 161, size = 161, vars = 0x7f9d0efb2540,
  last_var = 12, size_var = 16, T = 94, brk_cont_array = 0x7f9d0efb2530, last_brk_cont = 1, current_brk_cont = -1, try_catch_array = 0x0, last_try_catch = 0, static_variables = 0x0, start_op = 0x0,
  backpatch_count = 0, this_var = 4294967295, filename = 0x7f9d0ef9aed0 "/mnt/goserver/gslib/include/Zend/Locale.php", line_start = 1003, line_end = 1069,
  doc_comment = 0x7f9d0ef9af08 "/**\n     * Internal function, returns a single locale on detection\n     *\n     * @param  string|Zend_Locale $locale (Optional) Locale to work on\n     * @param  boolean", ' ' , "$strict (Optional) St"..., doc_comment_len = 362, early_binding = 4294967295, reserved = {0x1, 0x0, 0x0, 0x0}}
(gdb) p op_array->arg_info
$3 = (zend_arg_info *) 0x7f9d0ef9ae68
(gdb) p *(op_array->arg_info)
$4 = {name = 0x7f9d0ef9c3b0 "locale", name_len = 6, class_name = 0xffffffff , class_name_len = 0, array_type_hint = 0 '\000', allow_null = 1 '\001',
  pass_by_reference = 0 '\000', return_reference = 0 '\000', required_num_args = 0}
(gdb) frame 2
#2  0x00007f9d1e14f164 in vspprintf (pbuf=0x7f9d27034a98, max_len=0, format=, ap=) at /usr/src/debug/php-5.3.6/main/spprintf.c:797
797             xbuf_format_converter(&xbuf, format, ap);
(gdb) frame 1
#1  0x00007f9d1e14e274 in xbuf_format_converter (xbuf=0x7fffa8560f10, fmt=0x7f9d1e26d0df "s, %s%s given, called in %s on line %d and defined", ap=0x7fffa8560fb0)
    at /usr/src/debug/php-5.3.6/main/spprintf.c:574
574                                                             s_len = strlen(s);
(gdb) p s
$7 = 0xffffffff 

I'm thinking about failed mallocs or the like that were left unnoticed in the Zend engine code, but I don't know enough of Zend internals to confirm it. Anyone have any clues about this? Thanks.

---

I'm not a C programmer to know about the source code behind the PHP interpreter/Zend Engine. Yet I followed the Zend_Locale::_prepareLocale() method and in the debugging data you specified, I'm thinking a call to strlen() is causing the overflow. I'm thinking that:

the strlen() call in C is translated from a string related PHP function call. In the mentioned Zend_Locale method I'm seeing these PHP functions that might be related: explode(), strstr(), strlen().

By inspecting the Zend_Local::_prepareLocale() you could find that the data being passed to string functions is the $local variable. Since this locale is being constructed from different resources (Environment variables, User browser specified data, etc), this value could be causing the memory failure. If you could find out what is the state of the PHP program that caused it, debugging might be easier. May be editing the source code of Zend_Locale and print out the $locale variable on some lines to have a line-by-line debug.