'Apache Httpd process hangs, no new connections accepted

I'm using httpd 2.4.52 and openssl 1.1.1m on windows. Very often the httpd process hangs and does not accept any connection. Nothing gets written to the logs. Sometimes the parent process crashes but usually it has to be restarted.

It seems to be happening at times when child process exists (graceful restart on max connections) and there's a number of websocket connections that are present at that time. MaxConnectionsPerChild is set to 2M.

error: HEAP_CORRUPTION

Backtrace:

           ntdll.dll!NtWaitForSingleObject ()            Unknown
           ntdll.dll!RtlpWaitOnCriticalSection()        Unknown
           ntdll.dll!RtlpEnterCriticalSectionContended ()     Unknown
           ntdll.dll!RtlpFreeHeap() Unknown
           ntdll.dll!RtlFreeHeap()   Unknown
           ucrtbase.dll!_free_base()             Unknown
            libapr-1.dll!apr_allocator_destroy(apr_allocator_t * allocator) Line 193   C
           libapr-1.dll!apr_pool_destroy(apr_pool_t * pool) Line 1054           C
           libapr-1.dll!apr_pool_destroy(apr_pool_t * pool) Line 997             C
           libhttpd.dll!child_main(apr_pool_t * pconf, unsigned long parent_pid) Line 1283                C
           libhttpd.dll!winnt_run(apr_pool_t * _pconf, apr_pool_t * plog, server_rec * s) Line 1721 C
           libhttpd.dll!ap_run_mpm(apr_pool_t * pconf, apr_pool_t * plog, server_rec * s) Line 95  C
           httpd.exe!main(int argc, const char * const * argv) Line 819          C
           [Inline Frame] httpd.exe!invoke_main() Line 64  C++
           httpd.exe!__scrt_common_main_seh() Line 253 C++
           kernel32.dll!BaseThreadInitThunk ()       Unknown
           ntdll.dll!RtlUserThreadStart ()    Unknown

Log snippet:

ssl_engine_io.c(2411): [client 10.10.10.10:65555] OpenSSL: I/O error, 5 bytes expected to read on BIO#a2093f0ab0 [mem: a2130776f3]

ssl_engine_io.c(2411): [client 10.10.10.10:59999] OpenSSL: I/O error, 5 bytes expected to read on BIO#a209591490 [mem: a218f07023]

child.c(1175): AH00361: Child: 1452 threads blocked on the completion port

ssl_engine_io.c(2411): [client 10.10.10.10:58888] OpenSSL: I/O error, 5 bytes expected to read on BIO#a2093f1130 [mem: a216fe82f3]

ssl_engine_io.c(2401): [client 10.10.10.10:65555] OpenSSL: write 31/31 bytes to BIO#a2093efcb0 [mem: a21b75d133]

ssl_engine_io.c(2401): [client 10.10.10.10:59999] OpenSSL: write 31/31 bytes to BIO#a209592010 [mem: a2078cf583]

ssl_engine_kernel.c(2223): [client 10.10.10.10:65555] OpenSSL: Write: SSL negotiation finished successfully

ssl_engine_kernel.c(2223): [client 10.10.10.10:59999] OpenSSL: Write: SSL negotiation finished successfully

ssl_engine_io.c(1147): [client 10.10.10.10:65555] AH02001: Connection closed to child 833 with standard shutdown

ssl_engine_io.c(1147): [client 10.10.10.10:59999] AH02001: Connection closed to child 4177 with standard shutdown

ssl_engine_io.c(2401): [client 10.10.10.10:58888] OpenSSL: write 31/31 bytes to BIO#a2093f1830 [mem: a216fe82f3]

ssl_engine_kernel.c(2223): [client 10.10.10.10:58888] OpenSSL: Write: SSL negotiation finished successfully

ssl_engine_io.c(1147): [client 10.10.10.10:58888] AH02001: Connection closed to child 609 with standard shutdown

AH00363: Child: Terminating 1408 threads that failed to exit.

AH00364: Child: All worker threads have exited.

mod_socache_shmcb.c(570): AH00837: socache_shmcb_remove (0x41 -> subcache 1)

mod_socache_shmcb.c(585): AH00839: leaving socache_shmcb_remove successfully

ssl_engine_kernel.c(2061): Inter-Process Session Cache: request=REM status=OK id=41fa32c2f3782d8db1894c3bffdbb7b20bca8ba9bc200dae45e318240df0050e (session dead)

ssl_engine_io.c(1853): [client 10.10.10.10:54444] coalesce: have 0 bytes, adding 6 more (buckets=3)

ssl_engine_io.c(1916): [client 10.10.10.10:54444] coalesce: passing on 6 bytes

ssl_engine_io.c(1853): [client 10.10.10.10:62222] coalesce: have 0 bytes, adding 15 more (buckets=3)

ssl_engine_io.c(1916): [client 10.10.10.10:62222] coalesce: passing on 15 bytes

ssl_engine_io.c(1853): [client 10.10.10.10:59999] coalesce: have 0 bytes, adding 15 more (buckets=3)

ssl_engine_io.c(1916): [client 10.10.10.10:59999] coalesce: passing on 15 bytes

ssl_engine_io.c(1853): [client 10.10.10.10:57777] coalesce: have 0 bytes, adding 6 more (buckets=3) ssl_engine_io.c(1916): [client 10.10.10.10:57777] coalesce: passing on 6 bytes

ssl_engine_io.c(1853): [client 10.10.10.10:63333] coalesce: have 0 bytes, adding 6 more (buckets=3)

ssl_engine_io.c(1916): [client 10.10.10.10:63333] coalesce: passing on 6 bytes

Has anyone encountered a similar issue? Any ideas how to resolve?

Thanks



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source