NginX “service nginx upgrade” / kill SIGUSR2 results in shutdown() failed (54: Connection reset by peer)

Huge thanks to Ganbold Tsagaankhuu who finally got to the bottom of this issue.

NginX is a great web server, and is built from the ground up for high volume, zero down time environments.

One of the most basic features of such a service is the ability to restart completely (not just child processes, which is what apache does) without missing a request.

For FreeBSD users this is available via the service nginx upgrade command, for other OSes you can send the SIGUSR2 to the master process. SIGUSR2 is documented as such:

SIGUSR2          Upgrade the nginx executable on the fly.

We’ve used this for years, however a few weeks ago we ran into an issue under FreeBSD where performing this resulted in a number of errors, an unclean shutdown and ultimately NginX not coming back up because old processes were still tying up the listen() port but not actually functioning. The nginx-errors log entries looked like this:

2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 [error] 73757#0: shutdown() failed (54: Connection reset by peer)
2016/06/27 00:18:58 
 73753#0: worker process 73757 exited on signal 11
2016/06/27 00:18:56 [emerg] 74294#0: bind() to 1.2.3.4:443 failed (48: Address already in use)
nginx: [emerg] bind() to 1.2.3.4:443 failed (48: Address already in use)
2016/06/27 00:18:56 [emerg] 74294#0: still could not bind()
nginx: [emerg] still could not bind()

After much debugging and opening a FreeBSD PR with the port maintaner we discovered it was due to OpenSSL (from FreeBSD ports) being compiled with assembler optimizations.

Once we recompiled OpenSSL without this the problem went away. Hopefully this saves someone some hair pulling 🙂