Understanding NginX’s sendfile Parameter and Its Implications with NFS

In the world of web servers, NginX has emerged as a robust, high-performance option, powering some of the most trafficked sites on the internet. It’s praised for its scalability and its myriad of features, which allow for fine-tuned optimization. One such feature is the sendfile directive, a parameter within NginX’s configuration that can significantly affect how files are served to clients. Understanding sendfile is crucial for system administrators, especially in the context of using Network File System (NFS) storage.

tl;dr: Don’t use sendfile with NFS because it will cause NginX to randomly hang and freeze!

What is the sendfile Parameter?

The sendfile parameter in NginX is a boolean setting that enables or disables the use of the sendfile() system call in file serving operations. When enabled, sendfile allows NginX to transfer data directly between file descriptors without the need for the data to pass through user space.  By bypassing the user-space buffer, the server minimizes context switches and memory copies, which can have a substantial impact on performance when serving large files or a high volume of requests.

sendfile and NFS: A Conflict of Interests

While sendfile has distinct advantages, it has known compatibility issues with NFS. NFS, or Network File System, is a distributed file system protocol that allows a user on a client computer to access files over a network in a manner similar to how local storage is accessed.

The problem with using sendfile over NFS lies in the caching mechanism. The sendfile system call is designed to work with the kernel’s file cache. However, NFS operates with its own caching and file consistency mechanisms that can conflict with the kernel’s cache. This discrepancy can lead to outdated data being served to the client because sendfile may bypass the NFS client’s cache, which is responsible for ensuring that the data is up to date.  Under FreeBSD we’ve seen outright hangs occur, likely due to NFS deadlock conditions.

When to Turn Off sendfile

Turning off sendfile becomes necessary when serving files over an NFS mount because the potential for serving stale data or hangs outweighs the performance benefits. The lack of coherency between the kernel cache and NFS’s caching can cause users to receive outdated content or even corrupted files, which can lead to a poor user experience and potentially harmful scenarios, depending on the nature of the files being served.  Additionally, as mentioned previously, under FreeBSD we’ve seen NginX freeze up randomly under load or after time, making troubleshooting this issue a nightmare.

Conclusion

In environments where NFS is not used, sendfile remains a valuable asset for the performance-oriented configuration of NginX servers. However, when NFS is involved, the safe practice is to disable the sendfile directive, falling back on the traditional file serving methods that ensure data integrity and consistency. Careful consideration and testing should be employed when deciding on the use of sendfile to strike the right balance between performance and reliability. As always, the specific use case dictates the configuration: while sendfile can be a boon for local file deliveries, it must be approached with caution when network file systems are in play.