File descriptor leaks: how to spot and diagnose them

If you’ve ever had a script run fine, and then suddenly fail with cryptic messages like Too many open files, then there’s a good chance that you’ve ran into a file descriptor leak.

Quick recap on file descriptors

A file descriptor is a handle the operating system uses in order to track all opened files, as well as other processes using I/O, such as sockets and pipes. These processes use file descriptors to perform a variety of operations on the target file, such as reading, and writing.

A process opens the file, the operating system creates a representation of that file, and stores information about the opened file.

The command needed to list all file descriptors is lsof.

If you need the file descriptors in use by a specific process, the -p flag can be used to input a process id:

root@linuxpc:~# lsof -p 730751
COMMAND    PID         USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
vim     730751 somebody  cwd    DIR   8,18    20480 4587587 /home/somebody
vim     730751 somebody  rtd    DIR   8,18     4096       2 /
vim     730751 somebody  txt    REG   8,18  3787792 5767354 /usr/bin/vim.basic
vim     730751 somebody  mem    REG   8,18  6096560 5768536 /usr/lib/locale/locale-archive
...

Alternatively, you may try ls -l /proc//fd if lsof is taking a long time to produce an output.

Using lsof, you can also search for a specific file, if you have insight into what file(s) might be being worked on. For example, assume we have a non-empty file file.txt:

<?php

$example = fopen("file.txt", "r");

while (true) {
    usleep(100000);
}

?>

Run the following php snippet, and observe:

root@linuxpc:~/tmp2# lsof | grep file.txt
php       757383                                 root    4r      REG               8,18          0    4343172 /root/tmp2/file.txt

It shows that the php process is holding the file file.txt.

So, what is a file descriptor leak?

Let’s change the php script a little bit:

<?php

try {
    while (true) {
        $example = fopen("file.txt", "r");
        while (($line = fgets($example)) !== false) {
            echo $line;
        }
    }
} catch (Exception $e) {
    echo $e->getMessage() . "\n";
    echo $e->getTraceAsString() . "\n";
}

?>

Inspecting lsof this time will give multiple file handles:

root@linuxpc:~/tmp2# lsof | grep file.txt
php       780917                                 root    4r      REG               8,18         23    4343176 /root/tmp2/file.txt
php       780917                                 root    5r      REG               8,18         23    4343176 /root/tmp2/file.txt

Let’s break down why, and what happens here:

• -> The file file.txt is being opened repeatedly, but never closed

• -> File handles will keep accumulating, until the operating system runs out of available file descriptors

• -> The program may crash when this happens, or may only perform part of its tasks with fd failures being eaten silently.

This is obviously a very synthetic example, but depending on the complexity of your program, it might not be obvious that the reason is a file descriptor exhaustion.

Besides causing issues with the program itself, file descriptor exhaustion can interrupt other processes, since the file descriptor pool is a global limit (you can check the exact value on your system with ulimit -n).

Steps to avoid this problem

• -> add delays between operations if your program must open and close files frequently

Even giving a 10ms breathing room to the OS can make the difference:

<?php
$files = ['file1.txt', 'file2.txt', 'file3.txt'];

foreach ($files as $filename) {
    $file = fopen($filename, "r");
    // ... do stuff
    fclose($file);

    usleep(10000);
}
?>

• -> batching operations instead of doing them one by one

For example, use a buffer, flush every 8kB:

<?php
$file = fopen("file.txt", "a");
$buffer = '';
foreach ($lines as $line) {
    $buffer .= $line . "\n";
    if (strlen($buffer) > 8192) {
        fwrite($file, $buffer);
        $buffer = '';
    }
}
if ($buffer) {
    fwrite($file, $buffer);
}
fclose($file);
?>

• -> if the same file needs to be written to over and over, consider just keeping it open and reusing the file handle

• -> don’t use unnecessary file descriptors: if you need to read from a file, use a read file descriptor, don’t open with write mode as well

Conclusion

I once had an old installer script written in php segfault halfway through deploying a large application. It was doing rapid file operations on a mechanical HDD, and I never managed to find a definitive conclusion, but given that swapping to a clean VM hosted on an SSD fixed the issue, an unclosed-descriptor loop is a plausible culprit. When a process repeatedly opens files faster than the operating system can release them, resource exhaustion can trigger crashes that look like random instability.