MySQL not releasing temp file descriptors
- by Wakaru44
Since a few days ago, we’ve been experiencing some serious problems with our MySQL installation: MySQL keeps opening temporal files (normal behaviour) but these files are never released. The consequence is that, eventually, the disk space is exhausted and we have to restart the service and clean up /tmp manually.
Using lsof, we see something like this:
mysqld 16866 mysql 5u REG 8,3 0 692 /tmp/ibyWJylQ (deleted)
mysqld 16866 mysql 6u REG 8,3 0 707 /tmp/ibf5adsT (deleted)
mysqld 16866 mysql 7u REG 8,3 0 728 /tmp/ibGjPRyW (deleted)
mysqld 16866 mysql 8u REG 8,3 0 5678 /tmp/ibMQDLMZ (deleted)
mysqld 16866 mysql 13u REG 8,3 0 5679 /tmp/ibQAnM42 (deleted)
Maybe it's not related, but when we shutdown the server, the files are finally freed, and we can see the following warnings in the MySQL log:
121029 7:44:27 [Warning] /usr/local/mysql/bin/mysqld: Forcing close of thread 1333 user: 'xxx'
121029 7:44:27 [Warning] /usr/local/mysql/bin/mysqld: Forcing close of thread 1156 user: 'yyy'
121029 7:44:27 [Warning] /usr/local/mysql/bin/mysqld: Forcing close of thread 1151 user: 'zzz'
where 'xxx', 'yyy' and 'zzz' are distinct mysql users (and the only 3 users with active connections to the database).
We have a few theories:
There is a problem in the OS, that keeps file handlers open. Could it be possible that the OS "delete" operation blocks the threads until shutdown? This may explain the warning at shutdown and the fact that files are finally deleted when the process dies.
Until now, data sets were so small that temp files were relatively small and there was enough time to release the file handles without exhausting disk space.
We are using Mysql 5.5 on a RHEL 6.2 with the default kernel.
