{"id":353,"date":"2014-08-25T07:44:55","date_gmt":"2014-08-25T05:44:55","guid":{"rendered":"http:\/\/lunatic.no\/?p=353"},"modified":"2014-08-25T07:44:55","modified_gmt":"2014-08-25T05:44:55","slug":"missing-diskspace-in-linux","status":"publish","type":"post","link":"http:\/\/130.61.186.249\/index.php\/2014\/08\/25\/missing-diskspace-in-linux\/","title":{"rendered":"Missing diskspace in Linux"},"content":{"rendered":"

Today I had a problem with a server that had no more disk space. And I learned something new in the process.
\ndf -h<\/code> told me that 100% was in use. Which was about 29GB on that server.
\nBut if I checked the root partition with du -shx \/<\/code> i got about 9GB of used space.
\nSo off to checking where the space could have gone:<\/p>\n

Inode usage<\/h2>\n

I know “du” does not take account for inodes, etc. But according to dumpe2fs \/dev\/sdx1<\/code> my Inode size * Inode count = about 700MB.
\nSo that was not it.<\/p>\n

“Hidden” directories under mountpoints<\/h2>\n

“du” will not see used space of files located in a path that is later mounted to another file-system. For example, if you have files in \/home\/ on your root partition, but has later mounted \/home to its own partition. The files will be hidden behind the new mountpoint.
\nTo be able to check these files without unmounting anything in use, I did the following:
\n
\nmount --bind \/ \/mnt
\ndu -shx \/mnt
\n<\/code>
\nIf “du” would give me a different result now, I would have known that the files where hidden under one of my mountpoints. But sadly, they where not. I was starting to run out of options.<\/p>\n

Deleted files<\/h2>\n

If a process opens a file, and then you delete it by rm thefile<\/code> you will have the file deleted from the filesystem, but the inodes will not be freed before the process closes\/releases the file. This is something I love about Linux\/Posix systems, since that means that processes does not need to lock my files and I have full control over my files as opposed to other operating systems(windows)<\/small>. But I thought that when you delete a opened file, there is no easy way of knowing which deleted files are “held back” by processes. But there is!
\nlsof | grep deleted<\/code> quickly gave me a list of files that has been deleted, but is held open by processes, and their size. In my case a deleted log file of 17GB in size was held by asterisk. So i reloaded the logger module of asterisk, and hey presto! The diskspace was available again.
\nNow only 9GB was “in use” according to df -h.<\/p>\n","protected":false},"excerpt":{"rendered":"

Today I had a problem with a server that had no more disk space. And I learned something new in the process. df -h told me that 100% was in use. Which was about 29GB on that server. But if I checked the root partition with du -shx \/ i got about 9GB of used […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"class_list":["post-353","post","type-post","status-publish","format-standard","hentry","category-tips-tricks"],"_links":{"self":[{"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/posts\/353","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/comments?post=353"}],"version-history":[{"count":0,"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/posts\/353\/revisions"}],"wp:attachment":[{"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/media?parent=353"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/categories?post=353"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/130.61.186.249\/index.php\/wp-json\/wp\/v2\/tags?post=353"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}