A sysadmin gets a ticket: "Disk is full, but df -h shows 40% free space." A classic trap for anyone unfamiliar with inodes. The server refuses to create new files not because storage is exhausted — but because index nodes are. This is where a real understanding of how Linux stores data actually begins.
Two Layers of Storage
Most users think a file is just a name and some content. In reality, every file exists in two independent places.
The first is data blocks: physical disk sectors holding the actual content. The second is the inode (Index Node): a separate fixed-size structure storing all metadata about that content. And the filename is a third, entirely separate thing — stored in a directory entry that simply points to an inode number.
Remove the directory entry and the inode goes nowhere, as long as at least one reference to it remains. This is the foundation of how hard links work and why a file held open by a process can be recovered even after rm.
The indirect pointer hierarchy allows ext4 to address files up to 16 TB while keeping each inode at a fixed 256 bytes.
What an Inode Actually Contains
The filename is not in the inode. Everything else is: file size in bytes, storage device ID, owner UID and GID, permission mask (mode), file flags, three timestamps (atime, mtime, ctime), hard link count, and pointers to data blocks. Blocks are addressed hierarchically: direct pointers for small files, indirect for medium, doubly indirect for large.
Three Timestamps: What Is the Difference
Confusion between atime, mtime, and ctime causes subtle bugs in backup and monitoring scripts.
mtime changes when file content is modified. ctime changes when content or inode metadata changes (permissions, ownership, hard link count). atime updates on every read. On heavily loaded servers, atime accumulates unnecessary write operations. The fix is the noatime mount option in /etc/fstab:
/dev/sda1 / ext4 defaults,noatime 0 1
On VPS instances with SSD storage this reduces disk wear and I/O count without any loss of functionality.
Get the Inode Number and File Metadata
The quick approach uses the -i flag with ls:
ls -i /etc/nginx/nginx.conf
786435 /etc/nginx/nginx.conf
For a full picture, use stat:
stat /etc/nginx/nginx.conf
Output includes the inode number, all three timestamps, block count, and link count — useful for debugging scripts that rely on modification times.
Inode Exhaustion: Diagnosing and Finding the Cause
Every partition is created with a fixed inode limit set at format time. On VPS and dedicated servers, inode exhaustion is not exotic. A few hundred thousand PHP session temp files or a Postfix queue holding a million messages — and the system stops accepting new files entirely.
Check inode usage across all partitions:
df -i
The /var/spool partition at 100% means the mail server has stopped. There may be 50 GB of free disk space and it makes no difference.
Find the culprit — the directory with the most files:
find /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -15
Typical offenders: /var/spool/postfix/deferred, PHP session temp folders, CMS cache directories, npm and composer build artifacts.
Find a File by Inode Number
A file whose name contains non-printable characters or starts with a dash is found via inode:
find /home -inum 786435
Limiting to a specific partition speeds up the search significantly. Deletion works the same way:
find /home -inum 786435 -delete
Hard Links and the Link Counter
A hard link is a second name pointing to the same inode. The data on disk is singular; the link count increments:
ln /var/log/app.log /backup/app_hardlink.log
stat /var/log/app.log | grep Links
# Links: 2
The data is physically removed only when the counter reaches zero. This is why rm is technically called unlink — it disconnects a name from an inode. If no links remain and no process holds the file open, the kernel frees the blocks.
Practical consequence: deleting a log file while nginx is writing to it removes it from the directory but disk space is not freed until nginx restarts or receives a USR1 signal.
Recover a Deleted File via /proc
While a process holds a file open, the data lives in /proc/PID/fd/:
lsof | grep deleted | grep nginx
cp /proc/12345/fd/7 /var/log/app_recovered.log
This works only while the process is still running.
Quick Reference
| Task | Command |
|---|---|
| Get inode number | ls -i filename |
| Full inode metadata | stat filename |
| Inode usage per partition | df -i |
| Find file by inode | find /path -inum NUMBER |
| Delete file by inode | find /path -inum NUMBER -delete |
| Directories with most files | find /path -xdev -printf '%h\n' |
| Create a hard link | ln source linkname |
| Check link count | stat filename |
| Recover deleted open file | cp /proc/PID/fd/FD /path/recovered |
