We have enough already, for crying out loud, what we don’t need is another distributed filesystems. We need to consolidate and improve on what we do have.
Here is a list of distributed filesystems that I found while looking for one that fit my needs, and some quick notes on each. (Of course I haven’t tried many of these, but please let me know if you have strong thoughts about any of them, or if you know of some glaring omissions from the list).
I’ll also say that I’m not happy with any of these yet, they all still leave plenty to be desired. If you want to talk about these please catch me at the upcoming Google Scalability Conference 08.
POSIX compatible; supports data migration,re-replication when a host goes down
con: write on close; can’t use for databases
replaced by lustre ?
DRBD (Distributed Replicated Block Device)
limited to 2 servers; It is not a filesystem, but the filesystem lives on top of drbd.
used with heartbeat(http://www.linux-ha.org/) for failover
The GNU Cluster FileSystem (GlusterFS)
also here
no rebalancing or automated replication
files are stored as whole files, not as distributed chunks
there are problems with bringing an old file store online
Google File System (GFS)
Clearly anyone can’t just “use” this, but it deserves to be on the list because of it’s design principles, and for comparison
desgin
win32(not production ready, but soon),linux
written in java 1.5.x
not designed for low latency
write once files
no rebalancing [yet]
no automatic handling of namenode failure
use client library (python,…) or fuse
comparison with Hadoop File system
sounds promising; check back in 6 months when it’s a little more stable and full featured.
posix compliant; no single point of failure
active-active failover pairs
Uses fuse, not persistent unless using memcachedb
write once files, replication, flat namespace, not posix compliant
Oracle Cluster File System OCFS2
GPL, not posix compliant, specifically designed for oracle database files
Parallel Virtual File System (PVFS)
requires shared storage device for redundancy; no online adding/removing of servers
A new project. heavy development and doesn’t have enough features yet to be used in a production environment.
posix compliant, ok to use for databases
about the design
no hotspot detection via usage
not designed for a single user, as there is a lot of overhead in maintaining information about who as access to what files; replication is not an automatic process.
not a distributed filesystem, but it has a lot of similar benefits of distributed storage devices, and it has a management interface that would be amazing for a distributed filesystem. Also see: luster