Title: Scalable Distributed Meta-Data Management in Parallel NFS
Abstract: High performance computing (HPC) applications demand high I/O throughput from file systems to match their fast processing requirements. These HPC applications create large amounts of file meta-data that can overwhelm current single meta-data server file systems leading to performance bottlenecks. We address this problem with a multiple meta-data server (M-MDS) design using standard parallel NFS (pNFS). We utilize NFS directory referrals mechanism with hashing to efficiently distribute meta-data across a cluster of metadata servers. We show through a large scale setup in Amazon EC2 that our pNFS M-MDS can scale almost linearly and outperform Lustre CMD (Clustered Metadata) by up to 3 times in some of the file system meta-data operation benchmarks.
Publication Year: 2016
Publication Date: 2016-07-01
Language: en
Type: article
Indexed In: ['crossref']
Access and Citation
Cited By Count: 1
AI Researcher Chatbot
Get quick answers to your questions about the article from our AI researcher chatbot