AWS Storage Gateway
15 Jul 2019AWS Storage Gateway is a managed AWS service that simplifies storage management and reduce costs for our enterprise document management system Ellucian Banner Document Management.
We built a file gateway to create a file interface that enables us to store files as objects in Amazon S3 using the industry-standard NFS protocols. Files are transferred to S3 STANDARD_IA because the majority of our documents are infrequently accessed. Because they are native S3 objects, we can then apply our bucket policies for lifecycle management.
Not only does this create a very cost-effective document repository, storage gateway satisfies one of the goals of the project which was to have a detached storage device. Detached storage enables us to perform future upgrades independent of the the storage device. File gateway backed by S3 allows us to take advantage of cloud services to improve availability, durability, and performance.
At the outset, we planned on building a detached EBS volume. But EBS volumes only work on an older version of NFS, and prefer linux. Our document management system is a windows application. For windows file shares AWS recommends FSx, but we found it to be too expensive.
- applications benefit from a low-latency local cache of frequently used content.
- preserves the file metadata as object metadata
- preserves the directory structure by including it in the object name
- tiered cloud storage
All objects are stored in the storage class Standard-IA (Infrequent Access). All objects are encrypted via S3 KMS.
One pitfall we discovered is if objects are updated directly in S3, the metadata must be manually refreshed in the gateway cache. In our use case, this should only occur during the initial load from our legacy storage. Also, there is no console application for monitoring this process, so you have to setup an SNS topic to monitor it.