Another critical aspect of Data Management is consideration of which datasets are the most valuable and what protections need to be in place for them. Discussed in this section, are the different stages data could be in, what to consider before choosing a storage option, a comparison of the storage offerings from ARCC, and long-term planning for the data.
Stages of Data
During a research project, data takes on different stages of use each with a different storage requirement. Some research projects will use all of these stages, some will only use a few.
Potential Data - Data that are not yet collected, but there is a plan to store them.
Raw Data - This transitional stage includes everything that is collected from the potential stage into a place for processing or pre-processing.
Prepared Data - This stage describes the pre-processing of the raw data that prepares for a model or other processes.
Intermediate Data - This stage is the most temporary of all data, it could be a step in a process that creates these data before processing into final data or simulated data that helps train a model.
Final Data - The resulting data of a process. These data tell the story of the research and indicate the results.
Published Data - These data are the same as the final data, but are in a format optimized for sharing
Archived Data - This stage of data are no longer needed for ongoing research projects but are not deleted.
Assessing the Needs
Stage | Storage Need |
|---|---|
Potential | None yet, but a plan for raw is in place |
Raw | Could be stored in a temporary place or in a more permanent place if keeping raw is determined to be valuable |
Prepared | Should be stored or transferred to storage that will optimize the next process |
Intermediate | Should be stored in a highly performant in read and write operations and backups are not necessarily a requirement |
Final | Should be stored in a safe place if the process is difficult to re-do |
Published | May be in a different format for data sharing, possibly a compressed file stored on a repository |
Archived | Could be stored somewhere in “cold” storage in the most cost effective way possible |
Comparing Storage Options
Storage Type | Advantages | Disadvantages |
|---|---|---|
External Storage i.e., portable hard-drive or Laptop | Fully user controlled, can be encrypted, portable, and not accessible by others without physical access | Easily lost, vulnerable to damage, no extra copy, only as safe as the circumstances |
Cloud backed service i.e., Google Drive or Dropbox | User friendly, accessible from anywhere, interactive use of native files, shareable, sync-able | Possibly costly and subject to unexpected terms of service changes, potentially unauthorized access |
Cloud storage services i.e., AWS, GCP, or Azure | Robust, scaleable storage with customizable access and interoperability within the cloud environment | Potentially costly egress fees, terms of service changes |
Institutional Research storage service i.e., ARCC Data Portal | Free up to default limits, support for UWyo researchers, included backups and snapshots | Requires a UWyo based PI, does not include an offsite back up, or compliant data |
Institutional HPC Storage i.e., ARCC MedicineBow | Access to compute power, specialized directories for performance and collaboration, snapshots | Linux only permissions, not backed up, non support for interactive use of some file formats |
Specialized Institutional storage i.e., ARCC Pathfinder | Cloud-like backend and functionality with S3 protcol for sharing | Not backed up, no access to compute, requires specialized software clients to interact with data |
Considering Other Requirements
Two Column Tables are nice ways to separate content/ Background info along with an image example on the same “Slide”. Please notice the table width. This should stop scroll bars from appearing
|  |
How to Decide
Next Steps
Link to Previous sub-module or Home Module | Align left link to next sub-module or home |