Skip to contents

CIDA-data-storage-guidelines

Research Tools Committee

2022-11-30

data_storage.Rmd

CIDA Data Storage Tools

Introduction

This document provides guidelines, recommendations, and current best practices for where code, data, and other documents should be stored and backed up for members of CIDA. These guidelines are intended to explain the pros, cons, and best practices for the data storage tools available to CIDA members. However, a CIDA member may come across exceptions to these guidelines, and the responsibility for proper data handling within a particular project rests on the CIDA member involved.

Who does this apply to: All members of CIDA (Professor, RA, RI, Senior RI, PRA, Senior PRA)

Definitions

  • ‘Code’ refers to any R, SAS, Stata, etc. (scripts stored as plain text formatted files)
  • ‘Data’ refers to spreadsheets, tables, or other information used to run analysis on
  • ‘Report’ refers to any document which outlines results of analysis for sending to an investigator
  • GitLab is the Lhotse CIDA GitLab repository, available here (requires VPN connection)
  • ‘Local storage’ refers to any university provided computer with a physical hard drive
  • ‘External server’ refers to a server which houses data or runs analysis (except Lhotse)
  • ‘Eureka’ is the Virtual Machine run by Health Data Compass (HDC)
  • ‘OneDrive for Business’ refers to Microsoft’s file-syncing software licensed by the University
  • ‘HPC’ refers to “High Performance Computing”

Summary

  • The CIDA (P) Drive is the required ongoing storage location for project data. Other data storage locations may be used temporarily during the course of active work on a project.

Data Storage Locations Available to CIDA Members

Local Computer

  • All files for any current projects can be stored on a University provided PC or Macintosh, if and only if the computer’s hard drive is encrypted.
  • Data with PID should only be locally stored during active work on a project.
  • Pros: data and projects load more quickly when stored locally.
  • Cons: no natural/default backup process, can be a hassle to do best security practices manually.
  • Best Practices:
    • Only keep project data on your computer while working on it, and if you do this often, ensure you frequently save your data to the CIDA drive.
    • Prior to travel with a laptop, any old or unneeded projects should be removed from the computer drive and can be restored upon return to campus.

CIDA (P) Drive

  • Mapping depends on operating system. See instructions here for mapping drives.
    • MAC: smb://data.ucdenver.pvt/dept/SPH/SPH-CIDA/CIDA
    • Windows: \\data.ucdenver.pvt\dept\SPH\SPH-CIDA\CIDA
    • If you have recently started and are having trouble mapping this drive, contact SOM-IT.
  • CIDA project data must be stored here on a permanent basis (with certain exceptions, e.g., projects larger > 16 GB or if the collaboration dictates otherwise).
  • The storage under the CIDA Drive is set up as follows:
    • CIDA/Branches: Long-standing collaborations, including those operating under MOUs, are treated as Branches, and their data should be stored in a subdirectory of CIDA/Branches.
    • CIDA/Projects: Data for projects for the consulting arm (i.e., those with a P-number) should be stored in a subdirectory of CIDA/Projects. Use this form to create folders in the P-drive with specific permissions, or this form to update the permissions of an existing folder. These forms are directly sent to SOM IT.
    • CIDA/Shared
      • The CIDA/Shared directory is accessible to all CIDA members who have gained approval from a data manager. Folders are also accessible to external users who have been approved for access.
      • Files and directories that you create on the shared drive inherit their permissions from their parent folder. You cannot restrict access to specific directories in accordance with data use agreements without the help of IT.
  • Pros:
    • All files and directories are backed up nightly; backups are stored for 30 days.
    • Collaboration and transfers of data among CIDA members can be quick and easy.
    • Data and code are easily findable by CIDA administration and other team members in the case of a CIDA member’s continued absence.
  • Con: working directly on this drive can be slow, especially through a VPN.
  • Best Practices:
    • Up-to-date raw data for all projects should be available in their expected location on this drive at least weekly, and especially at project conclusion, or prior to a project not being actively worked on.
    • Eliminate redundancies and intermediate data sets in projects with “big” data; only store the data you need to make code and reports run.
    • It is OK not to work directly on the CIDA drive in cases where speed is a concern. If you do, know that all active projects should copy data over to the CIDA drive regularly (weekly), and especially prior to taking leave. The CIDAtools R package, located here, has the function BackupProject() that can streamline this process. By default, this function only updates folders/files in DataRaw/* and DataProcessed/* which have changed, so it should not take too long.
  • CIDA pays for this server storage on a per-GB-month basis, so be cognizant of the size of the data utilized by your projects. If possible, the project’s scope of work should charge more for data sets and project materials which are anticipated to fall above a threshold of 32 GB. The OIT’s central file storage rate should be applied and multiplied to account for anticipated duration of storage. See below.

Data storage costs on CIDA drive

Expected 10-year cost ($)
Data packet size (GB) P-drive OneDrive for Businness (per user)1 Eureka2
10 $30 $600 $28
100 $300 $600 $276
1,000 (1 TB) $3,000 $600 $2,760
10,000 (10 TB) $30,000 $1,200 $27,600

1Charged to University; not to CIDA
2Prices are based on the HDC website.

OneDrive for Business

  • OneDrive for Business is a hybrid local and cloud storage system that allows for up to 5TB of cloud storage. Individual file size is limited to 15 GB.
  • Unlike the CIDA drive, files and directories saved to the OneDrive directory are private by default and are backed up to the cloud.
  • OneDrive for Business is HIPAA compliant. OIT and SOM IT can access folders created on the directory, but only after a manager enters a data access request that gets approved by HR.
  • Files can be downloaded to your local computer from the cloud on an as-needed basis off-site, which ensures a limited amount of data is stored locally.
  • Files can also be shared with other individuals with @cuanschutz.edu email addresses via OneDrive for Business.
  • Things to be aware of:
    • Do not confuse this with a personal OneDrive account!
    • Do not install OneDrive for Business on an unencrypted or personal computer. Files may sync (and be downloaded) to the local computer, which would leave potentially sensitive data vulnerable.
    • Concurrent use of OneDrive for Business in directories tracked by Git can lead to some syncing issues with OneDrive. This can typically be solved by restarting OneDrive.
    • Relevant project data stored in OneDrive should be copied over to the CIDA drive regularly, especially when projects become inactive.
    • Certain institutions and groups may have policies against using OneDrive for Business, so please verify with your research group that it is acceptable for your data to live there.

External Servers

  • Depending on the project and the data of interest, the data may be housed in an external server. External servers exist in various forms and require varying workflows.
  • Care should be taken that data living on an external server remain on the external server. However, code, reports, and any other files necessary to the project should still be copied to the CIDA drive at regular intervals (exceptions may exist, e.g., if other processes are specified in CIDA’s memorandum of understanding with your research group).

Eureka by Health Data Compass (HDC)

  • Any patient level data on Eureka should only be moved off of the Google Cloud with explicit permission from HDC for that specific project and data.
  • Raw data or data with PID should not be copied over to any other location except those approved by HDC.
  • Backups of code, reports, and other files should still be copied over to the CIDA drive so that it is accessible to others in CIDA in case of an emergency.
  • For projects requiring Eureka, the Eureka Cost Estimator can be used to determine the expected costs of a particular project a priori. These costs should be charged to the project’s PI if possible.

GitHub

  • In general, GitHub should be used for everything, except for data. While tracking small data sets in Git and pushing to GitHub is usually harmless, tracking data sets in Git can snowball the storage needed in GitHub and also slow down your git commands.
  • Avoid tracking especially large binary data files (e.g. files with db, xls, xlsx, extensions). Instead, convert these files into a text file (e.g. tab-delimited txt file, csv file). This will speed up your git processes and help conserve storage space [see link for more information].
  • Instead of tracking data, one can build in a “data-check” in the code for the project to ensure the MD5 sums are as expected.
  • Please consult the “coding guidelines” document for more information on the best practices for tracking your work in Git and GitHub.
  • All repositories on GitHub should be updated frequently, but especially prior to leaving on vacation or any other extended time out of office.

Really, really big data

  • If your data for a particular project is particularly large, you may require or wish to have a data storage solution tailored to that particular project (and funded entirely by the project). For non-Eureka solutions, please contact the SOM IT.
  • Consider using CIDA’s HPC option (see below).

CIDA-BIOS High Performance Computing Server

  • The HPC was purchased by CIDA and funds from a U01 grant (Katerina Kechris and Debashis Ghosh are PIs). Therefore, primary priority is given to CIDA & research related to the U01 grant. Secondary priority is given to Ghosh & Kechris group members. [Tertiary access can be attained by requesting access from Dr. Kechris via smartsheet.]
  • The server is a Dell PowerEdge R740XD, with Intel Xeon Gold 6152 2.1G X (2) CPU, 44 cores, 1TB memory, 240 SSD X (2) mirrored disk operating system, and ~50TB of usable disk storage. The operating system is CentOS 7.x, and common research software are available such as R, R Studio, MatLab, Python, and Java.
  • Server is not HIPAA compliant and therefore no PHI can be stored on the server.
  • Backups are available via Wasabi: https://wasabi.com/hot-cloud-storage/.

Unapproved data storage options

  • Only the options listed above are approved by CIDA for temporary data storage, and only the CIDA drive is approved for ongoing data storage. Please follow the best practices and ask questions if you have them.
  • The following non-exhaustive list of data storage options are not approved by CIDA:
    • Dropbox
    • Google Drive
    • OneDrive (personal)
    • Unencrypted hard drive or flash drive

Data Transfer Options

Mode of transfer Notes
CIDA drive For transfers among CIDA members, the CIDA drive can be used for data transfer
OneDrive for Business University-preferred means of transferring data
Redcap Web-based, useful for ongoing projects where data updates are more frequent, ensures data format stays more consistent
External hard/flash drive Acceptable: CIDA’s 48 TB NAS station, or an encrypted flash drive. Unacceptable: Unencrypted drive, even if the file is password protected
Last resort: Email Email is not encouraged means of transferring data. In circumstances when no other approach is available, note that although email between CU-affiliated email addresses are automatically encrypted, this is not the case for external emails. You can manually encrypt by putting one of these keywords in brackets in the subject of an email: secure, safemail, or encrypt. Email from any other email system, such as gmail, is not acceptable.
Not acceptable Non-approved web-based systems including: Dropbox, Google Drive, … Unencrypted flash/hard drives, even if the file is password protected. Email from any other email system, such as gmail

Health Insurance Portability and Accountability Act (HIPAA):
http://www.hhs.gov/hipaa/for-professionals/index.html

Health Information Technology for Economic and Clinical Health (HITECH):
http://www.hhs.gov/hipaa/for-professionals/special-topics/HITECH-act-enforcement-interim-final-rule/index.html

Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule:
http://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html

VPN and Remote Desktop:
https://www1.ucdenver.edu/offices/office-of-information-technology/software/how-do-i-use/vpn-and-remote-access

GitLab:
http://cidagitlab.ucdenver.pvt/

OneDrive for Business:
https://www1.ucdenver.edu/offices/office-of-information-technology/software/how-do-i-use/onedrive https://www1.ucdenver.edu/docs/default-source/offices-oit-documents/how-to-documents/onedrive-staying-secure.pdf?sfvrsn=668bb7b8_4

Eureka:
https://www.healthdatacompass.org/cloud-analytics-infrastructure/using-eureka https://www.healthdatacompass.org/cloud-analytics-infrastructure/eureka-cost-estimator