NIH LIFE SCIENCES, IN PARTNERSHIP WITH VAST DATA
From a 7-year analysis to weeks: neuroimaging at scale
AT A GLANCE
- 1.4 billion brain images processed in one experiment alone
- A 7-year analysis completed in just a few weeks
- Further optimization is underway to reduce analysis time to hours.
- Infrastructure scalable for all AI, ML, and long-term scientific projects
THE CHALLENGE
Billions of small files. Existing systems could not keep up.
A spinTwo customer at NIH required robust computing infrastructure to handle billions of small files while integrating seamlessly with their existing HPC cluster. The work was crucial for accelerating research in immune analysis and drug discovery. Across multiple experiments, scientists captured 1.4 billion images of mouse brains to study their neurological activity.
Managing and processing this data was a monumental challenge. Traditional parallel file systems could not handle the large number of small files and the performance requirements the research demanded. The customer at NIH needed a filesystem that could not only present data to the existing HPC cluster but also scale to support future AI and ML workloads — all while remaining accessible to scientists, not just system administrators.
The core issue was the filesystem. The project required infrastructure that could handle not just massive computing power but also billions of small files simultaneously, presenting data for intensive computational analysis.
WHAT WE DID
Evaluate, select, integrate, and scale
After evaluating multiple options capable of integrating with traditional HPC systems, spinTwo selected VAST Data on behalf of their customer at NIH. VAST Data's architecture stood out for its ability to handle massive amounts of small files while delivering high performance and providing flexible data access.
- 1
Infrastructure assessment and technology evaluation
spinTwo evaluated multiple filesystem options capable of integrating with the NIH customer's existing HPC environment, identifying the need for a system that could handle billions of small files at the performance levels required for research. - 2
VAST Data selection and integration
spinTwo selected VAST Data on behalf of the customer at NIH and integrated it as a core storage layer of the HPC cluster. The platform provided access to the HPC cluster, high-performance workstations, and critical servers controlling scientific instrumentation. - 3
Scalable, future-proof infrastructure
spinTwo integrated the VAST Data Platform into a scalable system designed to support all AI and ML initiatives, large-scale data analytics, and long-term scientific projects beyond the initial engagement. - 4
NAIRR Secure Pilot integration
The spinTwo team played a major role in driving insights in collaboration with the NAIRR Secure Pilots, co-led by the DOE and NIH, integrating computational resources and AI-based workflows across HPC environments at both NIH NCATS and DOE ORNL.
THE RESULT
What used to take years can now be completed in weeks
The results transformed the NIH customer's research capabilities. Scientists accessed their data faster than ever before. The customer at NIH significantly improved its data processing speed and ability to scale complex life sciences research. The system allows scientists to focus on their research, free from the burden of data management and infrastructure logistics.
spinTwo integrated the VAST Data Platform into a scalable, future-proof system that continues to support all AI and ML initiatives, large-scale data analytics, and long-term scientific projects.
7yr → wks
A data analysis project estimated to take seven years completed in just a few weeks
1.4B
Brain images processed through a more efficient imaging analysis workflow
Weeks→hours
spinTwo is working to reduce analysis time from weeks to hours.
Faster
Scientists access data faster than ever, spending more time conducting research and less time waiting.
GOVERNMENT WORK
Driving insights across federal research programs
Building on this engagement, the spinTwo team has played a major role in driving insights in collaboration with the National Artificial Intelligence Research Resources (NAIRR) Secure Pilots, co-led by the Department of Energy and NIH.
The team has been pivotal in integrating computational resources and AI-based workflows across environments within both NIH and DOE organizations.
spinTwo facilitated the integration of large language model (LLM)-based workflows with the Frontier supercomputer at Oak Ridge National Laboratory (ORNL) and helped integrate JupyterHub to launch notebooks as traditional jobs running on an HPC cluster.
Hugo Hernandez has noted that the primary objective of the NAIRR Secure Pilots is to identify challenges and opportunities for interoperability between tools, infrastructure, software, and resources across NAIRR Secure enclaves, enabling both scientific and technical collaborations across the federal government while maintaining essential security measures.
TECHNICAL DETAILS
What was integrated
For buyers who want to go deeper into the technical scope of the engagement.
Storage platform
- VAST Data Platform
- VAST DataStore
- VAST DataBase
- VAST Data Catalog
Architecture
- DASE (Disaggregated and Shared-Everything)
- All-flash NAS
- AI-scale analytics
Integration points
- Existing HPC clusters
- High-performance workstations
- Scientific instrumentation servers
- LLM-based workflows
Government programs
- NAIRR Secure Pilots
- NIH
- JupyterHub
Working on a similar challenge?
Whether you need HPC infrastructure for large-scale data processing, AI-ready pipelines, or federal compliance, we start with a diagnosis before we touch anything.