The Impending Data Distribution Catastrophe

Simon SHEPHERD and Raymond A. GREENWALD
Johns Hopkins University/Applied Physics Laboratory, USA

The increased number of SuperDARN radars, the development of STEREO–mode operation, and the increased use of high temporal resolution data are rapidly bringing the data-distribution institutions to their knees. The data throughput has increased by factors approaching four and additional increases are likely. The daily data throughput may exceed 5 GB. Ultimately, we will be limited by our ability to push bits through the system. Within this context, exabyte tapes are proving to be increasingly cumbersome and do not have the long-term reliability that is necessary considering the total data storage requirement. Our hope that we may eventually be able to go to DVD technology may no longer be the best or most cost effective solution. In this context, we must consider ways in which we might reduce our storage demands, including approaches to limit data collected at the sites. We must also consider cost, reliability and manpower factors in our data distribution. In this context, we should discuss how we are currently processing data at APL and suggest a new alternative for data distribution.

Session Index

Proceedings Index