How to Estimate Storage Requirements?
Some rules of thumb
When the DMP asks for storage size and transfer speeds, it may be a daunting task to find a conservative real-world estimation of a starting value.
Of course, you may improve performance and efficiency by rethinking you data formats, compression algorithms and network protocols.
But let's start in a safe and sound conservative corner:
Bandwidth
Bandwidth is normally given in GBit/sec. Divide it by 10 to get the GByte/sec value. Yes, a Byte has 8 Bits, but protocol overhead, latency, timing etc. slow your transfers down. Having a 20% buffer in mind is not a bad idea.
The 2–5–9 Rule
The 2–5–9 rule of server side disk storage:
When you write, stream, or pipe data through a real-world network to some server infrastructure do not expect more than
2 GBit/sec on HDD (spinning rust Hard Drives)
5 GBit/sec on SSD (solid state disks with classical adapters such as SAS or SATA)
9 GBit/sec on NVME (solid state disks bound directly to a PCIE Bus)
Copying or Synching Data
When copying or syncing data over a network, always rethink if the network speed or the slowest participating disk tends to be the "bottleneck".
Concerning network speeds, do not expect more than
10 GBit/sec from lab to neighbouring lab
1 GBit/sec from office to lab
0,3 GBit/sec when WIFI is involved
30 MBit/sec when encrypted copy over single-threaded SSH is involved
7MBit/sec if a mobile phone line is involved
Uncompressed Media Streams
Uncompressed media streams are most demanding. You will want to start your estimations with:
1 GByte per second for a RAW high quality full color 4K video stream
1 GByte per hour for a RAW standard quality stereo audio stream
1 MByte per second for motion tracking data at 25 samples/sec per motion tracking marker and camera for coordinates and metadata
When it comes to improving performance, optimising your storage concepts or rethinking your setup, do not hesitate to contact your Research Data Managers Lena Dreher and Ilja Werner.