If anyone is wondering what DATA and WORK are there for in their home directory on the IDF RSP:
The convention we’ve been following for the system-test notebooks is that DATA is for input data sets you’re working with, and WORK is for intermediate results (temporary files you’re creating in the course of processing, that sort of thing).
Anything under your home directory is persistent, so really you can put whatever you like wherever makes sense to you, but those directories are intended to be suggestive. For instance, in the analysis of the scale testing I’ve been doing, I’ve been keeping my timing data sets under DATA/IDF-scale .
If you want a lot of temporary storage, you’re better off using /scratch. Think of it as a larger, shared /tmp (in the RSP, your /tmp is local to your own container and not shared between users). Don’t count on it for long-term storage–that’s what your homedir is for (for your own stuff) and what /project is for for collaborative efforts. It should be regarded as ephemeral, but generally not on the timescale of active work (and if you’re running something that runs for days, the RSP Notebook Aspect is probably not the right choice for that workload).