Migration to read-only cache

Changes and implications of migrating the `/cache` filesystem to read-only access on November 19, 2024:

What is Changing?

• Access: The /cache filesystem will become read-only side-wide (farm, ifarm, Globus, XRootD), with the exception of the following transfer methods.
• File Transfer Methods:
  • jcache: To read files from tape and write them to /cache.
  • jput: To write files to tape.  The `-cache` option will inform the system to put a copy in /cache as soon as it is written to tape.
  • jmirror: Designed to support data ingestion from experimental halls but also available for general use.  Also has a `-cache` option to place a copy of the file in /cache as soon as it is on tape.

Why is This Happening?

• Improved File Management: This change ensures timely movement of files to tape and maintains /cache as an accurate reflection of tape storage.
• Conflict Resolution: Reduces conflicts where files in /cache did not match those stored on tape, leading to slowdowns and data management confusion.
• Small File Handling: Addresses historical issues with small files that were not backed up or stored on tape.

Effects on Farm Job Workflows

• SWIF Workflows: Jobs using SWIF that specify output to `mss:/mss/...` and `/cache` will continue to function as before.
• Direct Writes: Jobs trying to write directly to /cache using standard commands (like `open()`, `cp`, `mv`) will fail.
• Slurm Jobs: Outputs from Slurm jobs not part of a SWIF workflow should go to `/volatile` and be manually moved to tape using `jput` on ifarm. It is recommended that Slurm workflows involving tape use SWIF.
• jput Availability: The `jput` command will not be available on non-interactive farm nodes to prevent job stalls.

Examples

• jmirror
  • Intended to copy/synchronize a directory tree to tape storage.  This was designed to support moving 'batches' of raw data files from the experimental Halls to long term storage, but can be used on the ifarm as well. NOTE: DO NOT abuse this tool by writing large volumes of small files to tape by accident!
  • If you want the file to be available under /cache, then use the '-cache' argument.
    

    # Usage: jmirror [-cache] <directory> <local-prefix> <stub-path>
    # The '-cache' option will inform the system to put a copy in /cache as soon as it is written to tape.
    
    # Example:
    jmirror -cache /super/duper/data/exp1 /super/duper /mss/halle
    
    # For further docs:
    man jmirror OR jmirror -h

• jput
  
  • Files will be copied to a staging area and queued for writing to tape.  They will not automatically show up under /cache by default.
  • If you want the file to be available under /cache, then use the '-cache' argument.
    

    # Usage: jput [-cache] <file...> <stub-path>
    # The '-cache' option will inform the system to put a copy in /cache as soon as it is written to tape.
    
    # Example:
    jput -cache file1 file2 /mss/halle/
    
    # For further docs:
    man jput OR jput -h

What is Not Changing?

• Cache Deletion Policy: Remains unchanged.
• Cache File Pinning: Continues to work as before.
• jcache client: Continues to work as before.
• SWIF outputs to /cache: Continues to work as before.

Summary

This transition aims to enhance data management efficiency and reliability by establishing a clear process for how files are stored and accessed, particularly with regard to tape storage. Users need to adapt their workflows accordingly, especially those using Slurm jobs.