CVMFS (CernVM-File System) at JLab

What is it?

The CernVM-File System (CernVM-FS) provides a scalable, reliable, low-maintenance software distribution platform. It was originally developed for High Energy Physics (HEP) collaborations to distribute software across globally distributed computing resources.

Key characteristics:

Implemented as a POSIX read-only filesystem in user space (FUSE module)
Hosts files and directories on standard web servers
Mounts content under the global namespace: /cvmfs
Optimized for distributing large, complex, frequently-updated software stacks

JLab Specifics

At Jefferson Lab, CVMFS is preconfigured on ifarm and farm nodes.

Local repository access is optimized through the squid cache:

CVMFS_HTTP_PROXY="http://sci-squid.jlab.org:3128"

Software is published using OASIS (OSG Application Software Installation Service).

Example Usage on ifarm

CVMFS repositories function like locally installed software. For example:

% ls /cvmfs/oasis.opensciencegrid.org/
accre  auger    csiu  fermilab  glow   ilc   ligo  nanohub  osg           sbgrid
atlas  cmssoft  enmr  geant4    gluex  jlab  mis   nova     osg-software  snoplussnolabca

For Scicomp-provided software environments, see: Environment Modules.

Getting Data Into OASIS

Data placed in specific source directories is synchronized and published to CVMFS.

Source: /scigroup/cvmfs/<project>
Destination: /cvmfs/oasis.opensciencegrid.org/jlab/<project>
Logs: /scigroup/cvmfs/logs/<project>.log
Update cadence: Every ~4 hours via cron

New top-level project directories require an update to the cronjob configuration. Users must submit a ServiceNow incident to request additions.

Changes (adds/removes) to the source directory are published at the next scheduled sync. After rsync completes, OSG processes the updated content and republishes it.

Log Files

Each project has a dedicated log file, which is rotated and purged periodically. Logs provide a reliable source for diagnosing:

rsync failures
permission errors
missing files
unintended deletions

Rsync Excludes

Each project directory includes an oasis_exclude.txt file. This file is passed to rsync’s --exclude-from option.

Important notes:

Items added to the exclude file after being published will be removed during the next sync.
Users should review exclude files carefully to avoid accidental deletion.

Restrictions (Effective January 20, 2026)

Beginning January 20, 2026, CVMFS access will be limited to a defined set of approved repositories. This change improves performance, reduces unnecessary mounts, and matches expected access patterns.

Impact

Only the following repositories will remain enabled:

eic.opensciencegrid.org
gluex.osgstorage.org
jlab.opensciencegrid.org
oasis.opensciencegrid.org
singularity.opensciencegrid.org

All other repositories will be disabled and will no longer auto-mount on login or compute nodes.

If software, data, or workflows depend on a repository outside the approved list, users must notify Scientific Computing Operations.

Requesting Additional Repositories

To request the re-enabling of a repository, provide:

Repository FQDN
Usage description (software, data, workflow)
User group or project needing access

Requests are reviewed on a case-by-case basis.

Need Additional Help?

Submit a ServiceNow incident for:

Adding a new OASIS project directory
Troubleshooting file updates
Requesting repository access changes
Reviewing rsync logs or permissions issues

References

CVMFS Documentation: https://cvmfs.readthedocs.io/en/stable/
OASIS Publishing Guide: https://opensciencegrid.org/docs/data/update-oasis/

CVMFS at JLab