Overview

The aim here is to demonstrate how a scitoken can be used to provide authentication and authorization to writeable storage via xrootd. See other Knowledge base articles (KBA) on getting enrolled for scitoken usage.

XRootD Example

Submission Script

The main thing to note is the addition of use_oauth_services. This gets set to the name of the token issuer, followed by the requested role. The role defines a specific set of policies and lifecycle specification.

xrootd_copy.submit

universe = vanilla
+projectname="GLUEX"

request_memory = 0.5GB

output = xrootd_copy.$(cluster).$(process).out
error  = xrootd_copy.$(cluster).$(process).err
log    = xrootd_copy.$(cluster).log

executable = xrootd_copy.sh

# Format <token issuer>_<role>
use_oauth_services = jlab_gluex

queue

Job Script

The payload of the job will have an environment variable pointing to the credential/token directory (this can vary site to site). This example shows what the contents and what they look like, but isn't necessary for production usage. The key line is setting BEARER_TOKEN_FILE to your access token for use with xrootd:

export BEARER_TOKEN_FILE=${_CONDOR_CREDS}/jlab_gluex.use

xrootd_copy.sh

#!/bin/bash -v
set -e

echo "=== Contents of credential directory ==="
ls -l ${_CONDOR_CREDS}
echo

echo "=== Contents of credential files ==="
for file in ${_CONDOR_CREDS}/*
do echo "--- ${file} ---" ;
cat ${file}
echo
done
echo

echo "=== Test xrdcp back to JLab ==="
export BEARER_TOKEN_FILE=${_CONDOR_CREDS}/jlab_gluex.use
echo "--- BEARER_TOKEN ---"
echo $BEARER_TOKEN_FILE

tmpfile=xrootd_copy.data
date > tmpfile
echo "xrootd_copy test" >> $tmpfile
USERNAME=`echo $OSGVO_SUBMITTER | cut -d@ -f1`
xrdcp -f $tmpfile xroots://dtn2002.jlab.org//gluex/xrootd_copy-${USERNAME}-${HOSTNAME}

exit 0

Job Submission

When submitting the first time or with an expired token, you will be prompted with a CILogon URL. Copy/paste that URL into any browser to authenticate (it does NOT have to be on submit node). Select Thomas Jefferson National Accelerator Facility as your identity provider.

Submit the job and monitor its progress. NOTE there is a 2min timeout for the provided URL. If that expires, you will get a clearly stated error message. Just resubmit the job and proceed to the new URL.

OPTIONAL: If you want to get and view the access token before job submission, you can do that by:

$> condor_vault_storer -v <issuer>_<role>
$> httokendecode -H

Otherwise, proceed to submit your job and get the URL prompt:

$> condor_submit xrootd_copy.submit

Submitting job(s)
Authentication needed for jlab_gluex
Attempting kerberos auth with https://htvault.jlab.org:8200 ... failed
Attempting OIDC authentication with https://htvault.jlab.org:8200

Complete the authentication at:
    https://cilogon.org/device/?user_code=D96-FJ4-KRG
No web open command defined, please copy/paste the above to any web browser
Waiting for response in web browser

Saving credkey to /home/wmoore/.config/htgettoken/credkey-jlab-gluex
Saving refresh token ... done
Attempting to get token from https://htvault.jlab.org:8200 ... succeeded
Storing bearer token in /var/run/user/4997/bt_u4997-jlab_gluex
Storing condor credentials for jlab_gluex
.
1 job(s) submitted to cluster 671993.

Monitoring Job and Output

Check job status. Once complete, you can check the xrootd output at ifarm:/work/test-xrootd/gluex/.

$> condor_q


-- Schedd: scosg20.jlab.org : <192.70.245.23:9615?... @ 09/27/22 10:16:08
OWNER  BATCH_NAME    SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
wmoore ID: 671993   9/27 10:14      _      1      _      1 671993.0

Total for query: 1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended 
Total for wmoore: 1 jobs; 0 completed, 0 removed, 0 idle, 1 running, 0 held, 0 suspended 
Total for all users: 273 jobs; 0 completed, 0 removed, 0 idle, 140 running, 133 held, 0 suspended


$> cat /work/test-xrootd/gluex/xrootd_copy-wmoore-scigrid6.physics.fsu.edu 
Tue Sep 27 10:15:47 EDT 2022
xrootd_copy test

Screenshots for CILogon usage

Navigate to the URL you are provided and select your identity provider. Keep in mind, it's TJNAF and not Jefferson Lab.

 

After hitting "Log On", you will be redirected to the JLab SSO page.

 

Once you are correctly authenticated, you will see the following. Then your terminal will show the successful job submission.