The Pelican protocol is a method for transferring data to and from Jlab via an osdf endpoint. This requires the use of a scitoken for authentication, but allows for writes back to JLab to be written with an indivudual's jlab username (which makes management easier).
Scitoken is a short lived access token with specifically enumerated privileges. They are requested from an issuer using the htgettoken command and can be interrogated using the httokendecode command.
To use scitokens you need to submit a servicenow incident stating which project you are a part.
You will then receive an email with a link to complete your registration. Note that since this uses the CILogon system it is important that you not accidentally use another institutions credentials to authenticate. As such it is recommended that you open an incognito browser, then copy the link to finish registration and paste it into the incognito browser. Ensure that you use select "Thomas Jefferson National Accelerator Facility" from the dropdown and use your jlab user name and password to finish authentication to complete your authentication.
Note: If this is being set up for a service account, and not an individual users account, you will need to have a password for that account and be able to receive emails for it. Put that in the request for registration.
Once enrollment is completed you can request a scitoken using the htgettoken command.
htgettoken -a (token issuer) -i (institution) -r (role)
At JLab the commands will be as follows, and can be run from the ifarm systems or the OSG APs
Gluex: htgettoken -a htvault.jlab.org -i jlab -r gluex
Clas12: htgettoken -a htvault.jlab.org -i jlab -r clas12
SOLID: htgettoken -a htvault.jlab.org -i jlab -r solid
EIC: htgettoken -a htvault.jlab.org -i eic
the first time you run this command it will generate three files.
/<your home directory>.config/htgettoken/credkey-<institution>-<role>
This file is generated using the email address the token was registered to
/tmp/vt_u<your numeric user id>
This is the "vault token", it is used to refresh your bearer token without having to use a user name and password, it has a lifespan of a few days
/run/user<your numeric user id>/bt_u<your numeric user id>
This is the token that is used to authenticate with systems. Your osg job will take this token with it. These tokens have a lifespan of a few hours, but the condor software will keep that token alive while the job is active.
You can then check the token by running the httokendecode command as follows (the -H converts the date into a human readable format), see below for an example of this command:
httokendecode -H{ "aud": "ANY", "sub": "http://cilogon.org/serverA/users/287151", "ver": "scitoken:2.0", "nbf": "Mon Apr 14 03:05:28 PM EDT 2025", "scope": "read:/eic/ write:/eic/", "iss": "https://cilogon.org/eic", "exp": "Mon Apr 14 06:05:33 PM EDT 2025", "iat": "Mon Apr 14 03:05:33 PM EDT 2025", "jti": "https://cilogon.org/oauth2/38df13f9f246e88aac68ce7a12803b93?type=accessToken&ts=1744657533434&version=v2.0&lifetime=10800000", "group": "eic"}
The jlab osdf endpoint can be accessed directly through the use of the pelican command, which is installed on ifarm/farm systems and osg access points. The object subcommand is then used to interact with files. Use the "pelican object -h" to see the currently supported options for interacting with files.
Once you have successfully generated a token use the service now incident you created to inform the operations team of the token's subject (the "sub" field). Your identity will then be added to the osdf endpoint, tying your token to your jlab user name. After this is done you can test reading and writing with your token using the pelican command.
The following base urls can be used to access file systems at jlab.
osdf:///jlab-osdf/gluex
osdf:///jlab-osdf/gluex
osdf:///jlab-osdf/gluex
osdf:///jlab-osdf/clas12
Under that will be directories for volaile, cache, and work areas.
For example:pelican object ls osdf:///jlab-osdf/clas12/volatile
shows the contents of /volatile/clas12 as seen from an ifarm system
To use tokens in the OSG you first need to add the following lines to your .bashrc
export XDG_RUNTIME_DIR=/run/user/10967
export BEARER_TOKEN_FILE=/var/run/user/10967/bt_u10967
line to your submit script:
Within the job itself you need to ensure that the environment variable for the token is explicitly defined, at which point the pelican command can be invoked. Provided below is an example script that can be used. Please adjust any paths within it to the paths for your particular project
#!/bin/bash# file name: test.sh
# written by: Kurt J. Strosahl (strosahl@jlab.org)
# This is a test file for writing back to jlab-osdf
# check that the Bearer_TOKEN_FILE has the proper project (jlab_solid, jlab_gluex, jlab_clas12, eic)export BEARER_TOKEN_FILE=${_CONDOR_CREDS}/jlab_solid.use#the following is used to detect network issues between the site running the job and jlab#echo "testing connection back to jlab"#curl -v https://dtn2304.jlab.org:8443#curl -v http://dtn2304.jlab.org:8443echo $HOSTNAMEwhich pelican
#test read from the pelican endpoint/usr/bin/pelican object ls osdf://jlab-osdf/solid/writetest/echo "a file to send $HOSTNAME" > send_testAPPEND=$((RANDOM % 1000))
#test write to the pelican endpoint, update the path to the proper path. Note that the permissions on that directory need to be set properly for the user to write into
# the -d is a debug flag that can be removed to make the logs less verbose/usr/bin/pelican -d object put send_test osdf://jlab-osdf/solid/writetest/send_test$APPENDecho ${_CONDOR_CREDS}echo ${_CONDOR_SCRATCH_DIR}