(Open)SSH configuration for Farm and QCD clusters


A security mandate now requires that you first SSH to a log-in gateway before logging in to an ifarm or qcdi host, regardless of whether you are starting on the on-site network or not (satisfying the two-factor authentication requirement without requiring it on ifarm/qcdi themselves).  You can automate that step, allowing you to "directly" ssh (but also ssh -X, scp, sftp, and rsync-over-SSH!) by configuring the ~/.ssh/config (creating the file, if necessary) on the host you are SSHing from (typically your laptop or desktop) with a ProxyJump, e.g.

# https://jlab.servicenowservices.com/kb?id=kb_article_view&sysparm_article=KB0014918
#
# Specify, in a restrictive way, the short names that should be
# assumed to be jlab.org., in order to avoid inadvertently
# applying this configuration to connections to other sites
# (or ending up repeating .jlab.org in the normalization,
# which e.g. ifarm* would do if given `ifarm.jlab.org`).
# CanonicalizeHostname wouldn't work because JLab limits
# which hostnames are published externally, and hostnames
# like ifarm and qcdi.jlab.org. are not included.
#
Host ifarm ifarm? ifarm???? qcdi qcdi??? qcdi????
  Hostname %h.jlab.org.

#
# Alternatively, if the user requests a full .jlab.org name,         
# normalize it to what is technically the absolute domain name, 
# so that the below Match-hosts don't have to be written both   
# ways to accommodate both. 
#
Host *.jlab.org
  Hostname %h.

#
# Prevent any later (e.g. wildcard) configuration from bothering (or 
# worse, creating a loop) ProxyJumping to hosts that don't require it.
# ProxyCommand is used because ProxyJump=none only became valid
# in OpenSSH 7.8, and EL7 has OpenSSH 7.4.
#
Match host login.jlab.org.,login?.jlab.org.,acclogin.jlab.org.,acclogin?.jlab.org.,scilogin.jlab.org.,scilogin?.jlab.org.,hallgw.jlab.org.,hallgw?.jlab.org.
  ProxyCommand none

#
# Now, having normalized several ways of referring to JLab hosts, we can
# apply User and ProxyJump configuration to all such references at once.
#
# `ifarm*.jlab.org.,qcdi*.jlab.org.` is appropriate for users who only
# want to apply ProxyJump to ifarm and qcdi.  Users who want to apply
# ProxyJump to more hosts can add them here (even, e.g. *.jlab.org.)
# or create another Match-host stanza to use different ProxyJumps
# for different hosts.
#
# Regardless, don't attempt a ProxyJump when SHELL is rzsh (like it is if       
# you actually log in to login.jlab.org. et al. instead of ProxyJumping         
# through them) because ProxyJump is implemented as a ProxyCommand,             
# ProxyCommand is implemented with an exec in $SHELL, and rzsh disables exec.  
#
# Also don't attempt a ProxyJump if the local and remote hostnames
# are .jlab.org and have a matching, pre-numeric alphabetic prefix,
# e.g. one ifarm???? (or qcdi) to another.
#
Match host ifarm*.jlab.org.,qcdi*.jlab.org. exec "test x$SHELL != x/bin/rzsh" exec "echo %l %h | grep -vE '^([a-z]+)[0-9]+(\.jlab\.org) \1[0-9]*\2\.'"
  ProxyJump login.jlab.org.
# Alternatively, prior to OpenSSH 7.3,
# ProxyCommand ssh -W %h:%p login.jlab.org.

#
# Separate stanza to set the same username for all jlab.org. hosts,
# regardless of ProxyJump setting.  Can be omitted completely if
# one's local username is the same as one's CUE username, e.g.
# if this configuration is used in one's CUE home directory.
#
Match host *.jlab.org.
  User your-CUE-username

When you next ssh ifarm, you will authenticate first to login.jlab.org. and then to ifarm.jlab.org..

lsh@laura-mac ~> ssh ifarm
lsh@login.jlab.org.'s password: 
(lsh@ifarm.jlab.org.) Password: 
[lsh@ifarm2401 ~]$

OpenSSH prior to version 8.5 did not prefix user@host, but authentication proceeds in the same order.

Re-using connections

To reduce the number of times you need to enter your password or perform two-factor authentication, you can add ControlMaster configuration (if you already have a Match host *.jlab.org. you can append the Control... lines to that stanza, or just have multiple Match host *.jlab.org. stanzas).

Match host *.jlab.org.
  ControlMaster auto
  ControlPath ~/.ssh/cm/%C.sock
# Use something like %l_%r@%h:%p instead of %C prior to OpenSSH 6.7.
  ControlPersist 10m
# If the directory portion of the specified ControlPath does not exist,
# you may see
#   kex_exchange_identification: Connection closed by remote host
#   Connection closed by UNKNOWN port 65535.

Be sure to create the directory you specified in ControlPath, after which your SSH connections will create and reuse sockets there.

lsh@laura-mac ~> mkdir -m 0700 ~/.ssh/cm
lsh@laura-mac ~> ssh ifarm
lsh@login.jlab.org.'s password: 
(lsh@ifarm.jlab.org.) Password: 
[lsh@ifarm2401 ~]$ logout
Shared connection to ifarm.jlab.org. closed.
lsh@laura-mac ~> ls ~/.ssh/cm
2a8ede34b2eb1d124f1aed889f8e9b825b79bd1f.sock=
65f19ccc6c191b7c3ff3617b1fe2d892d287b1a3.sock=
lsh@laura-mac ~> ssh ifarm
[lsh@ifarm2401 ~]$

See man page ssh_config(5) (ideally on the host you're SSHing from, but also available online) for more information about these directives.

Restarting a shared connection

You can have a problem, for example with X-forwarding, if you need different options for a connection but you're re-using an old one.

lsh@laura-mac ~> ssh ifarm
[lsh@ifarm2401 ~]$ logout
Shared connection to ifarm.jlab.org. closed.
lsh@laura-mac ~> ssh -X ifarm xclock
Error: Can't open display: 
lsh@laura-mac ~ [1]>

You can set new options if you first stop the old connection.

lsh@laura-mac ~ [1]> ssh -O stop ifarm
Stop listening request sent.
lsh@laura-mac ~> ssh -X ifarm xclock

Microsoft Windows

I have also written up instructions for PuTTY.  For Windows-native OpenSSH, the configuration file is placed under %userprofile% instead of ~ (e.g. C:/Users/lsh/.ssh/config -- create the .ssh directory if it doesn't already exist), and you will need to omit both

lest you see

'test' is not recognized as an internal or external command,
operable program or batch file.

and

getsockname failed: Bad file descriptor
channel_send_open: channel 0: unexpected internal error

respectively, instead of the expected result,

(c) Microsoft Corporation. All rights reserved.
J:\>ssh ifarm
Password:
Password:
[lsh@ifarm2401 ~]$

Visual Studio (VS) Code (using Microsoft's "Remote - SSH" extension)

As a prerequisite, make sure the OpenSSH configuration is working from the command prompt (whether that's Windows' cmd.exe, Linux/macOS' /bin/sh, or some other shell) as shown above.  Enable Remote.SSH: Remote Server Listen On Socket in your VSCode User Settings.  When you select Remote-SSH: Connect to Host... from the Command Palette, you should see the hosts configured in your .ssh/config already listed.

At the first password prompt, enter your two-factor PIN+OTP, and at the second, your CUE password.

GlueX have more detailed instructions for setting this up on their own wiki.