Introducing Caddy-SSH

Mohammed Al-Sahaf

Prologue

Much of the Internet infrastructure runs on software written in memory unsafe programming languages. Two independent studies by Microsoft and Google present the same figure of ~70% of bugs being rooted in memory safety issues. The ISRG estimates ~80% of the vulnerabilities exploited in the wild are memory safety bugs. The existing tools to save unsafe languages are bandaids, and we ought to do better by spending an ounce of prevention. The Internet and the collective community should not panic for another Heartbleed. The concern is grave enough for the ISRG to commission the Prossimo project “to move the Internet’s security-sensitive software infrastructure to memory safe code” 1. Prossimo’s current initiatives are concerned with Rusttls, the Linux kernel, curl, Apache mod_tls, NTP, and DNS.

There is lots of work left to do for the rest of us. There are plethora of supportive and functional software that runs the Internet. The myriad of web servers, ssh servers, OpenSSL, gpg, and the various VPN services and others are all serving users for critical needs. The efforts are underway to build modern and memory safe alternatives of legacy infrastructure. Caddy, age, Tailscale, and Algo are leading the way on some of those fronts.

Modern software should not only be flexible and modular, but also have sane, safe, and secure defaults. For instance, the HTTP server of Caddy is HTTPS by default, even for localhost. It also does not enable nor support TLS 1.1 or lower, and Caddy only enables the ciphers TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 by default, which are all part of the secure and recommended cipher suite list. Any other ciphersuite outside of the recommended list must be willfully and manually enabled by the user, but it is not part of the default configuration that is intended to be secure by default.

It is with those concerns that I built the ssh server app atop Caddy. Modular, memory safe, and secure by default: choose all three.

Et Voila: Caddy-SSH

Reveal: Caddy-SSH

Few days before the Caddy v2.0.0 release in 2020, I mused with the rest of the Caddy team about how hard or easy it will be to have a Caddy ssh app akin to its http app. Talk is cheap, so I wielded my editor to build the proof-of-concept.

Sure enough, it was easy to whip up a rough prototype within few days by standing on the shoulders of the giants: Caddy and github.com/gliderlabs/ssh. This is when Matt Holt, of Caddy, sent out the spoiler tweet.

At that time, the ssh server app was far from complete, but we know it is possible. I spent the next ~2 years working on the project, squeezing the time between $DAYJOB, school, and life to get it to the MVP stage. This is where we are now. The project currently resides on GitHub at mohammed90/caddy-ssh and you can also find it listed on the Caddy server website.

Users can currently login using ssh keys or password, backed by either the operating system (requires cgo) or via URLs referenced in the config (e.g. referring to https://github.com/<username>.keys). The authentication providers are Caddy modules, so they can be whatever you want as long as they adhere to the defined interface.

The default module for hostkey generation is called fallback, which load for any existing hostkeys but will generate only RSA 4096 and ed25519 if either is absent. The source of the signers is modular, which means it can source the hostkey anyway it sees fit, which may be Hashicorp Vault or some microservice endpoint.

An authorization mechanism exists, which is also modular and can take action based on any criteria, whether based on internal state (e.g. count of currently active sessions), user metadata, or other session-related data. The authorization criteria can be chained using the ssh.session.authorizers.chained module, which runs the authorizers as middlewares applying the authorization and deauthorization logic in FIFO. Users can spawn shells and tunnels and are subject to modular authorization mechanisms as well.

Being a Caddy app and inspired much by Caddy’s design, the connection configuration and actions (analogous to HTTP handlers) are all executed and applied based on the matchers defined along them.

What’s Next

Windows Support: The Windows support is lagging behind due to difficulties with the PTY management on Windows. The OS-backed password authentication was implemented, thanks to Justen Walker (jwalk) on the Gopher Slack for guidance, but the spawned sessions are always bound by the user running the server. This was initially implemented with dependency on the winpty project, which successfully spawns a session but bound by the current user running the ssh server. Later attempt to use ConPTY failed due to Windows security context and permissions which require me to deep-dive into the Windows sandbox models. At this point, I left the Windows parts WIP until the spawning of sessions using logged-in user context is implemented properly.

Config Adapter: The sshd_config adapter is theoretically possible, which will enable the caddy-ssh app to load the current sshd config and take over. This makes the transition smoother for most users. After some minor work on it I realized it requires a proper parser to effectively manage the match config scopes. I have pulled this part into a PR for a future visit.

Multi-Factor Authentication: This is blocked by an upstream issue in x/crypto/ssh. The workaround implemented by SFTPGo requires forking x/crypto/ssh, which I am not in favor of. It is best if the implementation is upstreamed to benefit the ecosystem instead. The other workaround requires an abuse of the keyboard-interactive authentication method, but it will only work with passwords. Another workaround is to rotate the passwords over time in similar manner to the second-factor TOTP that is often used with Google Authenticator, Microsoft Authenticator, or you favorite password manager app. I have implemented this to test the waters. You can find the working implementation in the linked PR.

Host-key Rotation: OpenSSH 6.8 (2015) was released with support for automatic hostkey rotation. I’m not aware of any blockers except for writing the code. I admit not having looked at the intricacies of how this works yet.

Rich Connection Context: The way x/crypto/ssh is implemented now does not expose the username at the time of applying the server configuration for an incoming connection. This hinders some of the advanced functions provided by OpenSSH, e.g. match on username to decide on custom hostkeys. Figuring this out requires deep-dive into the OpenSSH code as well as x/crypto/ssh.

You: Modular, open-source software lives and dies by its ecosystem. I have tried to build a good foundation for other modules to be developed. Feedback from the community is most valuable at this point to steward the project forward. File issues and feature requests, send pull requests, and discuss the design. Once we have a solid foundation, we can extract the proper bits outside the internal/ package for consumers.

For now, Matt can mark one of those as done :)

  1. https://www.memorysafety.org/about/ ↩︎