Refine the «Break-Glass»

upgrading disaster recovery mechanism

May 23, 2025

Evgenii Frikin

Exprience: from IaaS to SaaS

R&D: Kubernetes, OpenStack

I like: Fedora, GitLab, Go, C

Hobby: long-distance running

Denis Makrushin

Mission: save the world and building cybersecurity solutions

R&D: combating ransomware at MEPhI

Hobby: researching threats, hunting vulns, and sharing insights

What is «Break-Glass»

Break-Glass is …

Break-glass access refers to a procedure used in critical emergencies or exceptional cases, when a user with insufficient access is granted elevated access rights to bypass normal access controls. – SSH Academy (ssh.com)

Usage

Emergency:

PAM/IAM¹ do not work

Server is down
(DDoS or MFA doesn’t work)

Failure of Federation Server

Service outage

Classification

Emergency Access Account
(MS Azure, AWS, IAM)

Temporary Tokens/Keys
(HashiCorp Vault)

Temporary Roles

MFA Unlock

Approval-Based Access
(ServiceNow)

Comparison Criteria

Access Provision Method	Security Level	Implementation Complexity	Ease of Use	Flexebility	Audit Transperency	Abuse Risk
Emergency account	Medium	Low	High	Low	Medium	High
Temporary Token/Keys	High	Medium	High	High	High	Low
Temporary Roles	High	Medium	Medium	High	High	Low
MFA Unlock	Very High	High	Medium	Medium	High	Low
Approval-Based Access	High	High	Low	High	Very High	Very Low

Vulnerabilities

Weak access control

Inadequate monitoring

Single Point of Failure (SPOF)

Security policies in Multi-Cloud environments

SPOF: When PAM tools fail

This vulnerability involved improper management of SSH keys within PAM systems, which hackers exploited to gain access to cryptographic key pairs and enterprise servers. The failure to secure SSH keys effectively demonstrates how mismanagement can lead to significant security breaches – CVE-2024-31497

ManageEngine PAM360 Vulnerability: A specific vulnerability in ManageEngine’s PAM360 allowed attackers to bypass authentication checks, granting them unauthorized access to sensitive operations within the application. This incident illustrates how flaws in PAM tools can create serious security risks if not addressed promptly – CVE-2024-30051

Background

Privileged Access Management Architecture

SSH Architectrure

Problem Statements

Challenges in Managing SSH keys

Monitoring of the usage of the keys has revealed that typically about 90% of the authorized keys are unused. That is, they are access credentials that were provisioned years ago, the need for which ceased to exist or the person having the private key left, and the authorized key was never deprovisioned. Thus, the access was not terminated when the need for it ceased to exist.
...
In many organizations – even very security-conscious organizations – there are many times more obsolete authorized keys than they have employees. Worse, authorized keys generally grant command-line shell access, which in itself is often considered privileged. We have found that in many organizations about 10% of the authorized keys grant root or administrator access. SSH keys never expire. – Tatu Ylonen

Challenges in Managing SSH keys

Historically, most organizations have not touched the location of the authorized keys files. This means they are in each user’s home directory, and each user can configure additional permanent credentials for themselves and their friends. They can also add additional permanent credentials for any service account or root account they are able to log into. This has lead to massive problems in large organizations around managing SSH keys.
...
Enterprises should also pay attention to the AuthorizedKeysCommand and AuthorizedKeysCommandUser options. They are typically used when SELinux is enabled and to fetch SSH keys from LDAP directories or other data sources. Their use can make auditing SSH keys cumbersome and they can be used to hide backdoor keys from casual observation. – SSH Academy (ssh.com)

SSH keys Limitation

The problems with public key authentication are caused by key management – Teleport Blog

Limited metadata
Managing unused keys
Duplicate authorized keys
End users manipulating keys freely
Rotating/revoking trusted public keys
Tracking of keys usage/modifications
Scalability challenge related to keys

Figure 1: Trojan meme

Requirements

Main Points

Keyless/Passwordless

Based on well-known protocols

Independent on PAM platform

Server-side authorization

Independent on shared account(s) (Accountless)

Simple integration to exist infrastructure

Short-lived access

Critical component(s) rotation

Main Points

Server-side authorization
Independent on shared account(s) (Accountless)

Analysis

SSH Stack Protocols

SSH User Authentication: for client authentication to the server

SSH Connection: for multiplexing the encrypted tunnel into logical channel

SSH Transport: for authentication, confidentiality, integrity, compression

SSH Authentication Requests

If the requested ‘user name’ does not exist, the server MAY disconnect, or MAY send a bogus list of acceptable authentication ‘method name’ values, but never accept any. This makes it possible for the server to avoid disclosing information on which accounts exist. In any case, if the ‘user name’ does not exist, the authentication request MUST NOT be accepted.

Conversation between SSH and NSS¹

Pluggable Authentication Module

UsePAM Enables the Pluggable Authentication Module interface. If set to yes this will enable PAM authentication using KbdInteractiveAuthentication and PasswordAuthentication in addition to PAM account and session module processing for all authentication types.

Because PAM keyboard-interactive authentication usually serves an equivalent role to password authentication, you should disable either PasswordAuthentication or KbdInteractiveAuthentication.

Conversation between sshd, NSS and PAM¹

Figure 2

Figure 3

Pitfalls

getpwnam function

The getpwnam() function returns a pointer to a structure containing the broken-out fields of the record in the password database (e.g., the local password file /etc/passwd, NIS, and LDAP) that matches the username name.

struct passwd {
    char   *pw_name;       /* username */
    char   *pw_passwd;     /* user password */
    uid_t   pw_uid;        /* user ID */
    gid_t   pw_gid;        /* group ID */
    char   *pw_gecos;      /* user information */
    char   *pw_dir;        /* home directory */
    char   *pw_shell;      /* shell program */
;

enum nss_status_nss_getpwnam_r (const char *name, struct passwd *p, char *buffer,
                         size_t buflen, int *errnop) {
...
};

NSS Archtecture

Figure 4

$ id test123321
uid=61100(test123321) gid=61100(test123321) groups=61100(test123321)

$ id test567765
uid=60599(test567765) gid=60599(test567765) groups=60599(test567765)

$ id test890098
uid=61053(test890098) gid=61053(test890098) groups=61053(test890098)

$ getent passwd root
root:x:0:0:SuperUser:/root:/bin/bash

$ getent passwd test890098
test890098:!:61053:61053:tmpuser:/home/test890098:/bin/bash

systemd.exec

SYSTEMD_EXEC_PID
The PID of the unit process (e.g. process invoked by ExecStart=). The child process can use this information to determine whether the process is directly invoked by the service manager or indirectly as a child of another process by comparing this value with the current PID (similarly to the scheme used in sd_listen_fds(3) with $LISTEN_PID and $LISTEN_FDS).
Added in version 248.

Getting Service Name

Get SYSTEMD_EXEC_PID value
Get process name from /proc/PID/comm

/proc/pid/comm (since Linux 2.6.33)
This file exposes the process’s comm value-that is, the command name associated with the process.

In order to limit number of allowing username It’s necessary to use prefix or postfix

Figure 5

Getting Authentication Information

SSH_AUTH_INFO_0: expose authentication information to PAM. The environment variable is versioned.

Type: ssh-ed25519-cert-v01@openssh.com user certificate
Public key: ED25519-CERT SHA256:XoLe…
Signing CA: ED25519 SHA256:2whaY… (using ssh-ed25519)
Key ID: “ssh_v1:dev:users”
Serial: 1
Valid: from 2024-12-25T19:06:42 to 2024-12-25T20:07:12
Principals:
test.brkgl2s
Critical Options: (none)
Extensions:
permit-pty

For more details please read: openssh-portable/auth-pam.c

Naming Convention

resource_type:environment:sudo group

ssh_v1 dev users

The Key ID is logged by the SSH server and can be useful for auditing

High-Level Design

Known Limitations

Main Points

only ed25519 pubkey type is supported
user is assigned unique UID/GID each time
GID is not defined in /etc/groups
break-glass user is ephemeral
termination all the process related to the user is not implemented
If SSH server sends SSH_MSG_USERAUTH_FAILURE
(e.g. during authentification) user record is not deleted