Skip to content

Troubleshooting

Start with the direct local controller, then work outward through the service, reverse proxy, DNS, and browser. This separates an aseStack failure from an edge or client problem.

First five checks

Run these on the controller:

sudo ase status
sudo ase health
sudo ase version
sudo ase logs-app
df -h

ase health queries the listener from ASESTACK_ADDR, normally 127.0.0.1:15173. If it succeeds, the controller is running even when the public URL is unavailable.

From another machine, test the public endpoint:

curl -fsS https://control.example.com/api/v1/health

Record which boundary first fails:

Result Likely area
Service is inactive and local health fails Controller service, runtime configuration, permissions, port conflict, or host resources.
Service is active but local health fails Wrong listener address, startup loop, or local network/socket issue.
Local health works but public health fails Reverse proxy, DNS, TLS, firewall, or proxy-to-origin routing.
Public health works but login fails Browser cookies, trusted origins, rate limit, Turnstile, MFA, passkey, or security-entrance policy.
Login works but a host action does not Controller action guard, managed-host guard, agent connectivity, or missing host dependency.

The controller service will not start

Inspect the unit and the most recent journal entries:

sudo systemctl status asestack --no-pager
sudo journalctl -u asestack -n 200 --no-pager
sudo systemctl cat asestack

Check these common causes:

  • /etc/asestack/asestack.env exists, is readable by the service setup, and does not contain a malformed assignment;
  • /opt/asestack/bin/asestack-server and /opt/asestack/apps/web/dist exist;
  • /var/lib/asestack has space and the expected service ownership;
  • the configured PostgreSQL URL is reachable when PostgreSQL is enabled; and
  • no other process owns the configured address.
sudo ss -ltnp | grep ':15173'
sudo systemctl status postgresql --no-pager

After correcting the cause, restart and query health:

sudo systemctl restart asestack
sudo ase health

If unit files, permissions, helper links, the backup timer, or local-agent wiring were damaged, run:

sudo ase repair

Repair does not replace application files, overwrite the runtime environment, or restore lost state. See Updates and the CLI.

The public URL is down but local health works

Confirm the listener and test it directly:

sudo ase health
sudo ss -ltnp | grep ':15173'

Then check the reverse proxy:

  • its upstream target reaches the controller listener from the proxy's own network namespace;
  • it preserves the original Host header;
  • the certificate covers the public hostname and is not expired;
  • the public DNS record resolves to the intended proxy; and
  • the host firewall allows the proxy's HTTP/HTTPS entrypoints.

A containerized reverse proxy cannot necessarily reach the host's 127.0.0.1. Use a host-reachable private address or the proxy's supported host gateway when the proxy is not in the host network namespace.

Check the public response without printing credentials or cookies:

curl -v --max-time 15 https://control.example.com/api/v1/health

Follow the reverse-proxy guide before changing the controller bind address. Avoid binding the control plane directly to a public interface merely to work around a proxy problem.

The reverse proxy returns 502 or 504

502 Bad Gateway usually means the proxy could not connect to the upstream. 504 Gateway Timeout usually means it connected to an unresponsive or incorrect upstream.

  1. Run sudo ase health on the controller.
  2. Test the same upstream address from the reverse-proxy host or container.
  3. Confirm the proxy uses HTTP to the default local listener unless origin TLS was explicitly configured.
  4. Compare its upstream port with ASESTACK_ADDR.
  5. Review both the proxy error log and sudo ase logs-app at the same time.

Do not point the proxy at a security-entrance URL. The upstream is the bare controller listener; the entrance is a browser-facing request path.

Login loops, forbidden requests, or missing cookies

Check the browser address and controller settings:

  • use the final HTTPS hostname, not an IP address or an old hostname;
  • keep ASESTACK_SECURE_COOKIES=true for HTTPS;
  • confirm the reverse proxy preserves Host and forwarded scheme information;
  • include only required extra origins in ASESTACK_TRUSTED_ORIGINS; and
  • make sure the browser accepts cookies for the controller.

Cookie-authenticated unsafe API requests require aseStack's CSRF header. The dashboard sends it automatically. A custom API client that logs in with cookies must send X-aseStack-CSRF: 1 and use an accepted same-host or trusted origin.

Repeated failed logins are rate-limited using ASESTACK_LOGIN_WINDOW and ASESTACK_LOGIN_MAX_FAILURES. Stop retrying, verify the credentials and challenge policy, then wait for the configured window instead of repeatedly extending the failure pattern.

The security entrance hides the login page

When entrance mode is required, requests that have not used the secret entrance return the configured decoy status. That is expected and does not mean the controller is down. Keep the entrance URL out of screenshots, tickets, and public monitoring.

If the entrance was configured only in /etc/asestack/asestack.env, an operator with console access can disable it there and restart the controller. A value saved in the Settings workspace overrides the environment fallback, so an env change may not disable a saved entrance policy. In that case, use a known-good controller backup or the project's supported recovery process; do not hand-edit PostgreSQL or the fallback state while the service is running.

After recovery, rotate the entrance code and review audit history for exposure. See Security entrance.

Turnstile verification fails

Turnstile needs two working paths: the browser loads the challenge, and the controller submits the result to the verification endpoint. A visible widget proves only the browser path.

On the controller, check DNS, time, and outbound HTTPS:

getent hosts challenges.cloudflare.com
timedatectl status
curl -sS -o /dev/null -w '%{http_code}\n' \
  https://challenges.cloudflare.com/turnstile/v0/siteverify
sudo ase logs-app

Also confirm that both ASESTACK_TURNSTILE_SITE_KEY and ASESTACK_TURNSTILE_SECRET_KEY are configured for the same widget and hostname. The journal records sanitized provider status and error codes; it should not print the token or secret.

If Turnstile is intentionally disabled, set the passkey preflight expectation to match:

sudo ASESTACK_PASSKEY_CHECK_REQUIRE_TURNSTILE=false \
  ase passkey-check https://control.example.com

TOTP or recovery-code login fails

For TOTP failures:

  • verify the controller and authenticator device clocks;
  • use the current code only once it is fully visible;
  • try one unused recovery code; and
  • remember that recovery codes are single-use and an old set becomes invalid after regeneration.
timedatectl status

If the password is lost, reset it locally:

sudo /opt/asestack/deploy/reset-admin-password.sh

The reset helper does not clear an existing user's MFA enrollment. Setting ASESTACK_REQUIRE_MFA=false also does not remove per-user MFA. If all factors and recovery codes are lost, recover from the documented, tested backup path rather than editing users.json by hand.

Passkey registration or login fails

Run the packaged preflight against the final hostname:

sudo ase passkey-check https://control.example.com

If a security entrance is active, supply its complete URL only in the private terminal invocation. Do not include it in support output. The common failures are:

Symptom Check
SecurityError or relying-party mismatch Use HTTPS and the exact hostname on which the passkey was enrolled. Passkeys do not move between hostnames.
InvalidStateError The authenticator may already hold a credential for this account and origin. Review existing passkeys in Settings.
NotAllowedError The prompt was cancelled, timed out, or refused by the browser. Retry with the manual Use Passkey control.
Turnstile error during passkey login Complete a fresh challenge and verify controller outbound access.
Conditional prompt does not appear Use the same browser profile/password manager. Manual passkey success is the required test.

The automated preflight cannot approve an authenticator. Always complete a real browser-device registration, sign-in, and deletion test as described in Passkeys.

A managed server is offline

In Servers, check the last heartbeat, enrollment status, token expiry, agent version, and most recent command result. On the managed host:

sudo systemctl status asestack-agent --no-pager
sudo journalctl -u asestack-agent -n 200 --no-pager
timedatectl status

Confirm that the host can resolve and reach the controller URL and that its client certificate is not expired. A revoked or expired enrollment must be re-created from the Servers workspace. Treat the replacement install command as a secret because it contains token and mTLS material.

For the controller's automatically enrolled local agent, these are also useful:

sudo ase doctor
sudo ase logs-agent
sudo ase repair

Follow Enroll a managed server when re-enrollment is required.

A command remains queued or is rejected

Read-only inventory and preflight commands do not grant mutation permissions. For a guarded action, check both sides:

  1. the controller queue guard is enabled;
  2. the agent host reports the matching execution guard enabled;
  3. the agent heartbeat is current;
  4. the required binary, socket, path, and permissions pass host preflight; and
  5. the latest command result in Servers contains no allowlist or validation error.

Enabling a controller flag alone permits queueing but does not make a managed host execute the command. Review the paired matrix in Guarded actions instead of enabling several flags at once.

A site deploy, certificate, or gateway fails

On the target host, verify Nginx before and after inspecting aseStack logs:

sudo nginx -t
sudo systemctl status nginx --no-pager
sudo journalctl -u asestack-agent -n 200 --no-pager

Check the action-specific requirements:

  • Nginx mutations need both the controller queue guard and the host execution guard for a selected server;
  • the configured available/enabled directories and site root must exist or be creatable by the execution path;
  • certificate issuance needs certbot, a contact email, correct DNS, and a reachable ACME webroot;
  • HTTPS deployment expects the configured certificate and key paths to exist; and
  • a gateway needs network reachability to the managed server's private origin address.

Disabling or cleaning up a site does not remove DNS-provider records. Verify DNS and proxy state separately. See Sites.

Backup or restore fails

Check the timer, archive directory, and controller log:

sudo systemctl status asestack-backup.timer --no-pager
sudo systemctl list-timers asestack-backup.timer
sudo ls -lah /var/backups/asestack
sudo ase logs-app

Verify a local archive without restoring it:

sudo /opt/asestack/deploy/backup-state.sh verify \
  /var/backups/asestack/asestack-YYYYmmddTHHMMSSZ.tar.gz

For provider transfers, confirm the saved profile tests successfully and ASESTACK_BACKUP_PROVIDER_TRANSFER_ACTIONS=true is intentional. Automatic schedules additionally require ASESTACK_BACKUP_SCHEDULE_RUNNER=true. Retention remains a dry run unless ASESTACK_BACKUP_RETENTION_DELETE_ACTIONS=true.

Restore execution is disabled unless ASESTACK_RESTORE_ACTIONS=true. Do not enable it as a troubleshooting experiment. Verify the archive, checksum, database tooling, destination paths, and maintenance window first. Follow Backups and restore.

An update fails

Check whether the running service is still healthy and inspect the backup made before replacement:

sudo ase health
sudo ase version
sudo ase status
sudo ase logs-app
sudo ls -lah /var/backups/asestack

The updater verifies the downloaded installer checksum and preserves /etc/asestack/asestack.env. A failed update does not guarantee an automatic rollback. Correct network, disk, package, or service errors before running the updater again. Use sudo ase repair only for service wiring and permissions; use a verified backup for state recovery.

See Updates and the CLI for the supported update flow.

Disk full or permission errors

Inspect the main storage areas without printing file contents:

df -h
sudo du -sh /opt/asestack /var/lib/asestack /var/backups/asestack /www
sudo systemctl status asestack asestack-agent --no-pager

Do not recursively change ownership across /etc, /var/lib, or /www as a generic fix. Those trees contain files owned by different services. Identify the exact failing path from the journal, compare it with the configured root, and repair only that path. Use sudo ase repair for the standard aseStack service paths.

Collect safe diagnostic information

Useful non-secret information includes:

sudo ase version
sudo ase health
sudo ase status
sudo journalctl -u asestack -u asestack-agent \
  --since '30 minutes ago' --no-pager
df -h

Review output before sharing it. Never attach:

  • /etc/asestack/asestack.env;
  • /etc/asestack/agent/agent.env;
  • an enrollment install command;
  • a security-entrance URL or code;
  • cookies, WebAuthn challenges, recovery codes, or session IDs;
  • provider fixture files or database URLs; or
  • the controller CA private key or backup credential key.

When reporting a problem, include the affected operation, controller and agent versions, whether direct local health works, whether the public endpoint works, the first relevant sanitized error, and the time of the attempt.