Linux Linux Mystery: Disappearing Processes in systemd Services

[dEEpEst]

---

Linux Mystery: Disappearing Processes in systemd Services

Created for Hack Tools Dark Community

---

Disclaimer: This topic is intended for educational and debugging purposes in secure environments. Misconfiguration in service units can lead to unexpected behavior or vulnerabilities.

Issue: Vanishing Processes

You're running a daemon (e.g., mydaemon) on Ubuntu 22.04, started via a systemd unit.
systemctl status mydaemon says it’s active (running), yet:

• ps aux | grep mydaemon shows nothing
• top, htop, pgrep, pidof return no matches

But restarting it with systemctl restart mydaemon logs a clean start — no errors. Still no visible process.

---

Diagnosis

• Check the unit type:
  

  Type=simple
  ExecStart=/bin/bash -c 'sleep 9999'

• Run:
  

  systemctl show -p MainPID mydaemon

  This might return MainPID=0 or a PID that vanishes quickly.
• Why? With Type=simple, systemd assumes the parent process launched by ExecStart is the main one — in this case, bash. But bash -c 'sleep 9999' spawns sleep and exits, leaving systemd tracking a non-existent PID.

---

Solutions

1. Avoid shell wrappers:
   Instead of using bash -c, directly run the binary:
   

   ExecStart=/usr/bin/sleep 9999

2. Use Type=forking (if your daemon forks itself) and define a PIDFile:
   

   Type=forking
   PIDFile=/var/run/mydaemon.pid
   ExecStart=/usr/bin/mydaemon

3. Use Type=exec (available in systemd v240+), which tracks the last process in the chain:
   

   Type=exec
   ExecStart=/bin/bash -c 'sleep 9999'

4. Use Type=notify if the daemon supports sd_notify:
   

   Type=notify
   ExecStart=/usr/bin/mydaemon --notify-ready

---

Tips for Debugging systemd Services

• Check MainPID:
  

  systemctl show -p MainPID mydaemon

• Inspect running processes and hierarchy:
  

  ps -eo pid,ppid,cmd | grep mydaemon

• Redirect logs to a file for more clarity:
  

  StandardOutput=file:/var/log/mydaemon.log
  StandardError=file:/var/log/mydaemon.err

---

Have you encountered similar stealthy behaviors in your services? Join the discussion and share your debugging techniques or unusual systemd cases.