What killed my process and why?

Question

My application runs as a background process on Linux. It is currently started at the command line in a Terminal window. Recently a user was executing the application for a while and it died mysteriously. The text: > Killed was on the terminal. This happened two times. I asked if someone at a different Terminal used the kill command to kill the process? No. Under what conditions would Linux decide to kill my process? I believe the shell displayed "killed" because the process died after receiving the kill(9) signal. If Linux sent the kill signal should there be a message in a system log somewhere that explains why it was killed?

Answer 1

Background process, hey?

Well that eliminates a user inadvertently typing ctrl-C at keyboard ...

But was it also run with nohup ??

If this is a long-running process that's performing some critical / intensive processing, you might want to run it with nohup (which ignores the hangup signal generated by the terminal going away as well.

$ nohup process param1 param2 &

[1] 9512

nohup: ignoring input and appending output to 'nohup.out'

Then you can review the latest output that the process produced ... if any, before exiting.

I did a bit of research on spontaneous "killing" of user processes by the kernel ... it doesn't happen very often. But when it does, it's typically attributed to critical resource exhaustion.

I don't think I've ever dealt with that sort of thing over decades of Linux/UNIX app experience.

But you can "grep" through (or look through the following system log files for references to your program name (or references to "kill"):

1. /var/log/kern.log
2. /var/log/dmesg

You may need to do that as root/sudo (get the sys admin involved).