Say I have a CLI app that I want to do some work (e.g. process some media files). When I run one instance of the app it takes say 20% CPU. The rest 80% is idle.
I want some sort of process manager that will run as many app instances as it takes to use nearly all the CPU up. The app knows what to do and how not to interfere with its siblings, but it can't spawn/manage itself.
Is there any manager like that? I've been using supervisor to run fixed sets of different apps one instance each, but I don't know if it can be used to run variable amount of instances of the same app depending on CPU availability.
For maximum overkill there is software such as HTCondor though such frameworks typically have a high setup cost and longer learning curve than your case may need.
A simpler but less flexible solution would be to write a small script that checks the current CPU load and from that decides whether to wait for a process to exit or to spawn one or more jobs (and then to wait for a process to exit). This sort of script is probably where things like HTCondor evolved from.
Even simpler might be to have a Makefile and then use make -j 4 ... where one knows that each "job" takes up 20% - 25% of the CPU. This assumes you know how much CPU the jobs take up, and that you know the jobs and can list them all in a Makefile. So not very flexible, unless you maybe generate the Makefile from a jobs list.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments