NUMABIND vSMP Foundation Manual NAME numabind is a rule based optimal placement of processes and threads SYNOPSIS numabind --version numabind [--verbose] [--config ] numabind [--verbose] [--config ] --offset [--nooversub] SUMMARY numabind has two modes of operation: 1. "Normal" mode: Perform optimal placement for running processes based on rules provided in a configuration file. Requires privileges to set the affinity of the processes and threads matching the rules. 2. "Offset" mode: Provide, on request, a set of consecutive cpus that are the least busy, so that this cpu set can be used to perform explicit placement of an application about to be run. DESCRIPTION numabind is a tool intended to help achieve application best performance by placing processes optimally within a system aggregated using vSMP foundation. numabind analyzes the current process activity on the system and then uses that information to optimally place running processes and threads that have not been assigned any affinity. numabind will only place those processes and threads that match one of the placement rules within the configuration file. A recommended option for running numabind is periodically (e.g. once a minute) under cron. Arguments are: --version Display the current version of numabind. --verbose numabind will output considerably more information about the state of the system as numabind sees it and numabind inner processing. --config Use the specified configuration file to provide numabind with rules for processing. --offset Instruct numabind to work in "offset" mode. In this mode the normal rule processing and placement of processes does not take place. Instead, an entry named "offset" is read from the configuration file and the special "offset" rule is used to load a numabind module. The "offset" module looks for a set of consecutive cpus that fit the requested "count". The set found will be the best placement for "count" processes or threads. The number of the first cpu of the set is written to standard output so that the caller to numabind can get it, figure from which cpus are in this set and actively assign affinity within the found cpu set to processes or threads that it initiates. --nooversub An option for "offset" mode requesting only cpus that are not assigned to any process that has affinity. If no appropraite cpu set is found, an empty string is returned. Can be useful for job schedulers. CONFIGURATION The configuration file contains rules for placement - one rule per line. Each rule line contains several key=value pairs that describe the rule. Here are the supported keywords and the appropriate values: name= - A name for the rule. Recommended, but not necessary. pattern= - A regular expression to match process names to the rule. Any process that matches the regular expression will be considered a candidate for placement by the rule. Required. rule= - The name of the module (library) that contains the code that provides the knowhow for this rule. Required. numabind comes out of the box with 3 default rule modules (see 'Default Rules' below). verbose={0|1} - Indicates whether extended information about the rule processing will be provided or not. Optional, default is 0. flags= - A list of comma delimited keywords that define flags that are specific to each rule. Optional. process_allocation= - Defines the affinity unit used by the rule for entire processes. Value can be "cpu", "core", "socket", "node", "board" & "multi". Not all rules support all values. task_affinity= - Defines the affinity unit used by the rule for threads. Values can be "cpu", "core", "socket", "node" and "board". Not al rules support this keyword or all these values. NOTES Affinity units explained: cpu - a single cpu as seen by the operating system. core - a single core in the processor. If Hyperthreading is enabled, includes all threads of the core. socket - a single physical socket. Includes all cores within one physical package. node - a single NUMA node as seen by the operating system. board - a single board of this vSMP Foundation system. multi - a variable value indcating the actual number of cpus depends on the number of threads in the process and the size of the affinity unit requested for each thread. Default rules - these rules are distributed with numabind "offset" - A special rule for the "offset" mode. Does not perform any placement. Scans the system for the best (least used) set of cpus and returns that to the core of the numabind utility, which then outputs it for the benefit of the caller. Rule must be named "offset". Module name is RULE-offset.so. "throughput" - A rule for placing a single process (and all its threads) on a single affinity unit. Currently supported affinity units are cpu, node and board. Module name is RULE-throughput.so. "process-group" - A rule for placing mutiple processes and/or threads that all belong to a single process group (or a job). This rule fits most of the more complex requirements such as multi-threaded applications, OpenMP programs, various MPI and many other multi-process applications. Module name is RULE-procgroup.so. rule specific flags: ignore_idle - Do not set affinity to threads that have no (or very little) cpu time accummulated. EXAMPLES A variety of examples using numabind for dynamic placement can be found under /opt/ScaleMP/examples