Scroll to navigation

scrun(1) Slurm Commands scrun(1)

NAME

scrun - an OCI runtime proxy for Slurm.

SYNOPSIS

Create Operation

scrun [GLOBAL OPTIONS...] create [CREATE OPTIONS] <container-id>

Prepares a new container with container-id in current working directory.

Start Operation

scrun [GLOBAL OPTIONS...] start <container-id>

Request to start and run container in job.

Query State Operation

scrun [GLOBAL OPTIONS...] state <container-id>

Output OCI defined JSON state of container.

Kill Operation

scrun [GLOBAL OPTIONS...] kill <container-id> [signal]

Send signal (default: SIGTERM) to container.

Delete Operation

scrun [GLOBAL OPTIONS...] delete [DELETE OPTIONS] <container-id>

Release any resources held by container locally and remotely.

Perform OCI runtime operations against container-id per:
https://github.com/opencontainers/runtime-spec/blob/main/runtime.md

scrun attempts to mimic the commandline behavior as closely as possible to crun(1) and runc(1) in order to maintain in place replacement compatibility with DOCKER(1) and podman(1). All commandline arguments for crun(1) and runc(1) will be accepted for compatibility but may be ignored depending on their applicability.

DESCRIPTION

scrun is an OCI runtime proxy for Slurm. scrun will accept all commands as an OCI compliant runtime but will instead proxy the container and all STDIO to Slurm for scheduling and execution. The containers will be executed remotely on Slurm compute nodes according to settings in oci.conf(5).

scrun requires all containers to be OCI image compliant per:
https://github.com/opencontainers/image-spec/blob/main/spec.md

RETURN VALUE

On successful operation, scrun will return 0. For any other condition scrun will return any non-zero number to denote a error.

GLOBAL OPTIONS

Ignored.

Activate debug level logging.

Use specified slurm.conf for configuration.
Default: sysconfdir from configure during compilation

Show quick help on how to call scrun

Optional select format for logging. May be "json" or "text".
Default: text

Path to spool directory to communication sockets and temporary directories and files. This should be a tmpfs and should be cleared on reboot.
Default: /run/user/{user_id}/scrun/

Ignored. All scrun commands are always rootless.

Ignored.

Increase logging verbosity. Multiple -v's increase verbosity.

Print version information and exit.

CREATE OPTIONS

Path to the root of the bundle directory.
Default: caller's working directory

Optional path to an AF_UNIX socket which will receive a file descriptor referencing the master end of the console's pseudoterminal.
Default: ignored

Ignored.

Ignored.

Specify the file to lock and populate with process ID.
Default: ignored

Ignored.

DELETE OPTIONS

Ignored. All delete requests are forced and will kill any running jobs.

INPUT ENVIRONMENT VARIABLES

Set logging level.

Set logging level for standard error output only.

Set logging level for syslogging only.

Set logging level for log file only.

JOB INPUT ENVIRONMENT VARIABLES

See SLURM_ACCOUNT from srun(1).

See SLURM_ACCTG_FREQ from srun(1).

See SLURM_BURST_BUFFER from srun(1).

See SLURM_CLUSTER_CONSTRAINT from srun(1).

See SLURM_CLUSTERS from srun(1).

See SLURM_CONSTRAINT from srun(1).

See SLURM_ACCOUNT from srun(1).

See SLURM_CPU_BIND from srun(1).

See SLURM_CPU_FREQ_REQ from srun(1).

See SLURM_CPUS_PER_GPU from srun(1).

See SRUN_CPUS_PER_TASK from srun(1).

See SLURM_DELAY_BOOT from srun(1).

See SLURM_DEPENDENCY from srun(1).

See SLURM_DISTRIBUTION from srun(1).

See SLURM_EPILOG from srun(1).

See SLURM_EXACT from srun(1).

See SLURM_EXCLUSIVE from srun(1).

See SLURM_GPU_BIND from srun(1).

See SLURM_GPU_FREQ from srun(1).

See SLURM_GPUS from srun(1).

See SLURM_GPUS_PER_NODE from srun(1).

See SLURM_GPUS_PER_SOCKET from salloc(1).

See SLURM_GPUS_PER_TASK from srun(1).

See SLURM_GRES_FLAGS from srun(1).

See SLURM_GRES from srun(1).

See SLURM_HIST from srun(1).

See SLURM_JOB_NAME from srun(1).

See SLURM_JOB_NODELIST from srun(1).

See SLURM_JOB_NUM_NODES from srun(1).

See SLURM_LABELIO from srun(1).

See SLURM_MEM_BIND from srun(1).

See SLURM_MEM_PER_CPU from srun(1).

See SLURM_MEM_PER_GPU from srun(1).

See SLURM_MEM_PER_NODE from srun(1).

See SLURM_MPI_TYPE from srun(1).

See SLURM_NCORES_PER_SOCKET from srun(1).

See SLURM_NETWORK from srun(1).

See SLURM_NSOCKETS_PER_NODE from srun(1).

See SLURM_NTASKS from srun(1).

See SLURM_NTASKS_PER_CORE from srun(1).

See SLURM_NTASKS_PER_GPU from srun(1).

See SLURM_NTASKS_PER_NODE from srun(1).

See SLURM_NTASKS_PER_TRES from srun(1).

See SLURM_MODE from srun(1).

See SLURM_OVERCOMMIT from srun(1).

See SLURM_OVERLAP from srun(1).

See SLURM_PARTITION from srun(1).

See SLURM_POWER from srun(1).

See SLURM_PROFILE from srun(1).

See SLURM_PROLOG from srun(1).

See SLURM_QOS from srun(1).

See SLURM_REMOTE_CWD from srun(1).

See SLURM_REQ_SWITCH from srun(1).

See SLURM_RESERVATION from srun(1).

See SLURM_SIGNAL from srun(1).

See SLURMD_DEBUG from srun(1).

See SLURM_SPREAD_JOB from srun(1).

See SLURM_TASK_EPILOG from srun(1).

See SLURM_TASK_PROLOG from srun(1).

See SLURM_THREAD_SPEC from srun(1).

See SLURM_THREADS_PER_CORE from srun(1).

See SLURM_THREADS from srun(1).

See SLURM_TIMELIMIT from srun(1).

See SLURM_TRES_PER_TASK from srun(1).

See SLURM_UNBUFFEREDIO from srun(1).

See SLURM_USE_MIN_NODES from srun(1).

See SLURM_WAIT4SWITCH from srun(1).

See SLURM_WCKEY from srun(1).

See SLURM_WORKING_DIR from srun(1).

OUTPUT ENVIRONMENT VARIABLES

Advertised version of OCI compliance of container.

Value based as container_id during create operation.

PID of process used to monitor and control container on allocation node.

Path to container bundle directory.

Path to container bundle directory before modification by Lua script.

List of annotations from container's config.json.

Path to pid file that is locked and populated with PID of scrun.

Path to control socket for scrun.

Path to workspace for all temporary files for current container. Purged by deletion operation.

Path to container's config.json file at time of submission.

Name of user that called create operation.

Numeric ID of user that called create operation.

Name of user's primary group that called create operation.

Numeric ID of user primary group that called create operation.

See --root.

Path to container's root directory.

Path to container's root directory at submission time.

Path to scrun's log file during create operation.

Log format type during create operation.

JOB OUTPUT ENVIRONMENT VARIABLES

For a heterogeneous job allocation, the environment variables are set separately for each component.

Name of the cluster on which the job is executing.

OCI Bundle for job.

OCI id for job.

Number of CPUs requested per allocated GPU.

Number of CPUs requested per task.

Plane distribution size. Only set for plane distributions.

Distribution type for the allocated jobs.

Requested binding of tasks to GPU.

Requested GPU frequency.

Number of GPUs requested.

Requested GPU count per allocated node.

Requested GPU count per allocated socket.

Requested GPU count per allocated task.

Set to count of components in heterogeneous job.

Account name associated of the job allocation.

Count of CPUs available to the job on the nodes in the allocation, using the format CPU_count[(xnumber_of_nodes)][,CPU_count [(xnumber_of_nodes)] ...]. For example: SLURM_JOB_CPUS_PER_NODE='72(x2),36' indicates that on the first and second nodes (as listed by SLURM_JOB_NODELIST) the allocation has 72 CPUs, while the third node has 36 CPUs. NOTE: The select/linear plugin allocates entire nodes to jobs, so the value indicates the total count of CPUs on allocated nodes. The select/cons_res and select/cons_tres plugins allocate individual CPUs to jobs, so this number indicates the number of CPUs allocated to the job.

The UNIX timestamp for a job's projected end time.

The global GPU IDs of the GPUs allocated to this job. The GPU IDs are not relative to any device cgroup, even if devices are constrained with task/cgroup. Only set in batch and interactive jobs.

The ID of the job allocation.

List of nodes allocated to the job.

Total number of nodes in the job allocation.

Name of the partition in which the job is running.

Quality Of Service (QOS) of the job allocation.

Advanced reservation containing the job allocation, if any.

UNIX timestamp for a job's start time.

Bind tasks to memory.

Set to bit mask used for memory binding.

Set to "prefer" if the SLURM_MEM_BIND option includes the prefer option.

Sort free cache pages (run zonesort on Intel KNL nodes)

Set to the memory binding type specified with the SLURM_MEM_BIND option. Possible values are "none", "rank", "map_map", "mask_mem" and "local".

Set to "verbose" if the SLURM_MEM_BIND option includes the verbose option. Set to "quiet" otherwise.

Minimum memory required per usable allocated CPU.

Requested memory per allocated GPU.

Specify the real memory required per node.

Sets of node name, communication address and hostname for nodes allocated to the job from the cloud. Each element in the set if colon separated and each set is comma separated. For example: SLURM_NODE_ALIASES=ec0:1.2.3.4:foo,ec1:1.2.3.5:bar

Specify the number of tasks to run.

Request the maximum ntasks be invoked on each core.

Request that there are ntasks tasks invoked for every GPU.

Request that ntasks be invoked on each node.

Request the maximum ntasks be invoked on each socket.

Overcommit resources.

Enables detailed data collection by the acct_gather_profile plugin.

Number of GPU Shards available to the step on this node.

The hostname of the computer from which scrun was invoked.

Number of tasks to be initiated on each node. Values are comma separated and in the same order as SLURM_JOB_NODELIST. If two or more consecutive nodes are to have the same task count, that count is followed by "(x#)" where "#" is the repetition count. For example, "SLURM_TASKS_PER_NODE=2(x3),1" indicates that the first three nodes will each execute two tasks and the fourth node will execute one task.

This is only set if --threads-per-core or SCRUN_THREADS_PER_CORE were specified. The value will be set to the value specified by --threads-per-core or SCRUN_THREADS_PER_CORE. This is used by subsequent srun calls within the job allocation.

SCRUN.LUA

/etc/slurm/scrun.lua must be present on any node where scrun will be invoked. scrun.lua must be a compliant lua(1) script.

Required functions

The following functions must be defined.

• function slurm_scrun_stage_in(id, bundle, spool_dir, config_file, job_id, user_id, group_id, job_env)
Called right after job allocation to stage container into job node(s). Must return SLURM.success or job will be cancelled. It is required that function will prepare the container for execution on job node(s) as required to run as configured in oci.conf(1). The function may block as long as required until container has been fully prepared (up to the job's max wall time).
Container ID
OCI bundle path
Temporary working directory for container
Path to config.json for container
jobid of job allocation
Resolved numeric user id of job allocation. It is generally expected that the lua script will be executed inside of a user namespace running under the root(0) user.
Resolved numeric group id of job allocation. It is generally expected that the lua script will be executed inside of a user namespace running under the root(0) group.
Table with each entry of Key=Value or Value of each environment variable of the job.

• function slurm_scrun_stage_out(id, bundle, orig_bundle, root_path, orig_root_path, spool_dir, config_file, jobid, user_id, group_id)
Called right after container step completes to stage out files from job nodes. Must return SLURM.success or job will be cancelled. It is required that function will pull back any changes and cleanup the container on job node(s). The function may block as long as required until container has been fully prepared (up to the job's max wall time).

Container ID
OCI bundle path
Originally submitted OCI bundle path before modification by set_bundle_path().
Path to directory root of container contents.
Original path to directory root of container contents before modification by set_root_path().
Temporary working directory for container
Path to config.json for container
jobid of job allocation
Resolved numeric user id of job allocation. It is generally expected that the lua script will be executed inside of a user namespace running under the root(0) user.
Resolved numeric group id of job allocation. It is generally expected that the lua script will be executed inside of a user namespace running under the root(0) group.

Provided functions

The following functions are provided for any Lua function to call as needed.

• slurm.set_bundle_path(PATH)
Called to notify scrun to use PATH as new OCI container bundle path. Depending on the filesystem layout, cloning the container bundle may be required to allow execution on job nodes.

• slurm.set_root_path(PATH)
Called to notify scrun to use PATH as new container root filesystem path. Depending on the filesystem layout, cloning the container bundle may be required to allow execution on job nodes. Script must also update #/root/path in config.json when changing root path.

• STATUS,OUTPUT = slurm.remote_command(SCRIPT)
Run SCRIPT in new job step on all job nodes. Returns numeric job status as STATUS and job stdio as OUTPUT. Blocks until SCRIPT exits.

• STATUS,OUTPUT = slurm.allocator_command(SCRIPT)
Run SCRIPT as forked child process of scrun. Returns numeric job status as STATUS and job stdio as OUTPUT. Blocks until SCRIPT exits.

• slurm.log(MSG, LEVEL)
Log MSG at log LEVEL. Valid range of values for LEVEL is [0, 4].

• slurm.error(MSG)
Log error MSG.

• slurm.log_error(MSG)
Log error MSG.

• slurm.log_info(MSG)
Log MSG at log level INFO.

• slurm.log_verbose(MSG)
Log MSG at log level VERBOSE.

• slurm.log_verbose(MSG)
Log MSG at log level VERBOSE.

• slurm.log_debug(MSG)
Log MSG at log level DEBUG.

• slurm.log_debug2(MSG)
Log MSG at log level DEBUG2.

• slurm.log_debug3(MSG)
Log MSG at log level DEBUG3.

• slurm.log_debug4(MSG)
Log MSG at log level DEBUG4.

• MINUTES = slurm.time_str2mins(TIME_STRING)
Parse TIME_STRING into number of minutes as MINUTES. Valid formats:
• days-[hours[:minutes[:seconds]]]
• hours:minutes:seconds
• minutes[:seconds]
• -1
• INFINITE
• UNLIMITED

Example scrun.lua scripts

function slurm_scrun_stage_in(id, bundle, spool_dir, config_file, job_id, user_id, group_id, job_env)
	return slurm.SUCCESS
end
function slurm_scrun_stage_out(id, bundle, orig_bundle, root_path, orig_root_path, spool_dir, config_file, jobid, user_id, group_id)
	return slurm.SUCCESS
end
return slurm.SUCCESS

This is full example that will stage container as given by docker(1) or podman(1). Container's config.json is modified to remove unwanted functions that may cause container run to under crun(1) or crun(1). Script uses rsync to move container to an shared /home filesystem. 

local json = require 'json'
local open = io.open
local function read_file(path)
	local file = open(path, "rb")
	if not file then return nil end
	local content = file:read "*all"
	file:close()
	return content
end
local function write_file(path, contents)
	local file = open(path, "wb")
	if not file then return nil end
	file:write(contents)
	file:close()
	return
end
function slurm_scrun_stage_in(id, bundle, spool_dir, config_file, job_id, user_id, group_id, job_env)
	slurm.log_debug(string.format("stage_in(%s, %s, %s, %s, %d, %d, %d)",
		       id, bundle, spool_dir, config_file, job_id, user_id, group_id))
	local status, output, user, rc
	local config = json.decode(read_file(config_file))
	local src_rootfs = config["root"]["path"]
	rc, user = slurm.allocator_command(string.format("id -un %d", user_id))
	user = string.gsub(user, "%s+", "")
	local root = "/home/"..user.."/containers/"
	local dst_bundle = root.."/"..id.."/"
	local dst_config = root.."/"..id.."/config.json"
	local dst_rootfs = root.."/"..id.."/rootfs/"
	if string.sub(src_rootfs, 1, 1) ~= "/"
	then
		-- always use absolute path
		src_rootfs = string.format("%s/%s", bundle, src_rootfs)
	end
	status, output = slurm.allocator_command("mkdir -p "..dst_rootfs)
	if (status ~= 0)
	then
		slurm.log_info(string.format("mkdir(%s) failed %u: %s",
			       dst_rootfs, status, output))
		return slurm.ERROR
	end
	status, output = slurm.allocator_command(string.format("/usr/bin/env rsync --exclude sys --exclude proc --numeric-ids --delete-after --ignore-errors --stats -a -- %s/ %s/", src_rootfs, dst_rootfs))
	if (status ~= 0)
	then
		-- rsync can fail due to permissions which may not matter
		slurm.log_info(string.format("WARNING: rsync failed: %s", output))
	end
	slurm.set_bundle_path(dst_bundle)
	slurm.set_root_path(dst_rootfs)
	config["root"]["path"] = dst_rootfs
	-- Always force user namespace support in container or runc will reject
	if ((config["process"] ~= nil) and (config["process"]["user"] ~= nil))
	then
		-- purge additionalGids as they are not supported in rootless
		config["process"]["user"]["additionalGids"] = nil
	end
	if (config["linux"] ~= nil)
	then
		-- force user namespace to always be defined for rootless mode
		local found = false
		if (config["linux"]["namespaces"] == nil)
		then
			config["linux"]["namespaces"] = {}
		else
			for _, namespace in ipairs(config["linux"]["namespaces"]) do
				if (namespace["type"] == "user")
				then
					found=true
					break
				end
			end
		end
		if (found == false)
		then
			table.insert(config["linux"]["namespaces"], {type= "user"})
		end
		-- clear all attempts to map uid/gids
		config["linux"]["uidMappings"] = nil
		config["linux"]["gidMappings"] = nil
		-- disable trying to use a specific cgroup
		config["linux"]["cgroupsPath"] = nil
	end
	if (config["mounts"] ~= nil)
	then
		-- Find and remove any user/group settings in mounts
		for _, mount in ipairs(config["mounts"]) do
			local opts = {}
			if (mount["options"] ~= nil)
			then
				for _, opt in ipairs(mount["options"]) do
					if ((string.sub(opt, 1, 4) ~= "gid=") and (string.sub(opt, 1, 4) ~= "uid="))
					then
						table.insert(opts, opt)
					end
				end
			end
			mount["options"] = opts
		end
		-- Remove all bind mounts by copying files into rootfs
		local mounts = {}
		for i, mount in ipairs(config["mounts"]) do
			if ((mount["type"] ~= nil) and (mount["type"] == "bind") and (string.sub(mount["source"], 1, 4) ~= "/sys") and (string.sub(mount["source"], 1, 5) ~= "/proc"))
			then
				status, output = slurm.allocator_command(string.format("/usr/bin/env rsync --numeric-ids --ignore-errors --stats -a -- %s %s", mount["source"], dst_rootfs..mount["destination"]))
				if (status ~= 0)
				then
					-- rsync can fail due to permissions which may not matter
					slurm.log_info("rsync failed")
				end
			else
				table.insert(mounts, mount)
			end
		end
		config["mounts"] = mounts
	end
	-- Merge in Job environment into container
	if (config["process"]["env"] == nil)
	then
		config["process"]["env"] = {}
	end
	for _, env in ipairs(job_env) do
		table.insert(config["process"]["env"], env)
	end
	-- Remove all prestart hooks to squash any networking attempts
	if ((config["hooks"] ~= nil) and (config["hooks"]["prestart"] ~= nil))
	then
		config["hooks"]["prestart"] = nil
	end
	-- Remove all rlimits
	if ((config["process"] ~= nil) and (config["process"]["rlimits"] ~= nil))
	then
		config["process"]["rlimits"] = nil
	end
	write_file(dst_config, json.encode(config))
	slurm.log_info("created: "..dst_config)
	return slurm.SUCCESS
end
function slurm_scrun_stage_out(id, bundle, orig_bundle, root_path, orig_root_path, spool_dir, config_file, jobid, user_id, group_id)
	slurm.log_debug(string.format("stage_out(%s, %s, %s, %s, %s, %s, %s, %d, %d, %d)",
		       id, bundle, orig_bundle, root_path, orig_root_path, spool_dir, config_file, jobid, user_id, group_id))
	if (bundle == orig_bundle)
	then
		slurm.log_info(string.format("skipping stage_out as bundle=orig_bundle=%s", bundle))
		return slurm.SUCCESS
	end
	status, output = slurm.allocator_command(string.format("/usr/bin/env rsync --numeric-ids --delete-after --ignore-errors --stats -a -- %s/ %s/", root_path, orig_root_path))
	if (status ~= 0)
	then
		-- rsync can fail due to permissions which may not matter
		slurm.log_info("rsync failed")
	end
	return slurm.SUCCESS
end
slurm.log_info("initialized scrun.lua")
return slurm.SUCCESS

SIGNALS

When scrun receives SIGINT, it will attempt to gracefully cancel any related jobs (if any) and cleanup.

COPYING

Copyright (C) 2023 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

SEE ALSO

Slurm(1), oci.conf(5), srun(1), crun(1), runc(1), DOCKER(1) and podman(1)

Slurm Commands February 2023