External Resources

Some setup work creates things outside the container: an IBM i library to build into, or a scratch SQL schema to test against. Those things outlive the container unless something deletes them. cfres is a command, available in task, test and deployment containers, that registers such a resource with the CoderFlow server. Once the container is removed, the server deletes the resource itself, over one of the environment's connections.

CoderFlow uses the same mechanism for the IBM i task library.

How it works

1. A script in the container, such as setup.sh, a deployment profile script, or a build script, creates a resource and runs cfres to register it. The registration says what the resource is, which connection to delete it over, and the commands that delete it.
2. cfres writes the registration to a folder that the server mounts into the container. The server reads it and records which container owns it.
3. When the container is removed, the server runs the delete commands on the remote system over that connection, with a timeout for each attempt, and records the result. A failed attempt is retried, up to a limit.

Stopping a task's container does not remove it, so its resources stay while the container can still be started again. A task's container is removed when the task is deleted, rejected, approved and pushed, or marked as a loser, when its stopped-container retention expires, or when an administrator removes it; see Container Lifecycle. A deployment or test container is removed seconds after its script ends. The deletion usually starts as soon as the container is removed, and at the latest when the server's cleanup sweep, which runs every 10 minutes, notices that the container is gone.

A minimal registration, in setup.sh after creating the library:

cfres register \
  --kind ibmi-library \
  --description "Scratch build library for this task" \
  --connection dev \
  --delete-command 'system -b "DLTLIB MYLIB123"' \
  --success-pattern CPF2110

That is the whole contract: no trap, no file of names to drop, and no cleanup.sh.

cfres is installed by the CoderFlow base image. If a container reports cfres: command not found, its environment image was built from an older base image: rebuild the base image and the environments that depend on it.

The connection it needs

The server deletes a resource over one of the environment's connections, using the credential that connection gave the container. The connection named with --connection must be:

• An IBM i, PostgreSQL or MySQL connection. MySQL connections also support MariaDB. SQL Server and generic SSH connections can't be used.
• Available to the container. Tasks, for a task or test container; Deploy, for a deployment container. See Available For under Connections.
• Holding the credential the deletion uses. The SSH key, from the Build or SSH feature, for --executor ssh, the default. A password and user for --executor sql; an IBM i connection also needs the SQL feature. PostgreSQL and MySQL use --executor sql.

Inside the container, CODERFLOW_RESOURCE_GRANTS lists the connections and credentials it was given, for example objx-build:ssh,dev:sql. cfres claim and cfres register refuse a connection or executor outside that list.

Name the connection the way the container knows it. The server records which connection that is when it reads the registration, so renaming the connection later does no harm, and a different connection that later takes the old name is never used.

Removing a connection does affect its resources. Once a connection is deleted, or is no longer available to the kind of container that registered against it (for example, its Tasks scope is cleared), every resource registered against it is orphaned, without retries, when its deletion comes due, and has to be deleted by hand. The same happens when the environment is deleted or the connection loses the credential the deletion uses. PostgreSQL/MySQL cleanup also refuses changes to the recorded engine, host, port, user, database or TLS settings, even if a new password is entered. Renaming the connection or rotating its password for the same target is supported. Let the resources registered against a connection be deleted before you remove it.

Commands

Command	What it does
cfres claim --kind K --connection C --delete-command CMD [options]	Records a resource you are about to create (state claimed). Prints its id.
cfres confirm <id> [--attr NAME=VALUE]... [--delete-command CMD]...	You created it: claimed → active, adding the attributes and commands.
cfres register --kind K --connection C --delete-command CMD [options]	Records a resource that already exists (state active). Prints its id.
cfres get --kind K [--state S]... [--attr NAME=VALUE]... [--attr NAME] [--json]	Prints the id of the oldest matching record. Exits 2 when nothing matches.
cfres list [--kind K] [--json]	Every record this container has written, oldest first.
cfres release <id>	Delete it now, not when the container is removed (state pending-delete).
cfres forget <id>	You deleted it yourself; CoderFlow does nothing more (state forgotten).

Claim before you create, confirm after, so a failure in between still leaves a record. A claimed resource that was never created is deleted harmlessly when a --success-pattern covers "does not exist".

Run cfres --help in a container for the full reference.

Options for claim and register

Option	Meaning
--kind K	Required. What the resource is: a lowercase slug of letters, digits and hyphens, at most 64 characters (for example ibmi-library).
--description D	What it is, for people reading the registry and the server log. Default: empty.
--connection C	Required. The external connection, by name, the server uses to delete it. It must be granted to this container.
--executor ssh|sql	How the server runs the delete commands. Default: ssh. ssh: all commands as one shell script on the IBM i over SSH (PASE). Needs the connection's SSH key (an IBM i connection with the Build or SSH feature). sql: each command is one SQL statement, run in order, stopping at the first that fails. Needs the connection's password and user (and, for IBM i, the SQL feature).
--timeout N	Seconds one deletion attempt may take, 5 to 600. Default: 120.
--attr NAME=VALUE	Records an attribute, to find the record again with cfres get. Repeatable.
--delete-command CMD	Required, repeatable: 1 to 10 commands, each at most 4 KB. A command that runs DLTLIB or DROP SCHEMA on a name beginning with Q is refused, whatever the kind. System schemas and databases are also refused; see What the server will run.
--success-pattern P	Output containing P (literal text, any case) means deleted even when the command fails. Use it for "not found". Repeatable.
--source entrypoint|script	Who registered it. Default: script. Only the CoderFlow entrypoint passes entrypoint.

A deletion succeeds when its commands exit 0, or when their output contains one of the success patterns. What prints "not found" depends on the command:

Pattern	Printed by
CPF2110	DLTLIB (through system): library not found
SQLSTATE: 42704	QShell db2, for example DROP SCHEMA: not found
SQL0204	IBM i --executor sql: object not found
3F000 or does not exist	PostgreSQL: schema does not exist
1008 or database doesn't exist	MySQL/MariaDB: database does not exist

Prefer DROP SCHEMA IF EXISTS … CASCADE for PostgreSQL and DROP DATABASE IF EXISTS … for MySQL/MariaDB. They succeed when the scratch resource is already gone, without a success pattern. Numeric codes are more reliable than message text, which can vary by server language. PostgreSQL/MySQL results and logs retain fixed SQLSTATE/errno codes only; success patterns still match the original output during the attempt.

cfres confirm takes --attr NAME=VALUE to add or replace an attribute and --delete-command CMD to add a delete command, both repeatable.

Options for get and list

Option	Meaning
--kind K	get: required; only records of this kind. list: optional.
--state S	get only. Only records in state claimed, active, pending-delete or forgotten. Repeatable. Default: claimed and active.
--attr NAME=VALUE	get only. Only records whose attribute NAME is VALUE. Repeatable.
--attr NAME	get only. Prints that attribute's value instead of the id. A record without the attribute does not match. At most one.
--json	get: prints the whole record as JSON instead of the id. list: a JSON array, instead of one tab-separated line per record (id, state, kind, attributes, description).

cfres get exits 2 when nothing matches. Under set -e, write x=$(cfres get ...) || true where nothing matching is expected.

Exit status and environment

Exit status	Meaning
0	Done, or nothing to do because there is no $CODERFLOW_RESOURCE_DIR.
1	An error: a bad option, an invalid record, a connection or credential not granted, an unknown id. The message names the offending field.
2	get: nothing matched.

Variable	Meaning
CODERFLOW_RESOURCE_DIR	Where records are written. Default: /external-resources. When it does not exist, every command does nothing, prints one line on stderr and exits 0, so a script runs unchanged where it is not mounted (a portal task, a plain docker run).
CODERFLOW_RESOURCE_GRANTS	The connection:credential pairs this container was given, for example objx-build:ssh,dev:sql. claim and register refuse a --connection or --executor outside it. The server checks again when it reads the record; this only fails early. When unset, cfres skips the check.

Examples

A PostgreSQL scratch schema

Use a fresh name for each task. This example uses scratch_42 and a PostgreSQL connection named pg-dev. The connection's configured database must be the one containing the schema.

aitool sql --input '{"connection":"pg-dev","sql":"CREATE SCHEMA scratch_42"}'
cfres register --kind sql-schema --executor sql --connection pg-dev \
  --description 'Scratch schema for task 42' --attr schema=scratch_42 \
  --delete-command 'DROP SCHEMA IF EXISTS scratch_42 CASCADE'

For scripts that may fail during creation, use cfres claim before CREATE SCHEMA, then cfres confirm after it succeeds.

A MySQL or MariaDB scratch database

Use a MySQL connection named mysql-dev, configured against a persistent database rather than the scratch database being dropped. SCHEMA and DATABASE are synonyms in MySQL; this example uses DATABASE.

aitool sql --input '{"connection":"mysql-dev","sql":"CREATE DATABASE scratch_42"}'
cfres register --kind sql-database --executor sql --connection mysql-dev \
  --description 'Scratch database for task 42' --attr database=scratch_42 \
  --delete-command 'DROP DATABASE IF EXISTS scratch_42'

The account needs permission to create and drop the scratch schema or database. The server uses this connection's own password, port and TLS settings, including certificates configured under Server Settings → CA Certificates.

An IBM i scratch SQL schema

In setup.sh; $schema holds the schema's name. The deletion runs over the connection's SQL password, and SQL0204 makes a schema that is already gone count as deleted.

id=$(cfres claim \
  --kind ibmi-library \
  --description "aitool scratch schema for this task" \
  --connection dev \
  --executor sql \
  --attr schema="$schema" \
  --delete-command "drop schema $schema cascade" \
  --success-pattern SQL0204)
# ... create schema $schema ...
cfres confirm "$id"

A library a later step in the same task reuses

In a build script that can run more than once in the same container. cfres get finds the record an earlier run made, so the library is registered once however many times the script runs.

id=$(cfres get --kind ibmi-library --attr library="$OBJX_BUILD_LIBRARY") ||
  id=$(cfres claim \
    --kind ibmi-library \
    --description "objx build library $OBJX_BUILD_LIBRARY" \
    --connection objx-build \
    --attr library="$OBJX_BUILD_LIBRARY" \
    --delete-command "system -b \"dltlib $OBJX_BUILD_LIBRARY\"" \
    --success-pattern CPF2110)
# ... crtlib $OBJX_BUILD_LIBRARY, unless it already exists ...
cfres confirm "$id"

One library kept across container restarts

This is how the CoderFlow entrypoint keeps the task library across restarts. $schema is a new schema name, used only when there is nothing to reuse. setup.sh runs on every start of the container, not just the first, so the same pattern suits anything setup.sh creates that should survive a restart.

library=$(cfres get --kind ibmi-build-library --state active --attr library) || true
if [ -n "$library" ]; then
  schema=$(cfres get --kind ibmi-build-library --state active --attr schema)
  # ... reuse $library ...
else
  id=$(cfres claim \
    --kind ibmi-build-library \
    --description "CoderFlow build library for task $TASK_ID" \
    --connection "$IBMI_BUILD_CONN" \
    --timeout 180 \
    --attr schema="$schema" \
    --delete-command "qsh -c 'db2 \"DROP SCHEMA $schema CASCADE\"'" \
    --success-pattern 'SQLSTATE: 42704')
  # ... CREATE SCHEMA $schema, then look up the library name it was given ...
  cfres confirm "$id" --attr library="$library"
fi

ibmi-build-library is the kind the entrypoint uses for the task library, and ibmi-project-build the kind IBM i project builds use. A project build given to a child or reviewer is kept past its orchestrator's container until the last of their containers is removed. Give your own resources a kind of your own, so neither takes one of them for its library.

cfres or cleanup.sh

Use cfres for anything that lives outside the container. Use cleanup.sh only for teardown inside the container, such as stopping a local stack or flushing a cache.

	cfres	cleanup.sh
Runs	On the server, after the container is removed	Inside the container, when it is sent SIGTERM as it stops
Time allowed	The --timeout you set for each attempt (default 120 seconds)	Whatever is left of the few seconds Docker allows a stopping container
When it fails	Recorded and retried, up to a limit; then logged as orphaned	Not recorded and not retried
Container stopped and restarted	The resource is kept	Runs at every stop
Container killed or crashed	The resource is still deleted after removal	Does not run
Deployment containers	Supported	Does not run when the script finishes; see Register only what is scratch
Portal tasks	Not available; see Portal tasks	Supported

When a deletion fails

The server does not retry a deletion indefinitely:

• A failed attempt is retried after a growing wait. After the n-th failed attempt the server waits min(5 minutes × 2ⁿ⁻¹, 6 hours) before trying again: 5 minutes, then 10, 20, 40 and so on, never more than 6 hours. A timeout counts as a failed attempt, as does a command that exits non-zero and prints none of your success patterns.
• An unavailable database provider is a failed attempt. If the server's aitool does not support PostgreSQL or MySQL, the attempt records that fixed reason and follows the same retries. Update CoderFlow to a release whose aitool supports PostgreSQL and MySQL, then rebuild the base image and dependent environment images.
• The number of attempts is limited. After CODERFLOW_EXTERNAL_RESOURCE_MAX_ATTEMPTS failed attempts (default 10; an integer from 1 to 1000, set by the server administrator), the resource is marked orphaned. With the defaults the attempts span about a day.
• A deletion that cannot run at all is not retried. It is orphaned on its first attempt. This happens when the environment or the connection has been deleted, when the connection is no longer available to the kind of container that registered the resource, or when the connection no longer has the credential the deletion uses.
• An orphaned resource is not tried again on its own. The server logs one error that names it, with its kind, description, owner, environment, attributes and last error. A Server Admin, or anyone with environments:manage on its environment, can retry it, or delete it by hand and forget it; see Seeing what is stuck. Write a --description and --attr values that tell whoever reads that error what to delete.

Administrators find these errors under Server Administration → Logs; see Server Logs.

Seeing what is stuck

Server Admins see every registered resource under Server Administration → External Resources, one row per resource: its kind, description and attributes, environment, owner (the task, linked to its page, or the interactive or deployment container), state, age, number of attempts and last error. The newest are first. See External Resources for the whole tab.

Environment managers, anyone with environments:manage on an environment, see that environment's records on its External Resources tab in Environment Management: the same rows without the environment, with the same state buttons and actions. The tab appears only while the environment has records; see External Resources. It reaches no other environment's records, and the records of a deleted environment are listed only in Server Administration.

The state buttons choose what is listed, each with its count:

• Not deleted — Every resource not yet deleted or forgotten. This is the default.
• Orphaned — Resources the server gave up deleting.
• Failed — Resources whose last attempt failed and that wait for their next one.
• All — Every record, including deleted and forgotten ones, until the server prunes them 30 days later.

Each row has the actions that apply to its state:

• Delete now — Runs the resource's own registered deletion at once, over the same connection and with the same timeout, logging and attempt count as the server's own attempts. It applies to any resource not yet deleted or forgotten. A resource still in use by its container is never deleted: while the container that registered it exists, even stopped, CoderFlow refuses and says so, and deletes it once the container is removed. A failure that reaches the attempt limit leaves the resource orphaned again. It asks first.
• Retry — For a failed or orphaned resource: runs the deletion at once and starts its attempt count again, so a failure is retried with the usual waits, up to the full limit. Use it once the cause is fixed, for example after ending the job that held a lock.
• Forget — CoderFlow stops tracking the resource and never deletes it, not even when its container is removed; the resource itself is left where it is. Use it after you have deleted a resource by hand, or for one that should be kept. It asks first, and is refused while a deletion attempt is running.
• Show command — Shows the deletion as you would run it by hand: the connection by its current name, and the commands. For ssh, the script the server sends, with its PATH line; for sql, one statement per line. Copy command copies it. No credential is shown.

Only one deletion runs at a time for a resource: a second Delete now or Retry while one runs waits for that one. Both wait up to 20 seconds for the attempt; a longer one goes on in the background, and Refresh shows its result. With CODERFLOW_EXTERNAL_RESOURCES=off, the tab says so and only Forget and the commands are available.

The task page shows the same records for one task. External resources, in Task Details, lists what the task's container registered, with each one's state: deleted once the container is removed, waiting to be deleted, deleted and when, failed and when it is tried next, or orphaned. It shows no commands, attributes or errors. It appears only for a task whose container registered something, and anyone who can view the task sees it. Server Admins also get Manage in Server Administration, which opens the tab. Someone else with environments:manage on the task's environment gets Manage in Environment, which opens that environment's External Resources tab.

What the server will run

A registration asks the server to run commands, so the server only runs them within limits a container cannot change:

• Never on the CoderFlow server. Delete commands run only on the remote system, over SSH or SQL. There is no way to register a command that runs on the server host.
• Only with a credential the container already had. The connection must be one the server gave this container, and the executor must use the credential it gave the container for that connection: the SSH key for ssh, the SQL password for sql. A registration naming any other connection, or a credential the container did not receive, is refused, even when the environment has that connection. Registering a resource gives a container nothing it didn't already have; it only lets the deletion run after the container is gone.
• The server decides who owns it. The owning container, its environment and its credentials come from what the server recorded when it created the container, never from the registration. A container cannot register a resource against another environment's connection.
• The connection is found by identity, not by name. A renamed connection is still used; a different connection that takes the old name is never used; a deleted connection leaves the resource orphaned rather than deleted over something else.
• System libraries, schemas and databases are refused. The existing Q-name rule rejects DLTLIB and DROP SCHEMA names beginning with Q, case-insensitively, on every connection. This also refuses PostgreSQL DROP SCHEMA qa_scratch (and MySQL DROP SCHEMA with a Q-prefixed name). The refusal may mention an “IBM i system library” because that shared rule originated with IBM i; it does not mean the connection was treated as an IBM i connection. PostgreSQL system names are pg_catalog, information_schema, public and every pg_* name. MySQL/MariaDB system names are mysql, information_schema, performance_schema and sys. The SQL executor guard reserves these names across SQL connections, case-insensitively, including quoted names and backticks. The SSH executor retains the IBM i Q-name guard.
• Database cleanup must be unambiguous. For PostgreSQL, use one simple DROP SCHEMA per --delete-command, optionally with IF EXISTS and CASCADE or RESTRICT; DROP DATABASE is always refused. For MySQL/MariaDB, use one simple DROP DATABASE or its synonym DROP SCHEMA, optionally with IF EXISTS. Names may contain letters, digits, underscores and dollar signs, starting with a letter or underscore. Comments, escaped names, multiple targets in one statement and dynamic SQL are refused. This conservative guard is a safety net, not a SQL parser.

A successful cfres register or cfres claim command is not confirmation that the server accepted the registration. The container command writes a registration file and cannot identify the database engine from its grants. For PostgreSQL/MySQL, the server checks the stricter grammar when it reads that file. For example, DROP SCHEMA scratch /* note */ CASCADE and DROP SCHEMA "my-scratch" can pass the container check but are rejected by the server. A rejected registration is not added to External Resources and will not be cleaned up automatically; the rejection appears only as a warning under Server Administration → Logs. Use the simple statements and identifiers shown above, confirm the resource appears in the task's External Resources, and register it again with a supported statement if it was rejected. If the resource's name cannot be expressed in the supported grammar, delete it manually.

Register only what is scratch

Register a resource only if you would be happy for it to disappear the moment your script finishes. If it is the deployment's output, do not register it.

A deployment container is removed seconds after its script finishes, so the server deletes what it registered straight away. That is right for a scratch library the deployment builds in, and wrong for the libraries, instances or sandbox copies the deployment exists to produce. Leave those unregistered. The same timing applies to a test container.

A deployment profile has no cleanup.sh path at all. Its container exits normally when the script finishes, so the SIGTERM handler that runs cleanup.sh never fires, and the container is then removed. For anything a deployment profile creates outside its container that should be deleted afterwards, cfres is the only option. See Deployment Profiles for writing the scripts.

Portal tasks

Portal tasks have no cfres registration. A portal task's container gets no registration folder, so cfres prints one line on stderr, records nothing and exits 0, and the server deletes nothing for that task. A portal agent cannot use the environment's IBM i credentials itself, and a registration would let it have the server use them. A portal environment's setup.sh therefore keeps using cleanup.sh for anything it creates outside the container.