swf-epicprod

epicprod — the ePIC automated production system — is the production domain of the swf platform, and this repository is its home: a peer application of swf-testbed, holding the production-specific applications and documentation.

The platform epicprod runs on has two homes, and this repository deliberately contains neither: swf-common-lib provides the importable library layer (agent base, message bus, logging), and swf-monitor provides the common monitor, web, and database services (web face, REST, MCP server, SSE relay, action-stream machinery, system configuration, alarms engine). Production code here ships as installable Django applications consumed by the swf-monitor runtime through the shared virtual-environment chain, together with the production documentation set.

Platform components are consumed through their interfaces — REST, MCP, the message bus, or an swf_common_lib import — never by importing another application's source.

The architecture map, docs/ARCHITECTURE_MAP.md, records for each platform and production component its current home, destined home, and consumption interface. Components of the production domain migrate from swf-monitor to this repository per the map, each moved document leaving a permanent stub at its old path.

PCS — Physics Configuration System

PCS, the Physics Configuration System, manages production configuration and campaign records — physics, event-generation, simulation, reconstruction, and background tags; datasets and their composed identities; campaigns and their continuum across the monthly production cadence; production requests, tasks, and configurations. PCS is where physicists meet the production system. The pcs/ Django application is installed into the swf-monitor runtime.

Contributing

Open a pull request from your fork and merge it yourself; no review is required. Before the merge, the deploy-preview check on the pull request reports which deploy the merge will trigger: none, a quick web update, or a full deploy. The merge deploys to production at once, and the deploy is followed in the repository's Actions tab, where a refused or failed deploy shows red. A payload change bumps swf_epicprod/payload/VERSION; it applies to tasks submitted after the deploy. A change to PCS models or migrations is not deployed automatically: announce it, and it is deployed by hand on approval. The mechanism is docs/AUTODEPLOY.md.

Documentation

The epicprod documentation set lives in docs/. The official system-level documentation of the ePIC WFMS is https://epic-wfms-docs.readthedocs.io.

Architecture and interfaces:

• ARCHITECTURE_MAP.md — component homes and consumption interfaces across the platform.
• API_DOCUMENTATION.md — the REST API.

PCS:

• PCS.md — the system: tags, datasets, composed identities, campaigns, production configs, REST and MCP surfaces.
• PCS_DATASET_REQUEST_WORKFLOW.md — production request intake and the dataset request workflow.
• PCS_PERMANENT_IDENTITIES.md — permanent tags/configurations, database enforcement, deployment and recovery.
• PCS_BACKGROUND_TAG.md — the background tag axis.
• PCS_COMPOSED_NAME_INTEGRITY.md — composed-name collision repair and the doors closed against it.
• PCS_COMPOSED_NAME_FAMILIES.md — the collision-family backfill review.
• COMMISSIONING_RELAXATIONS.md — PCS rules relaxed during commissioning.

Campaigns:

• CAMPAIGN_CONTINUUM.md — the campaign continuum across the monthly cadence.
• CAMPAIGN_FAMILY.md — campaign families.
• CAMPAIGN_DELIVERY.md — the delivered-data record and its views.
• STORAGE.md — the placement record: production data on every RSE, its lifecycle per RSE, and the Storage view.
• CONTINUOUS_PRODUCTION.md — the ready queue, the dispatcher, and the tripwire.
• EPIC_JOB_THROTTLER.md — the queue-side regulator: a JEDI job throttler paced per PanDA queue.
• REGULATION_AUDIT_2026-10-22.md — the October regulation audit, corrective changes and deployment evidence.
• JEV.md — Jev, a decision model: the client, the measurement on PCS's request links, and configurations like this one.
• EPICPROD_NARRATIVES.md — campaign narratives.
• EPICPROD_ASSESSMENTS.md — campaign assessments: analytics library, rollup service, scheduled runs.
• EPICPROD_ASSESSMENTS_V1.md — assessments V1 implementation plan.
• EPICPROD_DASHBOARD.md — the master production dashboard.

Production operations:

• EPICPROD_OPS.md — operations: running, restarting, and monitoring the production services.
• EPICPROD_OPS_AGENT.md — the credentialed production operations agent.
• AUTODEPLOY.md — automatic deployment: a merge to main deploys itself, the deploy chosen from the changed paths.
• EPICPROD_TASK_CATALOG.md — the production task catalog.
• EPICPROD_DATA_LINEAGE.md — produced data gathered onto the catalog; also the home of the find page (/pcs/find/), the direct dataset lookup over the recorded corpus.
• EPICPROD_EVGEN_INPUTS.md — EVGEN inputs: assimilation, matching, marks, registration.
• PCS_INGEST.md — PC ingest: physics configurations from legacy submission lines.
• EPICPROD_QUESTIONNAIRE.md — the production request questionnaire.
• EPICPROD_VALIDATION.md — the validation interface.
• EPICPROD_ALARM_PAUSE.md — alarm-driven task pause.
• EPICPROD_LLM_OPERATIONS.md — LLM operations over epicprod.
• EPICPROD_SUCCESSION.md — succession: credentials, hosts, and hand-over.
• EPICPROD_PAYLOAD.md — the epicprod payload: the in-job runner and payload implemented end to end, cloned from the production team's run script and evolved here.
• DEVELOPING_THE_PAYLOAD.md — developing the payload: new behaviour behind an EPICPROD_SWITCH_* switch, tested with payload canaries from a pull request or branch.
• EPICPROD_INTERNAL_EVGEN.md — internal EVGEN: event generation as a payload stage, the steering derived from the tags, and the first trial.
• EPICPROD_RETRIES.md — retries and reruns: the four recovery levels, what each needs to still exist, the task-state gates, and the output-name collision rule.
• RUCIO_RESILIENCE.md — preventing Rucio registration losses.
• RUCIO_FAILOVER_STASH.md — the failover stash: outputs staged at BNL while JLab is unreachable, moved and registered at JLab by the registrar.
• RUCIO_REGISTRATION_CONTRACT.md — what every Rucio registration must carry: the event count on every file, by registration path.

Submission and execution:

• JEDI_INTEGRATION.md — direct task submission from PCS to JEDI.
• JEDI_EPIC_PROPOSAL.md — the original direct-submission proposal.
• OSG_SUBMISSION.md — how production reaches the OSG pool: hosts, harvester, submit description, container layering, levers; which pilot a queue runs, and our own canary pilot directory in CVMFS with the procedure for publishing to it.
• SCHEDCONFIG_OVERRIDES.md — the PanDA server's queue records: CRIC's, with our tracked overrides on top.
• PANDA_USER_JOBS.md — user jobs and the analysis share.
• PANDA_CAPABILITIES.md — PanDA capabilities and monitoring for GPU worker jobs.
• PANDA_ANCILLARY_AUDIT.md — audit of the PanDA ancillary systems.
• NODE_EVENT_DISPATCHER.md — event-range processing in fixed-lifetime allocations.
• GKE_PILOT_FLOW.md — the production operations agent runs the pilot pods of the Google Cloud Event Service queue, in step with its backlog.
• WORK_UNIT_CONTRACT.md — the work-unit contract.
• VOLUNTEER_GPU_PLAN.md — volunteer-class GPU computing under PanDA.
• NPPS0_WORKER.md — the GPU worker host behind BNL_NPPS_GPU.
• NPPS0_TEST_QUEUE.md — BNL_NPPS_GPU as the pilot and payload test queue: choosing the pilot, the program from the canary pilot through the Event Service path to prmon in the runner, and what the queue cannot represent.
• NERSC_PERLMUTTER.md — how a pilot starts on Perlmutter, the public worker record, CVMFS on the nodes, pilot-side queue configuration by file, and production operations' wrapper on the Perlmutter test queue.
• WINDOWS_WORKER.md — the Windows worker.
• DEVCLOUD_STAGEOUT.md — the devcloud stage-out endpoint.
• JOB_REPORTING.md — how a job's own account of itself reaches the production system, including from a job that fails.
• LOG_STAGEOUT_FALLBACK.md — a pilot log whose transfer fails is held in the stage-out bucket under a per-job grant, and does not fail the job.
• DEVCLOUD_SUCCESSION.md — epic-devcloud.org succession.

Validation records and audits:

• SYNRAD_VALIDATION.md — synchrotron-radiation GPU transport validation.
• ADEPT_AUDIT.md — the AdePT audit.