RTECO-1651 - Add agent skills e2e tests - #3634
Conversation
9d5f121 to
4459327
Compare
Cover jf agent skills publish/install/update/delete/list/search flows with CI wiring and dependency pins for the skills Artifactory API.
4459327 to
c300329
Compare
Drop unused publishTestSkill args/return and have waitForLocalListStatus return the matched row so callers assert RegistryLatest.
Pick up the Windows publish zip path/sort fix from RTECO-1723.
Use the main merge commit for the Windows publish zip fix and restore jfrog-cli-core to the master pin.
Expand skills e2e to cover implemented Artifactory CLI features (module/build env, signing-key, default project scope, update --path/--version, list sort/limit, repo discovery, server-id) and apply agent-plugins review lessons. Keep only the jfrog-cli-artifactory pin vs master. Apply Go KB and gosec fixes for safe temp-file handling, stronger assertions, and complete cleanup comments.
Add implementable open-plan scenarios and harden helpers for clearer assertions, shared plugins helper reuse, and idiomatic Go test style. Wire agent-skills into build-gate-success needs like other suites. Co-authored-by: Cursor <cursoragent@cursor.com>
Use a fixed filename inside the test-owned temporary directory so repository names cannot influence the config path.
Avoid the gosec WriteFile taint sink with CreateTemp, assert missing build-info locally like plugins, and publish into the target skills repo for multi-env install instead of relying on raw rt cp.
naveenku-jfrog
left a comment
There was a problem hiding this comment.
Some minor improvements comments.
Make the evidence negative-path assertions opt-in via JFROG_CLI_TEST_SKILLS_EVIDENCE_GATE instead of skipping whenever the gate does not fire, so a green run cannot hide a missing assertion. Fail the unreachable-Artifactory test against a closed loopback port rather than an unresolvable hostname, removing the dependency on CI resolver behavior and the blackhole connect timeout. Recreate the shared Skills repo when missing so a hard exit during the no-repo scenario cannot cascade into the rest of the suite, consolidate search/list JSON extraction on one helper, and align the SKILL.md fixture to 0600. Co-authored-by: Cursor <cursoragent@cursor.com>
1bf6653 to
64309db
Compare
📗 Scan Summary
|
at 🎯 Static Application Security Testing (SAST) VulnerabilityFull descriptionVulnerability Details
OverviewHardcoded credentials are usernames, passwords, API keys, or other secrets Vulnerable exampleIn this example, the database username and password for the frog pond are package main
import (
"database/sql"
"fmt"
"log"
_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)
func main() {
// VULNERABLE: Hardcoded database credentials for the frog pond.
frogUser := "pond_admin"
frogPassword := "LeapFlog123!"
pondName := "lilypad_db"
connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
frogUser, frogPassword, pondName)
lilypadDB, err := sql.Open("mysql", connStr)
if err != nil {
log.Fatalf("Error opening database: %v", err)
}
defer lilypadDB.Close()
err = lilypadDB.Ping()
if err != nil {
log.Fatalf("Error pinging database: %v", err)
}
fmt.Println("Successfully connected to the frog pond.")
}RemediationThe remediated code retrieves the database credentials from environment package main
import (
"database/sql"
"fmt"
"log"
"os"
_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)
func main() {
// SECURE: Retrieve credentials from environment variables.
frogUser := os.Getenv("FROG_DB_USER")
frogPassword := os.Getenv("FROG_DB_PASS")
pondName := os.Getenv("FROG_DB_NAME")
if frogUser == "" || frogPassword == "" || pondName == "" {
log.Fatal("DB credentials are not set in environment variables.")
}
connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
frogUser, frogPassword, pondName)
lilypadDB, err := sql.Open("mysql", connStr)
if err != nil {
log.Fatalf("Error opening database: %v", err)
}
defer lilypadDB.Close()
err = lilypadDB.Ping()
if err != nil {
log.Fatalf("Error pinging database: %v", err)
}
fmt.Println("Successfully connected to the frog pond.")
} |
at 🎯 Static Application Security Testing (SAST) VulnerabilityFull descriptionVulnerability Details
OverviewHardcoded credentials are usernames, passwords, API keys, or other secrets Vulnerable exampleIn this example, the database username and password for the frog pond are package main
import (
"database/sql"
"fmt"
"log"
_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)
func main() {
// VULNERABLE: Hardcoded database credentials for the frog pond.
frogUser := "pond_admin"
frogPassword := "LeapFlog123!"
pondName := "lilypad_db"
connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
frogUser, frogPassword, pondName)
lilypadDB, err := sql.Open("mysql", connStr)
if err != nil {
log.Fatalf("Error opening database: %v", err)
}
defer lilypadDB.Close()
err = lilypadDB.Ping()
if err != nil {
log.Fatalf("Error pinging database: %v", err)
}
fmt.Println("Successfully connected to the frog pond.")
}RemediationThe remediated code retrieves the database credentials from environment package main
import (
"database/sql"
"fmt"
"log"
"os"
_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)
func main() {
// SECURE: Retrieve credentials from environment variables.
frogUser := os.Getenv("FROG_DB_USER")
frogPassword := os.Getenv("FROG_DB_PASS")
pondName := os.Getenv("FROG_DB_NAME")
if frogUser == "" || frogPassword == "" || pondName == "" {
log.Fatal("DB credentials are not set in environment variables.")
}
connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
frogUser, frogPassword, pondName)
lilypadDB, err := sql.Open("mysql", connStr)
if err != nil {
log.Fatalf("Error opening database: %v", err)
}
defer lilypadDB.Close()
err = lilypadDB.Ping()
if err != nil {
log.Fatalf("Error pinging database: %v", err)
}
fmt.Println("Successfully connected to the frog pond.")
} |
at 🎯 Static Application Security Testing (SAST) VulnerabilityFull descriptionVulnerability Details
OverviewHardcoded credentials are usernames, passwords, API keys, or other secrets Vulnerable exampleIn this example, the database username and password for the frog pond are package main
import (
"database/sql"
"fmt"
"log"
_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)
func main() {
// VULNERABLE: Hardcoded database credentials for the frog pond.
frogUser := "pond_admin"
frogPassword := "LeapFlog123!"
pondName := "lilypad_db"
connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
frogUser, frogPassword, pondName)
lilypadDB, err := sql.Open("mysql", connStr)
if err != nil {
log.Fatalf("Error opening database: %v", err)
}
defer lilypadDB.Close()
err = lilypadDB.Ping()
if err != nil {
log.Fatalf("Error pinging database: %v", err)
}
fmt.Println("Successfully connected to the frog pond.")
}RemediationThe remediated code retrieves the database credentials from environment package main
import (
"database/sql"
"fmt"
"log"
"os"
_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)
func main() {
// SECURE: Retrieve credentials from environment variables.
frogUser := os.Getenv("FROG_DB_USER")
frogPassword := os.Getenv("FROG_DB_PASS")
pondName := os.Getenv("FROG_DB_NAME")
if frogUser == "" || frogPassword == "" || pondName == "" {
log.Fatal("DB credentials are not set in environment variables.")
}
connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
frogUser, frogPassword, pondName)
lilypadDB, err := sql.Open("mysql", connStr)
if err != nil {
log.Fatalf("Error opening database: %v", err)
}
defer lilypadDB.Close()
err = lilypadDB.Ping()
if err != nil {
log.Fatalf("Error pinging database: %v", err)
}
fmt.Println("Successfully connected to the frog pond.")
} |


Cover jf agent skills publish/install/update/delete/list/search flows with CI wiring and dependency pins for the skills Artifactory API.
masterbranch.go vet ./....go fmt ./....Summary
Adds
jf agent skillse2e coverage inagent_skills_test.go(~20 top-level suites with harness/flag subtests), wires a Skills local repo +-test.agentSkills, and hooks anagent-skillsjob into the build gate so Linux/Windows CI runs the suite.What the suite covers
--versionoverride, quiet/CI collision, prebuiltzip/{slug}_{version}.ziplayout, validation (missingSKILL.md/ name / frontmatter, invalid slug/semver, missing path), build-info module + build props--path, path vs harness mutual exclusion, unknown harness, missing skill/version, evidence quiet-failure gate (JFROG_SKILLS_DISABLE_QUIET_FAILURE),.jfrog/skill-info.json--force+ JSON summary, published-but-not-installed (summary detail), not-in-repoFileInfo), missing--version--repoJSON (Repo:source prefix), local harness list after install, empty local (no skills dir), flag matrix (neither mode / mutual exclusion / comma harness /--limit=0),--limiton repo list,--check-updatesbehind → current--propmatch, no matches, empty/blank queryJFROG_SKILLS_REPOenv, nonexistent repo (upload failed), multi-repo quiet/CI failskills-agentsinagent-config.jsonSKILL.md; quiet CI publish(+bp)→install→list→update--formatfalls back to table; unknown flags per subcommand; missing install slugNotable test improvements
waitForSkillIndexed, list/search retries) — storage upload alone is not enough for install/list/search/update/delete-dry-runassertErrorContainsAll(no loose OR matches)FileInfo(notGetItemProps, which maps “no properties” 404 → nil success)JFROG_SKILLS_DISABLE_QUIET_FAILURE(test Artifactory has no One-Model).cursor/skills/<slug>)CI / workflow
agentSkillsTests.yml: matrix ubuntu/windows/macos (macOS skipped — JGC-413),go test … --test.agentSkillsbuild-gate.yml:agent-skillscallable jobjfrog-cli-artifactory@5dfbbf6and compatiblejfrog-cli-corefor Skills API supporttestdata/skills_local_repository_config.json(packageType: skills) +testdata/agent_skills/test-skill