Skip to content

RTECO-1651 - Add agent skills e2e tests - #3634

Open
udaykb2 wants to merge 16 commits into
masterfrom
RTECO-1651-add-skills-e2e-tests
Open

RTECO-1651 - Add agent skills e2e tests#3634
udaykb2 wants to merge 16 commits into
masterfrom
RTECO-1651-add-skills-e2e-tests

Conversation

@udaykb2

@udaykb2 udaykb2 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Cover jf agent skills publish/install/update/delete/list/search flows with CI wiring and dependency pins for the skills Artifactory API.

  • All tests have passed. If this feature is not already covered by the tests, new tests have been added.
  • The pull request is targeting the master branch.
  • The code has been validated to compile successfully by running go vet ./....
  • The code has been formatted properly using go fmt ./....

Summary

Adds jf agent skills e2e coverage in agent_skills_test.go (~20 top-level suites with harness/flag subtests), wires a Skills local repo + -test.agentSkills, and hooks an agent-skills job into the build gate so Linux/Windows CI runs the suite.

What the suite covers

Area Focus
Publish Happy path + checksums, frontmatter version, --version override, quiet/CI collision, prebuilt zip/{slug}_{version}.zip layout, validation (missing SKILL.md / name / frontmatter, invalid slug/semver, missing path), build-info module + build props
Install Project harness matrix (cursor, claude-code, github-copilot, windsurf, codex, cross-agent, multi-harness), global cursor, latest via --path, path vs harness mutual exclusion, unknown harness, missing skill/version, evidence quiet-failure gate (JFROG_SKILLS_DISABLE_QUIET_FAILURE), .jfrog/skill-info.json
Update Dry-run, update to latest, --force + JSON summary, published-but-not-installed (summary detail), not-in-repo
Delete Dry-run, delete one version (storage absence via FileInfo), missing --version
List --repo JSON (Repo: source prefix), local harness list after install, empty local (no skills dir), flag matrix (neither mode / mutual exclusion / comma harness / --limit=0), --limit on repo list, --check-updates behind → current
Search Match, --prop match, no matches, empty/blank query
Repo resolution JFROG_SKILLS_REPO env, nonexistent repo (upload failed), multi-repo quiet/CI fail
Custom agent Install/list/update lifecycle under skills-agents in agent-config.json
Round-trips / CI Publish→install byte round-trip of SKILL.md; quiet CI publish(+bp)→install→list→update
Flags Unknown --format falls back to table; unknown flags per subcommand; missing install slug

Notable test improvements

  • Skills API indexing lag handled with shared backoff (waitForSkillIndexed, list/search retries) — storage upload alone is not enough for install/list/search/update/delete-dry-run
  • Stronger error assertions via assertErrorContainsAll (no loose OR matches)
  • Artifact presence/absence uses storage FileInfo (not GetItemProps, which maps “no properties” 404 → nil success)
  • Evidence gate exercised with/without JFROG_SKILLS_DISABLE_QUIET_FAILURE (test Artifactory has no One-Model)
  • Windows-safe path assertion: list JSON always uses forward slashes (.cursor/skills/<slug>)
  • Shared harness matrix mirrors skills install layout so e2e fails if agent path maps drift

CI / workflow

  • New agentSkillsTests.yml: matrix ubuntu/windows/macos (macOS skipped — JGC-413), go test … --test.agentSkills
  • build-gate.yml: agent-skills callable job
  • Pins jfrog-cli-artifactory @ 5dfbbf6 and compatible jfrog-cli-core for Skills API support
  • Fixture: testdata/skills_local_repository_config.json (packageType: skills) + testdata/agent_skills/test-skill

Cover jf agent skills publish/install/update/delete/list/search flows with CI wiring and dependency pins for the skills Artifactory API.
@udaykb2
udaykb2 force-pushed the RTECO-1651-add-skills-e2e-tests branch from 4459327 to c300329 Compare August 2, 2026 10:50
Drop unused publishTestSkill args/return and have waitForLocalListStatus return the matched row so callers assert RegistryLatest.
@udaykb2
udaykb2 requested review from a team, agrasth, bhanurp, fluxxBot, itsmeleela, naveenku-jfrog and reshmifrog and removed request for a team August 3, 2026 07:28
Pick up the Windows publish zip path/sort fix from RTECO-1723.
udaykb2 added 2 commits August 3, 2026 21:12
Use the main merge commit for the Windows publish zip fix and restore jfrog-cli-core to the master pin.
Expand skills e2e to cover implemented Artifactory CLI features (module/build env,
signing-key, default project scope, update --path/--version, list sort/limit,
repo discovery, server-id) and apply agent-plugins review lessons. Keep only the
jfrog-cli-artifactory pin vs master. Apply Go KB and gosec fixes for safe temp-file
handling, stronger assertions, and complete cleanup comments.
Add implementable open-plan scenarios and harden helpers for clearer
assertions, shared plugins helper reuse, and idiomatic Go test style.
Wire agent-skills into build-gate-success needs like other suites.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use a fixed filename inside the test-owned temporary directory so repository names cannot influence the config path.
Avoid the gosec WriteFile taint sink with CreateTemp, assert missing
build-info locally like plugins, and publish into the target skills repo
for multi-env install instead of relying on raw rt cp.

@naveenku-jfrog naveenku-jfrog left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor improvements comments.

Comment thread agent_skills_test.go Outdated
Comment thread agent_skills_test.go Outdated
Comment thread agent_skills_test.go Outdated
Comment thread agent_skills_test.go
Comment thread agent_skills_test.go
Comment thread agent_skills_test.go Outdated
Comment thread .github/workflows/agentSkillsTests.yml
Make the evidence negative-path assertions opt-in via
JFROG_CLI_TEST_SKILLS_EVIDENCE_GATE instead of skipping whenever the gate
does not fire, so a green run cannot hide a missing assertion.

Fail the unreachable-Artifactory test against a closed loopback port rather
than an unresolvable hostname, removing the dependency on CI resolver
behavior and the blackhole connect timeout.

Recreate the shared Skills repo when missing so a hard exit during the
no-repo scenario cannot cascade into the rest of the suite, consolidate
search/list JSON extraction on one helper, and align the SKILL.md fixture
to 0600.

Co-authored-by: Cursor <cursoragent@cursor.com>
@udaykb2
udaykb2 force-pushed the RTECO-1651-add-skills-e2e-tests branch from 1bf6653 to 64309db Compare August 6, 2026 07:42
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

🚨 Frogbot scanned this pull request and found the below:

📗 Scan Summary

  • Frogbot scanned for vulnerabilities and found 3 issues
Scan Category Status Security Issues
Software Composition Analysis ✅ Done Not Found
Contextual Analysis ✅ Done -
Static Application Security Testing (SAST) ✅ Done
3 Issues Found 3 Low
Secrets ✅ Done Not Found
Infrastructure as Code (IaC) ✅ Done Not Found

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
NpmLocalRepositoryConfig

at utils/tests/consts.go (line 96)

🎯 Static Application Security Testing (SAST) Vulnerability

Severity Finding
low
Low
Credentials for restricted resources included directly in source code
Full description

Vulnerability Details

Rule ID: go-hardcoded-credentials

Overview

Hardcoded credentials are usernames, passwords, API keys, or other secrets
embedded directly in source code. This practice, identified by CWE-798, is
highly insecure because it makes it easy for anyone with access to the code to
discover and misuse the credentials. If the code is publicly released, shared,
or leaked, the credentials will be exposed to unauthorized parties.

Vulnerable example

In this example, the database username and password for the frog pond are
hardcoded directly in the source code as string literals. This is a major
security risk, as anyone who can read this file can steal the credentials and
gain unauthorized access to the database.

package main

import (
	"database/sql"
	"fmt"
	"log"

	_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)

func main() {
	// VULNERABLE: Hardcoded database credentials for the frog pond.
	frogUser := "pond_admin"
	frogPassword := "LeapFlog123!"
	pondName := "lilypad_db"

	connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
		frogUser, frogPassword, pondName)

	lilypadDB, err := sql.Open("mysql", connStr)
	if err != nil {
		log.Fatalf("Error opening database: %v", err)
	}
	defer lilypadDB.Close()

	err = lilypadDB.Ping()
	if err != nil {
		log.Fatalf("Error pinging database: %v", err)
	}
	fmt.Println("Successfully connected to the frog pond.")
}

Remediation

The remediated code retrieves the database credentials from environment
variables instead of hardcoding them. This is a much more secure approach, as
it separates the secrets from the source code. This allows credentials to be
managed securely by deployment systems and rotated without changing the code.

package main

import (
	"database/sql"
	"fmt"
	"log"
	"os"

	_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)

func main() {
	// SECURE: Retrieve credentials from environment variables.
	frogUser := os.Getenv("FROG_DB_USER")
	frogPassword := os.Getenv("FROG_DB_PASS")
	pondName := os.Getenv("FROG_DB_NAME")

	if frogUser == "" || frogPassword == "" || pondName == "" {
		log.Fatal("DB credentials are not set in environment variables.")
	}

	connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
		frogUser, frogPassword, pondName)

	lilypadDB, err := sql.Open("mysql", connStr)
	if err != nil {
		log.Fatalf("Error opening database: %v", err)
	}
	defer lilypadDB.Close()

	err = lilypadDB.Ping()
	if err != nil {
		log.Fatalf("Error pinging database: %v", err)
	}
	fmt.Println("Successfully connected to the frog pond.")
}



@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
NpmLocalScopedRespositoryConfig

at utils/tests/consts.go (line 97)

🎯 Static Application Security Testing (SAST) Vulnerability

Severity Finding
low
Low
Credentials for restricted resources included directly in source code
Full description

Vulnerability Details

Rule ID: go-hardcoded-credentials

Overview

Hardcoded credentials are usernames, passwords, API keys, or other secrets
embedded directly in source code. This practice, identified by CWE-798, is
highly insecure because it makes it easy for anyone with access to the code to
discover and misuse the credentials. If the code is publicly released, shared,
or leaked, the credentials will be exposed to unauthorized parties.

Vulnerable example

In this example, the database username and password for the frog pond are
hardcoded directly in the source code as string literals. This is a major
security risk, as anyone who can read this file can steal the credentials and
gain unauthorized access to the database.

package main

import (
	"database/sql"
	"fmt"
	"log"

	_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)

func main() {
	// VULNERABLE: Hardcoded database credentials for the frog pond.
	frogUser := "pond_admin"
	frogPassword := "LeapFlog123!"
	pondName := "lilypad_db"

	connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
		frogUser, frogPassword, pondName)

	lilypadDB, err := sql.Open("mysql", connStr)
	if err != nil {
		log.Fatalf("Error opening database: %v", err)
	}
	defer lilypadDB.Close()

	err = lilypadDB.Ping()
	if err != nil {
		log.Fatalf("Error pinging database: %v", err)
	}
	fmt.Println("Successfully connected to the frog pond.")
}

Remediation

The remediated code retrieves the database credentials from environment
variables instead of hardcoding them. This is a much more secure approach, as
it separates the secrets from the source code. This allows credentials to be
managed securely by deployment systems and rotated without changing the code.

package main

import (
	"database/sql"
	"fmt"
	"log"
	"os"

	_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)

func main() {
	// SECURE: Retrieve credentials from environment variables.
	frogUser := os.Getenv("FROG_DB_USER")
	frogPassword := os.Getenv("FROG_DB_PASS")
	pondName := os.Getenv("FROG_DB_NAME")

	if frogUser == "" || frogPassword == "" || pondName == "" {
		log.Fatal("DB credentials are not set in environment variables.")
	}

	connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
		frogUser, frogPassword, pondName)

	lilypadDB, err := sql.Open("mysql", connStr)
	if err != nil {
		log.Fatalf("Error opening database: %v", err)
	}
	defer lilypadDB.Close()

	err = lilypadDB.Ping()
	if err != nil {
		log.Fatalf("Error pinging database: %v", err)
	}
	fmt.Println("Successfully connected to the frog pond.")
}



@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
NpmRemoteRepositoryConfig

at utils/tests/consts.go (line 98)

🎯 Static Application Security Testing (SAST) Vulnerability

Severity Finding
low
Low
Credentials for restricted resources included directly in source code
Full description

Vulnerability Details

Rule ID: go-hardcoded-credentials

Overview

Hardcoded credentials are usernames, passwords, API keys, or other secrets
embedded directly in source code. This practice, identified by CWE-798, is
highly insecure because it makes it easy for anyone with access to the code to
discover and misuse the credentials. If the code is publicly released, shared,
or leaked, the credentials will be exposed to unauthorized parties.

Vulnerable example

In this example, the database username and password for the frog pond are
hardcoded directly in the source code as string literals. This is a major
security risk, as anyone who can read this file can steal the credentials and
gain unauthorized access to the database.

package main

import (
	"database/sql"
	"fmt"
	"log"

	_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)

func main() {
	// VULNERABLE: Hardcoded database credentials for the frog pond.
	frogUser := "pond_admin"
	frogPassword := "LeapFlog123!"
	pondName := "lilypad_db"

	connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
		frogUser, frogPassword, pondName)

	lilypadDB, err := sql.Open("mysql", connStr)
	if err != nil {
		log.Fatalf("Error opening database: %v", err)
	}
	defer lilypadDB.Close()

	err = lilypadDB.Ping()
	if err != nil {
		log.Fatalf("Error pinging database: %v", err)
	}
	fmt.Println("Successfully connected to the frog pond.")
}

Remediation

The remediated code retrieves the database credentials from environment
variables instead of hardcoding them. This is a much more secure approach, as
it separates the secrets from the source code. This allows credentials to be
managed securely by deployment systems and rotated without changing the code.

package main

import (
	"database/sql"
	"fmt"
	"log"
	"os"

	_ "[github.com/go-sql-driver/mysql](https://github.com/go-sql-driver/mysql)"
)

func main() {
	// SECURE: Retrieve credentials from environment variables.
	frogUser := os.Getenv("FROG_DB_USER")
	frogPassword := os.Getenv("FROG_DB_PASS")
	pondName := os.Getenv("FROG_DB_NAME")

	if frogUser == "" || frogPassword == "" || pondName == "" {
		log.Fatal("DB credentials are not set in environment variables.")
	}

	connStr := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/%s",
		frogUser, frogPassword, pondName)

	lilypadDB, err := sql.Open("mysql", connStr)
	if err != nil {
		log.Fatalf("Error opening database: %v", err)
	}
	defer lilypadDB.Close()

	err = lilypadDB.Ping()
	if err != nil {
		log.Fatalf("Error pinging database: %v", err)
	}
	fmt.Println("Successfully connected to the frog pond.")
}



Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants