Geko [patch] Prevent graph traversal stack overflows - #117
Conversation
e8e44df to
d633697
Compare
| if visitedNodes[dependencyPath] != nil { return } | ||
| guard let graphDependency = graph.xcframeworks[dependencyPath] else { return } | ||
| let directDependencies = graph.dependencies[graphDependency] ?? [] | ||
| let transitiveDependencies = directDependencies.reduce(into: Set<GraphDependency>()) { acc, graphDependency in | ||
| if case let .xcframework(xcframework) = graphDependency { | ||
| transitiveXCFrameworkDependencies( | ||
| dependencyPath: xcframework.path, | ||
| graph: graph, | ||
| visitedNodes: &visitedNodes | ||
| guard | ||
| visitedNodes[dependencyPath] == nil, | ||
| let graphDependency = graph.xcframeworks[dependencyPath] | ||
| else { | ||
| return | ||
| } | ||
|
|
||
| let rootDependencies = Array(graph.dependencies[graphDependency] ?? []) | ||
| var activePaths = Set([dependencyPath]) | ||
| var stack = [ | ||
| XCFrameworkDependenciesFrame( | ||
| path: dependencyPath, | ||
| dependency: graphDependency, | ||
| directDependencies: rootDependencies, | ||
| nextDependencyIndex: 0, | ||
| result: Set(rootDependencies) | ||
| ), | ||
| ] | ||
|
|
||
| while !stack.isEmpty { | ||
| let frameIndex = stack.count - 1 | ||
| let frame = stack[frameIndex] | ||
|
|
||
| if frame.nextDependencyIndex < frame.directDependencies.count { | ||
| let dependency = frame.directDependencies[frame.nextDependencyIndex] | ||
| guard case let .xcframework(xcframework) = dependency else { | ||
| stack[frameIndex].nextDependencyIndex += 1 | ||
| continue | ||
| } | ||
|
|
||
| if let cachedDependencies = visitedNodes[xcframework.path]?.1 { | ||
| stack[frameIndex].nextDependencyIndex += 1 | ||
| stack[frameIndex].result.formUnion(cachedDependencies) | ||
| continue | ||
| } | ||
|
|
||
| if activePaths.contains(xcframework.path) { | ||
| stack[frameIndex].nextDependencyIndex += 1 | ||
| continue | ||
| } | ||
|
|
||
| guard let childDependency = graph.xcframeworks[xcframework.path] else { | ||
| stack[frameIndex].nextDependencyIndex += 1 | ||
| continue | ||
| } | ||
|
|
||
| let childDependencies = Array(graph.dependencies[childDependency] ?? []) | ||
| activePaths.insert(xcframework.path) | ||
| stack.append( | ||
| XCFrameworkDependenciesFrame( | ||
| path: xcframework.path, | ||
| dependency: childDependency, | ||
| directDependencies: childDependencies, | ||
| nextDependencyIndex: 0, | ||
| result: Set(childDependencies) | ||
| ) | ||
| ) | ||
| acc.formUnion(visitedNodes[xcframework.path]?.1 ?? []) | ||
| } else { | ||
| let completedFrame = stack.removeLast() | ||
| visitedNodes[completedFrame.path] = ( | ||
| completedFrame.dependency, | ||
| completedFrame.result | ||
| ) | ||
| activePaths.remove(completedFrame.path) | ||
|
|
||
| if let parentFrameIndex = stack.indices.last { | ||
| stack[parentFrameIndex].nextDependencyIndex += 1 | ||
| stack[parentFrameIndex].result.formUnion(completedFrame.result) | ||
| } | ||
| } | ||
| } | ||
| visitedNodes[dependencyPath] = (graphDependency, directDependencies.union(transitiveDependencies)) |
There was a problem hiding this comment.
Hi and thank you for contribution!
It seems to me, that it is pretty redundant to track next dependency index. There is no need to traverse dependencies in the same order. In fact, it is enough to just add dependencies into the stack as is, while keeping parent. In such case, dependencies will be traversed from last to first, which is fine. There still will be need to track that each dependency of current node is traversed, but in this case a simple boolean will be sufficient.
The code might look something like this
private func transitiveXCFrameworkDependencies(
dependencyPath: AbsolutePath,
graph: Graph,
visitedNodes: inout [AbsolutePath: (GraphDependency, Set<GraphDependency>)]
) {
if visitedNodes[dependencyPath] != nil { return }
guard let graphDependency = graph.xcframeworks[dependencyPath] else { return }
var stack: [(
dependencyPath: AbsolutePath,
graphDependency: GraphDependency,
dependencies: Set<GraphDependency>,
preOrderVisit: Bool
)] = [
(dependencyPath, graphDependency, graph.dependencies[graphDependency] ?? [], true)
]
while !stack.isEmpty {
let dependencyPath = stack[stack.count - 1].dependencyPath
let graphDependency = stack[stack.count - 1].graphDependency
let directDependencies = stack[stack.count - 1].dependencies
if stack[stack.count - 1].preOrderVisit {
guard visitedNodes[dependencyPath] == nil else {
stack.removeLast()
continue
}
stack[stack.count - 1].preOrderVisit = false
for dependency in directDependencies {
guard case let .xcframework(xcframework) = dependency else { continue }
stack.append((xcframework.path, dependency, graph.dependencies[dependency] ?? [], false))
}
} else {
visitedNodes[dependencyPath] = (graphDependency, [])
for case let .xcframework(xcframework) in directDependencies {
visitedNodes[dependencyPath]!.1.formUnion(
visitedNodes[xcframework.path]?.1 ?? []
)
}
stack.removeLast()
}
}
}The code becomes much more concise and readable. Could you please reiterate your changes using this idea?
There was a problem hiding this comment.
That sounds reasonable :) I wanted to keep the original semantics, but it's really not necessary here. I'll do it as suggested, thank you
There was a problem hiding this comment.
Done!
I decided to keep GraphAlgorithms.swift as-is, because these algorithms rely on the exact DFS order and current path for sorting and cycle detection
Description
Large valid Geko dependency graphs can crash
geko generatewith a stack overflow when graph processing code uses recursive traversal.This PR replaces the affected recursive traversals with iterative implementations that use explicit stacks:
topologicalSortandfindCycleinGekoSupportGraphTraverserModuleMapMapperStaticProductsGraphLinterSwiftModulesBuilderThe implementations preserve the existing traversal semantics, ordering, caching, collected metadata, and diagnostics where relevant, without growing the Swift call stack for every visited graph node.
The local
topologicalSortreplaces the implementation imported from TSC. Its upstream source (swiftlang/swift-tools-support-core) is deprecated, so fixing it there would not be useful for Geko.Related Issue
This is the Geko counterpart of tuist/tuist#11350, which fixes the same class of recursive graph traversal problems in Tuist.
Geko had already made its circular dependency linter stack-safe in geko-tech/geko#92, and it does not contain Tuist's
GraphCircularDetector, so those parts of the Tuist change did not need to be ported.Motivation and Context
We encountered this problem in our project while using Tuist. Removing a single dependency edge unexpectedly changed the graph traversal shape enough to produce a very deep valid acyclic graph, after which project generation started crashing with a stack overflow.
We are also evaluating and integrating Geko, which contains several of the same recursive graph-processing paths. Preventing the same failure mode here makes that integration safer for our large workspace and protects other sufficiently deep Geko projects from depending on the process call-stack limit.
Replacing recursion with explicit traversal stacks makes the affected code stack-safe without requiring changes to an otherwise valid dependency graph. During validation of the initial Geko changes, fixing one set of recursive paths exposed another stack overflow in Swift macro traversal, demonstrating how a deep graph can reach these risks one processing stage at a time.
How Has This Been Tested?
Automated tests
The following affected test suites were run:
Results:
GraphAlgorithmsTests: 6 passedGraphTraverserTests: 137 passedModuleMapMapperTests: 5 passedStaticProductsGraphLinterTests: 24 passedSwiftModulesBuilderTests: 2 passedTest environment:
arm64)The new long-chain tests exercise graph depths that previously risked overflowing the process stack. Smaller characterization tests verify that the iterative implementations preserve relevant results such as traversal order, transitive dependencies, metadata, conditions, platform propagation, caching behavior, and lint warnings.
End-to-end validation
I adapted the standalone fixture generator used for the related Tuist fix so that it can generate and validate the same large dependency graph using Geko manifests:
igooor-bb/tuist-large-dag-repro
The fixture generates a large valid DAG that is intentionally closer to a real large workspace than a single linked list: multiple projects, feature layers, shared infrastructure, bridge targets with high fan-in, and no dependency cycles.
Primary Geko validation command:
Focused-target validation path:
For more details about the fixture structure, available presets, and reproduction commands, see the fixture repository README.
Actual result:
geko generatecompletes successfully on the generated workspace;Screenshots (if appropriate):
Not applicable.
Types of changes
Checklist: