Skip to content

Drop world-sized polygons from fabric sources so they cannot stall tippecanoe - #719

Merged
iandees merged 1 commit into
masterfrom
fabric-drop-oversized-polygons
Oct 25, 2026
Merged

iandees merged 1 commit into
masterfrom
fabric-drop-oversized-polygons

Conversation

@iandees

@iandees iandees commented Oct 24, 2026

Copy link
Copy Markdown
Member

The parcels fabric job that started 2026-10-19 (job 151a65fd, with the overview/detail split from #716) did not hit the std::bad_alloc any more, but it failed on its 3 day timeout. Tiling took 64 hours: the overview pass took 6.3 hours, and in the detail pass 15 of 16 shards finished in 2.5 to 4.4 hours while shard 5 took 57.4 hours (07:13 on 10-20 to 16:38 on 10-22). It crept from 86.7% to 100% with only 0.8GB of memory in use. The join then ran out of the last 5 hours and was killed.

Shard 5 contains us/wa/statewide. That file has two parcels (pid "043-", near Spokane) whose outer ring is the whole-world rectangle (-180,-90 to 180,90) with the real parcel as a hole in it, which looks like a bad conversion upstream. At z13 and z14 tippecanoe has to emit a filled tile for every tile on Earth for each of them. I reproduced this locally with the same detail options on that one file:

input detail pass
us/wa/statewide, 3,322,111 features still at 87.3% after 20+ minutes, advancing about 0.1% a minute (stopped)
the same file minus those two features 133 seconds

Everything else in that file is ordinary: the next largest polygon bounding box is about 1.2 square degrees, and the mean is 22 vertices per feature.

This adds task/lib/extent.js and drops any Polygon or MultiPolygon whose bounding box is wider or taller than 5 degrees while a source is being fetched in get_source, logging a not ok - dropped N polygon(s) ... line with the job and source so we can see which sources have these. Points and lines are never dropped (a long centerline is legitimate and only costs the tiles along its path). A polygon crossing the antimeridian has a longitude span near 360 and is dropped too.

Run over the whole us/wa/statewide file, the check drops exactly those two features out of 3,322,111 and takes 12 seconds including parsing. Unit tests cover the world-with-hole shape, ordinary and large parcels, MultiPolygons, and non-polygons.

Not checked: the other 1,374 parcels sources and the other layers. The 15 other parcels shards finished normally, so none of them has one of these, and addresses, buildings and centerlines completed their last runs. A source that gets new broken geometry later would now log the drop instead of stalling the job.

The two bad parcels are probably worth reporting to whoever maintains the Washington source.

@iandees
iandees merged commit 7c73000 into master Oct 25, 2026
3 checks passed
@iandees
iandees deleted the fabric-drop-oversized-polygons branch October 25, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant