Skip to content

Per Domain Concurrency #1263

Closed
Closed
Per Domain Concurrency#1263
Labels
t-toolingIssues with this label are in the ownership of the tooling team.

Description

@Ehsan-U
Ehsan-U
opened on Jun 20, 2025

Does Crawlee support per-domain concurrency?

In this example, the first domain (paklap.pk) can't handle much load, but the second domain (centurycomputerpk.com) can.
Does Crawlee allow setting concurrency limits per domain, or is concurrency managed globally?

In Scrapy, this is possible through the download_slot mechanism. I’m wondering if there’s an equivalent in Crawlee.

crawler = ParselCrawler(concurrency_settings=ConcurrencySettings(
        desired_concurrency=1,
    ))

await crawler.run([Request.from_url("https://www.paklap.pk/laptops-prices.html", label="paklap_listing")])
await crawler.run([Request.from_url("https://centurycomputerpk.com/product-category/laptops", label="century_listing")])

Activity

  1. github-actions
    added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Jun 20, 2025

  2. Mantisus commented on Jun 20, 2025

    @Mantisus
    Mantisus
    on Jun 20, 2025
    Collaborator

    No, you cannot currently set different concurrency for different domains.

  3. B4nan commented on Jun 20, 2025

    @B4nan
    B4nan
    on Jun 20, 2025
    Member

    But you can have multiple crawler instances, one per each domain. You'd have to use different storage instances to isolate their contexts, not sure if we have a python example for that in the docs.

  4. apify
    locked and limited conversation to collaborators on Jun 30, 2025
  5. B4nan
    converted this issue into a discussion #1277 on Jun 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions