additional_urls field is not populated with the list of canonical URLs

[ravikumarmvs1]

Bug Description

After the crawl is performed using opencrawler, we don't seem find any field called additional_urls with the list of canonical URLs. We are very sure that the canonical URL exist on the pages.

To Reproduce

Steps to reproduce the behavior:

1. Perform crawl
2. Notice the crawled document in the index
3. additional_urls is not found in the crawled document

Expected behavior

additional_urls should list the canonical URLs in the crawled document

Screenshots

Environment

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context

[seanstory]

Hi @ravikumarmvs1 - the main url field should be what contains the canonical URL (my understanding is that there should be only 1 for a given page). The additional_urls are... additional, non-canonical, URLs that are also found for that page during the crawl.

Do you see your canonical URL in the url field?

[ravikumarmvs1]

Hi @seanstory , Thank you for the quick response.
Yes, URL field is populated appropriately. But we would like to understand how many pages are linked to the canonical version. We had used Elastic enterprise crawler earlier and that use to preserve the list of canonical URLs.

[ravikumarmvs1]

Also the same is mentioned in the documentation : https://github.com/elastic/crawler/blob/2d29f0862b93b84676583e0decc6d6a305f808a3/docs/features/CRAWLER_DIRECTIVES.md#:~:text=field%2C%20while%20the-,additional_urls,-field%20contains%20all

[seanstory]

Ah - so you're trying to find how many other pages have hyperlinks that end up on a specific page, but the Open Crawler is storing those in links to non-canonical URLs?

You should be able to get the list/count of pages pretty easily. Use a terms query, and enumerate all of the page's additional_urls in a query for matching links. For, if you had a page with cannonical URL http://acme.com/foo that could also show up as http://acme.com/foo_alt and http://acme.com/foo_other, you could find all pages that link to this page with a query like:

GET <your_index>/_search
{
  "query": {
    "terms": {
      "links": [
         "http://acme.com/foo",
        "http://acme.com/foo_alt",
        "http://acme.com/foo_other"
      ]
    }
  }
}

[ravikumarmvs1]

Ah - so you're trying to find how many other pages have hyperlinks that end up on a specific page, but the Open Crawler is storing those in links to non-canonical URLs?

You should be able to get the list/count of pages pretty easily. Use a terms query, and enumerate all of the page's additional_urls in a query for matching links. For, if you had a page with cannonical URL http://acme.com/foo that could also show up as http://acme.com/foo_alt and http://acme.com/foo_other, you could find all pages that link to this page with a query like:

GET <your_index>/_search
{
  "query": {
    "terms": {
      "links": [
         "http://acme.com/foo",
        "http://acme.com/foo_alt",
        "http://acme.com/foo_other"
      ]
    }
  }
}

Hi,

As per the documentation : The additional_urls field tracks other URLs where the duplicated content appears. But unfortunately I don't find any field named additional_urls in the crawled document.

[seanstory]

Are you certain that the crawl has found more than 1 URL for that page?
Is the site you're crawling public? If you share your crawl config, I'd be happy to try to reproduce.

[Jan-Kazlouski-elastic]

Hello @ravikumarmvs1

Fix in progress: PR #495

• url → canonical URL when page declares one
• additional_urls → alternate crawled URLs for the same page
• No separate crawl/index for the canonical URL when content is already fetched from a variant

Matches CRAWLER_DIRECTIVES.md. Also addresses sdh-search#1986.