feat: Add robots.txt and License file

[imtherealnaska]

Which issue does this PR close?

• Closes Improve search engine / agent discoverability for DataFusion website #23258

What changes are included in this PR?

There are 2 PR's for this issue. This one contains the changes for robots.txt

• added the robots.txt , added license header and allow rules.

Are these changes tested?

Yes. Tested by building the docs and verifying the files getting generated in the build/

Are there any user-facing changes?

No.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.17%. Comparing base (e76f1af) to head (d6eeaad).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24333      +/-   ##
==========================================
- Coverage   81.17%   81.17%   -0.01%     
==========================================
  Files        1109     1109              
  Lines      388033   388033              
  Branches   388033   388033              
==========================================
- Hits       314992   314975      -17     
- Misses      54501    54514      +13     
- Partials    18540    18544       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Jefffrey]

dependent on #24332 since i think we want to include the sitemap as part of the robots.txt

[Jefffrey]

not sure if we're supposed to have this license txt in a robots.txt?

[imtherealnaska]

I added this because , '#' will be ignored by crawlers also I had to add the robots.txt file name to rat_exclude_ files.txt. Im happy to remove it , fine either way .