Approach to store scrapped data in database (postgres)

[acanimal]

Hi, I recently discovered Crawlee and I'm trying to figure out how can I store the scraped data in database instead in local directorio storage.

Is there any plugin for that? How must I proceed to implement one? Must I code my own class that implements StorageClient interface? If so how must I injected later to be used.

Thanks!

[B4nan]

If you'd like to use postgres for all the storages, then yes, you'd have to implement new storage client with all the interface methods. We have alternative storage implementation called https://github.com/apify/apify-storage-local-js - it uses sqlite under the hood, so you could probably use that as a base for a postgres storage implementation.

In the long run, we want to simplify this interface and provide more storage implementations.

You could as well just use the memory storage for everything except datasets, and store data directly (via your own custom helper) to the postgres.

[aliraza948]

I was using postgresql by python

import asyncio,re

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.storage_clients import SqlStorageClient
from crawlee import service_locator, Request
from sqlalchemy.ext.asyncio import create_async_engine
from crawlee import ConcurrencySettings
from crawlee.router import Router
import xmltodict
router = Router[BeautifulSoupCrawlingContext]()

configuration = service_locator.get_configuration()
configuration.purge_on_start = False
page_number = 2
import json
from typing import Union, Dict
async def main() -> None:
    
   
  async with SqlStorageClient(
        # Create an `engine` with the desired configuration
        engine=create_async_engine(
            'postgresql+asyncpg://xxx:xxxx@127.0.0.1:5432/crawlee',
            future=True,
            pool_size=5,
            max_overflow=10,
            pool_recycle=3600,
            pool_pre_ping=True,
            echo=False,
        )
    ) as storage_client:
    crawler = BeautifulSoupCrawler(storage_client=storage_client,concurrency_settings=ConcurrencySettings(max_tasks_per_minute=50))
    # Define a request handler to process each crawled page
    # and attach it to the crawler using a decorator.
    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        # Extract relevant data from the page context.
        global page_number
        status=context.http_response.status_code
        context.log.info(f"Status is {status}")
        context.push_data(data)
       
      
 
    await crawler.run(['url'])


if __name__ == '__main__':
    asyncio.run(main())

That's awesome way to store in postgresql just not given example in docs of crawlee like this that how to store in postgresql

[janbuchar]

This is because the Python version of Crawlee has an SqlStorageClient implemented, but the Javascript one doesn't... yet. It's on the roadmap for Crawlee JS version 4, so stay tuned 🙂