VisConnect-CronJob – URL Warmup (Playwright)

Opens a list of URLs with Playwright and optionally waits for specific page elements. Useful to warm caches / trigger rendering jobs / validate that pages load.

Requirements

• Debian/Ubuntu Linux (other systems may work)
• screen (for the cron example)
• Python managed via pyenv
• Dependencies installed for Playwright browsers

Setup

Install pyenv

Run curl -fsSL https://pyenv.run | bash

In ~/.bashrc add

#pyenv
export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init - bash)"

Restart your shell (or source ~/.bashrc)

Install pipx

Run apt-get install pipx Run pipx ensurepath

Install pipenv

Run pipx install pipenv Run pipenv install in the folder

Install playwright browser + OS deps

Run pipenv shell Run playwright install Run sudo playwright install-deps

Configure via config.json

Change default behavior via config.json:

{
  "headless": true,
  "elementLocatingTimeoutInMs": 300000
}

Configure warmup urls via urls.txt

1. Rename urls.txt.example -> urls.txt
2. One URL warmup call per line.

Each line may start with an action list executed in sequence, followed by the URL. Actions let you wait for elements or skip lines based on conditions.

Skip if no progress bar appears: {notIf: "#chart_canvas", waitForShow:"#progressbar"}

Wait for progressbar to disappear: {waitForHide:"#progressbar"}

Example with both: [{notIf: "#chart_canvas", waitForShow:"#progressbar"},{waitForHide:"#progressbar"}],https://openknowledgemaps.org/search?type=get&service=orcid&orcid=[ORCID]&limit=200&academic_age_offset=0&enable_h_index=false&enable_teaching_mentorship=true

In "crontab -e" to run daily at 4am

0 4 * * * screen -dmS visconnectcronjob bash -c 'cd /home/[USER]/visconnect-cronjob && rm -f cron-log.log && /home/[USER]/.local/bin/pipenv run python3 script.py >> cron-log.log 2>&1'

Troubleshooting

Cron Headless

IMPORTANT: make sure config.json has headless set to true

Pipenv won’t start

Install pipenv via pipx (not apt): pipx install pipenv

Playwright missing system libraries (Debian/Ubuntu)

sudo apt-get install -y \
  libasound2 libedit-dev xz-utils \
  libbz2-dev libffi-dev libreadline-dev libssl-dev \
  build-essential libsqlite3-dev zlib1g-dev liblzma-dev