Google scholar scraper

Google Scholar Scraper

A skill for ethically collecting bibliometric data from Google Scholar, including search results, citation counts, author profiles, and related articles. Covers rate limiting, CAPTCHA avoidance, alternative APIs, legal considerations, data parsing, and practical workflows that balance data needs with responsible access.

Legal and Ethical Considerations

Before You Scrape

Google Scholar does not offer an official API, and its Terms of Service restrict automated access. Researchers must weigh their data needs against legal and ethical constraints.

Legal landscape:

Terms of Service:
  - Google's ToS prohibit automated queries
  - Violation can result in IP blocking (temporary or permanent)
  - Institutional IPs can be blocked, affecting all campus users
  - In some jurisdictions, ToS violations are not legally binding
    for non-commercial academic research, but this is debated

Ethical guidelines:
  - Minimize load: respect the server, use delays between requests
  - Cache aggressively: never request the same page twice
  - Use official alternatives first (see below)
  - Do not redistribute raw scraped data
  - Cite Google Scholar as your data source in publications
  - Consider whether your research question truly requires
    Google Scholar data, or if Web of Science, Scopus, or
    OpenAlex could answer it instead

Official and semi-official alternatives:
  - OpenAlex API: free, no key required, excellent coverage
  - OpenAlex API: free, comprehensive, well-documented
  - Crossref API: free, DOI-based metadata and citation counts
  - CORE API: free, full-text open access content
  - Google Scholar Alerts: manual but ToS-compliant monitoring
  - Publish or Perish (software): uses Google Scholar with built-in
    rate limiting, commonly used in bibliometric research

Data Collection Approaches

Using Scholarly (Python Library)

The scholarly Python library wraps Google Scholar access with built-in rate limiting and proxy support. It is the most commonly used tool for academic Google Scholar scraping.

from scholarly import scholarly, ProxyGenerator

def setup_scholarly_with_proxy():
    """
    Configure scholarly with a free proxy to reduce blocking risk.
    For heavy usage, consider ScraperAPI or similar paid services.
    """
    pg = ProxyGenerator()
    # Free proxy (less reliable, suitable for small jobs)
    pg.FreeProxies()
    scholarly.use_proxy(pg)


def search_scholar(query, max_results=20):
    """
    Search Google Scholar and collect structured results.

    IMPORTANT: Add delays between queries to avoid blocking.
    Recommended: 10-30 seconds between searches.
    """
    import time

    results = []
    search_query = scholarly.search_pubs(query)

    for i in range(max_results):
        try:
            result = next(search_query)
            parsed = {
                "title": result["bib"].get("title", ""),
                "author": result["bib"].get("author", []),
                "year": result["bib"].get("pub_year", ""),
                "venue": result["bib"].get("venue", ""),
                "abstract": result["bib"].get("abstract", ""),
                "citations": result.get("num_citations", 0),
                "url": result.get("pub_url", ""),
            }
            results.append(parsed)

            # Rate limiting: wait between result fetches
            time.sleep(2)

        except StopIteration:
            break

    return results


def get_author_profile(author_name):
    """
    Retrieve an author's Google Scholar profile.
    Includes h-index, i10-index, and publication list.
    """
    search_query = scholarly.search_author(author_name)
    author = next(search_query)
    author = scholarly.fill(author)

    profile = {
        "name": author.get("name", ""),
        "affiliation": author.get("affiliation", ""),
        "h_index": author.get("hindex", 0),
        "i10_index": author.get("i10index", 0),
        "cited_by": author.get("citedby", 0),
        "interests": author.get("interests", []),
        "publications": len(author.get("publications", [])),
    }

    return profile

Rate Limiting and Anti-Blocking

Best Practices

Rate limiting strategy:

1. Request delays:
   - Between search queries: 15-30 seconds minimum
   - Between profile lookups: 10-20 seconds
   - Between citation fetches: 5-10 seconds
   - Add random jitter: delay + random(0, 5) seconds

2. Session management:
   - Rotate user agents (maintain a list of 10+ real browser UAs)
   - Clear cookies periodically
   - Use residential proxies for large jobs (paid)
   - Limit sessions to 100-200 requests before rotating proxy

3. Caching:
   - Cache every response to disk (shelve, sqlite, or JSON)
   - Check cache before making any request
   - Set cache expiry (7-30 days for citation counts)

4. Batch scheduling:
   - Spread collection over days, not hours
   - Run during off-peak hours (late night UTC)
   - Process in batches of 50-100 queries per session

Handling CAPTCHAs and Blocks

import time
import random

def resilient_search(query, max_retries=3):
    """
    Search with exponential backoff on failures.
    When blocked, wait and retry with increasing delays.
    """
    for attempt in range(max_retries):
        try:
            results = search_scholar(query, max_results=10)
            return results
        except Exception as e:
            if "CAPTCHA" in str(e) or "429" in str(e):
                wait_time = (2 ** attempt) * 60 + random.randint(0, 30)
                print(f"Blocked. Waiting {wait_time}s before retry "
                      f"(attempt {attempt + 1}/{max_retries})")
                time.sleep(wait_time)
            else:
                raise e

    print("Max retries exceeded. Consider using a different proxy "
          "or waiting 24 hours before resuming.")
    return []

Data Processing and Storage

Structuring Collected Data

import pandas as pd
import json
from datetime import datetime

def save_results(results, output_dir, query_name):
    """
    Save scraped results in multiple formats with metadata.
    """
    # Add collection metadata
    metadata = {
        "query": query_name,
        "collected_at": datetime.now().isoformat(),
        "n_results": len(results),
        "source": "google_scholar",
    }

    # Save as JSON (preserves all structure)
    with open(f"{output_dir}/{query_name}_results.json", "w") as f:
        json.dump({"metadata": metadata, "results": results}, f, indent=2)

    # Save as CSV (for spreadsheet analysis)
    df = pd.DataFrame(results)
    df.to_csv(f"{output_dir}/{query_name}_results.csv", index=False)

    return f"Saved {len(results)} results for query: {query_name}"

Recommended Alternatives to Scraping

When Not to Scrape Google Scholar

Use these free APIs instead when possible:

OpenAlex (openalex.org):
  - Coverage: 250M+ works
  - API: REST, no key needed (polite pool with email)
  - Rate limit: 10 requests/sec (polite pool), 100K/day
  - Data: titles, abstracts, citations, authors, institutions
  - Best for: large-scale bibliometric analysis

OpenAlex (openalex.org):
  - Coverage: 250M+ works, all disciplines
  - API: REST, no key required
  - Rate limit: ~10 requests/sec polite
  - Data: titles, abstracts, citations, concepts, author profiles
  - Best for: cross-disciplinary analysis, open data research

Crossref (crossref.org):
  - Coverage: 130M+ DOIs
  - API: REST, no key needed (polite pool with email)
  - Data: metadata, reference lists, citation counts
  - Best for: DOI resolution, reference matching

Use Google Scholar scraping ONLY when:
  - You need Google Scholar-specific metrics (h-index by GS)
  - Your target papers are not indexed elsewhere
  - You need Google Scholar's ranking/relevance ordering
  - Small-scale collection (< 500 results)

Responsible data collection from Google Scholar requires balancing research needs with ethical obligations to shared infrastructure. When possible, prefer official APIs that are designed for programmatic access. When scraping is necessary, implement aggressive rate limiting, cache results, and keep total request volumes as low as your research question permits.