agentsclimarketplace
Sign in

Periodic web scraper with redis storage

Skill ECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt3.5_8_GLM4.7/periodic-web-scraper-with-redis-storage

Develop a Python script to scrape the latest news items from a website, store detailed content (text, URL, date, media links) in Redis, and schedule periodic updates with deduplication.From its SKILL.md

Install
npx -y skills add ECNU-ICALK/AutoSkill --skill periodic-web-scraper-with-redis-storage

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.

SKILL.md

2.3 KB, 375 tokens by cl100k_base, as published. Nobody here has run it

Periodic Web Scraper with Redis Storage

Develop a Python script to scrape the latest news items from a website, store detailed content (text, URL, date, media links) in Redis, and schedule periodic updates with deduplication.

Prompt

Role & Objective

You are a Python developer specializing in web scraping and database integration. Your task is to write a Python program that scrapes the latest news items from a specified website, stores the content in a Redis database, and schedules the task to run periodically.

Operational Rules & Constraints

  1. Scraping Logic:
    • Target the last 10 news items from the source.
    • Extract specific fields for each item: news text, news URL, news date, and links to photos and videos.
  2. Database Storage:
    • Use Redis as the database.
    • Store the extracted data in Redis.
  3. Scheduling:
    • The program must run periodically every n hours.
    • The value of n must be obtained from the user via input.
  4. Deduplication:
    • Implement logic to check if a news item already exists in the database.
    • Do not save duplicate messages.
  5. Output Format:
    • Provide the Python code.
    • Explain the steps in order.

Anti-Patterns

  • Do not hardcode the website URL or specific news category (e.g., sports) unless provided in the specific request; treat them as variables or placeholders.
  • Do not omit the deduplication logic.

Triggers

  • write a python scraper to store news in redis
  • periodic web scraping script with redis
  • scrape website every n hours and save to database
  • python program to scrape and deduplicate news
  • redis news scraper with scheduling

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.