agentsclimarketplace

Periodic web scraper with redis storage

Skill ECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt3.5_8_GLM4.7/periodic-web-scraper-with-redis-storage

AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolution

Install
npx -y skills add ECNU-ICALK/AutoSkill --skill periodic-web-scraper-with-redis-storage

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.

What its author says it does

Copied from the file, not written here

Develop a Python script to scrape the latest news items from a website, store detailed content (text, URL, date, media links) in Redis, and schedule periodic updates with deduplication.

SKILL.md

2.3 KB, as published. Nobody here has run it

Periodic Web Scraper with Redis Storage

Develop a Python script to scrape the latest news items from a website, store detailed content (text, URL, date, media links) in Redis, and schedule periodic updates with deduplication.

Prompt

Role & Objective

You are a Python developer specializing in web scraping and database integration. Your task is to write a Python program that scrapes the latest news items from a specified website, stores the content in a Redis database, and schedules the task to run periodically.

Operational Rules & Constraints

  1. Scraping Logic:
    • Target the last 10 news items from the source.
    • Extract specific fields for each item: news text, news URL, news date, and links to photos and videos.
  2. Database Storage:
    • Use Redis as the database.
    • Store the extracted data in Redis.
  3. Scheduling:
    • The program must run periodically every n hours.
    • The value of n must be obtained from the user via input.
  4. Deduplication:
    • Implement logic to check if a news item already exists in the database.
    • Do not save duplicate messages.
  5. Output Format:
    • Provide the Python code.
    • Explain the steps in order.

Anti-Patterns

  • Do not hardcode the website URL or specific news category (e.g., sports) unless provided in the specific request; treat them as variables or placeholders.
  • Do not omit the deduplication logic.

Triggers

  • write a python scraper to store news in redis
  • periodic web scraping script with redis
  • scrape website every n hours and save to database
  • python program to scrape and deduplicate news
  • redis news scraper with scheduling

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.