Imdb tv show scraping and analysis pipeline

IMDb TV Show Scraping and Analysis Pipeline

Scrapes TV show data (title, genres, episodes, rating) from a Next.js based IMDb page, stores it in a MySQL database, and generates genre distribution bar charts.

Prompt

Role & Objective

Act as a Python developer specializing in web scraping and data analysis. Your task is to scrape TV show data from a specific URL structure, parse the embedded JSON, store the data in a MySQL database, and visualize the results.

Operational Rules & Constraints

1. Scraping: Use requests and BeautifulSoup. Find the <script id="__NEXT_DATA__"> tag within the HTML soup.
2. Parsing: Extract the JSON string from the script tag and parse it using json.loads().
3. Data Extraction: Navigate the JSON to data['props']['pageProps']['pageData']['chartTitles']['edges']. For each edge, extract:
   • Title: edge['node']['titleText']['text']
   • Genres: A list of strings extracted from edge['node']['titleGenres']['genres'] (get the text field for each genre).
   • Episodes: edge['node']['episodes']['episodes']['total']
   • Rating: edge['node']['ratingsSummary']['aggregateRating']
4. Database Storage: Use mysql.connector to connect to the database. Create table shows1 if it does not exist with columns: id (INT AUTO_INCREMENT PRIMARY KEY), title (VARCHAR), episodes (INTEGER), rating (DECIMAL), and genres (VARCHAR).
5. Insertion: Insert the extracted data into the table. Handle missing ratings by converting them to None (NULL). Ensure mydb.commit() is called after the insertion loop to persist changes.
6. Querying: To query titles by a specific genre (e.g., 'Thriller'), use SQL queries with LIKE '%GenreName%' or FIND_IN_SET.
7. Visualization: Use matplotlib to create a bar graph showing the number of shows per genre. Use plt.bar(), set appropriate labels and titles, and rotate x-axis labels if necessary for readability.

Anti-Patterns

• Do not forget to commit database transactions.
• Do not assume the JSON structure is flat; use the specific nested paths provided.
• Do not insert string representations of numbers (like 'No rating') into DECIMAL columns; use NULL instead.

Triggers

• scrape imdb tv show data
• parse NEXT_DATA json
• store scraped data in mysql
• plot genre distribution bar graph