Web automation workflow discovery

Web Automation Workflow Discovery

Learn how a human performs a task on a website, then build a robust automation script for it.

Workflow Overview

Discovery -> Exploration -> Element Mapping -> Script Build -> Test & Refine -> Package

Phase 1: Discovery

Gather the essential context before touching any code:

1. Target website URL — The entry point
2. Human workflow — Step-by-step what a human does (click, type, upload, etc.)
3. Input data — What files/data does the process consume?
4. Success criteria — How do you know the task completed successfully?
5. Frequency — One-time, daily, or triggered by events?
6. Security constraints — Should credentials be saved? Can others use this?

If the user has not provided these, ask before proceeding.

Phase 2: Exploration

Use Playwright to explore the website automatically. Do NOT rely on the user to manually click through for you.

2.1 Open and screenshot

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False, slow_mo=100)
    page = browser.new_page(viewport={"width": 1600, "height": 1000})
    page.goto("https://target-site.com/path")
    page.screenshot(path="screenshot_01_entry.png", full_page=True)
    with open("page_01_entry.html", "w", encoding="utf-8") as f:
        f.write(page.content())

2.2 Auto-navigate through the workflow

Write a small exploration script that clicks likely buttons and saves a screenshot at each step:

def explore_step(page, action_name):
    """Execute next action and save state."""
    page.screenshot(path=f"explore_{action_name}.png", full_page=True)
    with open(f"explore_{action_name}.html", "w", encoding="utf-8") as f:
        f.write(page.content())

Use keyword-based selectors to find elements (text content, placeholder, etc.) rather than fragile CSS paths.

2.3 Analyze screenshots and HTML

For each saved screenshot and HTML file:

• Identify the page type: login, dashboard, form, modal, grid/list, upload
• Note any loading states, overlays, or AJAX patterns
• Record the exact text of buttons/links you need to interact with

Key page types to recognize:

Page Type	Characteristics	Handling Strategy
SPA (Single Page App)	URL doesn't change, content replaced dynamically	Use wait_until="domcontentloaded", detect AJAX completion
Login Modal	#modal-login, form inside overlay	Fill inputs inside modal, wait for modal to disappear
Grid/Table (yiiGridView, DataTables)	table, pagination, filter inputs	Use Enter key for filtering, wait for .grid-view-loading to disappear
File Upload (plupload, Dropzone)	Hidden input[type="file"], progress bars	set_input_files() on hidden input, look for <a class="plupload_start"> not <button>
Popup Window	window.open()	Use page.expect_popup()

Phase 3: Element Mapping

Build a selector reference table. Test each selector with page.locator(sel).count() to confirm it resolves to exactly the expected number of elements.

Element                | Selector                                    | Count Expected
-----------------------|---------------------------------------------|---------------
Username input         | input[name="LoginForm[user]"]               | 1
Password input         | input[name="LoginForm[pwd]"]                | 1
Login button           | button:has-text("Login")                    | 1
Held Shipments menu    | a:has-text("Held Shipments")                | 1
Tracking filter        | #task_grid_view input#ImParcel_ref          | 1
View link              | a.tracking-modal-link                       | >=0
Upload Document link   | a:has-text("Upload Document")               | >=0
File input (plupload)  | input[type="file"]                          | 1
Start Upload button    | a.plupload_start:not(.plupload_disabled)    | 1

Selector priority (most to least reliable):

1. id or stable data-testid
2. input[name="exact_name"]
3. Text-based: button:has-text("Exact Text"), a:has-text("Partial")
4. Scoped selector: #parent_id input#child_id (prevents matching wrong elements)
5. CSS class (only if unique and semantic): .tracking-modal-link
6. XPath (last resort)

Phase 4: Script Build

Build the automation script incrementally. Do not write the entire script at once.

4.1 Structure the script

def main():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False, slow_mo=120)
        page = browser.new_page(viewport={"width": 1600, "height": 1000})

        # Step 1: Navigate to entry page
        # Step 2: Login (if needed)
        # Step 3: Navigate to target page
        # Step 4: Loop through input data
        # Step 5: For each item: search -> open -> interact -> verify
        # Step 6: Completion dialog
        browser.close()

4.2 SPA navigation — prefer direct goto over clicks

For Single Page Applications, clicking dashboard menu items is unreliable because AJAX timing is unpredictable. Instead:

# Good: direct navigation
page.goto("https://site.com/shipment/heldList.app",
          timeout=120000, wait_until="domcontentloaded")

# Avoid: clicking SPA menu buttons
# page.click("a:has-text('Held Shipments')")  # Unreliable

4.3 Wait strategies

# For AJAX grid updates
def wait_for_grid_update(page, timeout=30):
    start = time.time()
    while time.time() - start < timeout:
        loading = page.locator("#loading-container").is_visible()
        grid_loading = page.locator(".grid-view-loading").count() > 0
        if not loading and not grid_loading:
            time.sleep(0.8)
            return True
        time.sleep(0.3)
    return False

# For modal content loading
modal.wait_for(state="visible", timeout=15000)

4.4 Login detection with dual criteria

SPA logins often replace content without redirecting. Use multiple signals:

logged_in = False
while time.time() - start < 600:
    try:
        login_form_gone = page.locator('input[name="LoginForm[user]"]').count() == 0
        has_welcome = page.locator("text=Welcome").count() > 0
        if login_form_gone or has_welcome:
            logged_in = True
            break
    except Exception:
        time.sleep(0.5)
        continue
    time.sleep(1)

4.5 Exception handling in wait loops

Critical: In login wait loops, use continue on exception, never break. A page refresh during the wait is normal.

# WRONG — exits early on any error
except:
    break

# CORRECT — retries on transient errors
except Exception as e:
    log(f"Retry after error: {e}")
    time.sleep(0.5)
    continue

4.6 Duplicate detection

Before uploading or creating records, check if they already exist:

uploaded_rows = upload_page.locator("#grid-id table tbody tr").all()
for row in uploaded_rows:
    if filename in row.inner_text():
        log(f"Already exists: {filename}, skipping")
        return

Phase 5: Test & Refine

5.1 Run and observe

Run the script. If it fails:

1. Check the last saved screenshot — what did the browser actually see?
2. Check the last saved HTML — what elements were present?
3. Read the error traceback — is it a selector mismatch, timeout, or unexpected page state?

5.2 Fix common failures

Symptom	Cause	Fix
Timeout 60000ms exceeded on page load	Slow SPA, large assets	wait_until="domcontentloaded", timeout=120000
Selector resolves to N elements (N>1)	Ambiguous selector	Add parent scope: #parent input#id
Executable doesn't exist (exe only)	Chromium not bundled	Bundle with --add-data "path/to/chromium;chromium" or detect system install
Button not found	Wrong tag type	plupload uses <a>, not <button>
No popup detected	Popup blocked or same-tab	Fallback: upload_page = page
File not actually uploaded	Hidden input not wired	Use set_input_files() on input[type="file"] inside the plupload container

5.3 Add visual feedback for the user

When running as an exe, the user may not see console output. Inject a completion dialog into the browser:

// Injected via page.evaluate()
const div = document.createElement('div');
div.innerHTML = `
  <div style="position:fixed;top:0;left:0;width:100%;height:100%;
      background:rgba(0,0,0,0.75);z-index:999999;
      display:flex;align-items:center;justify-content:center;">
      <div style="background:#fff;padding:40px;border-radius:14px;text-align:center;">
          <h2>All tasks completed!</h2>
          <p>Closing in <span id="countdown">30</span>s</p>
          <button id="close-btn">Close Now</button>
      </div>
  </div>
`;
document.body.appendChild(div);

Then wait for the dialog to disappear:

page.wait_for_selector("#dialog-id", state="detached", timeout=30000)

Phase 6: Package

6.1 PyInstaller resource path

Make the script work both as .py and as PyInstaller .exe:

def resource_path(relative_path):
    if hasattr(sys, '_MEIPASS'):
        return os.path.join(sys._MEIPASS, relative_path)
    return os.path.join(os.path.abspath("."), relative_path)

6.2 Bundle Chromium for standalone distribution

py -m PyInstaller --onefile \
  --add-data "config.html;." \
  --add-data "C:\Users\...\ms-playwright\chromium-XXXX;chromium" \
  --name "MyAutomation" script.py

In the script, check bundled Chromium first:

if hasattr(sys, '_MEIPASS'):
    bundled = os.path.join(sys._MEIPASS, "chromium", "chrome-win", "chrome.exe")
    if os.path.exists(bundled):
        return bundled

6.3 Desktop logging

Always write logs to a file on the Desktop so users can diagnose issues even if the console closes:

LOG_FILE = os.path.join(os.path.expanduser("~"), "Desktop", "automation_log.txt")

def log(msg):
    line = f"[{time.strftime('%H:%M:%S')}] {msg}"
    print(line)
    try:
        with open(LOG_FILE, "a", encoding="utf-8") as f:
            f.write(line + "\n")
    except Exception:
        pass

Security Checklist

Before delivering to others:

• Credentials are NOT hardcoded
• Credentials are NOT saved to disk (only configuration like paths/URLs)
• If credentials must be entered each run, provide a local HTML config page
• Log files do NOT contain passwords

Additional Resources

• For detailed Playwright patterns and selector reference, see reference.md
• For a complete worked example (customs upload portal), see examples.md