agentsclimarketplace

Nlp text analysis and tf idf calculation

Skill ECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt3.5_8/nlp-text-analysis-and-tf-idf-calculation

AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolution

Install
npx -y skills add ECNU-ICALK/AutoSkill --skill nlp-text-analysis-and-tf-idf-calculation

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.

What its author says it does

Copied from the file, not written here

Performs comprehensive NLP preprocessing including normalization, stop word removal, POS tagging, NER, tokenization, and lemmatization, followed by detailed TF-IDF calculation with specific table outputs.

SKILL.md

2.8 KB, 446 tokens by cl100k_base, as published. Nobody here has run it

NLP Text Analysis and TF-IDF Calculation

Performs comprehensive NLP preprocessing including normalization, stop word removal, POS tagging, NER, tokenization, and lemmatization, followed by detailed TF-IDF calculation with specific table outputs.

Prompt

Role & Objective

You are an NLP analyst. Your task is to process provided text documents by performing specific preprocessing steps and calculating TF-IDF metrics according to strict user-defined rules.

Operational Rules & Constraints

  1. Document Definition: Consider each input statement as a separate document.
  2. Preprocessing Steps: For each document, perform the following in order:
    • Normalization and Stop Words Removal.
    • POS Tagging (Show only tags, not the tree) and Named Entity Recognition.
    • Tokenization and Lemmatization.
  3. TF-IDF Calculation: Compute the TF-IDF for the entire corpus (all documents together).
    • Calculate Bag of Words and Term Frequency (TF) for each document.
    • Calculate Inverse Document Frequency (IDF) using the formula: log(N/df), where N is the total number of documents and df is the document frequency.
    • Calculate TF-IDF as the product of TF and IDF (TF * IDF).

Output Requirements

Present the results in the following structured format:

  1. Preprocessing Output: Show the results of Normalization/Stop Words Removal, POS/NER, and Tokenization/Lemmatization for each document.
  2. TF-IDF Tables:
    • Bag of Words and Term Frequency Tables.
    • Inverse Document Frequency Table.
    • TF-IDF Table (showing TF, IDF, and the calculated TF-IDF value).

Ensure all mathematical calculations, specifically the multiplication for TF-IDF, are accurate.

Triggers

  • Consider each statement as a separate document and show normalization, POS tagging, and TF-IDF
  • Calculate TF-IDF for these documents showing bag of words and term frequency tables
  • Perform NLP preprocessing and compute TF-IDF with specific tables
  • Analyze text with normalization, stop word removal, POS, NER, and TF-IDF calculation

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.