SyntaxLens

A comparative NLP analysis tool for text and speech that visualizes POS tags and parse trees alongside Bag-of-Words vectors, with a built-in experiment on how reordering words affects each representation.

Give it a sentence — typed, spoken into a mic, or from an audio file — and it will tag, parse, vectorize, and explain what each representation does and doesn't capture about the sentence's meaning.

Features

• A) POS Tagging — Tokenizes the input and tags every token with NLTK's Penn Treebank tagger, then groups tokens into readable categories: Noun, Verb, Adjective, Adverb, Conjunction, Preposition, Pronoun, Determiner, Other.

• B) Parsing (Constituency + Dependency)

  • Constituency: a rule-based RegexpParser chunk grammar (NP / PP / ADVP / VP / CLAUSE) builds and prints a phrase-structure tree, saved as an SVG (svgling) or plain-text fallback.
  • Dependency: spaCy's dependency parser produces word → POS → relation → head rows, saved as an SVG via displacy.
  • Dependency labels (cc, conj, mark, advcl, ccomp, xcomp, acl, relcl) are inspected to describe whether clauses in the sentence are coordinated or subordinated to one another.

• C) Bag of Words — Builds a BoW representation with scikit-learn's CountVectorizer, printing the vocabulary and each sentence's word-frequency vector.

• D) Analysis and Interpretation — Auto-generates a written discussion of:

  • what BoW preserves (vocabulary, frequency, coarse topical content)
  • what BoW loses (word order, grammatical roles, phrase/clause structure, attachment ambiguity)
  • why parsing captures structural information BoW cannot

• E) Higher-Order Challenge — Reverses the word order of the input sentence and re-runs POS tagging, BoW, and dependency parsing on the scrambled version, showing that the BoW vector is unchanged while POS tags and the dependency ROOT shift — proof that BoW is blind to order while parsing is not.

• Speech input — Accepts a pre-recorded audio file (.wav, .aiff, .flac natively; .mp3/.m4a via pydub + ffmpeg conversion) or records live from the microphone (sounddevice), transcribing with Google's Web Speech API.

Tech Stack

Purpose	Library
POS tagging & tokenization	nltk
Constituency parsing	nltk.RegexpParser, svgling
Dependency parsing & visualization	spacy, en_core_web_sm, displacy
Bag-of-Words	scikit-learn (CountVectorizer)
Speech-to-text	SpeechRecognition, pydub, sounddevice

Setup

git clone https://github.com/Naman-bh/SyntaxLens.git
cd SyntaxLens

pip install nltk spacy scikit-learn svgling SpeechRecognition pydub sounddevice numpy
python -m spacy download en_core_web_sm

NLTK's punkt and averaged_perceptron_tagger models are downloaded automatically on first run.

For non-WAV audio input, ffmpeg must also be installed and on your system PATH.

Usage

Interactive mode (prompts you to type a sentence or record from the mic):

python SOP.py

Analyze a typed sentence:

python SOP.py --text "The old man saw the boy with a telescope and he smiled."

Analyze an audio file:

python SOP.py --audio path/to/speech.wav

Record live from the microphone:

python SOP.py --record --record-seconds 5

Also run the word-reorder experiment (Part E):

python SOP.py --text "The cat chased the mouse." --reorder

CLI Flags

Flag	Description
--text	A sentence to analyze directly
--audio	Path to an audio file to transcribe and analyze
--record	Record live speech from the microphone instead
--record-seconds	Max seconds to record from the mic (default: 5)
--reorder	Also run the Part E word-order-change experiment

Output

Running the script prints POS tags, both parse trees, the BoW vocabulary/vector, and the written analysis directly to the console. Visual parse trees (constituency + dependency) are saved as .svg files in ./output/, along with any converted or recorded audio.

Sample Output (excerpt)

--- A) POS TAGGING -----------------------------------------------
Token          POS Tag
-------------------------
The            DT
old            JJ
man            NN
saw            VBD
...

--- C) BAG OF WORDS --------------------------------------------------
Vocabulary (9 unique words): ['boy', 'he', 'man', 'old', 'saw', ...]
Word-frequency vector:
 the           : 3
 old           : 1
 ...

Why This Project

Built to compare structural (parsing) vs. distributional (Bag-of-Words) representations of language head-to-head on the same sentence — showing concretely, with real output, what syntax captures that a word-count vector never can.