Refactor parseArtSci function and add tests for newly introduced parseDepartmentList by r-weng · Pull Request #1725 · Courseography/courseography

r-weng · 2026-06-07T16:00:56Z

Proposed Changes

This pull request refactors the parseArtSci function in app/WebParsing/ArtSciParser.hs by introducing a new function, parseDepartmentList, to handle parsing of department lists. parseDepartmentList now ignores department pages with no courses, duplicate courses, and program areas belonging to a college. Additionally, this pull request adds test for the newly introduced parseDepartmentList function.

Type of Change

(Write an X or a brief description next to the type or types that best describe your changes.)

Type	Applies?
🚨 Breaking change (fix or feature that would cause existing functionality to change)
✨ New feature (non-breaking change that adds functionality)
🐛 Bug fix (non-breaking change that fixes an issue)
🎨 User interface change (change to user interface; provide screenshots)
♻️ Refactoring (internal change to codebase, without changing functionality)	X
🚦 Test update (change that only adds or modifies tests)
📦 Dependency update (change that updates a dependency)
🔧 Internal (change that only affects developers or continuous integration)

Checklist

(Complete each of the following items for your pull request. Indicate that you have completed an item by changing the [ ] into a [x] in the raw text, or by clicking on the checkbox in the rendered description on GitHub.)

Before opening your pull request:

I have performed a self-review of my changes.
- Check that all changed files included in this pull request are intentional changes.
- Check that all changes are relevant to the purpose of this pull request, as described above.
I have added tests for my changes, if applicable.
- This is required for all bug fixes and new features.
I have updated the project documentation, if applicable.
- This is required for new features.
If this is my first contribution, I have added myself to the list of contributors.
I have updated the project Changelog (this is required for all changes).

After opening your pull request:

I have verified that the CircleCI checks have passed.
I have requested a review from a project maintainer.

Questions and Comments

(Include any questions or comments you have regarding your changes.)

coveralls · 2026-06-07T16:10:40Z

Coverage Report for CI Build 0

Coverage increased (+0.4%) to 58.998%

Details

Coverage increased (+0.4%) from the base build.
Patch coverage: 1 uncovered change across 1 file (17 of 18 lines covered, 94.44%).
No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
app/WebParsing/ArtSciParser.hs	18	17	94.44%

Coverage Regressions

No coverage regressions found.

Coverage Stats


Relevant Lines:	3965
Covered Lines:	2405
Line Coverage:	60.66%
Relevant Branches:	964
Covered Branches:	503
Branch Coverage:	52.18%
Branches in Coverage %:	Yes
Coverage Strength:	163.81 hits per line

💛 - Coveralls

david-yz-liu

@r-weng This is good, but please write tests for this new function

…i-function

david-yz-liu · 2026-06-17T14:12:13Z

 ### 🔧 Internal changes

+- Refactor `parseArtSci` function in `app/WebParsing/ArtSciParser.hs` by introducing `parseDepartmentList`
+- Add test cases for the `parseDepartmentList` function in `backend-test/WebParsing/ArtSciParserTests.hs`


It isn't necessary to add a second entry describing test cases added (this is implied as part of the first change)

david-yz-liu · 2026-06-17T14:19:22Z

+                        "Research Opportunity/Research Excursions (299/398/399)"]
+    bodyTags <- httpBodyTags url
+    let deptList = getDeptList bodyTags
+    let cleaned = map (BF.second (T.replace "\160" " ")) deptList


First, move this into getDeptList (i.e., we don't need to separate out the "cleaning" step). That function currently only calls T.strip, so it can be simplified by using getDeptList anyways.

Second, rather than do this replacing, please use the existing cleanText. That said, replacing "\160" with " " is better than replacing it with "", so you can also go ahead and modify cleanText itself. (See here for a reference for the '\160' char.)

david-yz-liu · 2026-06-17T14:20:43Z

+    bodyTags <- httpBodyTags url
+    let deptList = getDeptList bodyTags
+    let cleaned = map (BF.second (T.replace "\160" " ")) deptList
+    return $ filter (\(deptPage, deptName) -> "/" `T.isPrefixOf` deptPage && deptName `notElem` ignoredDepts && not (" College)" `T.isSuffixOf` deptName)) cleaned


As was mentioned at our meeting yesterday, this filtering is a great candidate for a local helper function. The idiomatic way to define this is to use a where block in this function:

return filterDepartments deptList where filterDepartments :: [(T.Text, T.Text)] -> [(T.Text, T.Text)] ...

david-yz-liu · 2026-06-17T14:21:22Z

@@ -1,11 +1,12 @@
 module WebParsing.ArtSciParser
-    (parseCalendar, getDeptList) where
+    (parseCalendar, getDeptList, parseDepartmentList) where


While parseDepartmentList is good to be exported for testing purposes, getDeptList isn't used anywhere else in the codebase so we can remove it from this export list.

… refactor-parseArtSci-function

…ng a local helper function

david-yz-liu · 2026-06-23T20:07:15Z

 import Data.List (findIndex, nubBy)
 import Data.Maybe (fromMaybe, mapMaybe)
 import qualified Data.Text as T
+import qualified Data.Bifunctor as BF


Put this in alphabetical order (i.e., above the Data.List)

david-yz-liu · 2026-06-23T20:08:42Z

+                        "Data Science",
+                        "Faculty of Arts and Science Programs (299/398/399)",
+                        "Laboratory Medicine and Pathobiology)", -- Displayed as "Pathobiology (see Laboratory Medicine and Pathobiology)" on program areas page
+                        "Research Opportunity/Research Excursions (299/398/399)"]


This is good, I just realized there's one more section to exclude: "Writing in the Faculty of Arts & Science".

david-yz-liu · 2026-06-23T20:10:17Z

+-- Exclude departments with no courses, duplicate courses, and program areas belonging to a college.
+parseDepartmentList :: String -> IO [(T.Text, T.Text)]
+parseDepartmentList url = do
+    let ignoredDepts = ["ASIP (Arts & Science Internship Program)",


please move this into the where clause as well, since it's only used by the helper

r-weng added 3 commits June 7, 2026 11:51

Refactor parseArtSci function

417223d

Update parseDepartmentList to ignore department names

622c635

Update CHANGELOG

6f84688

r-weng requested a review from david-yz-liu June 7, 2026 16:37

Add comment for parseDepartmentList

d208de1

david-yz-liu reviewed Jun 7, 2026

View reviewed changes

r-weng added 6 commits June 9, 2026 09:09

Update indentation for ignoredDepts variable

ec39261

Add parseDepartmentList to module export list

309e313

Update department name in ignoredDepts

67684a4

Add test for WebParsing.ArtSciParser

6752a68

Update parseDepartmentList in WebParsing.ArtSciParser

caa36b4

Update CHANGELOG

12d1582

r-weng changed the title ~~Refactor parseArtSci function~~ Refactor parseArtSci function and add tests for newly introduced function Jun 14, 2026

r-weng changed the title ~~Refactor parseArtSci function and add tests for newly introduced function~~ Refactor parseArtSci function and add tests for newly introduced parseDepartmentList Jun 14, 2026

r-weng requested a review from david-yz-liu June 14, 2026 20:48

r-weng added 3 commits June 14, 2026 16:52

Merge remote-tracking branch 'origin/master' into refactor-parseArtSc…

acc4003

…i-function

Update CHANGELOG again

7965eb6

Update CHANGELOG bullet tense

38f79d6

david-yz-liu reviewed Jun 17, 2026

View reviewed changes

r-weng added 4 commits June 17, 2026 11:16

Merge branch 'master' of https://github.com/r-weng/courseography into…

6f6013a

… refactor-parseArtSci-function

Revert CHANGELOG

d63629a

Remove getDeptList export

433c269

Refactor parseDepartmentList by moving code to getDeptList and creati…

10e8a0f

…ng a local helper function

r-weng requested a review from david-yz-liu June 23, 2026 00:06

r-weng added 2 commits June 22, 2026 20:36

Remove extra blank line

e747852

Add comments for clarity

5b71452

david-yz-liu reviewed Jun 23, 2026

View reviewed changes

Add styling and other changes

43eb734

r-weng requested a review from david-yz-liu June 24, 2026 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor parseArtSci function and add tests for newly introduced parseDepartmentList#1725

Refactor parseArtSci function and add tests for newly introduced parseDepartmentList#1725
r-weng wants to merge 20 commits into
Courseography:masterfrom
r-weng:refactor-parseArtSci-function

r-weng commented Jun 7, 2026 •

edited

Loading

Uh oh!

coveralls commented Jun 7, 2026 •

edited

Loading

Uh oh!

david-yz-liu left a comment

Uh oh!

david-yz-liu Jun 17, 2026

Uh oh!

david-yz-liu Jun 17, 2026

Uh oh!

david-yz-liu Jun 17, 2026

Uh oh!

david-yz-liu Jun 17, 2026

Uh oh!

david-yz-liu Jun 23, 2026

Uh oh!

david-yz-liu Jun 23, 2026

Uh oh!

david-yz-liu Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

r-weng commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Type of Change

Checklist

Questions and Comments

Uh oh!

coveralls commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for CI Build 0

Coverage increased (+0.4%) to 58.998%

Details

Uncovered Changes

Coverage Regressions

Coverage Stats

💛 - Coveralls

Uh oh!

david-yz-liu left a comment

Choose a reason for hiding this comment

Uh oh!

david-yz-liu Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

david-yz-liu Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

david-yz-liu Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

david-yz-liu Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

david-yz-liu Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

david-yz-liu Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

david-yz-liu Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

r-weng commented Jun 7, 2026 •

edited

Loading

coveralls commented Jun 7, 2026 •

edited

Loading