diff --git a/README.md b/README.md
index 36c81e3..8cfc030 100644
--- a/README.md
+++ b/README.md
@@ -254,6 +254,40 @@ Notes:
- All aggregators operate over per‑observation scores where non‑confident observations are already clipped to 0.
- The default `skewness` remains a good choice when user activity volume varies widely.
+## Simulation-based tuning
+
+Which aggregator and hyperparameters work best depends on your data, so it helps to measure them on labeled examples. The `sentinel.simulation` module is a small, numpy‑only harness for exactly that — no Ray, S3, or experiment trackers required.
+
+Give it groups of observations with known labels (`1` = rare/positive source, `0` = common/negative source), score them once, then compare every aggregator (and hyperparameter) cheaply:
+
+```python
+import pandas as pd
+from sentinel.simulation import LabeledGroup, score_groups, compare_aggregators, run_grid_search
+
+groups = [
+ LabeledGroup(name="source_a", label=1, observations=["...", "..."]), # known rare-class source
+ LabeledGroup(name="source_b", label=0, observations=["...", "..."]), # known common-class source
+ # ...
+]
+
+# Expensive step (runs the model) — done once.
+scored = score_groups(index, groups, top_k=5)
+
+# Cheap step — compare all six aggregators on the same scores.
+pd.DataFrame(compare_aggregators(scored))
+
+# Or sweep hyperparameters as well.
+pd.DataFrame(run_grid_search(index, groups, top_k_values=[3, 5, 10], min_score_values=[0.0, 0.1, 0.25]))
+```
+
+Each result row reports three families of evaluation metrics, so you can tune for whatever matters to your use case:
+
+- Ranking: `roc_auc`, `recall_at_n`, `precision_at_n`, `rank_ratio` (do known positives rank at the top?)
+- Threshold / classification: `precision`, `recall`, `f1`, `false_positive_rate` at a chosen (or automatically best‑F1) cutoff
+- Separation / distribution: `mean_separation`, `cohens_d`, `ks_statistic` (threshold‑free)
+
+See [examples/sentinel_against_hate.ipynb](examples/sentinel_against_hate.ipynb) for a worked example comparing aggregators on hate‑speech data.
+
## How It Works
Sentinel uses a two-step process to detect rare classes of text, focusing on high recall for realtime applications:
diff --git a/docs/source/modules.rst b/docs/source/modules.rst
index 0859667..82a26d7 100644
--- a/docs/source/modules.rst
+++ b/docs/source/modules.rst
@@ -5,3 +5,4 @@ API Reference
:maxdepth: 4
sentinel
+
diff --git a/docs/source/sentinel.rst b/docs/source/sentinel.rst
index b0f78d7..398fbf5 100644
--- a/docs/source/sentinel.rst
+++ b/docs/source/sentinel.rst
@@ -34,6 +34,14 @@ sentinel.sentinel_local_index
:undoc-members:
:show-inheritance:
+sentinel.simulation
+-------------------
+
+.. automodule:: sentinel.simulation
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
Subpackages
----------
diff --git a/examples/Example_Threshold_Script.py b/examples/Example_Threshold_Script.py
index 3f9d498..c085eb3 100644
--- a/examples/Example_Threshold_Script.py
+++ b/examples/Example_Threshold_Script.py
@@ -33,6 +33,11 @@
import time
from typing import Dict, List
+# Path to a saved Sentinel index. By default this points at the hate-speech
+# index built by examples/sentinel_against_hate.ipynb. Change it to your own
+# index path (local directory or s3:// URI) if needed.
+DEFAULT_INDEX_PATH = "./hate_speech_model"
+
def create_user_profiles() -> Dict[str, List[str]]:
"""Create 10 different user profiles with varying speech patterns."""
@@ -163,9 +168,9 @@ def test_thresholds_and_ratios(review_mode: bool = False,
# Time data loading
load_start = time.time()
index = SentinelLocalIndex.load(
- path="path/to/local/index",
+ path=DEFAULT_INDEX_PATH,
negative_to_positive_ratio=ratio,
- Cache_Model=True & ~no_cache
+ cache_model=not no_cache,
)
load_time = time.time() - load_start
total_load_time += load_time
diff --git a/examples/sentinel_against_hate.ipynb b/examples/sentinel_against_hate.ipynb
index bdf9338..5936edc 100644
--- a/examples/sentinel_against_hate.ipynb
+++ b/examples/sentinel_against_hate.ipynb
@@ -113,20 +113,6 @@
"Found 249 markdown files.\n"
]
},
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "6ff47bdd94334db8970505b6b189dc05",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Processing markdown files: 0%| | 0/249 [00:00, ?it/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
{
"name": "stdout",
"output_type": "stream",
@@ -297,7 +283,7 @@
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 5,
"id": "f95affd4",
"metadata": {},
"outputs": [
@@ -305,7 +291,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "DataFrame shape: (2515, 2)\n",
+ "DataFrame shape: (1516, 2)\n",
"Random 30 rows:\n"
]
},
@@ -336,223 +322,223 @@
" \n",
"
\n",
" \n",
- " | 617 | \n",
- " center-immigration-studies | \n",
- " Washington Times | \n",
+ " 51 | \n",
+ " richard-bertrand-spencer | \n",
+ " “We’re going to be back here, and we’re going to humiliate all of these people who opposed us. We’ll be back here 1,000 times if necessary. I always win. Because I have the will to win, I keep going until I win.” | \n",
"
\n",
" \n",
- " | 927 | \n",
- " louis-beam | \n",
- " â “New World Order,” 1999 essay by Beam | \n",
+ " 168 | \n",
+ " tim-wildmon | \n",
+ " â[Islam] is in fact a religion of war, violence, intolerance, and physical persecution of non-Muslims.â | \n",
"
\n",
" \n",
- " | 942 | \n",
- " bill-white | \n",
- " In August 2004, White was convicted of assaulting a woman when she was handing out flyers that identified him as a neo-Nazi landlord. During the court proceedings, White cursed at the woman from the witness stand and was held in contempt of court by the judge. In December 2009, a federal jury found White guilty of threatening several people through intimidating phone calls and internet postings. He was sentenced to 2 ½ years in prison at an April 2010 hearing. In January 2011, White was found guilty of using his website to encourage violence against a jury foreman in the trial of another white supremacist, | \n",
+ " 2468 | \n",
+ " misogynist-incels | \n",
+ " “Jews heavily pushed feminism. Roasties have such high standards because they are ‘liberated’. You probably would have gotten your d*** wet by now if it weren’t for feminism and for Jews.” | \n",
"
\n",
" \n",
- " | 973 | \n",
- " william-pierce | \n",
- " â Pierce, quoted in a biography of him called Fame of a Dead Man’s Deeds, on the power of narrative | \n",
+ " 1040 | \n",
+ " kyle-bristow | \n",
+ " â Quoted in the | \n",
"
\n",
" \n",
- " | 1967 | \n",
- " tucker-carlson | \n",
- " a reporter later exposed | \n",
+ " 422 | \n",
+ " raymond-cattell | \n",
+ " âSuppose, as may well be the case, that one of these races is naturally courageous, self-sacrificing and enterprising and the other less so. The group will continue to prosper owing to the activities of inventors and explorers of the first race, who, as is generally the rule, will not pass on the usual number of children to the next generation. The nation will be successful in war because the same race has actively responded to the call to arms and to self-sacrifice. Throughout these activities, this first race will on an average be giving more to the group than it can itself recoup. Eventually only the second race will inherit the group advantages acquired largely by the first racial compound. Then like a huge parasite which has devoured its host, will the nation be bereft of all the qualities that gave it power, remain a monstrous frustration of evolution, a biological abortion able in virtue of its inherited wealth, to do untold damage to neighboring races naturally more capable. The hatred and abhorrence which many peoples feel for the Jewish (and to some extent Mongolian) practice of living in other nations, instead of forming an independent, self-sustained group of their own, comes from a deep intuitive feeling that somehow it is not âplaying the game.â Because our unbiologically-minded civilization cannot perceive or appreciate any intellectual causes for these feelings they are readily branded as âprejudiceâ by would-be intellectuals.â | \n",
"
\n",
" \n",
- " | 1116 | \n",
- " stefan-molyneux | \n",
- " Collective Guilt for Fun and Profit”, | \n",
+ " 1479 | \n",
+ " joseph-francis-farah | \n",
+ " Polygamy? Sure. | \n",
"
\n",
" \n",
- " | 1375 | \n",
- " malik-zulu-shabazz | \n",
- " Audience: “Jews!” | \n",
+ " 585 | \n",
+ " bill-white | \n",
+ " â Headline for 2008 article about then-presidential candidate Barack Obama who was depicted in the middle of a rifle scope on the cover of White’s American National Socialist Workers Party’s magazine | \n",
"
\n",
" \n",
- " | 170 | \n",
- " paul-nehlen | \n",
- " Nehlenâs 2016 campaign for Congress was ideologically similar to Trumpâs and was buoyed by FOX News and other outlets with large, national audiences. Breitbart.com ran several stories on Nehlen, for example. Right-wing pundits such as | \n",
+ " 220 | \n",
+ " lydia-brimelow | \n",
+ " âAs the President of the VDARE Foundation, I work alongside my husband, Editor of VDARE.com, Peter Brimelow, to defeat the post-1964 Immigration Act disaster that has set America on a course of destruction.â | \n",
"
\n",
" \n",
- " | 2025 | \n",
- " tucker-carlson | \n",
- " “Tucker is ultimately on our side. He can get millions and millions of boomers to nod along with talking points that would have only been seen on VDare or American Renaissance a few years ago,” Greer said on his podcast in the Spring of 2021, | \n",
+ " 2467 | \n",
+ " misogynist-incels | \n",
+ " – Comment on an incel forum post titled “‘They are animals…’ ER was right,” | \n",
"
\n",
" \n",
- " | 321 | \n",
- " ronald-doggett | \n",
- " “Here in the state of Virginia our Confederate History Month proclamations became so watered down with nods towards the non-Whites they weren’t even worth supporting. We can’t ever have anything just for Whites…I wonder what a true Confederate (White Supremacist) brought to today’s time would think of the cowardly defenders of his cause.” | \n",
+ " 2439 | \n",
+ " james-lindsay | \n",
+ " If anyone causes one of these little onesâthose who believe in meâto stumble, it would be better for them to have a large millstone hung around their neck and to be drowned in the depths of the sea.â) | \n",
"
\n",
" \n",
- " | 1000 | \n",
- " scott-lively | \n",
- " “It is not mere coincidence that the emperors of Rome in its horrific final days were homosexual; that Adolf Hitler’s inner circle were mostly homosexual; and that nearly all of the most prolific serial killers in U.S. history were homosexual. It is not mere coincidence that America’s cultural decline parallels the rise of ‘gay rights.â” | \n",
+ " 1535 | \n",
+ " jeff-berry | \n",
+ " â 1998 speech at a Klan rally in Jasper, Texas, after the truck-dragging murder there of James Byrd Jr. | \n",
"
\n",
" \n",
- " | 2331 | \n",
- " proud-boys | \n",
- " “I am not afraid to speak out about the atrocities that whites and people of European descent face not only here in this country but in Western nations across the world. The war against whites, and Europeans and Western society is very real and it’s time we all started talking about it and stopped worrying about political correctness and optics.” – Kyle Chapman, who formed the Fraternal Order of Alt-Knights, a paramilitary wing of the Proud Boys, Unite America First Peace Rally, Sacramento, California, July 8, 2017 | \n",
+ " 49 | \n",
+ " harry-cooper | \n",
+ " “We must remember that official government sources state definitely that Hitler committed suicide in the bunker and that the body found not far from the Chancellery was indeed that of Martin Bormann. … Naturally we must believe what the governments tell us, like John F. Kennedy was shot by one madman using a worthless rifle when nobody with any sense buys that story except the Warren Commission. … The list of lies by the world’s governments could by themselves, fill a major size book but you already know that.” | \n",
"
\n",
" \n",
- " | 408 | \n",
- " moms-liberty | \n",
- " âGender dysphoria is a mental health disorder that is being normalized by predators across the USA. California kids are at extreme risk from predatory adults. Now they want to âliberateâ children all over the country. Does a double mastectomy on a preteen sound like progress?â | \n",
+ " 535 | \n",
+ " david-irving | \n",
+ " â 1991 speech to a Canadian audience | \n",
"
\n",
" \n",
- " | 651 | \n",
- " center-immigration-studies | \n",
- " American Renaissance | \n",
+ " 175 | \n",
+ " jack-posobiec | \n",
+ " “StoptheSteal 2020 is coming.” – Twitter, Sept. 7, 2020, a full two months before news outlets declared Joe Biden the winner of the 2020 presidential election | \n",
"
\n",
" \n",
- " | 1614 | \n",
- " tomislav-sunic | \n",
- " Postmortem Report: Cultural Examinations from Postmodernity | \n",
+ " 1464 | \n",
+ " willis-carto | \n",
+ " The Barnes Review | \n",
"
\n",
" \n",
- " | 1135 | \n",
- " stefan-molyneux | \n",
- " “If we could just get people to be nice to their babies for five years straight, that would be it for war, drug abuse, addiction, promiscuity, sexually transmitted diseases. Almost all would be completely eliminated, because they all arise from dysfunctional early childhood experiences, which are all run by women.” | \n",
+ " 405 | \n",
+ " proud-boys | \n",
+ " “All the heroes of BLM and Antifa are degenerate criminal lowlifes or pedophile rapists. I don’t lose any sleep when they are justly removed from society.” – A Telegram channel associated with the Proud Boys, Sept. 22, 2020 | \n",
"
\n",
" \n",
- " | 582 | \n",
- " center-immigration-studies | \n",
- " (CCC), which Charleston shooter | \n",
+ " 846 | \n",
+ " greg-johnson | \n",
+ " — “Dealing with the Holocaust | \n",
"
\n",
" \n",
- " | 1961 | \n",
- " tucker-carlson | \n",
- " served as a platform | \n",
+ " 598 | \n",
+ " atomwaffen-division | \n",
+ " âPro f—– propaganda is working! AIDS is spreading like wildfire. Dead f—— couldnât make us happier! Hail AIDS!â â Atomwaffen Division website | \n",
"
\n",
" \n",
- " | 1640 | \n",
- " michael-flynn | \n",
- " “We the people are proud to proclaim that the United States of America is ‘One Nation under God’ – in this public profession of faith in God, we recognize his Lordship over our country, and we proudly stand beneath the banner of Christ and our flag in which millions have sacrificed their very lives for. In scripture through the strength and commitment of Matthew, he said, ‘Whoever is not with Me is against Me.’” – Blog post titled | \n",
+ " 2493 | \n",
+ " mark-weber | \n",
+ " “Around the world awareness is growing that the âHolocaust’ campaign is a major weapon in the Jewish-Zionist arsenal, that it is used to justify otherwise unjustifiable Israeli policies, and as a powerful tool for blackmailing enormous sums of money from Americans and Europeans.” | \n",
"
\n",
" \n",
- " | 1073 | \n",
- " paul-elam | \n",
- " “P—- is the only real empowerment women will ever know. Put all the hopelessly wishful thinking of feminist ideology aside and what remains is the fact that it is men, and pretty much men only, who draw power from accomplishment, who invent technology, build nations, cure disease, create empires and generally advance civilization. Women – whether acknowledging it makes us feel warm and fuzzy or not – depend on men for all of that and the only tool they have at their disposal to have any sort of influence on any of it is the power of p—- and p—- is powerful indeed…Sexual robotics may well prove to be the best thing that ever happened to women from the standpoint of their humanity…. what would that do to the vast majority of women who would suddenly have to prove their worth as human beings beyond simply being the owners of said p—-?” – Paul Elam, An Ear for Men, Sex Robots: Part 3 – Disempowering P—-, October 2017 | \n",
+ " 1052 | \n",
+ " gays-against-groomers | \n",
+ " “The modern trans movement is radicalizing activists into terrorists. […] Trans terrorism is on the rise. […] This sends a message: if you don’t let us mutilate children’s genitals, then we will shoot yours.” – GAG then-Secretary David Leatherwood, “Real America with Dan Ball,” in response to the Covenant School shooting in Nashville, Tennessee, | \n",
"
\n",
" \n",
- " | 1582 | \n",
- " mike-cernovich | \n",
- " Despite making completely unsubstantiated accusations of pedophilia, Cernovich discussed the allegations that aspiring Alabama senator Roy Moore had sexually abused underage girls cautiously. After tweeting that “If it’s true, string the guy up, man. I got no problem with that,” he later retweeted individuals casting doubt on the validity of the accusers, writing: “When you’re lied about in the news daily, as I am, you pause when 40-year-old accusations surface one month away from an election where WaPo endorsed the other candidate.” Taking it further, however, Cernovich then recorded a podcast giving “scientific” advice to Roy Moore, | \n",
+ " 2485 | \n",
+ " john-de-nugent | \n",
+ " âThey [the Jews] control the media and so it spews lies against the enemy they fear most. And the Jews now misrule our beloved country of America, bringing it down by huge bribes, media brainwashing and blackmail of our officials, and police, and military officers both high and low. And worst of all, by a national network of organized pedophiles, child molesting and child-murdering adults who once involved in that network, one orgy at a time, are photographed there, captured forever on film. That film is stored away for future blackmail use. Those officials cannot quit even if they want to.â | \n",
"
\n",
" \n",
- " | 907 | \n",
- " nation-islam | \n",
- " âPedophilia and sexual perversion institutionalized in Hollywood and the entertainment industries can be traced to Talmudic principles and Jewish influence. Now Jewish influence â satanic influence under the name of Jew…The wicked practices that govern their industries are largely justified and influenced by such Talmudic principles. The pervasive rape culture, Hollywoodâs casting couch, sex trafficking and prostitution, the age-old buck breaking process that emasculates Black men and corrupts Black women.â | \n",
+ " 203 | \n",
+ " andrew-anglin | \n",
+ " “Look, I hate women. I think they deserve to be beaten, raped and locked in cages.” – Daily Stomer, July 2018 | \n",
"
\n",
" \n",
- " | 613 | \n",
- " center-immigration-studies | \n",
- " No such reprimand occurred when Steinlight, | \n",
+ " 1271 | \n",
+ " lieutenant-general-william-g-jerry-boykin-ret | \n",
+ " “If we don’t take a stand here, it’s only a matter of time before biological men will be able to share showers and locker rooms with women and girls all across America! … Many on the Left believe that biological men have a right to share showers, locker rooms, and bathrooms with women and little girls as long as they identify as female at that particular point in time — and that anyone who disagrees with that should be punished!” | \n",
"
\n",
" \n",
- " | 2039 | \n",
- " tucker-carlson | \n",
- " the digital publication. | \n",
+ " 297 | \n",
+ " alt-right | \n",
+ " “Immigration is a kind of proxy war—and maybe a last stand—for White Americans, who are undergoing a painful recognition that, unless dramatic action is taken, their grandchildren will live in a country that is alien and hostile.” | \n",
"
\n",
" \n",
- " | 962 | \n",
- " edgar-steele | \n",
- " “Without a pressure release valve, as open racism once provided, an explosion of epic proportions at some time in the future is guaranteed. There will be a race war, the initial skirmishes of which already are being fought in America’s streets, that will bring an altogether new meaning to the concepts of race war and genocide, courtesy of those who claim to abhor racism.” | \n",
+ " 453 | \n",
+ " garrett-hardin | \n",
+ " —“How Diversity Should be Nurtured,” | \n",
"
\n",
" \n",
- " | 2060 | \n",
- " tucker-carlson | \n",
- " Tucker Carlson hosted male supremacist Andrew Tate on August 5, 2022. (Twitter) | \n",
+ " 367 | \n",
+ " alex-jones | \n",
+ " âImagine how bad she [Hillary Clinton] smells, man? Iâm told her and Obama, just stink, stink, stink, stink. You canât wash that evil off, man. Told thereâs a rotten smell around Hillary. Iâm not kidding, people say, they say â folks, Iâve been told this by high-up folks. They say listen, Obama and Hillary both smell like sulfur. ⦠Iâve talked to people that are in protective details, theyâre scared of her. And they say listen, sheâs a frickinâ demon and she stinks and so does Obama. I go, like what? Sulfur. They smell like hell.â â âThe Alex Jones Show,â Oct. 10, 2016 | \n",
"
\n",
" \n",
- " | 1627 | \n",
- " james-timothy-turner | \n",
- " “I am not in jail for violating the law. I am in jail because I stood for righteousness and truth in government. Pray that God will crumble the foundations and break the power and strength of the corporation and restore his righteous government in America.” | \n",
+ " 1055 | \n",
+ " gays-against-groomers | \n",
+ " “Every year, THOUSANDS of children are exposed to displays of sexual depravity at Pride events, and MILLIONS applaud it. This year, Gays Against Groomers will not stand for this destruction of childhood innocence. Be seeing you REAL soon.” – GAG Illinois X (formerly Twitter), | \n",
"
\n",
" \n",
- " | 2186 | \n",
- " matt-walsh | \n",
- " In addition to spreading disinformation about transgender identity and discredited pseudoscience, Walsh’s film relies on propagandist tactics of narrative manipulation in suggesting legal protections for transgender people will lead to people being prosecuted for using the wrong pronouns. And in making numerous | \n",
+ " 237 | \n",
+ " chuck-baldwin | \n",
+ " âI believe homosexuality is moral perversion and deserves no special consideration under the law. ⦠I believe the South was right in the War Between the States, and I am not a racist.â | \n",
"
\n",
" \n",
- " | 866 | \n",
- " barry-black | \n",
- " â A 1998 comment to a Virginia newspaper | \n",
+ " 1553 | \n",
+ " craig-cobb | \n",
+ " âNov. 16, 2013, videotaped rant against a resident while âpatrollingâ the streets of Leith | \n",
"
\n",
" \n",
- " | 471 | \n",
- " david-yerushalmi | \n",
- " â âOffensive and Defensive Lawfare: Fighting Civilization Jihad in Americaâs Courts,â Center for Security Policy press release, | \n",
+ " 677 | \n",
+ " family-research-council | \n",
+ " “The reality is, homosexuals have entered the Scouts in the past for predatory purposes.” | \n",
"
\n",
" \n",
"\n",
""
],
"text/plain": [
- " filename \\\n",
- "617 center-immigration-studies \n",
- "927 louis-beam \n",
- "942 bill-white \n",
- "973 william-pierce \n",
- "1967 tucker-carlson \n",
- "1116 stefan-molyneux \n",
- "1375 malik-zulu-shabazz \n",
- "170 paul-nehlen \n",
- "2025 tucker-carlson \n",
- "321 ronald-doggett \n",
- "1000 scott-lively \n",
- "2331 proud-boys \n",
- "408 moms-liberty \n",
- "651 center-immigration-studies \n",
- "1614 tomislav-sunic \n",
- "1135 stefan-molyneux \n",
- "582 center-immigration-studies \n",
- "1961 tucker-carlson \n",
- "1640 michael-flynn \n",
- "1073 paul-elam \n",
- "1582 mike-cernovich \n",
- "907 nation-islam \n",
- "613 center-immigration-studies \n",
- "2039 tucker-carlson \n",
- "962 edgar-steele \n",
- "2060 tucker-carlson \n",
- "1627 james-timothy-turner \n",
- "2186 matt-walsh \n",
- "866 barry-black \n",
- "471 david-yerushalmi \n",
+ " filename \\\n",
+ "51 richard-bertrand-spencer \n",
+ "168 tim-wildmon \n",
+ "2468 misogynist-incels \n",
+ "1040 kyle-bristow \n",
+ "422 raymond-cattell \n",
+ "1479 joseph-francis-farah \n",
+ "585 bill-white \n",
+ "220 lydia-brimelow \n",
+ "2467 misogynist-incels \n",
+ "2439 james-lindsay \n",
+ "1535 jeff-berry \n",
+ "49 harry-cooper \n",
+ "535 david-irving \n",
+ "175 jack-posobiec \n",
+ "1464 willis-carto \n",
+ "405 proud-boys \n",
+ "846 greg-johnson \n",
+ "598 atomwaffen-division \n",
+ "2493 mark-weber \n",
+ "1052 gays-against-groomers \n",
+ "2485 john-de-nugent \n",
+ "203 andrew-anglin \n",
+ "1271 lieutenant-general-william-g-jerry-boykin-ret \n",
+ "297 alt-right \n",
+ "453 garrett-hardin \n",
+ "367 alex-jones \n",
+ "1055 gays-against-groomers \n",
+ "237 chuck-baldwin \n",
+ "1553 craig-cobb \n",
+ "677 family-research-council \n",
"\n",
- " paragraph \n",
- "617 Washington Times \n",
- "927 â “New World Order,” 1999 essay by Beam \n",
- "942 In August 2004, White was convicted of assaulting a woman when she was handing out flyers that identified him as a neo-Nazi landlord. During the court proceedings, White cursed at the woman from the witness stand and was held in contempt of court by the judge. In December 2009, a federal jury found White guilty of threatening several people through intimidating phone calls and internet postings. He was sentenced to 2 ½ years in prison at an April 2010 hearing. In January 2011, White was found guilty of using his website to encourage violence against a jury foreman in the trial of another white supremacist, \n",
- "973 â Pierce, quoted in a biography of him called Fame of a Dead Man’s Deeds, on the power of narrative \n",
- "1967 a reporter later exposed \n",
- "1116 Collective Guilt for Fun and Profit”, \n",
- "1375 Audience: “Jews!” \n",
- "170 Nehlenâs 2016 campaign for Congress was ideologically similar to Trumpâs and was buoyed by FOX News and other outlets with large, national audiences. Breitbart.com ran several stories on Nehlen, for example. Right-wing pundits such as \n",
- "2025 “Tucker is ultimately on our side. He can get millions and millions of boomers to nod along with talking points that would have only been seen on VDare or American Renaissance a few years ago,” Greer said on his podcast in the Spring of 2021, \n",
- "321 “Here in the state of Virginia our Confederate History Month proclamations became so watered down with nods towards the non-Whites they weren’t even worth supporting. We can’t ever have anything just for Whites…I wonder what a true Confederate (White Supremacist) brought to today’s time would think of the cowardly defenders of his cause.” \n",
- "1000 “It is not mere coincidence that the emperors of Rome in its horrific final days were homosexual; that Adolf Hitler’s inner circle were mostly homosexual; and that nearly all of the most prolific serial killers in U.S. history were homosexual. It is not mere coincidence that America’s cultural decline parallels the rise of ‘gay rights.â” \n",
- "2331 “I am not afraid to speak out about the atrocities that whites and people of European descent face not only here in this country but in Western nations across the world. The war against whites, and Europeans and Western society is very real and it’s time we all started talking about it and stopped worrying about political correctness and optics.” – Kyle Chapman, who formed the Fraternal Order of Alt-Knights, a paramilitary wing of the Proud Boys, Unite America First Peace Rally, Sacramento, California, July 8, 2017 \n",
- "408 âGender dysphoria is a mental health disorder that is being normalized by predators across the USA. California kids are at extreme risk from predatory adults. Now they want to âliberateâ children all over the country. Does a double mastectomy on a preteen sound like progress?â \n",
- "651 American Renaissance \n",
- "1614 Postmortem Report: Cultural Examinations from Postmodernity \n",
- "1135 “If we could just get people to be nice to their babies for five years straight, that would be it for war, drug abuse, addiction, promiscuity, sexually transmitted diseases. Almost all would be completely eliminated, because they all arise from dysfunctional early childhood experiences, which are all run by women.” \n",
- "582 (CCC), which Charleston shooter \n",
- "1961 served as a platform \n",
- "1640 “We the people are proud to proclaim that the United States of America is ‘One Nation under God’ – in this public profession of faith in God, we recognize his Lordship over our country, and we proudly stand beneath the banner of Christ and our flag in which millions have sacrificed their very lives for. In scripture through the strength and commitment of Matthew, he said, ‘Whoever is not with Me is against Me.’” – Blog post titled \n",
- "1073 “P—- is the only real empowerment women will ever know. Put all the hopelessly wishful thinking of feminist ideology aside and what remains is the fact that it is men, and pretty much men only, who draw power from accomplishment, who invent technology, build nations, cure disease, create empires and generally advance civilization. Women – whether acknowledging it makes us feel warm and fuzzy or not – depend on men for all of that and the only tool they have at their disposal to have any sort of influence on any of it is the power of p—- and p—- is powerful indeed…Sexual robotics may well prove to be the best thing that ever happened to women from the standpoint of their humanity…. what would that do to the vast majority of women who would suddenly have to prove their worth as human beings beyond simply being the owners of said p—-?” – Paul Elam, An Ear for Men, Sex Robots: Part 3 – Disempowering P—-, October 2017 \n",
- "1582 Despite making completely unsubstantiated accusations of pedophilia, Cernovich discussed the allegations that aspiring Alabama senator Roy Moore had sexually abused underage girls cautiously. After tweeting that “If it’s true, string the guy up, man. I got no problem with that,” he later retweeted individuals casting doubt on the validity of the accusers, writing: “When you’re lied about in the news daily, as I am, you pause when 40-year-old accusations surface one month away from an election where WaPo endorsed the other candidate.” Taking it further, however, Cernovich then recorded a podcast giving “scientific” advice to Roy Moore, \n",
- "907 âPedophilia and sexual perversion institutionalized in Hollywood and the entertainment industries can be traced to Talmudic principles and Jewish influence. Now Jewish influence â satanic influence under the name of Jew…The wicked practices that govern their industries are largely justified and influenced by such Talmudic principles. The pervasive rape culture, Hollywoodâs casting couch, sex trafficking and prostitution, the age-old buck breaking process that emasculates Black men and corrupts Black women.â \n",
- "613 No such reprimand occurred when Steinlight, \n",
- "2039 the digital publication. \n",
- "962 “Without a pressure release valve, as open racism once provided, an explosion of epic proportions at some time in the future is guaranteed. There will be a race war, the initial skirmishes of which already are being fought in America’s streets, that will bring an altogether new meaning to the concepts of race war and genocide, courtesy of those who claim to abhor racism.” \n",
- "2060 Tucker Carlson hosted male supremacist Andrew Tate on August 5, 2022. (Twitter) \n",
- "1627 “I am not in jail for violating the law. I am in jail because I stood for righteousness and truth in government. Pray that God will crumble the foundations and break the power and strength of the corporation and restore his righteous government in America.” \n",
- "2186 In addition to spreading disinformation about transgender identity and discredited pseudoscience, Walsh’s film relies on propagandist tactics of narrative manipulation in suggesting legal protections for transgender people will lead to people being prosecuted for using the wrong pronouns. And in making numerous \n",
- "866 â A 1998 comment to a Virginia newspaper \n",
- "471 â âOffensive and Defensive Lawfare: Fighting Civilization Jihad in Americaâs Courts,â Center for Security Policy press release, "
+ " paragraph \n",
+ "51 “We’re going to be back here, and we’re going to humiliate all of these people who opposed us. We’ll be back here 1,000 times if necessary. I always win. Because I have the will to win, I keep going until I win.” \n",
+ "168 â[Islam] is in fact a religion of war, violence, intolerance, and physical persecution of non-Muslims.â \n",
+ "2468 “Jews heavily pushed feminism. Roasties have such high standards because they are ‘liberated’. You probably would have gotten your d*** wet by now if it weren’t for feminism and for Jews.” \n",
+ "1040 â Quoted in the \n",
+ "422 âSuppose, as may well be the case, that one of these races is naturally courageous, self-sacrificing and enterprising and the other less so. The group will continue to prosper owing to the activities of inventors and explorers of the first race, who, as is generally the rule, will not pass on the usual number of children to the next generation. The nation will be successful in war because the same race has actively responded to the call to arms and to self-sacrifice. Throughout these activities, this first race will on an average be giving more to the group than it can itself recoup. Eventually only the second race will inherit the group advantages acquired largely by the first racial compound. Then like a huge parasite which has devoured its host, will the nation be bereft of all the qualities that gave it power, remain a monstrous frustration of evolution, a biological abortion able in virtue of its inherited wealth, to do untold damage to neighboring races naturally more capable. The hatred and abhorrence which many peoples feel for the Jewish (and to some extent Mongolian) practice of living in other nations, instead of forming an independent, self-sustained group of their own, comes from a deep intuitive feeling that somehow it is not âplaying the game.â Because our unbiologically-minded civilization cannot perceive or appreciate any intellectual causes for these feelings they are readily branded as âprejudiceâ by would-be intellectuals.â \n",
+ "1479 Polygamy? Sure. \n",
+ "585 â Headline for 2008 article about then-presidential candidate Barack Obama who was depicted in the middle of a rifle scope on the cover of White’s American National Socialist Workers Party’s magazine \n",
+ "220 âAs the President of the VDARE Foundation, I work alongside my husband, Editor of VDARE.com, Peter Brimelow, to defeat the post-1964 Immigration Act disaster that has set America on a course of destruction.â \n",
+ "2467 – Comment on an incel forum post titled “‘They are animals…’ ER was right,” \n",
+ "2439 If anyone causes one of these little onesâthose who believe in meâto stumble, it would be better for them to have a large millstone hung around their neck and to be drowned in the depths of the sea.â) \n",
+ "1535 â 1998 speech at a Klan rally in Jasper, Texas, after the truck-dragging murder there of James Byrd Jr. \n",
+ "49 “We must remember that official government sources state definitely that Hitler committed suicide in the bunker and that the body found not far from the Chancellery was indeed that of Martin Bormann. … Naturally we must believe what the governments tell us, like John F. Kennedy was shot by one madman using a worthless rifle when nobody with any sense buys that story except the Warren Commission. … The list of lies by the world’s governments could by themselves, fill a major size book but you already know that.” \n",
+ "535 â 1991 speech to a Canadian audience \n",
+ "175 “StoptheSteal 2020 is coming.” – Twitter, Sept. 7, 2020, a full two months before news outlets declared Joe Biden the winner of the 2020 presidential election \n",
+ "1464 The Barnes Review \n",
+ "405 “All the heroes of BLM and Antifa are degenerate criminal lowlifes or pedophile rapists. I don’t lose any sleep when they are justly removed from society.” – A Telegram channel associated with the Proud Boys, Sept. 22, 2020 \n",
+ "846 — “Dealing with the Holocaust \n",
+ "598 âPro f—– propaganda is working! AIDS is spreading like wildfire. Dead f—— couldnât make us happier! Hail AIDS!â â Atomwaffen Division website \n",
+ "2493 “Around the world awareness is growing that the âHolocaust’ campaign is a major weapon in the Jewish-Zionist arsenal, that it is used to justify otherwise unjustifiable Israeli policies, and as a powerful tool for blackmailing enormous sums of money from Americans and Europeans.” \n",
+ "1052 “The modern trans movement is radicalizing activists into terrorists. […] Trans terrorism is on the rise. […] This sends a message: if you don’t let us mutilate children’s genitals, then we will shoot yours.” – GAG then-Secretary David Leatherwood, “Real America with Dan Ball,” in response to the Covenant School shooting in Nashville, Tennessee, \n",
+ "2485 âThey [the Jews] control the media and so it spews lies against the enemy they fear most. And the Jews now misrule our beloved country of America, bringing it down by huge bribes, media brainwashing and blackmail of our officials, and police, and military officers both high and low. And worst of all, by a national network of organized pedophiles, child molesting and child-murdering adults who once involved in that network, one orgy at a time, are photographed there, captured forever on film. That film is stored away for future blackmail use. Those officials cannot quit even if they want to.â \n",
+ "203 “Look, I hate women. I think they deserve to be beaten, raped and locked in cages.” – Daily Stomer, July 2018 \n",
+ "1271 “If we don’t take a stand here, it’s only a matter of time before biological men will be able to share showers and locker rooms with women and girls all across America! … Many on the Left believe that biological men have a right to share showers, locker rooms, and bathrooms with women and little girls as long as they identify as female at that particular point in time — and that anyone who disagrees with that should be punished!” \n",
+ "297 “Immigration is a kind of proxy war—and maybe a last stand—for White Americans, who are undergoing a painful recognition that, unless dramatic action is taken, their grandchildren will live in a country that is alien and hostile.” \n",
+ "453 —“How Diversity Should be Nurtured,” \n",
+ "367 âImagine how bad she [Hillary Clinton] smells, man? Iâm told her and Obama, just stink, stink, stink, stink. You canât wash that evil off, man. Told thereâs a rotten smell around Hillary. Iâm not kidding, people say, they say â folks, Iâve been told this by high-up folks. They say listen, Obama and Hillary both smell like sulfur. ⦠Iâve talked to people that are in protective details, theyâre scared of her. And they say listen, sheâs a frickinâ demon and she stinks and so does Obama. I go, like what? Sulfur. They smell like hell.â â âThe Alex Jones Show,â Oct. 10, 2016 \n",
+ "1055 “Every year, THOUSANDS of children are exposed to displays of sexual depravity at Pride events, and MILLIONS applaud it. This year, Gays Against Groomers will not stand for this destruction of childhood innocence. Be seeing you REAL soon.” – GAG Illinois X (formerly Twitter), \n",
+ "237 âI believe homosexuality is moral perversion and deserves no special consideration under the law. ⦠I believe the South was right in the War Between the States, and I am not a racist.â \n",
+ "1553 âNov. 16, 2013, videotaped rant against a resident while âpatrollingâ the streets of Leith \n",
+ "677 “The reality is, homosexuals have entered the Scouts in the past for predatory purposes.” "
]
},
"metadata": {},
@@ -563,167 +549,162 @@
"output_type": "stream",
"text": [
"\n",
- "Number of unique extremists: 159\n",
+ "Number of unique extremists: 154\n",
"Unique extremist quotes:\n",
- "tucker-carlson 324\n",
- "center-immigration-studies 286\n",
- "matt-walsh 152\n",
- "mike-cernovich 123\n",
- "paul-nehlen 114\n",
- "stefan-molyneux 28\n",
- "greg-johnson 24\n",
- "robert-spencer 24\n",
- "gays-against-groomers 24\n",
- "jack-posobiec 23\n",
- "james-lindsay 23\n",
- "veterans-patrol 22\n",
- "roy-moore 22\n",
- "alex-jones 21\n",
- "faith-education-commerce 20\n",
- "moms-liberty 19\n",
- "act-america 19\n",
- "august-kreis 19\n",
- "family-research-council 19\n",
- "pamela-geller 18\n",
- "lieutenant-general-william-g-jerry-boykin-ret 17\n",
- "proud-boys 17\n",
- "david-yerushalmi 17\n",
- "richard-bertrand-spencer 16\n",
- "frazier-glenn-miller 15\n",
- "malik-zulu-shabazz 15\n",
- "pickup-artists-alpha-males-self-help 15\n",
- "david-duke 15\n",
- "roger-pearson 15\n",
- "william-h-regnery-ii 15\n",
- "center-security-policy 15\n",
- "oath-keepers 14\n",
- "joseph-francis-farah 14\n",
- "michael-levin 14\n",
- "cody-rutledge-wilson 14\n",
- "elmer-stewart-rhodes 14\n",
- "michael-flynn 14\n",
- "david-lane 14\n",
- "nation-islam 13\n",
- "thomas-rousseau 13\n",
- "matt-hale 12\n",
- "louis-farrakhan 12\n",
- "david-irving 12\n",
- "scott-lively 12\n",
- "peter-brimelow 12\n",
- "base 12\n",
- "john-tanton 12\n",
- "andrew-anglin 12\n",
- "bill-white 11\n",
- "andrew-weev-auernheimer 11\n",
- "charles-murray 11\n",
- "nick-fuentes 11\n",
- "male-supremacy 11\n",
- "misogynist-incels 11\n",
- "garrett-hardin 11\n",
- "ernst-zundel 11\n",
- "don-black 11\n",
- "william-shockley 11\n",
- "tom-metzger 10\n",
- "matthew-heimbach 10\n",
- "larry-klayman 10\n",
- "vdare 10\n",
- "ray-redfeairn 10\n",
- "understanding-threat 10\n",
- "james-mason 10\n",
- "tomislav-sunic 10\n",
- "linda-gottfredson 10\n",
- "fred-phelps 10\n",
- "michael-hill 10\n",
- "raymond-cattell 10\n",
- "paul-mullet 10\n",
- "lydia-brimelow 10\n",
- "asatru-folk-assembly 9\n",
- "paul-elam 9\n",
- "alt-right 9\n",
- "michael-ralph-tubbs 9\n",
- "jeff-berry 9\n",
- "david-barton 9\n",
- "craig-cobb 9\n",
- "barnes-reviewfoundation-economic-liberty-inc 9\n",
- "kyle-bristow 9\n",
- "krisanne-hall 9\n",
- "jean-philippe-rushton 9\n",
- "louis-beam 9\n",
- "edgar-steele 9\n",
- "tim-wildmon 8\n",
- "richard-butler 8\n",
- "sam-francis 8\n",
- "bo-gritz 8\n",
- "tony-perkins 8\n",
- "dan-stein 8\n",
- "nathan-benjamin-damigo 8\n",
- "daryush-roosh-valizadeh 8\n",
- "bradley-dean-griffin 8\n",
- "nationalist-social-club-nsc-131 8\n",
- "remembrance-project 8\n",
- "henry-harpending 8\n",
- "kevin-strom 8\n",
- "bryan-fischer 8\n",
- "barry-black 8\n",
- "paul-ray-ramsey 8\n",
- "kevin-macdonald 8\n",
- "frank-gaffney-jr 8\n",
- "hal-turner 8\n",
- "richard-lynn 7\n",
- "jared-taylor 7\n",
- "billy-roper 7\n",
- "james-timothy-turner 7\n",
- "johnny-monoxide-aka-john-ramondetta 7\n",
- "atomwaffen-division 7\n",
- "willis-carto 7\n",
- "arthur-jensen 7\n",
- "chuck-baldwin 7\n",
- "aryan-brotherhood 7\n",
- "wayne-lutton 6\n",
- "american-freedom-party 6\n",
- "alex-linder 6\n",
- "chaya-raichik 6\n",
- "thomas-robb 6\n",
- "boyd-cathey 6\n",
- "proenglish 6\n",
- "gary-demar 6\n",
- "gary-gerhard-lauck 6\n",
- "ronald-doggett 6\n",
- "radical-hebrew-israelites 6\n",
- "michael-enoch-peinovich 6\n",
- "stephen-miller 6\n",
- "committee-open-debate-holocaust 5\n",
- "michael-brian-vanderboegh-0 5\n",
- "virginia-abernethy 5\n",
- "april-gaede 5\n",
- "larry-pratt 5\n",
- "federation-american-immigration-reform 5\n",
- "jeff-schoep 5\n",
- "jamie-kelso 5\n",
- "james-edwards 5\n",
- "william-pierce 5\n",
- "paul-cameron 5\n",
- "kyle-rogers 5\n",
- "james-wickstrom 5\n",
- "glenn-spencer 4\n",
- "barbara-coe 4\n",
- "shaun-walker 4\n",
- "william-daniel-johnson 4\n",
- "ron-edwards 4\n",
- "john-de-nugent 4\n",
- "erich-gliebe 4\n",
- "mark-weber 4\n",
- "aryan-freedom-network 4\n",
- "james-orien-allsup 4\n",
- "kevin-lamb 4\n",
- "paul-fromm 3\n",
- "roan-garcia-quintana 3\n",
- "cliff-kincaid 3\n",
- "tom-deweese 3\n",
- "jt-ready 3\n",
- "vincent-bertollini 3\n",
- "gordon-baum 3\n",
- "harry-cooper 2\n",
+ "stefan-molyneux 28\n",
+ "robert-spencer 24\n",
+ "gays-against-groomers 24\n",
+ "greg-johnson 24\n",
+ "jack-posobiec 23\n",
+ "james-lindsay 23\n",
+ "veterans-patrol 22\n",
+ "roy-moore 22\n",
+ "alex-jones 21\n",
+ "faith-education-commerce 20\n",
+ "family-research-council 19\n",
+ "august-kreis 19\n",
+ "act-america 19\n",
+ "moms-liberty 19\n",
+ "pamela-geller 18\n",
+ "proud-boys 17\n",
+ "lieutenant-general-william-g-jerry-boykin-ret 17\n",
+ "david-yerushalmi 17\n",
+ "richard-bertrand-spencer 16\n",
+ "pickup-artists-alpha-males-self-help 15\n",
+ "david-duke 15\n",
+ "frazier-glenn-miller 15\n",
+ "center-security-policy 15\n",
+ "malik-zulu-shabazz 15\n",
+ "roger-pearson 15\n",
+ "william-h-regnery-ii 15\n",
+ "joseph-francis-farah 14\n",
+ "oath-keepers 14\n",
+ "elmer-stewart-rhodes 14\n",
+ "michael-levin 14\n",
+ "david-lane 14\n",
+ "cody-rutledge-wilson 14\n",
+ "michael-flynn 14\n",
+ "nation-islam 13\n",
+ "thomas-rousseau 13\n",
+ "matt-hale 12\n",
+ "david-irving 12\n",
+ "john-tanton 12\n",
+ "base 12\n",
+ "louis-farrakhan 12\n",
+ "peter-brimelow 12\n",
+ "andrew-anglin 12\n",
+ "scott-lively 12\n",
+ "bill-white 11\n",
+ "garrett-hardin 11\n",
+ "misogynist-incels 11\n",
+ "nick-fuentes 11\n",
+ "charles-murray 11\n",
+ "don-black 11\n",
+ "william-shockley 11\n",
+ "male-supremacy 11\n",
+ "ernst-zundel 11\n",
+ "andrew-weev-auernheimer 11\n",
+ "vdare 10\n",
+ "raymond-cattell 10\n",
+ "james-mason 10\n",
+ "linda-gottfredson 10\n",
+ "matthew-heimbach 10\n",
+ "understanding-threat 10\n",
+ "ray-redfeairn 10\n",
+ "tom-metzger 10\n",
+ "paul-mullet 10\n",
+ "lydia-brimelow 10\n",
+ "tomislav-sunic 10\n",
+ "larry-klayman 10\n",
+ "fred-phelps 10\n",
+ "michael-hill 10\n",
+ "louis-beam 9\n",
+ "paul-elam 9\n",
+ "asatru-folk-assembly 9\n",
+ "michael-ralph-tubbs 9\n",
+ "craig-cobb 9\n",
+ "barnes-reviewfoundation-economic-liberty-inc 9\n",
+ "edgar-steele 9\n",
+ "jean-philippe-rushton 9\n",
+ "alt-right 9\n",
+ "kyle-bristow 9\n",
+ "david-barton 9\n",
+ "jeff-berry 9\n",
+ "krisanne-hall 9\n",
+ "bo-gritz 8\n",
+ "nathan-benjamin-damigo 8\n",
+ "tim-wildmon 8\n",
+ "richard-butler 8\n",
+ "remembrance-project 8\n",
+ "barry-black 8\n",
+ "daryush-roosh-valizadeh 8\n",
+ "frank-gaffney-jr 8\n",
+ "bradley-dean-griffin 8\n",
+ "dan-stein 8\n",
+ "kevin-strom 8\n",
+ "bryan-fischer 8\n",
+ "kevin-macdonald 8\n",
+ "paul-ray-ramsey 8\n",
+ "henry-harpending 8\n",
+ "sam-francis 8\n",
+ "nationalist-social-club-nsc-131 8\n",
+ "tony-perkins 8\n",
+ "hal-turner 8\n",
+ "willis-carto 7\n",
+ "arthur-jensen 7\n",
+ "atomwaffen-division 7\n",
+ "billy-roper 7\n",
+ "james-timothy-turner 7\n",
+ "johnny-monoxide-aka-john-ramondetta 7\n",
+ "richard-lynn 7\n",
+ "jared-taylor 7\n",
+ "chuck-baldwin 7\n",
+ "aryan-brotherhood 7\n",
+ "american-freedom-party 6\n",
+ "chaya-raichik 6\n",
+ "thomas-robb 6\n",
+ "gary-gerhard-lauck 6\n",
+ "radical-hebrew-israelites 6\n",
+ "boyd-cathey 6\n",
+ "ronald-doggett 6\n",
+ "stephen-miller 6\n",
+ "michael-enoch-peinovich 6\n",
+ "gary-demar 6\n",
+ "alex-linder 6\n",
+ "proenglish 6\n",
+ "wayne-lutton 6\n",
+ "michael-brian-vanderboegh-0 5\n",
+ "jamie-kelso 5\n",
+ "kyle-rogers 5\n",
+ "virginia-abernethy 5\n",
+ "james-wickstrom 5\n",
+ "jeff-schoep 5\n",
+ "william-pierce 5\n",
+ "paul-cameron 5\n",
+ "larry-pratt 5\n",
+ "committee-open-debate-holocaust 5\n",
+ "federation-american-immigration-reform 5\n",
+ "james-edwards 5\n",
+ "april-gaede 5\n",
+ "john-de-nugent 4\n",
+ "erich-gliebe 4\n",
+ "glenn-spencer 4\n",
+ "shaun-walker 4\n",
+ "aryan-freedom-network 4\n",
+ "kevin-lamb 4\n",
+ "james-orien-allsup 4\n",
+ "william-daniel-johnson 4\n",
+ "barbara-coe 4\n",
+ "ron-edwards 4\n",
+ "mark-weber 4\n",
+ "roan-garcia-quintana 3\n",
+ "vincent-bertollini 3\n",
+ "tom-deweese 3\n",
+ "cliff-kincaid 3\n",
+ "gordon-baum 3\n",
+ "jt-ready 3\n",
+ "paul-fromm 3\n",
+ "harry-cooper 2\n",
"Name: filename, dtype: int64\n"
]
}
@@ -753,7 +734,7 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 6,
"id": "81c31a4a",
"metadata": {},
"outputs": [
@@ -762,7 +743,7 @@
"output_type": "stream",
"text": [
"Max tokens in any paragraph: 322\n",
- "Average tokens per paragraph: 42.7\n",
+ "Average tokens per paragraph: 49.7\n",
"Number of paragraphs exceeding 512 tokens: 0\n",
"\n",
"Longest paragraph (322 tokens):\n",
@@ -789,7 +770,7 @@
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 7,
"id": "609a3723",
"metadata": {},
"outputs": [
@@ -797,7 +778,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "Segmented dataset shape: (2515, 4)\n"
+ "Segmented dataset shape: (1516, 4)\n"
]
},
{
@@ -829,37 +810,37 @@
" \n",
" \n",
" \n",
- " | 617 | \n",
- " center-immigration-studies | \n",
- " washington times | \n",
+ " 51 | \n",
+ " richard-bertrand-spencer | \n",
+ " “ we ’ re going to be back here, and we ’ re going to humiliate all of these people who opposed us. we ’ ll be back here 1, 000 times if necessary. i always win. because i have the will to win, i keep going until i win. ” | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 927 | \n",
- " louis-beam | \n",
- " a “ new world order, ” 1999 essay by beam | \n",
+ " 168 | \n",
+ " tim-wildmon | \n",
+ " a [ islam ] is in fact a religion of war, violence, intolerance, and physical persecution of non - muslims. a | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 942 | \n",
- " bill-white | \n",
- " in august 2004, white was convicted of assaulting a woman when she was handing out flyers that identified him as a neo - nazi landlord. during the court proceedings, white cursed at the woman from the witness stand and was held in contempt of court by the judge. in december 2009, a federal jury found white guilty of threatening several people through intimidating phone calls and internet postings. he was sentenced to 2 a½ years in prison at an april 2010 hearing. in january 2011, white was f... | \n",
+ " 1469 | \n",
+ " misogynist-incels | \n",
+ " “ jews heavily pushed feminism. roasties have such high standards because they are ‘ liberated ’. you probably would have gotten your d * * * wet by now if it weren ’ t for feminism and for jews. ” | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 973 | \n",
- " william-pierce | \n",
- " a pierce, quoted in a biography of him called fame of a dead man ’ s deeds, on the power of narrative | \n",
+ " 926 | \n",
+ " kyle-bristow | \n",
+ " a quoted in the | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 1967 | \n",
- " tucker-carlson | \n",
- " a reporter later exposed | \n",
+ " 422 | \n",
+ " raymond-cattell | \n",
+ " asuppose, as may well be the case, that one of these races is naturally courageous, self - sacrificing and enterprising and the other less so. the group will continue to prosper owing to the activities of inventors and explorers of the first race, who, as is generally the rule, will not pass on the usual number of children to the next generation. the nation will be successful in war because the same race has actively responded to the call to arms and to self - sacrifice. throughout these act... | \n",
" 0 | \n",
" 1 | \n",
"
\n",
@@ -871,37 +852,37 @@
" ... | \n",
" \n",
" \n",
- " | 866 | \n",
- " barry-black | \n",
- " a a 1998 comment to a virginia newspaper | \n",
+ " 1164 | \n",
+ " craig-cobb | \n",
+ " anov. 16, 2013, videotaped rant against a resident while apatrollinga the streets of leith | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 471 | \n",
- " david-yerushalmi | \n",
- " a aoffensive and defensive lawfare : fighting civilization jihad in americaas courts, a center for security policy press release, | \n",
+ " 677 | \n",
+ " family-research-council | \n",
+ " “ the reality is, homosexuals have entered the scouts in the past for predatory purposes. ” | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 1174 | \n",
- " kevin-strom | \n",
- " “ the aryan race, by dint of its intelligence and creativity and character has managed to drag itself up to a state of civilization and some degree of scientific understanding of the universe about us. but what dr. pierce could clearly see, and what the more jingoistic racialists cannot see, is that that state of civilization is but a few inches above the slime of universal savagery. a¦ the journey has just begun and the danger of falling back is very great. ” | \n",
+ " 123 | \n",
+ " base | \n",
+ " “ most of our members are national socialists and / or fascists, although we also have some run - of - the - mill white nationalists … we have a strong revolutionary and militant current running through the base. most of our members are pretty hardcore in that sense. you ’ re going to be stepping into probably the most extreme group of pro - white people that you can probably come across. ” – | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 56 | \n",
- " david-barton | \n",
- " “ and we don ’ t want racism in america. but then as you started watching, that ’ s not what this was about. this was about a hate america movement. ” in reference to cancel culture, the 1619 project, and covid - era anti - racism protests. – the elephant heard podcast, april 27, 2021 | \n",
+ " 706 | \n",
+ " scott-lively | \n",
+ " “ the gay movement is an evil institution [ whose ] goal is to defeat the marriage - based society and replace it with a culture of sexual promiscuity in which thereas no restrictions on sexual conduct except the principle of mutual choice. ” | \n",
" 0 | \n",
" 1 | \n",
"
\n",
" \n",
- " | 1457 | \n",
- " barnes-reviewfoundation-economic-liberty-inc | \n",
- " ajewish involvement in the communist revolution by john wear, j. d. radical jewish activists, political instigators, and criminals were the main protagonists of the arussiana revolution, which overthrew the tzar, resulting in the brutal murder of the royal family and millions of gentiles. john wear highlights and documents the overwhelming evidence that jews played the leading role in this tragedy, using a variety of sources. a | \n",
+ " 602 | \n",
+ " jean-philippe-rushton | \n",
+ " “ whites have, on average, more neurons and cranial size than blacks … blacks have an advantage in sport because they have narrower hips — but they have narrower hips because they have smaller brains. ” | \n",
" 0 | \n",
" 1 | \n",
"
\n",
@@ -911,44 +892,44 @@
""
],
"text/plain": [
- " filename \\\n",
- "617 center-immigration-studies \n",
- "927 louis-beam \n",
- "942 bill-white \n",
- "973 william-pierce \n",
- "1967 tucker-carlson \n",
- "... ... \n",
- "866 barry-black \n",
- "471 david-yerushalmi \n",
- "1174 kevin-strom \n",
- "56 david-barton \n",
- "1457 barnes-reviewfoundation-economic-liberty-inc \n",
+ " filename \\\n",
+ "51 richard-bertrand-spencer \n",
+ "168 tim-wildmon \n",
+ "1469 misogynist-incels \n",
+ "926 kyle-bristow \n",
+ "422 raymond-cattell \n",
+ "... ... \n",
+ "1164 craig-cobb \n",
+ "677 family-research-council \n",
+ "123 base \n",
+ "706 scott-lively \n",
+ "602 jean-philippe-rushton \n",
"\n",
" paragraph_segment \\\n",
- "617 washington times \n",
- "927 a “ new world order, ” 1999 essay by beam \n",
- "942 in august 2004, white was convicted of assaulting a woman when she was handing out flyers that identified him as a neo - nazi landlord. during the court proceedings, white cursed at the woman from the witness stand and was held in contempt of court by the judge. in december 2009, a federal jury found white guilty of threatening several people through intimidating phone calls and internet postings. he was sentenced to 2 a½ years in prison at an april 2010 hearing. in january 2011, white was f... \n",
- "973 a pierce, quoted in a biography of him called fame of a dead man ’ s deeds, on the power of narrative \n",
- "1967 a reporter later exposed \n",
+ "51 “ we ’ re going to be back here, and we ’ re going to humiliate all of these people who opposed us. we ’ ll be back here 1, 000 times if necessary. i always win. because i have the will to win, i keep going until i win. ” \n",
+ "168 a [ islam ] is in fact a religion of war, violence, intolerance, and physical persecution of non - muslims. a \n",
+ "1469 “ jews heavily pushed feminism. roasties have such high standards because they are ‘ liberated ’. you probably would have gotten your d * * * wet by now if it weren ’ t for feminism and for jews. ” \n",
+ "926 a quoted in the \n",
+ "422 asuppose, as may well be the case, that one of these races is naturally courageous, self - sacrificing and enterprising and the other less so. the group will continue to prosper owing to the activities of inventors and explorers of the first race, who, as is generally the rule, will not pass on the usual number of children to the next generation. the nation will be successful in war because the same race has actively responded to the call to arms and to self - sacrifice. throughout these act... \n",
"... ... \n",
- "866 a a 1998 comment to a virginia newspaper \n",
- "471 a aoffensive and defensive lawfare : fighting civilization jihad in americaas courts, a center for security policy press release, \n",
- "1174 “ the aryan race, by dint of its intelligence and creativity and character has managed to drag itself up to a state of civilization and some degree of scientific understanding of the universe about us. but what dr. pierce could clearly see, and what the more jingoistic racialists cannot see, is that that state of civilization is but a few inches above the slime of universal savagery. a¦ the journey has just begun and the danger of falling back is very great. ” \n",
- "56 “ and we don ’ t want racism in america. but then as you started watching, that ’ s not what this was about. this was about a hate america movement. ” in reference to cancel culture, the 1619 project, and covid - era anti - racism protests. – the elephant heard podcast, april 27, 2021 \n",
- "1457 ajewish involvement in the communist revolution by john wear, j. d. radical jewish activists, political instigators, and criminals were the main protagonists of the arussiana revolution, which overthrew the tzar, resulting in the brutal murder of the royal family and millions of gentiles. john wear highlights and documents the overwhelming evidence that jews played the leading role in this tragedy, using a variety of sources. a \n",
+ "1164 anov. 16, 2013, videotaped rant against a resident while apatrollinga the streets of leith \n",
+ "677 “ the reality is, homosexuals have entered the scouts in the past for predatory purposes. ” \n",
+ "123 “ most of our members are national socialists and / or fascists, although we also have some run - of - the - mill white nationalists … we have a strong revolutionary and militant current running through the base. most of our members are pretty hardcore in that sense. you ’ re going to be stepping into probably the most extreme group of pro - white people that you can probably come across. ” – \n",
+ "706 “ the gay movement is an evil institution [ whose ] goal is to defeat the marriage - based society and replace it with a culture of sexual promiscuity in which thereas no restrictions on sexual conduct except the principle of mutual choice. ” \n",
+ "602 “ whites have, on average, more neurons and cranial size than blacks … blacks have an advantage in sport because they have narrower hips — but they have narrower hips because they have smaller brains. ” \n",
"\n",
" segment_id label \n",
- "617 0 1 \n",
- "927 0 1 \n",
- "942 0 1 \n",
- "973 0 1 \n",
- "1967 0 1 \n",
+ "51 0 1 \n",
+ "168 0 1 \n",
+ "1469 0 1 \n",
+ "926 0 1 \n",
+ "422 0 1 \n",
"... ... ... \n",
- "866 0 1 \n",
- "471 0 1 \n",
- "1174 0 1 \n",
- "56 0 1 \n",
- "1457 0 1 \n",
+ "1164 0 1 \n",
+ "677 0 1 \n",
+ "123 0 1 \n",
+ "706 0 1 \n",
+ "602 0 1 \n",
"\n",
"[33 rows x 4 columns]"
]
@@ -986,18 +967,15 @@
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 8,
"id": "d5092eff-7e7d-4129-8e01-bbf9b28ee016",
"metadata": {},
"outputs": [
{
- "name": "stderr",
+ "name": "stdout",
"output_type": "stream",
"text": [
- "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
- "To disable this warning, you can either:\n",
- "\t- Avoid using `tokenizers` before the fork if possible\n",
- "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
+ "Running with: ~/workspace/Sentinel/.venv/bin/python\n"
]
},
{
@@ -1005,23 +983,11 @@
"output_type": "stream",
"text": [
"Loading neutral podcast dataset directly from Hugging Face...\n",
- "Using cached file at /Users/evonck/.cache/huggingface/neutral/lex-fridman-podcastUsing cached file.parquet\n",
+ "Using cached file at ~/.cache/huggingface/neutral/lex-fridman-podcastUsing cached file.parquet\n",
"Successfully loaded data with 346 rows\n",
- "Processed 346 episodes with segments\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Extracting segments: 100%|██████████| 316/316 [00:00<00:00, 3474.10it/s]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Collected 750250 segments before sampling\n",
+ "Processed 346 episodes with segments\n",
+ "Extracting segments: 100%|█████████████████| 316/316 [00:00<00:00, 3296.62it/s]\n",
+ "Collected 761946 segments before sampling\n",
"Randomly sampled 15000 segments\n",
"Saved 15000 training segments to neutral-segments-training.csv\n",
"Saved 30 evaluation episodes to neutral-episodes-eval.parquet\n"
@@ -1029,26 +995,48 @@
}
],
"source": [
- "%%bash\n",
- "# Change directory into the 'scripts' folder\n",
- "cd scripts\n",
- "\n",
- "# Run your Python script using poetry run\n",
- "# This ensures all your Poetry-managed dependencies are available\n",
- "poetry run python fetch-neutral-examples-data.py\n",
- "\n",
- "# The script will create two files:\n",
- "# 1. `neutral-segments-training.csv`: Contains ~15,000 individual segments (10x our positive examples) for training\n",
- "# 2. `neutral-episodes-eval.parquet`: Contains 30 full episodes for evaluation\n",
- "# You can customize the number of segments and evaluation episodes:\n",
- "# ```bash\n",
- "# python fetch-neutral-examples-data.py -n 20000 -e 50 # 20k segments, 50 eval episodes\n",
- "# ```"
+ "# ============================================================================\n",
+ "# ORIGINAL CODE (kept for reference, commented out).\n",
+ "# It failed with \"poetry: command not found\" because a %%bash cell starts a\n",
+ "# plain non-login shell that never reads ~/.zshrc, so ~/.local/bin (where\n",
+ "# poetry is installed) is missing from PATH.\n",
+ "# ============================================================================\n",
+ "# %%bash\n",
+ "# # Change directory into the 'scripts' folder\n",
+ "# cd scripts\n",
+ "#\n",
+ "# # Run your Python script using poetry run\n",
+ "# # This ensures all your Poetry-managed dependencies are available\n",
+ "# poetry run python fetch-neutral-examples-data.py\n",
+ "#\n",
+ "# # The script will create two files:\n",
+ "# # 1. `neutral-segments-training.csv`: Contains ~15,000 individual segments (10x our positive examples) for training\n",
+ "# # 2. `neutral-episodes-eval.parquet`: Contains 30 full episodes for evaluation\n",
+ "# # You can customize the number of segments and evaluation episodes:\n",
+ "# # ```bash\n",
+ "# # python fetch-neutral-examples-data.py -n 20000 -e 50 # 20k segments, 50 eval episodes\n",
+ "# # ```\n",
+ "\n",
+ "# ============================================================================\n",
+ "# WORKING VERSION\n",
+ "# ============================================================================\n",
+ "import sys\n",
+ "\n",
+ "# This notebook's kernel already runs inside the Poetry virtualenv\n",
+ "# (Sentinel/.venv), so sys.executable has every dependency the script needs and\n",
+ "# the `poetry run` wrapper is unnecessary.\n",
+ "print(\"Running with:\", sys.executable)\n",
+ "\n",
+ "# The script writes two files into examples/data/ :\n",
+ "# 1. neutral-segments-training.csv -> ~15,000 individual segments for training\n",
+ "# 2. neutral-episodes-eval.parquet -> 30 full episodes for evaluation\n",
+ "# To change those amounts, append flags, e.g. `... -n 20000 -e 50`\n",
+ "!cd scripts && {sys.executable} fetch-neutral-examples-data.py"
]
},
{
"cell_type": "code",
- "execution_count": 8,
+ "execution_count": 9,
"id": "922bed03",
"metadata": {},
"outputs": [
@@ -1056,36 +1044,22 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "Number of positive examples: 2515\n",
- "Number of negative examples to extract: 25150\n",
+ "Number of positive examples: 1516\n",
+ "Number of negative examples to extract: 15160\n",
"Loading neutral podcast dataset directly from Hugging Face...\n",
- "Using cached file at /Users/evonck/.cache/huggingface/lex-fridman-podcast/neutral-episodes-eval.parquet\n",
+ "Using cached file at ~/.cache/huggingface/lex-fridman-podcast/neutral-episodes-eval.parquet\n",
"Successfully loaded data with 346 rows\n",
"Processed 346 episodes with segments\n",
"Dataset loaded with 346 episodes\n",
"Extracting random segments...\n"
]
},
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "4968b5e5b3804c72b40b1ab265589e5c",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Extracting segments: 0%| | 0/346 [00:00, ?it/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collected 832839 segments before sampling\n",
- "Extracted 25150 negative examples\n",
+ "Extracted 15160 negative examples\n",
"\n",
"Sample of negative examples:\n"
]
@@ -1121,35 +1095,35 @@
" \n",
" | 0 | \n",
" neutral_podcast | \n",
- " And I feel like the imagination is a really powerful tool | \n",
+ " and gave people a big massive lesson in civics. | \n",
" 0 | \n",
" 0 | \n",
"
\n",
" \n",
" | 1 | \n",
" neutral_podcast | \n",
- " Oh, that was like the smaller one, like the firefly. | \n",
+ " I'm humiliated about that kind of stuff | \n",
" 1 | \n",
" 0 | \n",
"
\n",
" \n",
" | 2 | \n",
" neutral_podcast | \n",
- " Computers weren't really great at that. | \n",
+ " I don't know. | \n",
" 2 | \n",
" 0 | \n",
"
\n",
" \n",
" | 3 | \n",
" neutral_podcast | \n",
- " to sort of pretend that there's something that they're not in order to understand what's | \n",
+ " Yeah, like it made me imagine | \n",
" 3 | \n",
" 0 | \n",
"
\n",
" \n",
" | 4 | \n",
" neutral_podcast | \n",
- " to math dot square root. | \n",
+ " but people love it when I ask it, | \n",
" 4 | \n",
" 0 | \n",
"
\n",
@@ -1158,19 +1132,12 @@
""
],
"text/plain": [
- " filename \\\n",
- "0 neutral_podcast \n",
- "1 neutral_podcast \n",
- "2 neutral_podcast \n",
- "3 neutral_podcast \n",
- "4 neutral_podcast \n",
- "\n",
- " paragraph_segment \\\n",
- "0 And I feel like the imagination is a really powerful tool \n",
- "1 Oh, that was like the smaller one, like the firefly. \n",
- "2 Computers weren't really great at that. \n",
- "3 to sort of pretend that there's something that they're not in order to understand what's \n",
- "4 to math dot square root. \n",
+ " filename paragraph_segment \\\n",
+ "0 neutral_podcast and gave people a big massive lesson in civics. \n",
+ "1 neutral_podcast I'm humiliated about that kind of stuff \n",
+ "2 neutral_podcast I don't know. \n",
+ "3 neutral_podcast Yeah, like it made me imagine \n",
+ "4 neutral_podcast but people love it when I ask it, \n",
"\n",
" segment_id label \n",
"0 0 0 \n",
@@ -1188,7 +1155,7 @@
"output_type": "stream",
"text": [
"\n",
- "Combined dataset shape: (27665, 4)\n",
+ "Combined dataset shape: (16676, 4)\n",
"Sample of combined dataset:\n"
]
},
@@ -1221,38 +1188,38 @@
" \n",
"