Add nlp exercise to exercise 3 (#111)

Author: williamma12

environment.yml

@@ -19,4 +19,6 @@ dependencies:
     - lxml
     - scikit-learn
     - setproctitle
+    - spacy
+    - wikipedia
     - atoma

exercises/exercise03-Nested_Remote_Functions.ipynb

@@ -191,6 +191,119 @@
    "source": [
     "ray.timeline(filename=\"timeline03.json\")"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Application: NLP Pipeline\n",
+    "\n",
+    "To demonstrate the practical applications of nested remote functions, we create a NLP pipeline that analyzes some interesting Wikipedia pages and we speed up this pipeline using nested remote functions. Though this is only a toy example, this example can easily be scaled up with very little changes! \n",
+    "\n",
+    "For this NLP pipeline, we first use the function `parse_wikipedia` to parse a Wikipedia page on a given topic. Within `parse_wikipedia`, we first tokenize the page using the `tokenize` function and then feed the tokens to `entity_recognizer`, a named entity recognizer. Not satisfied with just the named entities of each language, we also search for the Wikipedia pages of the first 10 unique named entities, recursively until we hit a recursion depth of 2. Finally, we return a pandas dataframe containing all the named entities we found.\n",
+    "\n",
+    "As an example, we parse the wikipedia entries for python, java, and c++, the languages that are used in ray!\n",
+    "\n",
+    "**NOTE:** We use the `en_core_web_sm`, a pre-trained model provided by `spacy`. Make sure you have it downloaded by running the following piece of code within a jupyter notebook cell.\n",
+    "\n",
+    "```\n",
+    "!python -m spacy download en_core_web_sm\n",
+    "```\n",
+    "After running the above code, you may need to restart the notebook kernel."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import modin.pandas as pd\n",
+    "import spacy\n",
+    "import wikipedia\n",
+    "\n",
+    "MAX_LINKS = 2\n",
+    "MAX_DEPTH = 2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def tokenize(text):\n",
+    "    time.sleep(2)\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    return nlp(text)\n",
+    "    \n",
+    "def entity_recognizer(tokens, topic):\n",
+    "    time.sleep(2)\n",
+    "    results = []\n",
+    "    for token in tokens.ents:\n",
+    "        results.append([topic, token.text, token.lemma_, token.label_])\n",
+    "    \n",
+    "    return results\n",
+    "\n",
+    "def recursive_wiki_scraper(topic, depth=0):\n",
+    "    try:\n",
+    "        wiki_page = wikipedia.page(topic)\n",
+    "    except:\n",
+    "        return []\n",
+    "    wiki_links = wiki_page.links[:MAX_LINKS]\n",
+    "\n",
+    "    page_tokens = tokenize(wiki_page.content)\n",
+    "    topic_result = entity_recognizer(page_tokens, topic)\n",
+    "    result = []\n",
+    "    \n",
+    "    if depth < MAX_DEPTH:\n",
+    "        for link in wiki_links:\n",
+    "            result.extend(recursive_wiki_scraper(link, depth+1))\n",
+    "    \n",
+    "    result.extend(topic_result)\n",
+    "    return result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now let's try and get some information on the languages that ray is built on!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "start = time.time()\n",
+    "\n",
+    "languages = [\"Python\", \"Java programming\", \"C++\"]\n",
+    "results = []\n",
+    "for lang in languages:\n",
+    "    results.extend(recursive_wiki_scraper(lang))\n",
+    "    \n",
+    "duration = time.time() - start\n",
+    "print(\"Constructing the dataframe took {} seconds.\".format(duration))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Exercise:** Speed up the above NLP pipeline using ray and its nested remote functions. To do so, it is recommended that you only make `tokenize` and `entity_recognizer` be remote functions and not the `recursive_wiki_scraper`. Try and understand why this is the case. Below you should find the a sample of the results. Feel free to explore the data to find interesting associations to the programming languages that ray is written in!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.DataFrame(results, columns=[\"topic\", \"text\", \"lemma\", \"label\"])\n",
+    "df.sample(10)"
+   ]
   }
  ],
  "metadata": {