[TASK] update README and documentation in Settings.yaml

Author: skurfuerst

Configuration/Settings.yaml

@@ -2,10 +2,13 @@ Flowpack:
   ElasticSearch:
     ContentRepositoryAdaptor:
 
+      # API. name of the ElasticSearch index to use
       indexName: typo3cr
 
+      # we use batch indexing, in order to reduce the number of HTTP requests while indexing
       indexingBatchSize: 100
 
+      # configuration of the ElasticSearch logfile
       log:
         backendOptions:
           fileBackend:

README.md

@@ -1,11 +1,229 @@
 # TYPO3 Neos ElasticSearch Adapter
 
+Created by Sebastian Kurfürst; contributions by Karsten Dambekalns and Robert Lemke.
+
 This project connects the TYPO3 Content Repository (TYPO3CR) to ElasticSearch; enabling two
 main functionalities:
 
-* Full-Text Indexing of Pages and other Documents (of course including the full content)
 * finding Nodes in TypoScript / Eel by arbitrary queries
+* Full-Text Indexing of Pages and other Documents (of course including the full content)
+
+
+## Building up the Index
+
+The node index is updated on the fly, but during development you need to update it frequently.
+
+In case of a mapping update, you need to reindex all nodes. Don't worry to do that in production;
+the system transparently creates a new index, fills it completely, and when everything worked,
+changes the index alias.
+
+
+```
+./flow nodeindex:build
+
+ # if during development, you only want to index a few nodes, you can use "limit"
+./flow nodeindex:build --limit 20
+
+ # in order to remove old, non-used indices, you should use this command from time to time:
+./flow nodeindex:cleanup
+```
+
+
+## Doing Arbitrary Queries
+
+We'll first show how to do arbitrary ElasticSearch Queries in TypoScript. This is a more powerful
+alternative to FlowQuery. In the long run, we might be able to integrate this API back into FlowQuery,
+but for now it works well as-is.
+
+Generally, ElasticSearch queries are done using the `ElasticSearch` Eel helper. In case you want
+to retieve a *list of nodes*, you'll generally do:
+```
+nodes = ${ElasticSearch.query(site)....execute()}
+```
+
+In case you just want to retrieve a *single node*, the form of a query is as follows:
+```
+nodes = ${q(ElasticSearch.query(site)....execute()).get(0)}
+```
+
+All queries search underneath a certain subnode. In case you want to search "globally", you will
+search underneath the current site node (like in the example above).
+
+Furthermore, the following operators are supported: 
+
+* `nodeType("Your.Node:Type")`
+* `exactMatch(key, value)`; supports simple types: `exactMatch('tag', 'foo')`, or node references: `exactMatch('author', authorNode)`
+* `sortAsc('propertyName')` and `sortDesc('propertyName')` -- can also be used multiple times, e.g. `sortAsc('tag').sortDesc(`date')` will first sort by tag ascending, and then by date descending.
+* `limit(5)` -- only return five results. If not specified, the default limit by ElasticSearch applies (which is at 10 by default)
+* `from(5)` -- return the results starting from the 6th one
+
+Furthermore, there is a more low-level operator which can be used to add arbitrary ElasticSearch filters:
+
+* `queryFilter("filterType", {option1: "value1"})`
+
+In order to debug the query more easily, the following operation is helpful:
+
+* `log()` log the full query on execution into the ElasticSearch log (i.e. in `Data/Logs/ElasticSearch.log`)
+
+### Example Queries
+
+#### Finding all pages which are tagged in a special way and rendering them in an overview
+
+Use Case: On a "Tag Overview" page, you want to show all pages being tagged in a certain way
+
+Setup: You have two node types in a blog called `Acme.Blog:Post` and `Acme.Blog:Tag`, both
+inheriting from `TYPO3.Neos:Document`. The `Post` node type has a property `tags` which is
+of type `references`, pointing to `Tag` documents.
+
+TypoScript setup:
+
+```
+ # for "Tag" documents, replace the main content area.
+prototype(TYPO3.Neos:PrimaryContent).acmeBlogTag {
+	condition = ${q(node).is('[instanceof Acme.Blog:Tag]')}
+	type = 'Acme.Blog:TagPage'
+}
+
+ # The "TagPage"
+prototype(Acme.Blog:TagPage) < prototype(TYPO3.TypoScript:Collection) {
+	collection = ${ElasticSearch.query(site).nodeType('Acme.Blog:Post').exactMatch('tags', node).sortDesc('creationDate').execute()}
+	itemName = 'node'
+	itemRenderer = Acme.Blog:SingleTag
+}
+prototype(Acme.Blog:SingleTag) < prototype(TYPO3.Neos:Template) {
+	...
+}
+```
+
+
+## Fulltext Search / Indexing
+
+When searching in a fulltext index, we want to show Pages, or, generally speaking, everything
+which is a `Document` node. However, the main content of a certain `Document` is often not stored
+in the node itself, but inside its (`Content`) child nodes.
+
+This is why we need some special functionality for indexing, which *adds the content of the inner
+nodes* to the `Document` nodes where they belong to, to a field called `__fulltext` and
+`__fulltextParts`.
+
+Furthermore, we want that a fulltext match e.g. inside a headline is seen as *more important* than
+a match inside the normal body text. That's why the `Document` node not only contains one field with
+all the texts, but multiple "buckets" where text is added to: One field which contains everything
+deemed as "very important" (`__fulltext.h1`), one which is "less important" (`__fulltext.h2`),
+and finally one for the plain text (`__fulltext.text`). All of these fields add themselves to the
+ElasticSearch `_all` field, and are configured with different `boost` values.
+
+In order to search this index, you can just search inside the `_all` field with an additional limitation
+of `__typeAndSupertypes` containing `TYPO3.Neos:Document`.
+
+**Currently, this package does not contain a plugin for searching, though we might provide one lateron.**
+
+
+## Advanced: Configuration of Indexing
+
+**Normally, this does not need to be touched, as this package supports all TYPO3 Neos data types natively.**
+
+Indexing of properties is configured at two places. The defaults per-data-type are configured
+inside `Flowpack.ElasticSearch.ContentRepositoryAdaptor.defaultConfigurationPerType` of `Settings.yaml`.
+Furthermore, this can be overridden using the `properties.[....].elasticSearch` path inside 
+`NodeTypes.yaml`.
+
+This configuration contains two parts:
+
+* Underneath `mapping`, the ElasticSearch property mapping can be defined.
+* Underneath `indexing`, an Eel expression which preprocesses the value before indexing has to be 
+  specified. It has access to the current `value` and the current `node`.
+
+Example (from the default configuration):
+```
+ # Settings.yaml
+Flowpack:
+  ElasticSearch:
+    ContentRepositoryAdaptor:
+      defaultConfigurationPerType:
+      
+        # strings should, by default, not be included in the _all field; and
+        # indexing should just use their simple value.
+        string:
+          mapping:
+            type: string
+            include_in_all: false
+          indexing: '${value}'
+```
+
+```
+ # NodeTypes.yaml
+'TYPO3.Neos:Timable':
+  properties:
+    '_hiddenBeforeDateTime':
+      elasticSearch:
+
+        # a date should be mapped differently, and in this case we want to use a date format which
+        # ElasticSearch understands
+        mapping:
+          type: date
+          include_in_all: false
+          format: 'date_time_no_millis'
+        indexing: '${(node.hiddenBeforeDateTime ? node.hiddenBeforeDateTime.format("Y-m-d\TH:i:s") + "Z" : null)}'
+```
+
+There are a few indexing helpers inside the `ElasticSearch` namespace which are usable inside the 
+`indexing` expression. In most cases, you don't need to touch this, but they were needed to build up
+the standard indexing configuration:
+
+* `ElasticSearch.buildAllPathPrefixes`: for a path such as `foo/bar/baz`, builds up a list of path
+  prefixes, e.g. `['foo', 'foo/bar', 'foo/bar/baz']`.
+* `ElasticSearch.extractNodeTypeNamesAndSupertypes(NodeType)`: extracts a list of node type names for
+  the passed node type and all of its supertypes
+* `ElasticSearch.convertArrayOfNodesToArrayOfNodeIdentifiers(array $nodes)`: convert the given nodes to
+  their node identifiers.
+
+
+
+## Advanced: Fulltext Indexing
+
+In order to enable fulltext indexing, every `Document` node must be configured as *fulltext root*. Thus,
+the following is configured in the default configuration:
+
+```
+'TYPO3.Neos:Document':
+  elasticSearch:
+    fulltext:
+      isRoot: true
+```
+
+A *fulltext root* contains all the *content* of its non-document children, such that when one searches
+inside these texts, the document itself is returned as result.
+
+In order to specify how the fulltext of a property in a node should be extracted, this is configured
+in `NodeTypes.yaml` at `properties.[propertyName].elasticSearch.fulltextExtractor`.
+
+An example:
+
+```
+'TYPO3.Neos.NodeTypes:Text':
+  properties:
+    'text':
+      elasticSearch:
+        fulltextExtractor: '${ElasticSearch.fulltext.extractHtmlTags(value)}'
+
+'My.Blog:Post':
+  properties:
+    title:
+      elasticSearch:
+        fulltextExtractor: ${ElasticSearch.fulltext.extractInto('h1', value)}
+```
+
+
+## Fulltext Searching / Search Plugin
+
+There is currently no fulltext search plugin included, though we might add one lateron.
+
+
+## Debugging
 
-TODO Sebastian: this needs improved documentation!
+In order to understand what's going on, the following commands are helpful:
 
-If you need this package, ping Sebastian Kurfuerst or Karsten Dambekalns and bug them to update documentation :-)
+* use `./flow nodeindex:showMapping` to show the currently defined ElasticSearch Mapping
+* use the `.log()` statement inside queries to dump them to the ElasticSearch Log
+* the logfile `Data/Logs/ElasticSearch.log` contains loads of helpful information.