Photon currently applies a hard-coded set of token filters:
|
final var NORMALIZATION_FILTERS = List.of( |
|
"lowercase", |
|
"asciifolding", |
|
"german_normalization" |
|
); |
It needs to be rather conservative with filtering because Photon cannot make any assumptions regarding script and languages. Already the given list is causing issues in some languages. So we'd rather want to get rid of most of the normalisations here.
However, when running Photon with a single-language dataset then more filtering and normalization for that specific language might be wanted. Ascii-normalization might or might not be wanted. Or, the posessive s problem could be handled with a pattern replacement filter.
The set of filter is normally set at import time making it a property of the database. So introducing a configurable set of filters mostly doesn't need to wait for a major release. Except, there is the synonym handling. This rewrites the index mapping. So we'd need to see how to introduce synonyms without changing the filter configuration.
Photon currently applies a hard-coded set of token filters:
photon/src/main/java/de/komoot/photon/opensearch/IndexSettingBuilder.java
Lines 169 to 173 in d151869
It needs to be rather conservative with filtering because Photon cannot make any assumptions regarding script and languages. Already the given list is causing issues in some languages. So we'd rather want to get rid of most of the normalisations here.
However, when running Photon with a single-language dataset then more filtering and normalization for that specific language might be wanted. Ascii-normalization might or might not be wanted. Or, the posessive s problem could be handled with a pattern replacement filter.
The set of filter is normally set at import time making it a property of the database. So introducing a configurable set of filters mostly doesn't need to wait for a major release. Except, there is the synonym handling. This rewrites the index mapping. So we'd need to see how to introduce synonyms without changing the filter configuration.