Natural language based search API
Siets Server software supports rich set of linguistic search options using only natural language words or phrases in queries.
Below are some examples of typical search query syntax for Siets Server XML database when using API command "SEARCH".
Free context search as in web search engines (default)
Free text natural language terms, case insensitive (default querying mode):
a little john in london
would return all documents that contain combination of relevant words "little", "John” and “London”, but ignoring words "a" and "in".
Siets Server search engine software can process data in 160 languages in the same XML document store. Multiple language content can be stored
in one XML document formatted with UTF-8 character set encoding.
Search queries need to be in the same languages that are actually present in the customer data.
Capitalizing the first letter in a word for case-sensitive search
Intuitive default 'rule of thumb' for making case sensitive free text queries is to capitalize the first letter in a word:
windows
would find both words "windows" and "Windows". However,
Windows
would find only capitalized words "Windows".
Free context search, case sensitive
Free text natural language terms, case sensitive:
a Little John in London
would return all documents that contain combination of relevant capitalized words "Little", "John” and “London”, ignoring "a" and "in",
and ignoring words "little", "john" and "london".
Context relevance is King
Contextually best matches based on natural language text analytics performed during every search query,
will surface Siets Server results with more relevant results upfront, prioritizing natural language analytics vs algorithmic data sorting.
For the previous example free text query:
a little john in london
document with the following text content:
Little John went to the nearest underground station in London.
would be sorted upfront (surfaced) vs the following document with this text content:
John went to the nearest underground station. It was a little more distance away from his hotel. It was time to go to London.
The second document will also be found and will be present in Siets search results. Yet it will be listed only after the
first result as being less contextually meaningful document from the natural language user point of view.
One can say that the first document is more relevant than the second document for the particular query context.
Standard rules for Siets Server language analytics used per each query are based on fast linguistic data pattern matching. Software is taking
into account language based factors that determine human relevance of search results. Among pattern matching criteria used by Siets Server are:
- relevant words making the query context
- irrelevant or ignored words making the query context
- matching word location positions per document
- matching related patterns of word forms such as with stemming, synonyms etc
- matching word actual frequency per entire database
- matching word actual frequency per document
- estimating density of matching words in the textual content, where query terms are found close to each other
- ranking of the particular data content part matching the query
- ranking of the document against other documents in search results
- matching trainable ranking thresholds applied on the data model by customer
The most relevant document context matching the search query terms by combined above text analytics criteria will be surfaced by Siets Server,
sorting the results by decreasing contextual relevance and providing small fragments of text snippets from documents where the best context
matches were found.
Additionally transactional pagination for results will be used by Siets Server to avoid information overload in web browsers and limit number of
results per one page (eg 20, 50, 100 etc), all parameters configurable for every Siets API query.
Siets Server enforces natural language content relevance sorting rules for each free text query, if not specified otherwise.
That enables to surface the most relevant results upfront by the best matching linguistic information in the textual content.
Contextual relevance ranking rules can also be flexibly further customized by the owner for different content surfacing rules, custom defined for
each Siets Server database through its search index ranking policy. Please kindly read other sections in this website about Siets unique
information ranking model and methods how to apply ranking weights to sort, group and order search results.
Finally, search results based on standard text analytics sorting by context relevance can be combined with other information ordering rules
through Siets API, for example, enabling results sorting by numeric values or dates like in classic SQL databases.
There are potentially millions of sort orders possible in a single Siets Server database. When the software is responding
to free text ad hoc queries coming from millions of users, it is prioritizing results by the best ranked context match first,
and only then by other sort orders.
Free natural language phrase search, case insensitive
Free text natural language phrase, case insensitive:
"a little john in london"
would return all documents that contain a context phrase "a little John in London” exactly in this order of all words, including "a" and "in",
and matching all required words in upper and lower case, where "john" and "John", "london" and "London" etc are equal matches.
Free natural language phrase search, case-sensitive
Free text natural language phrase, case-sensitive:
"A Little John in London"
would return all documents that contain a context phrase "A little John in London” exactly in this order of all words, including "A" and "in",
with words started with capitalized-letters matched in case-sensitive way, matching "Little", but no "little", "John", but no "john" etc.
Free text search with enforced all words matches
Use plus + symbol as the word prefix to require strong match irrespectively of this word presence in the ignored word list.
Free text with enforced all search terms matches in a natural language query:
+A little John +in London
would return all documents that contain combination of relevant words "little", "John” and “London” with usually ignored words "A" and "in".
Free text search with stemming for some natural language words
Enclose word in dollar sign $ symbols to request word substitution with a list of word forms derived from stemming and Boolean OR among all of them.
Free text with natural-language grammar rules stemming:
little $john$
would return all documents that contain combination of "little" with either word "John” or all stemming forms like “John’s” etc.
Free text search with synonyms for some natural language words
Enclose the word in percentage % symbols % to request this word substitution with a list of synonyms and Boolean OR among all of them.
Free text with synonyms using pre-loaded for each XML database file with a simple list of synonyms, used with boolean OR logic in a query:
little jonh %country%
would find terms "little" and "john" in combination with either word "country", or word "region", or word "area", provided that all
3 synonym words were listed in the synonym file line, comma-delimited, that started with word "country": "country, region, area".
Free text search with different word suffixes
Use asterix * symbol as a wildcard template for words.
Free text with a wildcard template for the word part in suffix:
little Jon*
would find all combination matches of word "little" with either "Jones", or "Jonson", or "Jonathan" etc.
Free text search with different word prefixes
Free text with a wildcard template for the word part in the prefix:
little *athan
would find all combination matches of word "little" with either "Jonathan", "Bathan", "Rathan" etc.
Free text search with middle-word wildcard
Free text with a wildcard template for the part in the middle of word:
little jo*n
would find all combination matches of word "little" with either "Jonathan", "John", "join" etc.
Free text search with letter positioning template
Use a question mark ? symbol in a free text query as a specific word letter wildcard
for designated letter positions per word (or per any character string that is not a delimiter symbol):
little J?s?n
would find all combination matches of word "little" with either "Jason", "Josin", "Jasin", "Jeson" etc.
Free text search with letter selector template for the position
Use square brackets [ ] in a free text query to specify allowed letters in that letter position in the word.
Free text with a selector for only specified letters template in a particular position in the word:
Little Jonn[iy]
would find all combination matches of case-sensitive word "Little" with either "Jonny" or "Jonni" etc, but not "Jonna", "Jonne" etc.
Free text search with wildcards in phrases
Natural language phrases with wildcards use for individual phrase terms:
"software develop*"
would find either phrase "software developer", or phrase "software developers" or phrase "software development" etc.
Ignored words detection and use in natural language queries
Siets Server automatically detects ignored words above certain threshold in natural language queries.
By default, Siets ignores common words and characters such as "a", "the", “and”, ”where”, “how” etc,
as well as certain single characters and single letters, because they tend to slow down the search without improving the search results.
The SIETS server detects words that appear in the SIETS storage most often and gradually adds them to the ignored words
list when loading and indexing large amounts of data.
It is possible to place reasonable restrictions (percentage thresholds, word lengths etc) on ignored words list for each Siets Server data store collection
so that limitations best fit the application business logic, user search requirements and type of data content.
Please note that SIETS server still creates the full-text index with all ignored words and their positions. Ignored words are actually being
included into the index exactly as they occur in natural language text content, so that a user can request search queries with normally
ignored word matches, if necessary.
If a common word or a character was ignored during search query by Siets Server, yet it is essential to
getting the results you want, you can include it by preceding it with a plus sign +:
John +and Anna
will find only results with all three words present: "John", "and" and "Anna".
Proximity search of words at certain distance from each other
Enclose two or more words using @ symbol followed with a number, specifying nearby textual distance
in maximum number of words to be searched for matches:
Proximity search for two or more words within nearby textual distance:
@ 4 John Smith @
would find document with content "John Armitage Smith", and "John Henry Armitage Smith", but would not find
document with content "John went to the nearest underground station. Smith was not there...".
Boolean search AND, OR, NOT in natural language queries
Combination of all above described natural language search options per single search query
with ( ) as AND, { } as OR and ~ as NOT.
Boolean AND: use ( ) brackets
John Smith
(John Smith)
would return documents that contain both words “John” or “Smith” in any order.
Boolean ( ) as AND is a default free text search criteria
and can be skipped in simple one level queries with no nested other terms.
Boolean OR: use { } curly brackets
{John Smith}
would return documents that contain either the word “John” or the word “Smith”.
Boolean NOT: use symbol ~ tilde
John ~Smith
would return documents that contain the word “John", but do not contain word “Smith”.
Search only specific XML data fields using free text natural language terms
Combination meta-data search using free text natural language context is specific data fields only:
<name>john</name> <address>piccadilly london</address>
would find all documents with "John" in <name> field where <address> field is
mandatory containing both text terms "Piccadilly" and "London" in any order.
Siets customers can freely choose to provide this type of easy to grasp programmable XML data filtering, slicing and pivoting
into different result subsets for their intelligent end-users for powerful search driven analytics and reporting.
Search across all XML data fields using free text natural language terms
Siets Server can be instructed with "index=all" policy to index all content from all individual XML data
fields into a single common full-text index.
The following query example would also work to provide reasonably good search results, if XML database
has been indexed to perform search across all unstructured text and through the entire XML data model.
piccadilly london john
Above example would probably be less precise than content search per specific XML field only, still it will yield the list of results
for review to user with text snippets showing context and allowing user to decide which result is most useful.
Search across all XML text content can be combined with meta data search in XML data fields.
Sanitizing natural language query options
Siets customers can flexibly sanitize all FTS queries for greater safety just leaving allowed options to perform plain text
and phrase queries for users.
Then anyone with basic web type search skills and not being aware about underlying XML data model
can easily search database retrieving results by default sort orders and limited to pre-programmed access logic.
Multi-level nesting of search queries with Boolean logic to build complex analytics
Siets software Boolean AND ( ), OR { } and NOT ~ query syntax can be nested in multiple levels to make different logical
combinations of FTS queries with XML field level context queries.
Complex business logic, if required by customer application, can be easily created. A simple example:
{(John Smith) (Abby Brown)}
would return documents that either contains the word “John” and the word “Smith”, or the word “Abby” and the word “Brown”.
Similar text content search in natural language to another content
Additionally unstructured text analytics is possible by textual given content similarity to another textual content with
tunable number of significant word occurance and concurrance thresholds for the best results in large data sets with billions of objects.
Spell-checking support with 'Did you mean that?' option
Another helpful text analytics feature is Did you mean that? type of spell-check correction function returning a list of similar by spelling
words from the actual your XML database index.
Multi-language data store
Users can store and process data in multiple languages within the same XML document, avoiding tons of localization efforts in their application software code.
Fast character set transformations in server code
Additional 'perk' is fast server-side XML data conversion between national ISO charachter sets and UTF-8 data store if requested by client API.
All data is always stored in standard UTF-8 data store on Siets Server and can be queried either using UTF-8 or a national ISO character set.
Nearly effortless linguistic search experience
Natural language based querying and search paradigm is the holistic approach of Siets Server software design.
Siets Server powered customer software applications do not require from end-users to learn more
complex query syntax forms than those described above.
It enables Siets Server software customers and application developers to start building applications where users need
to have only basic skills how to search for relevant information.
Most of users are already pretty familiar with basics of this knowledge today from their web and corporate database search experience.
Default linguistic search options in Siets Server search query syntax will work well enough just using plain text natural
language terms.
Siets Server customers can start providing easy, intuitive, fast and relevant user search experience in their own databases similar to
customer satisfaction when using the world's leading web search engines.
Siets users are free to use plain text words or phrases for relevant information search in data, being the most intuitive query terms based on
everyone's language knowledge.
Relevant ranking for natural language content
Siets Server provides a unique policy tool how to rank customer database search index policy through trainable system of ranking
weights, uniquely applied on the custom XML data model for data fields and on the natural language text content that those fields contain.
Most web and mobile users can instantly query and analyze large data volumes at the back-end application services systems, run by Siets Server,
using just the web browser and free text natural language query terms: few words or a known context phrase. If necessary, users can
expanding the NLP-only query with a bit of additional and very easy to learn syntax options described above.
There is no need to learn SQL or similar complex querying languages in order to retrieve information from vast data volumes
and rearrange it into grouped, sorted and ordered way, even up to the precise positioning of individual search result entries.
This Siets as a ranking engine feature enables to replace with NLP terms more complex SQL syntax queries for combined
full-text and structured search, that would typically in SQL syntax look like this:
SELECT ... LIKE ... GROUP BY ... ORDER BY ... JOIN
with Siets Server ranking policy (set of relative weightings for XML data model), that will instruct Siets Server to index all
customer XML data in application-specific data sorting way by desired relevance,
so that complex information sorting, grouping and ordering (relevance of search results) is automatically performed when users are doing search
with free text terms in natural language.
Essentially the index ranking policy enables Siets Server customers to "merge" contextual search and sorted structured data search into
one single "most relevant result set" from a user point of view, when "information relevance" is being estimated in relation to the application
data model and specific business need.
In Siets Server system customers, using policy tool for index ranking, can flexibly govern Siets Server search behavior aligning
it with desired free text search relevance, when data is queried and analyzed in plain natural language terms. Siets Server will
automatically enforce ranking policy changes during data modification and would also make content sorting rules uniformly
available for all applications without the need to change application software code in all applications using Siets platform.
This relevance ranking engine feature in combination with natural language processing (NLP) at search is probably one of the most
powerful Siets Server capability. It enables Siets customers to build massively scalable distributed data stores
with blazing-fast and relevant search enabled in any types of customer databases with text-rich language content.
Please see more details about all NLP syntax search options, indexing and relevance "policy" methods in Developer documentation.
Read more about SIETS API specification in Developer Guide:
Developer Guide / Siets API
