SIETS Solutions: Use Cases, Advantages, Benefits

All-in-one software platform productivity

Siets Server is a highly scalable data management software platform that enables to efficiently store, manage, search and analyze large amounts of digital information, accessing it from web applications and using free text natural language terms as queries.

Siets blends together 3 fundamental data management functionalities into one software platform:

• full text search (FTS) engine supporting queries in free text language ()
• XML database server with UTF-8 and ISO character sets ()
• distributed computing architecture with REST API over http/https ()

Instead of a typical IT infrastructure stack, consisting from 2 or 3 integrated and possibly supplied by different vendors software platforms:

one can use Siets Server as a single platform software product to reduce IT infrastructure complexity:

Blazing-fast linguistic search at any scale

SIETS main competitive advantage in search engine software world is nearly linear scalability for free text search queries in growing data volumes of digital information assets, where some of data is stored as unstructured text content in natural language text or text-rich meta data such as in XML documents and XML-formatted messages.

Siets Server provides linguistic search queries by analyzing all language terms in the user query context and then finding the best contextually matching documents for this query, sorted in decreasing relevancy with respect of the most meaningful language context found in XML data.

Using just Siets Server software installed on commodity PC hardware servers with OS and connected through TCP/IP network, customers can scale out underlying IT infrastructure from a single server to multiple server farm with no other software necessary.

By distributing the total data volume and transactional web search queries from users among multiple hardware servers, Siets Server will deliver sub-second search response time and provide relevant results from MBs, GBs and even TBs of data:

MB

GB

TB

Siets platform's REST API enables software developers to build their code without dependency on underlying data distribution in a particular cluster.

The same application software code, using Siets REST API, will work with a single Siets Server deployment running on a dedicated hardware server, as well as with Siets Software installed on a fleet of multiple cluster servers.

Software can be used as sole or complementary data storage platform along existing SQL databases.

Instantly relevant search experience

The best use of Siets software is to power back-end IT solutions for modern web and mobile applications that need easy and fast search in large amounts of text-rich data.

Customers can flexibly mix all textual data required for search in custom made XML documents, that can easily contain unstructured data (text as in articles, books), semi-structured meta data (as names, addresses, classifiers etc) and even tabular SQL data exported from databases into XML data.

Siets Server provides search across the entire database content: both in text data and in meta data values within XML markup.

Siets Server enables customers to provide to their users simplicity of search. They can do free text search queries in plain language terms, entering just few known words or phrases per query:

text

search
using
words
or
phrases

XML

SQL

Siets Server will find the most relevant content in milliseconds, matching the end-user search query context, and sorting the result list of all documents found in ranked way by descending relevance order. The list of results could be further improved with snippets of text where context matches were found.

SIETS software delivers perfect information search experience to users and high customer satisfaction.

Distributed data and computing solutions

Multiple Siets Server instances, working together in a cluster deployment, support management of the total data volume of a large database split in smaller parts (shards):

Part 1

Part 2

Part 3

Each database part will be located on its own hardware server and Siets Server software will use local storage and RAM of that server to process local database shard at the maximum performance.

Database sharding into multiple parts will prevent encountering situations when a single hardware server, if used to process the entire database data, crashes abnormally being out of free resources.

Reasonable database clustering in smaller size shards prevents hardware hitting its storage and free RAM limits per each server. Distributing data and managing risk of potential overload failures is one of key principles how to build and operate reliable computing services.

In Siets Server cluster data processing is performed on two or more times smaller size database shards, essentially, on each server in parallel, delivering as many times more overall performance efficiency.

Double or multiply search performance

Proportionally to the number of cluster nodes with Siets Server running an n-th times smaller part of the entire database on each of servers, overall search query performance would increase:

response
time

1 sec

0.5 sec

0.3 sec

Siets Server can be configured in cluster for optimal performance delivering search results to users with sub-second search query response times even in very large size databases.

Cut data recovery, load and index times

Proportionally to the number of shared database cluster nodes with Siets Server running on each of them, indexing performance would increase about the same number of times:

indexing

1 hour

30 min

20 min

A massive clustering is going to deliver substantial time-savings for performing bulk update, reindexing, backup/restore or other maintenance job on large size databases.

Enjoy high availability fault tolerant search

In particular use cases where only a certain number of the most relevant results must be recommended to users from the entire database, a massively distributed database architecture is going to better satisfy quality and availability requirements.

A mission-critical web service such as Internet search could be a fairly good example. Whenever a hardware node fails in a cluster, the rest of the cluster nodes will still be available for Siets Server operations and the rest of a database shards will be available for user queries:

Part 1

failure



Part 2

Part 3

End-users using Siets Server search services would hardly notice temporary unavailability of data from the failed hardware node, while system engineers are performing replacement of the malfunctioning hardware.

Solidify data assets using multi-replication

Siets Server software supports internal and real-time synchronized replication of database (by shards) into two or more fully mirrored redundant copies for high availability, fault-tolerance and search query workload sharing in a cluster deployment.

Part 1

Part 2

Part 3

Part 1

Part 2

Part 3

Part 1

Part 2

Part 3

A shard can be replicated in two or more copies on another hardware or storage device, enabling to replicate the entire database in multiple operational copies on different hardware sets.

Reduction of the individual size of Siets Server database shard would also be helpful. Siets Server replicates shard to shard between replicas: it never mixes replicated data taken from two or more shards into one new shard.

This keep it simple 1:1 by each shard replication to another shard design principle guarantees high consistency of database information in all replicas and in the entire database.

In case of emergency data recovery or planned major equipment upgrades, that approach would also enable to synchronize database shards more easily, quickly discover inconsistency errors per shard basis in replicas, or bulk update information per each shard.

Simplify software with cluster-wide API

In a cluster deployment each hardware server is running a copy of Siets Server instance and all Siets Server software instances together deliver cluster-wide data management and search operations to API clients.

cluster
API

MB

GB

TB

Single mode and cluster API flexibility

Each Siets Server instance in the cluster of N nodes with its local data store shard (when a shard contains 1/N-th part of the total database) can be queried or updated either fully autonomously from API or cluster-wide, by using single mode or cluster mode parameter in client API requests.

Part 1

single
mode
API



Part 2

Part 3

Single or cluster mode API calls offer reasonable flexibility for Siets Server software developers to program their own desired application level side business logic with respect of cluster configuration and enable to enforce precise location of mission-critical data in the cluster, when it is needed.

Fast single mode updates in API calls enable to avoid typical performance problems arising with automatic cluster data rebalancing in other platforms.

Many uses cases require to properly manage parts of the entire database, controlling exact location of types of XML documents in a cluster by time, by content type, by access rights or by relevance.

In particular, partitioning a very large database storage by business logic enables to perform quick emergency restore only on parts of data, or limit user access only to certain parts of the entire data store.

Safe zero-knowledge access to data assets

Software developers could have zero knowledge about the actual Siets Server cluster deployment configuration and location of data in back-end systems, while still doing their job: writing applications, testing etc.

All internal clustering is taken care by Siets Server software automatically, without requiring application software developers to spend their time learning or adjusting their application software code to particular cluster deployment configurations.

All they need to do their job is API access, given to them by system administrators on a "need to know" basis, with roles limiting access only to subset of authorized API calls.

API

MB

GB

TB

This is the best safety practice for companies with high value digital information assets to protect.

The resulting "total" Siets data storage and computing capacity is presented to application software as a one "giant and big" database server and search platform in secure and safe to use way: through Siets API.

Availability replicas in geo-location zones

For large databases servicing millions of users it is recommended to replicate the entire Siets Server software run databases both in multiple copies in the same data center and also put additional copies in geographic locations close to users during peak workloads (normally in day-time).

Replicated databases can be located either nearby in the same data center, for example, to address peak workloads by load-balancing, or even in different data centers far away one from another for extra backup or safe redundancy.

New York

London

Tokio

Databases replicated by Siets Server software among different physical locations must have access to reasonably fast and good communication links between data centers, to be able to haul all traffic of updates when Siets Server software is synchronizing replica sets.

Obviously databases would not need very fast links under normal OLTP workload, but in case of bulk data upload, cluster re-configuration or other maintenance tasks requiring processing of significant amount of data, slow links among replica sets could start hindering Siets Server performance and cause delays in search query response times.

To avoid this situation, running two or more replica sets in each data center is recommended. Then the entire cluster setup in one data center can be replicated in another data center and used through load-balancing software to service all nearby users following global availability zones by geography.

Using manual or scheduled jobs policy, individual updated copies of a database all shards could be transferred one by one from one data center to another as simple file transfer to recreate the full snapshot cluster database and its last updated state elsewhere, without repeated reloading and slow reindexing of all content in other location.

Siets Server data store synchronization tasks can be performed over standard Internet VPN connection.

Ranking engine for recommendations

Siets Server can do recommendation engine job in XML databases, improving it with a novel indexing architecture based on ranking the index by policy.

Indexing policy is a tool in Siets Server that allows to set ranking rules how to adjust search results relevance (ranking) weights for custom data parts stored in XML data format with respect of other parts of data or in relation to the entire text content in XML. This policy (a set of all ranking rules) will determine Siets Server behavior at search. For each XML database a different policy can be established depending on business needs.

Policy in Siets Server is implemented as a set of relative weights having human-friendly percentage values from 0% to 100%, with 0% the least relevant data, and 100% the most relevant:

The example XML ranking policy model above assigns maximum relevance to search hits in the <Title> tag content, less relevance to hits in <Body> part content, and, finally, the least relevance for any content found in <Footer> tag.

Using familiar to most people percentage values as ranking weights delivers perfect ranking simplicity to customers, with easy to understand and to apply rules how to mark up XML data parts for the desired significance at free text search.

Most search engines have contextual relevancy determination tools based on a number of word occurrences and word positions in the documents. Statistical text analytics applied on inverted index evaluates free language terms in queries and determines document ranking position in the result set. Better contextual matches to user search query terms are always surfaced upfront as a result of this statistical ranking or scoring of data by search engine.

This is the basic principle how most recommendation engines work, when matching query terms against natural language content (text).

Siets Server allows to define flexibly customizable relevancy determination rules for individual XML document parts in relation to other XML data parts in addition to contextual relevancy calculations.

Application developers can assign different fixed relevance weightings or ranges of min-max weightings only to those parts of XML documents that are important for search according to business requirements. The rest of XML data can be made none-searchable, if necessary.

When querying the database with full text search query, search results will be sorted relatively according to the desired relevancy sorting by ranking specific XML parts, if actual hits are found in those parts, rather than just by the frequency or word positions in full text content. Frequency and word positions become active only when min-max weighting range is specified in policy file for a particular XML tag, effectively being threshold values of rankings for that group.

The rule of thumb for determining ultimate relevancy of search results in Siets Server is the following one: when contextual relevancy of search results is equal, ranking by policy weightings applied to XML tags determines the order how the results will be surfaced to user. If both ranking dimensions are equal too, then contextual ranking by weighting interval using specified min-max ranking values determines results sorting (as in classic search engines).

All relevancy rules within XML policy can be combined for the desired best sorting, grouping and ordering of search results for a particular search query context.

At first this might sound complicated to use 3 different ranking dimensions combined, but trying it out on big data set will quickly show efficiency and quality of this ranking method.

Most importantly, this seems to be a good solution how to scale the same uniform index ranking policy across the entire XML content in a giant size distributed database. As a result SIETS customer can enjoy high search performance in a distributed database with nearly the same latency and sub-second query time as doing search on a single server.

Relevance sorting by policy-defined ranking weights for certain XML parts perfectly works in cluster-wide search solutions. In fact, this indexing method makes Siets Server capable of powering massively scalable search engines.

Applying several ranking dimensions across the entire index essentially is partitioning the giant inverted index in many relatively small parts. Access to those smaller parts is a very efficient computing method using local data only.

Whenever a particular user query terms are matching the content within specified XML tags, stored as inverted index entries locally and pre-sorted by relevancy weightings from the policy file, Siets Server will automatically combine all local ranking dimensions from all cluster servers together and then sort the aggregated results for the most relevant final results to be delivered to user.

Relevancy ranking by policy instrument probably is one of the most powerful Siets search engine's features how to tune and adjust search relevancy to your XML document data model. It enables to quickly improve your application search performance, solves scalability issue, and provide excellent return on investment through high user satisfaction with search results.

Technically in Siets Server architecture contextual relevancy determination mechanism follows the indexing policy of schema free XML document model.

Indexing policy taps into the actual XML schema as it was created by you, without the need to define XML document structure explicitly through DTDs or elsewhere in your software. It can be a very simple policy too. For example, a basic policy "index=all" can be applied to your XML document root tag to make all XML data searchable as in classic search engines, all without going deeper into Siets Server rich data ranking facility.

The only mandatory tag in your XML data model must be document ID tag, used similarly as URL is used on the web: as a unique key to store and find XML documents in the Siets Server database by its ID. ID is a simple text string with unique value (eg web link, ip address, social security number etc).

To learn more please visit section: Developers Guide / Relevance

Web-scale search engine platform

Siets Global Internet Crawler spidering software, accompanying Siets Server, uses generic Siets Server clustering capacity to index and store massive volumes of data needed for establishing and operation of a country-wide or a global scale Internet search engine.

Siets Server handles large capacity of multi-lingual web data with a less number of servers due to its all-in-one platform design architecture.

If you are running a large scale country-wide or a global Internet search engine, by replacing existing resource demanding systems and by installing Siets software technology you can save more than 50% from the current administrative and maintenance costs.

Subsections below describe in more details how Siets Global Internet Crawler can benefit from distributed cluster architecture deployments.

Web scale search requires large cluster deployment

By joining several Siets Server computers in cluster configuration it is possible to increase indexing performance several times.

Total time needed for indexing of all data will be decreased proportionally to the total number of cluster computers. See figures below where it has been illustrated.

Figure 1. Indexing of large text collections on a single server computer can take many hours.

Figure 2. Siets Server indexing performance can be scaled linearly using multiple cluster computers to reduce indexing times for large document collections.

This approach greatly benefits many applications where search database can be split among multiple computers. Large scale Internet crawling and indexing use case is the best example.

Major Internet search engines use the system of distributed crawlers and indexers based on cluster configuration. Siets Server was designed to allow to implement Internet search applications in similar way.

Split the entire web index into smaller parts

Using Siets cluster configuration, the full database update frequency can be increased substantially because complete updates are needed for parts of the database which are relatively small.

Internet crawler and search applications frequently need to update their search databases and search indexes. So does Siets Global Internet Crawler software.

In case of very large data sets such indexing can be done once per day or per week depending on how many servers work in parallel in the cluster.

Never-lost reliability if some data is missing

Clever clustering can provide for higher business availability of search and indexing services.

In case of malfunction of a server in the cluster the system as whole continues to operate. Search results are being received and processed from the rest of working servers. Siets Global Internet Crawler software is doing the same.

There will be missing some documents in the search results from the malfunctioning server. Nevertheless until the problem will be fixed the search service to end-users will continue to work. In this way corporations can avoid total service interruption.

Internet search applications can greatly benefit from this cluster enabled fault tolerance against total service disruptions.

End-users almost never discover that some documents from search results are missing. Even if some end-users suspect some missing data and retry the same search later, service provider has time to replace the malfunctioning server and restore backup copy of the missing data.

Generally there will be little or no customer complaints about service unavailability and higher quality of end user experience using business search services running in cluster environments.

Most leading Internet search engines are operating in similar way, using generic fault tolerance of cluster environment. Siets Global Internet Crawler software is doing the same.

Asymmetric distribution of index updates

Besides symmetric data distribution among all cluster nodes or replicas, it is possible to implement asymmetric data distribution for indexing of very large data sets in cluster configurations.

The asymmetrical indexing method can be used when large portion of data should be updated relatively less frequently than other smaller portion of data.

In this case the largest data set with seldom updates should be indexed on one cluster server, and small data set with frequent updates should be indexed on the other cluster server.

This solution has two-fold benefits. First is the economy of resources because all frequent updates are being done for the database of rather small size, which provides higher total performance of the system and less frequent disk equipment reads/writes.

Secondly, this is higher reliability environment: there are much less chances to break something due to equipment malfunctioning or software errors in a relatively small data set that is being modified.

Typically archived data portions do not change at all but occupy most of the resources. Even if the smaller portion of data on one cluster node will be damaged, the largest archived data portion on other nodes should not be even re-indexed.

Savings can be tens and even hundreds of hours of working time on maintaining the large database updates.

Organized data store for efficient backups

Similar efficiency savings are for performing data backup copying and restoring.

In cluster configuration with organized data distribution (known location per nodes) backup and restore tasks require much less time from system administrators compared to one large database or a distributed database where location of data per cluster nodes is not known.

Initial backup copy for the large unchanging archived portion of the data can be made immediately after its creation on one of the cluster servers.

Backup for the smaller frequently changing data on other cluster server can be made as frequently as needed, for example, every day or even every hour small database size does not require a lot of resources to do it.

The same time savings are present when it is needed to restore full working database from the backup copies. Mostly only smaller size backup copy will be needed to restore in case of problems.

This kind of clustering solution is the most crucial for building Internet search solutions that can be reliably and quickly serviced for such mundane tasks as backup, recovery, replacement upon hardware repair, and incremental web site data updating.

Automatic cluster balancing tools in some other search products sounds a good solution, unfortunately those products fail to scale on web-scale tasks since any intelligent update operations on data become too expensive.

The controlled management of database parts by organizing clever application-driven data distribution across large cluster is not an obvious Siets Server cluster software advantage at the first glance.

Therefore let's imagine backup or a recovery task of a single website with some small set of 100 pages in a billion document database cluster on 100 servers. When one does not even know which server out of 100 to approach to do this pretty frequent in web search backup task, the only solution available is to back up / restore either all 100 server data or to do a cluster-wide search for every document and then update one by one, overloading all 100 servers and network with unnecessary traffic and indexing job on all servers. More problems come after: all 100 servers needs to be fully backed up again to reflect consistent search results after just this one smallish web site update. This type of escalating search index consistency maintenance nightmare quickly becomes unmanageable problem where automatic distribution of data across all cluster nodes is performed by search engine platform.

This is key reason why Siets Server software does not automatically rebalance and redistribute data in cluster, unbeknownst to the application owner. Siets Global Internet Crawler software does controlled data partitioning for website data collected to avoid problems of "automatic data rebalancing" that ruins scalability at very large scale.

Save by using commodity off-the-shelf hardware

Distributing a large web search database among several cluster computers, each hardware computer can be having less expensive and simple hardware specifications compared to the heavy-duty power-server situation, when the entire database runs on a single powerful server computer.

For example, suppose a database is running on an enterprise level 2-processor Xeon server with SCSI RAID disk array with 2GB RAM memory.

About the same cost will be to install 4-5 more custom computers (1-Pentium processor, 1GB RAM, no RAID, one fast 80GB IDE or SATA disk) and use them in cluster configuration.

In this way organizations can have all above mentioned cluster benefits which will not be available if the whole database are running on a single and powerful server.

Making hardware upgrade cost-efficient and easy

Great benefit of clustering on many low-cost hardware is upgrade scalability. In order to increase data volumes there is no need to look for possibilities how to upgrade existing enterprise server or how to replace it with more powerful hardware. Enterprise server upgrades in combination of RAID/SCSI disk equipment usually are substantial expenses for every corporation.

Another inconvenience of big powerful servers is need to stop all services in case of hardware upgrades while hardware components are being replaced and reconfigured.

Very often it causes unnecessary business service unavailability not only for the primary application but for some other business applications running on the same server.

On the contrary, if the system already works in the cluster configuration, it is sufficient to add a new relatively custom (commodity) computer to the existing cluster and start to store additional new data on the new cluster node without any service interruptions.

Scale out vs scale up cost-efficiency

Another problem of large servers are their technical scalability limits. Their upgrade options usually are limited physically. One can not put more memory or more disk devices than a server equipment manufacturer has reserved as free physical space for limited upgrade needs in a single server.

Cluster hardware environment can be easily upgraded up to tens and even hundreds of computers, creating in this way computing platform with giant workload capability and reliability. This principle is used by many leading Internet search portals.

Interchangeable use of hardware

Replacement of outdated hardware is less problematic in cluster environments compared with big server environments.

Large technically outdated enterprise servers are hard to reconfigure for other use. Outdated cluster servers can still serve as employee PCs.

Simplify support with inexpensive hardware

In cluster environments administrators do not spend their time for installation and configuration of large enterprise servers.

It can be time consuming process because of many differences between enterprise server models, proprietary device drivers etc.

Every cluster hardware equipment is relatively simple computer and can be installed and configured in about 20-30 minutes.

Time is being saved both for installations in the data center environment and solving of technical problems of cluster nodes.

Commodity hardware can be served by technically less skilled people.

Manage very large databases cost-efficiently

All of techniques described above for running web-scale Internet crawling and search service as a reliable 365/24 operation can be also applied to your corporate data volumes of giant size.

Replacement solution for SQL complexity

Siets Server customers can benefit from a significant CAPEX and OPEC cost reduction in corporate search applications where large size data volumes are stored in SQL database silos.

A search engine platform typically then runs in parallel to SQL database platform with some custom integration application or middleware software gluing all systems together.

OEM and integrated systems solutions

Siets Server based search engine technology can be licensed to OEM equipment manufactures and solution providers.

Partners with different OEM licensing use cases can benefit:

• selling Siets Server software together with hardware equipment as a search appliance product
• adding value to low-margin equipment sales with bundled search driven applications
• making customized search solutions on top of Siets Search engine (sold as integrated solutions)
• adding instantly relevant search to document management or email archiving platforms
• improving user search functionality in corporate database applications
• and many other use cases

Siets Server open architecture based on XML document storage and http based API messaging allows to quickly build and deploy virtually any applications.

Original Siets Server technology source code licensing is available, including licensing of the complete Siets Server search engine software source codes and the Siets Enterprise Manager software source code.

SIETS Server and Siets Crawler source code has been written in C and C++ programming languages, with management GUI modules written in PHP.

Optionally SIETS source code for large scale distributed Internet crawler software can be licensed for building and operating web scale Internet search engines.

Hosted search solutions for websites

Siets Server can be used to remotely host search service for multiple corporate websites per single server.

Internet service providers and web hosting providers can add extra service to their portfolio of hosting products.

With every Siets Server license also comes Siets Enterprise Manager and Siets Crawler software where one can easily define and schedule crawler tasks for indexing specific Web domains.

Hosting vendors can create separate searchable Siets databases (storages) for each Web site and define a custom indexing policy for sorting and ordering data there.

Additionally hosting vendors can provide customized Siets API scripts for remote Web site owners to include Siets search results in their user interface of the owner's Web site.

Since Siets API is so easy and simple to use, being just REST API over http/https, almost any programming language environment or scripting language can easily interface with a hosted Siets Search engine at the back-end following published Siets Server API Documentation and using provided client code Samples as reference scripts.

Hosting vendors can manage and supervise multiple searchable databases (indexes) within a single Siets Server and Siets Enterprise Manager deployment, provided hosting vendor has hardware capacity for all client websites.

For each customer a hosting vendor can create a separate isolated user account on a Siets Server to offer them to remotely manage a hosted search solution data content for search securely, while the same vendor hardware resources will be shared among multiple customers using the same hardware.

In this way hosting vendors can increase return from their hardware, while website owners do not need to maintain their own search applications and more complex infrastructure in-house.

In many small and medium businesses outsourcing of quality web search services to hosting vendors usually having fast and reliable Internet communication lines can be more cost-effective than to keep a dedicated IT personnel with own servers in-house.