Database virtualization

Database virtualization is the decoupling of the database layer, which lies between the storage and application layers within the application stack. Virtualization of the database layer enables a shift away from the physical, toward the logical or virtual. Virtualization enables compute and storage resources to be pooled and allocated on demand. This enables both the sharing of single server resources for multi-tenancy, as well as the pooling of server resources into a single logical database or cluster. In both cases, database virtualization provides increased flexibility, more granular and efficient allocation of pooled resources, and more scalable computing.

Virtual Data Partitioning

The act of partitioning data stores as a database grows has been in use for several decades. There are two primary ways that data has been partitioned inside legacy data management solutions:

I. Shared-Data Databases–an architecture that assumes all database cluster nodes share a single partition. Inter-node communications is used to synchronize update activities performed by different nodes on the cluster. Shared-data data management systems are limited to single-digit node clusters.

II. Shared-Nothing Databases–an architecture in which all data is segregated to internally managed partitions with clear, well-defined data location boundaries. Shared-nothing databases require manual partition management.

In virtual partitioning, logical data is abstracted from physical data by autonomously creating and managing large numbers of data partitions (100s to 1000s). Because they are autonomously maintained, resources required to manage the partitions are minimal. This kind of massive partitioning results in:

partitions that are small, efficiently managed and load balanced; and
systems that do not required re-partitioning events to define additional partitions, even when hardware is changed

“Shared-data” and “shared-nothing” architectures allow scalability through multiple data partitions and cross-partition querying and transaction processing without full partition scanning.

Horizontal Data Partitioning

Partitioning database sources from consumers is a fundamental concept. With greater numbers of database sources, inserting a horizontal data virtualization layer between the sources and consumers helps address this complexity. Rick van der Lans, author of multiple books on SQL and relational databases, has defined data virtualization as "the process of offering data consumers a data access interface that hides the technical aspects of stored data, such as location, storage structure, API, access language, and storage technology." ^[1]