VirtualizeMe

*aka ‘vME’

DevOps that Data! You will never have to worry about getting realistic databases for dev, test and CICD again.

vME, pronounced /’vi:mi/, a member of Enov8’s “Test Data Management Suite”, is a database provisioning technology using “cloud native” containerization & cloning (database virtualization) methods. A solution that lets you “economically” spin up dev, test environments in seconds & circumvents traditionally slow data operations & storage issues.

The Problem

Enterprise Data Management can often be challenging due to its complexity, size, and sophistication, leading to issues such as long processing times, high costs, low data quality, difficult serviceability, and inadequate security.

• Cadence:The deployment of test data and environments can take anywhere from weeks to months, slowing down software development and analytics projects. This can be especially problematic for urgent Business as Usual endeavors.
• Cost: Data Provisioning can be expensive due to the large amount of data that applications require. Data sources of tens of terabytes may require high-performance storage and disks, and specialized professionals may need to be accessed, leading to further costs.
• Quality: Data Inconsistencies and Inaccuracies can occur when data is stored in multiple, disconnected silos, is prepared in inconsistent or non-standardised ways, or is allowed to become outdated. This can have negative effects on Analytical, Engineering, and Quality efforts.
• Serviceability. Development and Test teams often lack the control over their data, causing them to have to rely on specialized DBAs or Data SMEs to meet their needs. This often leads to delays in the process, as these experts have numerous other tasks to attend to. This can also inhibit their ability to rollback, refresh, and retest.
• Security: Organizations must protect their sensitive data and comply with data privacy regulations. Without adequate data security measures, organizations can risk data breaches and non-compliance penalties.

All of which adds up when you consider that every application needs between five and ten test environments, such as System Test, System Integration, User Acceptance, Staging, and Production Support. Plus dedicated environments for users, tribes and project teams.

Under the Hood

Under the hood, Enov8 vME is an open virtual appliance* (OVA) designed to simplify the DataOps process and virtualize database infrastructure.

*Think of an appliance as a plug & play toaster.

You can employ one appliance, or you can horizontally scale, by having multiple appliances spread across your domain, clouds or teams.

vME DataOps Platform is built by combining two key technologies into a single unified DataOps usecase. The first technology used is containerisation. Through the use of containers, vME provides an efficient and consistent environment for databases, reducing space requirements, improving agility,manageability, and performance. This results in faster and more reliable datadeployment compared to traditional virtual or physical machines. The second technology used is the Zettabyte File System (ZFS), an advanced volume manager that allows us capabilities like data compression, data pooling, point-in-time snapshotting and clones.

Enov8 vME workflow can be described in 4 simple steps:

1. Ingest you Data Source
2. Snaphshot your Data Source *Point in Time
3. Clone the Snapshot
4. Provision & Use the Clone

Supported Data Platforms

Our vME appliance manages the database servers through containers and the underlying data at the file system block level. Simply put, that means our underlying engine is 100% database or file type agnostic.

vME supports all the most popular database types including AWS RDS, Azure Databases, Couchbase, DB2, MariaDB, Microsoft SQL, MongoDB, MySQL, Oracle, Sybase, Postgres, Files & Directories. And can be easily configured for other data types.

1. Data Ingestion

In Enov8 vME, data ingestion refers to the process of bringing data into the Enov8 vME platform from various sources, such as databases, filesystems, and cloud environments.

This process involves extracting a copy of the data from the data source (for example a backup-file or database.SQL) and loading (ingesting) it into the Enov8 vME platform. The ingested data is then used to create virtual copies, which can be provisioned to development, testing, and analytics teams.

During the data ingestion process, Enov8 vME can also performs various tasks on the data, for example: Data Profiling, Data Masking or Encryption, Compliance Validation, Fabrication through the Enov8 TDM Engine.

3. Data Cloning

In vME, Data Cloning in is a process of creating an exact ‘point-in-time’ copy of an existing datasource (typically a database or set of files) that has undergone snaphotting.

Each clone is an exact copy of the source database, but it is stored in a smaller, lighter-weight format. This makes it much faster and easier to deploy and manage multiple copies of the same database. From all intensive purposes a clone sometimes refered to as a Virtual Database (VDB) is a fully functioning copy of the dataset and is ready for provisioning and consumption.

DataOps Automation

In addition to vME offering a light and intuitive web interface for easy administration of your data. vME also provides a complete API and Command Line Interface (CLI) for automation of your data operations.

Easy to learn, the Enov8 API & CLI can be easily included in your favourite scripting language or CICD* tool. Thus supporting displacement of manual data operations and allowing task scheduling.

*vME integrates easily with all CICD tools, including: Jenkins, Travis CI, CircleCI, GitLab CI/CD, Azure DevOps, AWS CodeDeploy, Buddy, Bitbucket Pipelines, GitLab CI/CD, and CodeShip.