iWay Big Data Integrator (iBDI) enables the following
natively in a Hadoop or Spark ecosystem:
- Spark pipelines enable sequences of processes for obtaining,
transforming, and persisting data in multiple combinations. Pipeline
chains allow for complex processing without the need to write custom
code.
- Change Data Capture (CDC) is unique to iBDI. iBDI provides extended
Hadoop native tools to support CDC for limit file transfer sizes
when possible.
- iBDI provides two modes of deployment:
- Deployment for
interactive runtime sessions.
- Publish for batch and streaming sessions.
- iBDI provides the ability to transform data (Spark pipelines)
through the Transformer tool or in native HDFS using the Mapper
tool.
- Metadata can be extracted from iBDI-provisioned deployments
in JSON format that can be joined with one another to describe:
- Where the data originated.
- Where the data was sent.
- How the data was processed.
- iBDI allows you to:
- Import data into Spark or Hadoop
from wherever it exists (database, stream, Kafka, file, or other
sources).
- Provide metadata so that the data can be consumed by Spark or
native Hadoop modules.
- Transform data so that it can be consumed by other enterprise applications.