iWay Big Data Integrator User's Guide > Getting Started With iWay Big Data Integrator > Creating a New Project

Creating a New Project

Topics:

How to:

A project contains all of the metadata for an iWay Big Data Integrator (iBDI) working set and all connection information for the runtime objects. A project may contain only one object or contain multiple objects depending on the requirements of the organization and design patterns. Pre-defined categories of Big Data operations appear in the project automatically as the operations are implemented, such as Pipelines or Flumes.

You can create projects by right-clicking anywhere within the Project Explorer pane, selecting New from the context menu, and then clicking Project. You can also click File from the main menu, select New from the context menu, and then click Project. When the project is created, a new entry appears in the Project Explorer pane for the project and subfolders appear for each of the iBDI operations.

Creating a pipeline opens the pipeline editor, which enables all new Spark-based functionality. Non-Spark or Hadoop operations are enabled by creating a new operation of the type from the project.

This section describes how to create a new project in iBDI.

Procedure: How to Create a New Project

  1. Right-click anywhere in the Project Explorer panel, select New from the context menu and then click Project, as shown in the following image.

    The New Project dialog opens, as shown in the following image.

  2. Select Big Data Project from the list of wizards and then click Next.

    The New Big Data Project dialog opens, as shown in the following image.

  3. Specify a unique name for your new project in the Project name field.
  4. Specify a location for your project or accept the default, which will save your project to the workspace directory by default.
  5. Click Finish.

    Your new project (for example, Sample_iBDI_Project) is listed in the Project Explorer panel with corresponding subfolders, as shown in the following image.

    Properties for your project can be viewed in the Properties tab, as shown in the following image.

Project Components

There are various project components that are supported by iWay Big Data Integrator (iBDI) for each project. Project components are stored in the corresponding subfolders that are created for an iBDI project, as shown in the following image.

Multiple project componentsan iBDI project

The following table lists and describes the supported project components.

Project Component / Subfolder

Description

DDL

Data Definitions. You can export a model from any RDBMS and iBDI will translate it to HQL and create the tables in Hive.

Flumes

Streaming inbound data.

IDS

Input Document Specifications, for use in Mapper table creation.

Mappings

Hive transformations.

Pipelines

Spark pipelines for multiple sources and transformations.

Sqoops

Relational inbound data.

Wranglers

Apply schemas to transform HDFS sources.

Depending on your requirements, multiple project components can be configured in a single iBDI project.

Note: Each component in an iBDI project consumes memory when this project is opened. For example, a large iBDI project containing many project components will be slower to open.

You can create a new project component by clicking File from the main menu, selecting New from the context menu, and then clicking Other, as shown in the following image.

The New - Select a wizard dialog opens, as shown in the following image.

Select the type from the list or type the name in the Wizards field.

The project component can also be created directly from the iBDI project by right-clicking the project, selecting New from the context menu, and then clicking Other, as shown in the following image.

The New - Select a wizard dialog opens, as shown in the following image.

Select the type from the list or type the name in the Wizards field.