iWay Big Data Integrator User's Guide > Defining Run Configurations > Defining Run Configurations for a Sqoop or a Flume

Defining Run Configurations for a Sqoop or a Flume

This section describes how to define run configurations for non-Spark pipeline types (for example, a Sqoop or a Flume).

Once a Sqoop or Flume configuration has been created and saved, right-click the Sqoop or Flume project in the Project Explorer, select Run As from the context and then click Run Configurations. From the main menu, you can also select Run and then click Run Configurations.

The Run Configurations dialog opens, as shown in the following image.

The Run Configurations dialog has two views. The left pane provides a list of applications. The right pane provides a series of instructions. You can resize the width of either pane accordingly by clicking and dragging the vertical bar that separates each pane.

Perform the following steps to define a run configuration for a Sqoop or Flume:

  1. Right-click iWay Big Data Integrator Deploy or iWay Big Data Integrator Publish in the left pane of the Run Configurations dialog and select New from the context menu.
    • Selecting iWay Big Data Integrator Deploy deploys to the edge node and runs.
    • Selecting iWay Big Data Integrator Publish only publishes to the edge node.

      Note: When defining a run configuration for a Flume, you must select the iWay Big Data Integrator Publish option from the context menu.

    Tip: If an existing run configuration is available, you can right-click that instance and select Duplicate from the context menu. You can then modify only those parameter values that are required without having to configure all of the parameters.

    A new run/build configuration pane opens.

  2. Enter a descriptive name for the new Sqoop or Flume run configuration in the Name field. Avoid using dashes or slash characters in the name.
  3. Click Browse to the right of the Deployment Build Target field.

    The Build Target Location dialog opens, as shown in the following image.

  4. Expand the main project node and then the Sqoops or Flumes subfolder.
  5. Click OK.
  6. Enter the following connection information:
    • Host Name
    • User Name
    • Password
  7. In the Deployment Location field, enter the path to the iBDI provision location where the run time object will be deployed.
  8. To specify the value for the iWay Big Data Integrator Processes area, click the green plus sign icon (+), which opens the iWay Big Data Integrator Processes Selection dialog, as shown in the following image.

    Expand the main project node, Sqoops or Flumes, and then select the Sqoop or Flume project (for example, sq2.sqoop).

    Note: More than one process can be added in one single deploy or publish operation.

  9. Click OK.
  10. Click Apply to save the information that you specified.

    To clear all of the information that you specified in this dialog, click Revert.

  11. Click Run to run the Sqoop or Flume configuration.

    The Console pane opens and displays the results for this deployment.