Configuring Apache NiFi

How to:

This section describes how to configure Apache NiFi. You can specify Apache NiFi as a streaming source stage in a Pipeline configuration.

For more information on installing Apache NiFi, see the iWay Big Data Integrator Provisioning Guide.

Procedure: How to Configure Apache NiFi

  1. Enter the following URL in your browser to access your instance of Apache NiFi:
    http://hostname:port/nifi

    where:

    hostname

    Is where Apache NiFi is installed and running.

    port

    Is the port number that was assigned to be used by Apache NiFI during the installation process. For more information, see the iWay Big Data Integrator Provisioning Guide.

    The Apache NiFi web-based user interface (UI) opens.

  2. In the Apache NiFi UI, drag a Processor object onto the canvas.
  3. Select ExecuteProcess as the type for this Processor object.
  4. Right-click the Processor object and select Configure from the context menu.

    The Processor Details dialog opens, as shown in the following image.

  5. Specify the values for the configuration properties as listed in the following table.

    Property

    Value

    Command

    tail

    Command Arguments

    /opt/nifi/nifi-1.1.2/logs/nifi-apps.log

    Batch Duration

    20sec

    Redirect Error Stream

    false

  6. Drag an Output Port object onto the canvas, and name it nifi2bdi.
  7. Click on the arrow icon in the Processor object and drag it to the Output Port object, as shown in the following image.

    A connection object will be created to complete the FlowFile, as shown in the following image.

  8. Click the red box icon within each object until it turns green, which indicates that the FlowFile has started.
  9. Return to the iWay Big Data Integrator (iBDI) Eclipse GUI.
  10. Create a new Pipeline where NiFI is specified as the Source.

    Specify the URL of your Apache NiFi instance (the server where it is running) and the name of the Output Port object that was created for Spark (nifi2bdi).

  11. For testing purposes, select console as the Target of the Pipeline.
  12. Save the Pipeline and create a new Run Configuration using the iWay Big Data Integrator Publish option.
  13. Publish the Pipeline to the edge node.
  14. Connect to the edge node using SSH or a similar tool.
  15. Navigate to the folder that was created using the iBDI Eclipse GUI. For example:
    home/myuserid/bdipipelines/mynifi/
  16. Change the directory to the /work directory:
    cd /work
  17. Edit the run.sh script file using vi (visual editor) or a similar editor.
  18. Add the following line under the cp statements section, which is located at the top of the script:
    cp ${HDM_HOME}/lib/nifi-*.jar .

    Note: Ensure that there is a space between jar and the period character (.)

  19. Save the run.sh script file and run the script using the following command:
    ./run.sh

    The NiFi streaming should begin.

    Note: If an Unsupported Major.Minor Version exception should occur, then wrong Apache NiFi libraries have been deployed for the runtime Java version.

    Check the installed version of Java on the edge node by typing which java at the command line to show the folder where Java is installed.

    Check the version of Java by typing java -version.