iWay Big Data Integrator User's Guide > Creating a Flume Configuration > Flume Parameters Reference

Flume Parameters Reference

Topics:

This section provides a reference for the Flume parameters that are available in the three tabs of the Flume Details pane (Source, Channel, and Sink), as shown in the following image.

Common Parameters

The following parameters are common to HTTP and Avro source types:

  • Enable SSL check box (on | off)
  • Key Store
  • Key Store Password
  • Key Store Type
  • Port

Source Parameters

Topics:

This section provides a reference for the Flume parameters that are available in the Source tab of the Flume Details pane.

Avro

  • Bind
  • Compression Type (blank or deflate)
  • Threads
  • Integer max number of worker threads to accept from clients

HTTP

  • Bind

Exec

  • Batch Size
  • Batch Timeout
  • Charset
  • Command
  • Log Standard Error check box (on | off)
  • Restart check box (on | off)
  • Restart Throttle
  • Shell

SpoolDirSource

  • Batch Size
  • Delete Policy
  • Serializer
  • File Header Key
  • File Suffix
  • Ignore Pattern
  • Input Charset r
  • Is File Header check box (on | off)
  • Tracker Directory

ExecSource

  • Batch Size
  • Batch Timeout
  • Charset
  • Command
  • Log Standard Error check box (on | off)
  • Restart check box (on | off)
  • Restart Throttle
  • Shell

JMSSource

  • Batch Size
  • Connection Factory
  • Converter Charset
  • Converter Type
  • Destination Name r
  • Destination Type r
  • Initial Context Factory r
  • Message Selector
  • Password File
  • Provider Url r
  • User Name

Channel Parameters

Topics:

This section provides a reference for the Flume parameters that are available in the Channel tab of the Flume Details pane.

Memory Channel

  • Byte Capacity
  • Capacity
  • Keep Alive
  • Transaction Capacity

File Channel

  • Backup Checkpoint Dir
  • Checkpoint Interval
  • Data Dirs
  • Keep Alive
  • Max File Size
  • Minimum Required Space
  • Transaction Capacity
  • Use Dual Checkpoints check box (on | off)
  • capacity
  • CheckpointDir

Sink Parameters

Topics:

This section provides a reference for the Flume parameters that are available in the Sink tab of the Flume Details pane.

Avro Sink

  • Batch Size
  • Compression Level
  • Compression Type
  • Connection Timeout
  • Exclude Protocols
  • HDFS Path check box (on | off)
  • Host Name r
  • Max IO Workers
  • Port r
  • Request Timeout
  • Reset Connection Interval
  • SSL check box (on | off)
  • Truststore
  • Truststore Password
  • Truststore Type

Kafka Sink

  • Broker List
  • Topic

HDFS Sink

  • Batch Size
  • Call Timeout
  • CodeC
  • File Prefix
  • File Suffix
  • File Type
  • Idle Timeout
  • In use Prefix
  • In use Suffix
  • Kerberos key tab
  • Kerberos Principal
  • Max Open files
  • Path
  • Proxy User
  • Roll Count
  • Roll Interval
  • Roll Timer Pool Size
  • Round check box (on | off)
  • Round Unit
  • Round Value
  • Threads Pool Size
  • Time Zone
  • Use Local TimeStamp check box (on | off)
  • Serializer

Flume File Name Escape Sequences

The following table lists and describes the Flume file name escape sequences.

Escape Sequence

Description

%{host}

Substitute value of event header named host. Arbitrary header names are supported.

%t

Unix time in milliseconds

%a

Locale's short weekday name (Mon, Tue, etc.)

%A

Locale's full weekday name (Monday, Tuesday, etc.)

%b

Locale's short month name (Jan, Feb, etc.)

%B

Locale's long month name (January, February, etc.)

%c

Locale's date and time (Thu Mar 3 23:05:25 2005)

%d

Day of month (01)

%e

Day of month without padding (1)

%D

Date, same as %m/%d/%y

%H

Hour (00...23)

%I

Hour (01...12)

%j

Day of year (001...366)

%k

Hour (0...23)

%m

Month (01...12)

%n

Month without padding (1...12)

%M

Minute (00...59)

%p

Locale's equivalent of AM or PM

%s

Seconds since 1970-01-01 00:00:00 UTC

%S

Second (00...59)

%y

Last two digits of year (00...99)

%Y

Year (2010)

%z

+hhmm numeric timezone (for example, -0400)