Avro File Emit Service

Syntax:

com.ibi.agents.XDAvroFileEmitAgent

Description:

This service converts an XML or JSON document into Avro binary format and writes the result in an Avro container file.

If the input document is XML, the root element in the input document can be <av:avro>, in which case multiple objects will be written to the container, one per <av:item> sub-element. If the root element is <av:item>, a single object will be written to the container. Any other root element will cause a failure. The av prefix stands for the XML Namespace http://iwaysoftware.com/avro. The actual name of the prefix is immaterial. The rules to convert from XML to Avro are strict and can be read backwards from the Avro to XML rules as described in the Avro File Read Service.

If the input document is JSON, the value will typically be an array to write multiple objects to the container, one per array item. Other value types will cause a single object to be written to the container. To write a single array object, wrap it in an outer JSON array. The conversion from JSON back to Avro is lenient. The rules are described below.

The resulting container can be compressed to reduce its size. The choices for the compression algorithm are:

No Compression
BZip2
Deflate
Snappy
XZ

The default is No Compression.

Avro requires the presence of a schema. The Avro File Emit service can use the schema specified in its parameter, or it can use the schema stored in the input document as the default. The only way to store a schema in the document is to use another Avro component that produces XML, for example the Avro File listener, the Avro File Read service or the Avro File Iterator service.

The output document returned can be the input document or a status document depending on the Return property.

Parameters:

The following table describes the parameters of the Avro File Emit service.

Parameter	Description
Avro Writer Schema	Path to the Avro Schema file. If absent, the schema stored with the document will be used.
Compression Codec	Algorithm used to compress the data.
Target Directory	The target output directory.
File Pattern	The output file name, which can contain an asterisk (*) which gets expanded to a fine timestamp.
Avoid Preemitter	Indicate which preemitter(s) should be avoided.
Return	Whether the out document will be the input document or a status document
Respect Transactionality	If set, the emit respects the transactionality of the channel. If not set, the emit is always done.

Edges:

The following table describes the edges that are returned by the Avro File Emit service.

Edge	Description
success	The Avro data was successfully converted to XML.
fail_parse	An iFL expression could not be evaluated.
fail_operation	The operation could not be completed successfully.

JSON to Avro Conversion

The rules to convert from JSON to Avro are described in the following table. The natural JSON representation can be read backwards in the Avro to JSON conversion rules as described in the Avro File Read Service.

Often, other JSON data types are supported with an extra conversion.

Avro Type to Create	Accepted JSON Representation
null	null
boolean	JSON booleans map naturally. Integer numbers other than 0 map to true, 0 maps to false, and the true and false Strings map to the corresponding values. Objects and Arrays map to false.
int	JSON numbers are coerced using default Java rules. Booleans convert to 0 (false) and 1 (true), and Strings are parsed using default Java language integer parsing rules. Objects and Arrays map to 0.
long	JSON numbers are coerced using default Java rules. Booleans convert to 0 (false) and 1 (true), and Strings are parsed using default Java language integer parsing rules. Objects and Arrays map to 0.
float	Numbers are coerced using default Java rules. Booleans convert to 0.0 (false) and 1.0 (true), and Strings are parsed using default Java language integer parsing rules. Objects and Arrays map to 0.
double	Numbers are coerced using default Java rules. Booleans convert to 0.0 (false) and 1.0 (true), and Strings are parsed using default Java language integer parsing rules. Objects and Arrays map to 0.
string	Value nodes map to the string representation of the value. Objects and Arrays map to the empty string.
enum	Value nodes map to the string representation of the enum symbol. Objects and Arrays are invalid.
bytes	Value nodes map to the string representation of the value. Objects and Arrays map to the empty string. The string is then converted to bytes using ISO8859-1 encoding.
fixed	Value nodes map to the string representation of the value. Objects and Arrays map to the empty string. The string is then converted to bytes using ISO8859-1 encoding. The number of bytes must match the length of the fixed data type.
record	A JSON Object.
array	A JSON Array.
map	A JSON Object.
union	A JSON Object with a single field. The field name is the full name of the selected type. The field value is the value of the union. A JSON null is also accepted if the union contains the null type.