Tag Archives: FIX

Use case: FIX messages processing. Part 2: writing and optimizing a FIX gateway

by Mikhail Vorontsov

Now, having a fast parsing method from the first part of this article, let’s try to implement a FIX gateway. Its purpose is to filter out some messages based on a some criteria. We wouldn’t discuss filtering in this article – it is very straightforward and very task-dependent. Instead we will see what we can do in order to optimize the gateway processing loop. For the beginning, let’s suppose that it is “parse-filter-compose message” loop. We have a parser, you have written a filter, now we need a compose method, which takes a list of parsed fields and returns a message string.

The following method uses a StringBuilder with initial size of 1024 bytes (it makes sense to use either average or maximal message length for StringBuilder initialization) to compose a message. It is rather straightforward, but contains a problem with double values: if values are big enough, they will be formatted in scientific “e”-notation, which means that we will not get an original message. Though, depending on the gateway clients, it may not be a problem – general purpose double parsers support scientific notation. You can read more about double -> String -> double conversion in BigDecimal vs double article.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
//compose a message from a list of parsed fields
private static String compose( final List<Field> fields )
{
    boolean first = true;
    final StringBuilder sb = new StringBuilder( 1024 );
    for ( final Field fld : fields )
    {
        if ( first )
            first = false;
        else
            sb.append( FIELD_SEPARATOR_CHAR );
        sb.append( fld.id ).append( VALUE_SEPARATOR_CHAR );
        if ( DATE_FIELDS.get( fld.id ) )
            sb.append( DATE_FORMAT.get().format( fld.value ) );
        else if ( INT_FIELDS.get( fld.id ) )
            sb.append( Integer.toString( ( Integer ) fld.value ) );
        else if ( DOUBLE_FIELDS.get( fld.id ) )
            sb.append( Double.toString( ( Double ) fld.value ) );
        else
            sb.append( fld.value );
    }
    return sb.toString();
}
//compose a message from a list of parsed fields
private static String compose( final List<Field> fields )
{
    boolean first = true;
    final StringBuilder sb = new StringBuilder( 1024 );
    for ( final Field fld : fields )
    {
        if ( first )
            first = false;
        else
            sb.append( FIELD_SEPARATOR_CHAR );
        sb.append( fld.id ).append( VALUE_SEPARATOR_CHAR );
        if ( DATE_FIELDS.get( fld.id ) )
            sb.append( DATE_FORMAT.get().format( fld.value ) );
        else if ( INT_FIELDS.get( fld.id ) )
            sb.append( Integer.toString( ( Integer ) fld.value ) );
        else if ( DOUBLE_FIELDS.get( fld.id ) )
            sb.append( Double.toString( ( Double ) fld.value ) );
        else
            sb.append( fld.value );
    }
    return sb.toString();
}

Continue reading

Use case: FIX message processing. Part 1: Writing a simple FIX parser

by Mikhail Vorontsov

In this article we will see a “real life” example: we will describe how to parse a tag-based FIX message, how to improve original parsing code. The second part of this article will be dedicated to implementing a simple gateway for FIX messages and finding out why parse-compose logic is very bad from performance point of view.

FIX messages consist of a number of fields. Each field has a name (it is decimal numerical in FIX) and a value (its datatype depends on message name). Fields are separated with 0x01 and name is separated from value with =. This is textual message format, so field 45 with value ‘test’ will look like ’45=test’. FIX also defines some binary fields, consisting of field name, field length and raw data, which may contain 0x01, but for the sake of simplicity we will not discuss them.

Message parsing: naive approach

Let’s start writing a message parser. Just for ease of reading, field separator 0x01 was replaced by semicolon in the source code. It doesn’t change any logic, only makes a message literal more readable. I’ve also replaced real FIX fields with very fake ones and left only date/int/double/string field formats. Adding more of them is straightforward, but not beneficial for this article.

The following code reads a message 20K times in the beginning – in order to compile test code and 10M times after that – for the actual test. It parses a “FIX” message string into a list of Field objects, which are field id plus field value.

Note: the actual code for this article (see link at the end of the article) is more object oriented 🙂

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
public class FixTests {
    private static final int ITERS = 10000000;
 
    private static final String MESSAGE = "1=123;5=test data;7=20120815;8=another data field, this one is rather long;" +
            "8=and one more field, looks like a repeating one;14=20120101;9=4444;21=20111231;48=one more string field to parse;" +
            "5=another field 5, why does it repeat itself?;1=123;5=test data;7=20120815;8=another data field, this one is rather long;100=144.82;102=2.25";
 
    public static void main(String[] args) {
        test( 20000 ); //to compile a method
        test( ITERS );
    }
 
    private static void test( final int iters )
    {
        long cnt = 0;
        final long start = System.currentTimeMillis();
        for ( int i = 0; i < iters; ++i )
        {
            final List<Field> fields = parse( MESSAGE );
            cnt += fields.size();
        }
        final long time = System.currentTimeMillis() - start;
        if ( iters >= 100000 )
            System.out.println( "Time to parse " + iters + " messages = " + time / 1000.0 + " sec, cnt = " + cnt );
    }
 
    private static Set<Integer> set( final int... values )
    {
        final Set<Integer> res = new HashSet<Integer>( values.length );
        for ( final int i : values )
        res.add( i );
        return res;
    }
 
    //numbers of non-string fields
    private static final Set<Integer> DATE_FIELDS = set( 7, 14, 21 );
    private static final Set<Integer> INT_FIELDS = set( 7, 14, 21 );
    private static final Set<Integer> DOUBLE_FIELDS = set( 100, 102 );
 
    private static final String FIELD_SEPARATOR = ";";
    private static final String VALUE_SEPARATOR = "=";
 
    private static final class Field
    {
        public final int id;
        public final Object value;
 
        private Field(int id, Object value) {
            this.id = id;
            this.value = value;
        }
    }
 
    //SimpleDateFormat objects are not threadsafe, so such wrapper will save us from multithreading issues
    private static final ThreadLocal<SimpleDateFormat> DATE_FORMAT = new ThreadLocal<SimpleDateFormat>()
    {
        @Override
        protected SimpleDateFormat initialValue() {
            final SimpleDateFormat sdf = new SimpleDateFormat( "yyyyMMdd" );
            sdf.setLenient( true );
            return sdf;
        }
    };
 
    private static List<Field> parse( final String str )
    {
        final String[] parts = str.split( FIELD_SEPARATOR );
        final List<Field> res = new ArrayList<Field>( parts.length );
        for ( final String part : parts )
        {
            final String[] subparts = part.split( VALUE_SEPARATOR );
            final int fieldId = Integer.parseInt( subparts[ 0 ] );
            if ( DATE_FIELDS.contains( fieldId ) )
            {
                try {
                    res.add( new Field( fieldId, DATE_FORMAT.get().parse( subparts[ 1 ] ) ) );
                } catch (ParseException e) {
                    //not production code, so ignore failure, like with numbers
                }
            }
            else if ( INT_FIELDS.contains( fieldId ) )
                res.add( new Field( fieldId, Integer.parseInt( subparts[1]) ) );
            else if ( DOUBLE_FIELDS.contains( fieldId ) )
                res.add( new Field( fieldId, Double.parseDouble( subparts[ 1 ] ) ) );
            else //string
                res.add( new Field( fieldId, subparts[ 1 ] ) );
        }
        return res;
    }
}
public class FixTests {
    private static final int ITERS = 10000000;

    private static final String MESSAGE = "1=123;5=test data;7=20120815;8=another data field, this one is rather long;" +
            "8=and one more field, looks like a repeating one;14=20120101;9=4444;21=20111231;48=one more string field to parse;" +
            "5=another field 5, why does it repeat itself?;1=123;5=test data;7=20120815;8=another data field, this one is rather long;100=144.82;102=2.25";

    public static void main(String[] args) {
        test( 20000 ); //to compile a method
        test( ITERS );
    }

    private static void test( final int iters )
    {
        long cnt = 0;
        final long start = System.currentTimeMillis();
        for ( int i = 0; i < iters; ++i )
        {
            final List<Field> fields = parse( MESSAGE );
            cnt += fields.size();
        }
        final long time = System.currentTimeMillis() - start;
        if ( iters >= 100000 )
            System.out.println( "Time to parse " + iters + " messages = " + time / 1000.0 + " sec, cnt = " + cnt );
    }

    private static Set<Integer> set( final int... values )
    {
        final Set<Integer> res = new HashSet<Integer>( values.length );
        for ( final int i : values )
        res.add( i );
        return res;
    }

    //numbers of non-string fields
    private static final Set<Integer> DATE_FIELDS = set( 7, 14, 21 );
    private static final Set<Integer> INT_FIELDS = set( 7, 14, 21 );
    private static final Set<Integer> DOUBLE_FIELDS = set( 100, 102 );

    private static final String FIELD_SEPARATOR = ";";
    private static final String VALUE_SEPARATOR = "=";

    private static final class Field
    {
        public final int id;
        public final Object value;

        private Field(int id, Object value) {
            this.id = id;
            this.value = value;
        }
    }

    //SimpleDateFormat objects are not threadsafe, so such wrapper will save us from multithreading issues
    private static final ThreadLocal<SimpleDateFormat> DATE_FORMAT = new ThreadLocal<SimpleDateFormat>()
    {
        @Override
        protected SimpleDateFormat initialValue() {
            final SimpleDateFormat sdf = new SimpleDateFormat( "yyyyMMdd" );
            sdf.setLenient( true );
            return sdf;
        }
    };

    private static List<Field> parse( final String str )
    {
        final String[] parts = str.split( FIELD_SEPARATOR );
        final List<Field> res = new ArrayList<Field>( parts.length );
        for ( final String part : parts )
        {
            final String[] subparts = part.split( VALUE_SEPARATOR );
            final int fieldId = Integer.parseInt( subparts[ 0 ] );
            if ( DATE_FIELDS.contains( fieldId ) )
            {
                try {
                    res.add( new Field( fieldId, DATE_FORMAT.get().parse( subparts[ 1 ] ) ) );
                } catch (ParseException e) {
                    //not production code, so ignore failure, like with numbers
                }
            }
            else if ( INT_FIELDS.contains( fieldId ) )
                res.add( new Field( fieldId, Integer.parseInt( subparts[1]) ) );
            else if ( DOUBLE_FIELDS.contains( fieldId ) )
                res.add( new Field( fieldId, Double.parseDouble( subparts[ 1 ] ) ) );
            else //string
                res.add( new Field( fieldId, subparts[ 1 ] ) );
        }
        return res;
    }
}

Continue reading