Non-Technical Incident Response for Quantitative Hedge Funds

The image of brokers shouting across a chaotic room to fill orders is long gone. Now trading happens through some of the most sophisticated high performing systems. The systems are run by big banks and hedge funds connect to them to handle their orders and manage their positions.

A lot of hedge funds base their trading strategy on human discretion, but a segment of them known as quantitative hedge funds take a complex automated model with limited human involvement. Humans are more prone to errors than machines and let their judgment get clouded by emotions – this is the fundamental notion that quantitative hedge funds are built around. Almost every aspect of a quant fund is automated. Hundreds of programs and validations are built around the core strategy to avoid errors. The infrastructure is enormous and needs to handle errors in real time. Minor delays can cause books to be off-hedged and can have disastrous consequences. Millions of dollars can be lost in a matter of seconds.

Needless to say, the technology stack demands a sophisticated incident response system. The system is managed by the development, IT and operations team. Each of these units contributes their own programs and scripts to the overall platform. However, the requirement for incident response is not limited to technical matters in the operations team.

Operations team brings a different flavor to the table. They are more versatile and manage a range of fully automated and semi-automated processes that ensure the systematic flow of platform. Majority of the processes managed by the operations team either runs prior to when trading begins or after trading ends. The pre-trading processes are extremely crucial as they must succeed prior to market opening. If these processes fail and are not fixed in due time, trading will not be able to start accordingly and can force the firm to decide against trading at all for the whole day. Not only will the firm lose out on potential income from trading that day, but will also have to make changes to their strategy to adjust for the missed day of trading as positions need to be adjusted daily based on market data.

The list of processes owned and managed by the operations team is fairly long. Here are some examples:

• Locate file checks
• Data vendor file checks like Barra and Factset
• Corporate action adjustments
• BOD positions check
• Trade reconciliations
• EOD positions check

These processes are more prone to failing than the rest as they rely on external files and usually require manual interference for repairs. Sometimes fixes may require the operations team to communicate with brokers and data vendors. It cannot be emphasized enough how delays can be costly. That is why it is imperative that failures in these processes are handled by an incident response service.

Let’s take locate files check as an example. Locate file contains the quantities and rates at which a list of names can be shorted by the fund if they wanted to.

• The first process entails the hedge fund sending a list of names and their respective quantities that it wants to short on a given day to its prime broker by FTP. The file is generated automatically by the quant strategy and uploaded on a storage system from where another process picks it up and sends by FTP to the primer broker. This lookup process runs at a certain time. If it does not find the file then it should crash.
• The second process runs some time later to look for the locate approval file after it is dropped on the same FTP site by the broker.

Failures in either of these processes are usually not technical in nature. They are the result of files not being present at a known location at an expected time. Delays in processing them will hold up the rest of the flow as the quant strategy will not be able to create its trade list without the locates.

That is precisely where the incident response service steps in. Alerts can be raised from the processes directly when they exit without success and be sent to the incident response system by email or API. The incident response system then dispatches the alert to the correct on-call responders persistently by email, push notifications, SMS, voice calls and chat-ops integrations like Slack.

A typical quant fund trades in multiple structured markets divided by regions – Americas, Europe and Asia. The difference in time zones is handled by splitting the team to work around the clock. Incident response services allow this to be systematically put together to set up a dynamic schedule. The schedule can be added to an escalation policy to assign back up support. And these escalation policies enable the incident response systems to determine the correct on-call responders at a time of crisis and ensure that the raised incident is addressed immediately.

Although incident response is mostly tied to technical issues, the scope of usage is far broader. In quantitative hedge fund operations it plays a key role to reduce system delays by catalyzing real time identification of incidents that arise outside the scope of a programming error or IT infrastructure failure. Whenever the need to identify an issue is real-time, whatever its nature may be, an incident response service can be of immense help. It will save you the precious minutes that matter.