Citadel’s inaugural ‘Data Open’ datathon series – a data analysis marathon comprised of regional ‘Data Opens’, culminating in the ‘Data Open Championship’ held in November 2017 at the New York Stock Exchange – was “a fantastic first year and a huge success.We have met exceptional candidates and already made dozens of job offers. The Data Open has more than delivered on its aims of hiring the most talented data science and STEM students,” says Citadel’s Head of Talent Strategy, Justin Pinchback.

Founded in 1990, Citadel is one of the world’s oldest and most successful hedge fund firms. The multi-strategy firm manages over USD 27 billion in capital for a variety of partners, including corporate pensions, endowments, foundations, public institutions, and sovereign wealth funds.

US and global roll out
The ground-breaking talent discovery and assessment programme drew students from roughly twenty top-tier universities in the US, Canada, UK and Ireland. The concept is now being rolled out in additional cities across the US and globally. In 2018, Citadel will be reaching out to a wider group of US universities, and adding some in continental Europe and Asia, including top institutions in France, Switzerland and China. “Talent has no borders. We want the best intellectual athletes and we are relentless in pursuit of that talent,” enthuses Pinchback, who previously worked at Bridgewater, Bain, Goldman Sachs and Moody’s. In 2017, all datathons were conducted in English, but Citadel, which has a long history of hiring talented people from around the world, expects local languages, such as Mandarin in China, or French in France, may be used in 2018.

Customised assessment
The Data Open was the brainchild of Citadel’s Chief People Officer, L.J. Brock, who was recruited from Red Hat in early 2016. Brock was involved in every step of the Data Open’s design, while its architecture and materialisation were also driven by Pinchback, and Talent Strategist, Candice Berger, inside Citadel and by Correlation One, outside Citadel. “I came up with the idea and others then made it grander and executed it in an outstanding fashion,” says Brock.

Correlation One develops quantitative assessments for a variety of companies and developed both the problem statement for the Data Open and the custom test administered to students applying for the Datathon. “We were very objective and careful in making sure those who attend have taken the time to sit the Data Science exam, which is customised to Citadel. It tests statistics, probability, coding, and other knowledge,” says Berger.

Immersive interview experience
“Despite all its success, Citadel has a constant focus on not becoming complacent. The Data Open is perhaps the most innovative recruitment technique in the field. It is a more immersive candidate and interview experience,” says Brock. “This is accomplished via a talent audition,” explains Pinchback. “Young STEM students get an immersive experience to know what it is like at Citadel, to work on a team under time pressure and constraints, and to have a really big opportunity to create value, compete and win,” he goes on. The participants work intensely from eight in the morning to three in the afternoon, submit their work, and the judges announce winners at five the same day. (Those who made it to the Data Open Championship were given a week to complete their submissions).

Time pressure
The amount of time pressure at the individual Data Opens and the Data Open Championship was much more intense than what Soren Kunzel – who was on the champion UC Berkeley team – experienced in regular study. Prioritisation of tasks and ongoing revision of priorities was crucial. Says Kunzel, “To be precise, we knew that there was not enough time for all the things we wanted to do. And to perform well and avoid a lot of time pressure, in the end, we had to time each task very carefully”. He adds, “the most important things I learned was how to not pursue a good idea in order to have time and resources to pursue a great idea.”

Non-traditional testing
The Data Open aims to test candidates in ways that traditional methods, such as SATs, GMATs and interviews, do not. Once candidates have passed the Data Science exam, “the datathon then tests how they perform under uncertainty, and tests their creativity under unpredictable conditions,” explains Correlation One Co-Founder and CEO, Sham Mustafa. “Students are not used to dealing with open-ended problems that have no clear right answer, and require them to think creatively,” he adds. “Thus, the Datathon introduces an element that is not controlled, and tests how they think spontaneously,” he goes on.

Team players
Additionally, the datathons are designed to give an insight into team building skills. Says Kunzel “it is crucial to work efficiently as a team. It is not enough that every team member is working very hard on what they are good at, but instead, working on what is most useful for the team. For example, we all had experience in statistical modelling and machine learning but it was strategically not wise for us to all be working on machine learning at the same time.”

The Datathon funnel
Over 10,000 candidates applied for the 2017 datathon, of which 2,000 were selected across around 20 universities. At each university’s datathon, 25 or 30 teams, each composed of four students, competed, meaning that there were about 600 teams in total. Only 20 of the top teams – less than 100 students and less than 1% of applicants – made it to the final Data Open Championship. “Though certain universities are more represented at the datathon than others, those studying anywhere can apply,” says Berger. Undergraduates, postgraduate and doctoral students were all represented at the Championship, including one first year undergraduate student from MIT (known as a ‘freshman’ in the US).

Problem-solving skills
Many datathon candidates study STEM (Science, Technology, Engineering and Maths) subjects including medicine and biology (where, incidentally, a higher proportion of students have historically been women). From the winning team, Kunzel is a Ph.D. candidate in the Department of Statistics at UC Berkeley. He describes his research focus as “machine learning, causal inference, and experimental design”.

The primary datasets used need not be drawn from finance as Citadel is testing raw ability rather than specific knowledge. “We want to test how you think and problem solve rather than specialised knowledge,” says Berger. So, medical data can be used and the Data Open Championship used educational data. Citadel views the datathons partly “as a way to pique interest in finance in a diverse range of students who might not previously have contemplated a career in finance,” says Berger.

Freestyle and extracurricular
Teams can choose which analytical techniques they employ to come up with solutions. They need to show all workings, including code, submit a 20 page technical report, with graphs, and a powerpoint presentation. Citadel does not prescribe specific programming languages or modes of output. Languages used include Python, C++, R and LaTeX. “One team developed an App,” says Berger.

The winning team from UC Berkeley used both techniques that they had formally studied and those that they had not. Says Kunzel, “for example, we used theory about multiple testing, matching, and regression adjustment, and all of these concepts are important in causal inference, which is one of my core research interests. Also, we used an idea coming from stability to select important features which we have designed with the help of subject knowledge. This is directly related to some research I have done on random forests.”

Additionally, Kunzel drew on his extracurricular interests: “We also used an ensemble estimator consisting of many different estimators which are not related to my research nor have I formally studied them in a university course. I have learned about these estimators and the potential of ensemble learning mainly because of personal interests.”

Assessment criteria
The assessment process is based on multiple criteria, including methodology and substance of work. A panel of ten judges, selected together by Citadel and Correlation One, “pore over submissions, reviewing every single line of code, every word, and questioning students,” says Pinchback.

Students are tested on “their application, their performance on the entry exam, how they collaborate with their team, their creativity, rigour, and technical ability. We need to get a good read on all of these things,” says Berger. Assessment utilises Citadel and Correlation One staff and external experts. The judges included Professor Bin Yu, Chancellor’s Professor in the Departments of Statistics and of Electrical Engineering & Computer Sciences at the University of California, Berkeley; James Yeh, Citadel Head of Quantitative Strategies; and Eric Haller, Executive Vice President of Experian.

NYSE partnership
The final Data Open was held at the New York Stock Exchange, where Citadel Securities, a global market maker also founded by Ken Griffin, has become the largest designated market maker, facilitating price discovery and providing liquidity. The NYSE was a pioneer as one of the first users of telegraphs in the nineteenth century. Citadel was also a pioneer in quantitative investing. In his remarks at the event, Citadel founder and CEO, Kenneth Griffin, who taught himself to program on one of the first available PCs, recalled how the application of technology to financial markets was relatively new when he first started his career in the industry.

Citadel’s broader talent strategy
“We use a myriad of strategies and approaches for talent acquisition and are very inventive,” says Pinchback, who describes his Talent Strategy and Solutions team as “an internal strategy and innovation hub for long-term human capital solutions.”

“The Data Open is part of Citadel’s broader campus strategy, which include internships that cut across investment, data science, trading, technology and other areas. A good number of interns end up working full time at Citadel upon graduation,” says Pinchback.

Citadel is taking a multi-year view with the Data Open, and expects some of those exposed to the Datathons, especially students still early in their studies, may end up joining Citadel a few years down the line.

Not only STEM
The firm is also using the Data Open series to support its search for tech talent. “One of the big drivers for me being here and creating the role was the evolution of the competitive set. We have always competed with Facebook, Google, Amazon, etc. in the quant and data science space, but now every company in every industry, including retail, consumer products, and healthcare, is diving in to find the same talent,” says Brock.

Brock himself was hired from a technology firm, Red Hat, but in fact there was no hard-wired job specification that candidates had to come from a technology background. “Citadel wanted the best person possible to align with its culture of meritocracy and there were no false inhibitors saying they have to come from a specific sector,” he explains.

Additionally, the emphasis on technology does not mean that everyone needs a STEM background. Brock does not have a technical background, nor do some other staff. “Strong intellectual curiosity, learning agility, and great problem-solving ability are the attributes we seek, regardless of a potential candidate’s chosen degree,” explains Brock.

Citadel’s talent culture
The Data Open illustrates key tenets of Citadel’s talent culture, including: meritocracy, innovation, immediate impact, and collaboration. Says Brock: “Two things really define Citadel. Firstly, a true meritocratic environment is the centrepiece of empowering people to add value. Secondly, flatter, less hierarchic and less bureaucratic organisational and team structures let the best ideas bubble to the top and win. The culture is really focused on quickly taking advantage of opportunities.”

Meritocracy and women
The Data Open is a level playing field that typifies Citadel’s culture. “We have a flat structure with less hierarchy and bureaucracy,” reiterates Brock. “A talent audition is a true representation of a meritocratic climate. All students, regardless of gender, come onto the playing field. We don’t care who you are or where you come from. We just want the best athletes,” says Pinchback. All submissions to the judges are anonymised. “We feel really confident that the talent audition levels the playing field for trailblazing women in data science,” says Pinchback. For instance, the Princeton team at the Data Open contained three women and one man. Citadel has an internal group to help advance women’s careers, and there are many women in senior roles at the firm, including Chief Risk Officer, Joanna Welsh.

Three women from Citadel featured in The Hedge Fund Journal’s 2017 ’50 Leading Women in Hedge Funds’ report published in association with EY, namely Tania Reif, Associate Portfolio Manager at Citadel New York, Sarah Illingworth, Macro Trader at Citadel in London and Lisa Schirf, Chief Operating Officer of the Central Data Organisation and AI Research.

Innovation
Brock wants to run a talent ecosystem that gives Citadel an advantage in the talent marketplace. “We cannot get a competitive advantage in the talent ecosystem by applying the same HR techniques as everyone else.” To this end Citadel acquired a HR technology company, Gild, which uses data science and predictive analytics, about a year ago, and “is building out proprietary technology in the recruiting and talent lifecycle,” says Brock. “Our firm’s leadership is outstanding at supporting any reasonable request to ensure we have the top engine in the sector,” he adds.

Collaboration and pods
Whether or not all offices are strictly speaking, physically open plan, “collaboration is encouraged through offices organised as integrated, multi-disciplinary pods,” explains Brock. Collaboration between offices is also important. “Citadel has invested in the infrastructure to drive collaboration, which works better than anywhere else I have seen in my career,” says Brock.

Rapid impact
This means that technology staff can really make things happen on a daily basis. Tech and data staff sit near the fundamental investment staff and have a lively dialogue. One attraction of joining Citadel is the chance to have “greater impact out of the gate, and advance faster, than might be the case in larger organisations that are massive machines,” says Berger. Tech staff “can craft solutions that add value. It is a practical model and not a theoretical model,” says Brock. In common with many tech firms, casual clothes are allowed.

IQ, EQ and RQ
Citadel has a holistic view of recruitment, which includes assessing IQ (Intelligence Quotient), EQ (Emotional Quotient) and RQ (Rationality Quotient), a cognitive assessment tool developed by Correlation One. “The balance between these three is always important. Much effort goes into innovating and improving our ability to evaluate and assess all three areas in a balanced way. It is one of our hallmarks to find people with all three,” says Brock. Those who aspire to work at Citadel can check out the twitter feed @CitadelCareers or the firm’s blog.

Correlation One
Correlation One is a technology company that helps businesses win the race for top data and technical talent. It creates innovative talent attraction strategies and custom testing solutions.

“Correlation One has been a partner in the true sense of the word, absolutely phenomenal to work with and we really appreciate their entrepreneurial spirit,” says Pinchback. In turn, Correlation One “wants to partner with companies that have a futuristic focus on data science and artificial intelligence,” says Mustafa, who spotted a gap in the market because, “the technology and the process currently used to evaluate and engage data science talent is very outdated.” Correlation One developed the Datathon concept and the assessments, in partnership with Citadel. They ran several pilots to refine the competition model before it was rolled out. “We ran several smaller projects and beta tested each element of the competition to calibrate and build a model that is scalable globally,” he says.

Correlation One works with other firms from industries including financial services, insurance, healthcare and technology.