Data Engineer
My global financial client are looking for a Data Engineer to help deliver an ecosystem of enriched and protected sets of data - created from raw, structured and unstructured sources. My client has over 300 sources and a rapidly growing book of work. We are utilising the latest technologies to solve business problems and deliver value and truly unique insights. They are looking for Data Engineers that will work on the collecting, storing, processing, and analysing of large sets of data.
The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company and to help build out some core services that power Machine Learning and analytics systems.
Key Responsibilities: Data Engineer:
* Ability to process and rationalize structured data, message data and semi/unstructured data and ability to integrate multiple large data sources and databases into one system
* Proficient understanding of distributed computing principles and of the fundamental design principles behind a scalable application
* Strong knowledge of the Big Data eco system, experience with Hortonworks/Cloudera platforms
* Practical experience in using HDFS
* Practical expertise in developing applications and using querying tools on top of Hive, Spark (PySpark)
* Strong Scala skills
* Experience in Python, particularly the Anaconda environment and Python based ML model deployment
* Experience of Continuous Integration/Continuous Deployment (Jenkins/Hudson/Ansible)
* Experience with using Git/GITLAB as a version control system.
Nice to Haves
* Knowledge of at least one Python web framework (preferably: Flask, Tornado, and/or twisted)
* Basic understanding of front-end technologies, such as JavaScript, HTML5, and CSS3 would be a plus
* Good understanding of global markets, markets macrostructure and macro economics
* Knowledge of Elastic Search Stack (ELK)
* Experience with Google Cloud Platform (Data Proc / Dataflow)
Domain Knowledge:
* Knowledge of and experience using data models and data dictionaries in a Banking and Financial Markets context.
* Knowledge of Trade Finance or Securities Services particularly useful.
* Knowledge of one or more of the following domains (including market data vendors):
* Party/Client
* Trade
* Settlements
* Payments
* Instrument and pricing
* Market and/or Credit Risk Need Experience using below languages/tools:
* Java
* HQL, SQL
* Querying tools on top of Hive, Spark (PySpark)
* Scala
* Python, particularly the Anaconda environment
* Git/GITLAB as a version control system