Data Engineer (2 positions)
Tencent (Thailand) Company Limited
Job Responsibilities and Qualifications :
- Create and Maintain Hadoop/Spark/Elastic Database to support 1,000,000,000,000 data records a year.
- Create data Model, data pipe-line for big data/real time system.
- Query and Analysis data based on business requirements.
- Implement/Improvement inhouse big data database (implemented by using HPC C++)
- Perform C++ parallel programming for Big data processing (over 1,000,000,000,000 records) on multiple machines, multiple cores and multiple GPUs.
- Optimize the Parallel programming Application, improve performance and consume less resources.
- Tuning database/coding to reduce processing time.
- Design the system to be able to process the Big data in Realtime (more than 90,000 TPS) per server with 32 logical cores with 256 GB of memory and HD Solid State Drive 500MBs I/O read/write.
- Optimize the Realtime data processing to minimize the resource consuming.
- Create REST API and dashboard(using Grafana, etc..) web page(using Python Django) to display the processed data
- In depth knowledge of C++ programming language both Windows and UNIX.
- In depth Knowledge of Python, numpy and panda.
- In depth Knowledge Hadoop, HDFS, HIVE, Spark and Elastic and have an experience to work with 1,000 billions of data records.
- Knowledge of such as PHP or Python DJango Web Development or .NET or .NET core.
- Advance knowledge of Parallel programming with traditional multithreading and OpenMP and OpenCL or Cuda development.
- Advance Knowledge of Parallel programming speed optimization.
- Knowledge of Statistic and Analytic Math.