When the Big Data Architect designs their solution, the Big Data Engineer builds it. Big data engineers develop, maintain, test and evaluate big data solutions within organisations. Data Engineers are involved in the design of big data solutions, they have Hadoop based technologies such as MapReduce, Hive MongoDB or Cassandra. A Big Data Engineer builds large-scale data processing systems as well as being an expert in data warehousing solutions and works with the latest (NoSQL) database technologies. You'll find the latest Edinburgh Big Data Engineer jobs all in one place, all online at The Big Data Jobs Board.
A Big Data Engineer usually has experience in software engineering before moving to the field of big data. Object-oriented design, coding and testing pattern experience is required as well as engineering software platform and large-scale data infrastructure experience. Big Data Engineers should also architect highly scalable distributed systems, use open source tools. Data Engineers should know how algorithms work and be able to build high-performance algorithms.
A Big Data Engineer should be able to deal with large data sets on a daily basis, be able to solve big data problems and to develop innovative big data solutions. They should have knowledge in Java, Linux, C++, PHP, Ruby, Phyton and/or R. The Data Engineer should also be experienced in MongoDB or Redis. They should be able to build data processing systems with Hadoop and Hive using Java or Python.
A Big Data Engineer implements complex big data projects, collecting, parsing, managing, analysing and visualizing large sets of data with a view to turning information into insights using multiple platforms. The Big Data Engineer organises hardware and software design needs and develops prototypes and proof of concepts.
A Data Engineer should be proficient in designing efficient and robust ETL workflows. Hadoop solution experience is essential. The Big Data Engineer requires substantial expertise in many software development and programming fields. They should be able to develop big data solutions on premises or in the cloud. If you are looking for a Data Developer career, look no further, we have a great selection of the latest and best Big Data Engineer jobs in Edinburgh.