Data Engineer

Job description

This is what you will do…

We are working on a multi-year big program where we completely redefine the way we build (data)products, including our current architecture and organisation. We do this using a domain driven architecture, with the goal of improving and aligning all our data processing and technical foundation. Part of successfully transforming the organization is to focus more on boosting our data engineering capabilities.

You will work on ingesting and processing and integrating data products for a specific domain. Some of the tasks that you will be involved with include:

• Design and implement data modelling architecture.
• Build production grade data pipelines and different processing paradigms (streaming, batch, and graph)
• Ingest data from a wide variety of sources with different sizes and formats.
• Use automated deployment tools and templates to deploy the solutions! (CI/CD)
• Work with Spark Structured Streaming, Delta Live Tables and Change Data Capture to refresh data products that require low latency!

Our stack is mainly composed of the following tools: Python, Databricks, PySpark, AWS S3, EventBridge and Lambdas.

Culture / the team …

You will be working together in a passionate team that continues to strive for being the first/best in the market, modern management, no hierarchy, and an informal atmosphere. Our Product & Tech department consists of 40 people. We are all passionate about our products, technology, and data.

Currently we are transitioning towards working in domain-based teams, with skill & expertise based chapters. We are looking for an experienced data engineer to strengthen our domain teams. In domain teams we aim to have a combination of data engineers, back-end engineers and a flexible pool of data scientists and analysts.

Currently the data engineering chapter consists of five data engineers, who work together in one of the domain teams. We expect the data engineering chapter to grow to 6-8 people in the next 2 years. In terms of management style, the organization is flat, relaxed and clear. The atmosphere is open, pleasant, social and fun. In this role you will have the opportunity to specialise in specific knowledge areas, and e.g. become a technical lead for this domain. So lot’s of opportunities for personal and technical development and growth!

This is what you’ll get from us…

We offer you a dynamic environment, international oriented with innovative, challenging, and diverse projects, using cutting-edge technology and directly benefiting our clients and our business managers at the same time.

• € 78.000 - € 86.000 euro per year including 8% holiday allowance, based on 40 hours per week
• Profit sharing
• Great premium free pension scheme
• Flexbile working hours en working in a hybrid way
• We will make sure you have all the software and hardware (your MacBook of choice) required to perform at your highest level
• Because we work with the latest technologies, we offer the opportunity to keep on developing yourself by taking courses, attending (and presenting at) conferences, e.g. free full access to Databricks Academy
• Very modern working environment, the office is new and specially furnished to the wishes of Company.info. The office is centrally located in Amsterdam and easily accessible by public transport

Functie-eisen

You already have this knowledge and experience…

• You have at least 5+ years of relevant working experience in the IT or data sector
• Extensive experience in production grade data ingestion, processing, and modelling
• Experience with cloud environments (preferably AWS)
• Preferably you have a Bachelor or Master in Computer Science, Software engineering, AI or a related field

Key technical skills and requirements

• Proficiency with Spark (preferably Databricks platform)
• Extensive experience with software development (Python)
• Experience with shipping production grade software with CI/CD
• Experience with AWS services (Lambda, Eventbridge), data stores (MySQL, PostgreSQL, Elasticsearch), and infrastructure as code (Terraform, CloudFormation)

Competencies

• Strong sense of ownership and great problem-solving skills
• Drive to learn our data domains and develop innovative solutions
• Like to coach more junior members and achieve team goals
• Great communication skills (written and spoken), in English
  

At Company.info, we recognize the value of diverse backgrounds, ideas, and experiences, and strive to create a work environment where everyone feels accepted and appreciated. We encourage applicants from all backgrounds to be part of our team, as we believe this contributes to a dynamic and inspiring work culture.
A social media check is an integral part of the selection process.

Company Info
Company.info is a Dutch company specializing in providing business information and data analysis tools. The company collects, manages, and provides comprehensive data about companies and organizations. This information is valuable for businesses and professionals involved in business decision-making, risk analysis, market research, and other business activities. Some of the key services and activities of Company.info include: company information, credit information, and data analysis tools.