We are looking for an experienced data engineer to join our growing team of data analytics experts. As a data engineer at [add company name], you will be responsible for developing, maintaining, and optimizing our data warehouse, data pipeline, and data products. The data engineer will support multiple stakeholders, including software developers, database architectures, data analysts, and data scientists, to ensure an optimal data delivery architecture. The ideal candidate should possess strong technical abilities to solve complex problems with data, a willingness to learn new technologies and tools if necessary, and be comfortable supporting the data needs of multiple teams, stakeholders, and products.
### Our main datasources are:
- APIs
- Web scraping
- RSS feeds
## Responsibilities
- Design, build & optimize database design/schema for relevant data collection tasks
- Design, build and maintain data pipelines in production.
- Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources - ensuring the highest data quality standards.
- Develop ETL processes to help extract and manipulate data from multiple sources, ensuring it is de-duped, linked, and creates golden records.
- Automate data workflows such as data ingestion, aggregation, and ETL processing.
- Automate data quality measures such as writing software tests, CICD operations, automated testing, etc.
- Monitor data systems performance and implement optimization strategies.
- Leverage data controls to maintain data privacy, security, compliance, and quality for allocated areas of ownership.
## Experience
A proven track record of completing all the above mentioned tasks.
### Minimum Qualifications
- Advanced SQL skills and experience with relational databases and database design.
- Experience working with data ingestion tools
- Strong proficiency in data pipeline and workflow management tools (e.g., Airflow)
- Strong project management and organizational skills.
- Excellent problem-solving, communication, and organizational skills.
- Proven ability to work independently and with a team.
## How to apply
If you're interested in this job, here is the application process:
### Phase 1: Reply to the post
1. Read/watch the job description in its entirety. Both the written content + video (if supplied)
2. Record a loom video discussing your relevant experience to this position and how you would approach this project.
3. Include your hourly rate, timezone, working hours, availability
4. Read our [Team collaboration & How we work](ttps://gist.github.com/devinschumacher/4d5c6fd5b40270b83af74a03bbe8eadb)policy
5. Read our [Definition of Done](https://github.com/serpcompany/public/discussions/8#discussion-6390391)
### Phase 2:
If selected from Phase 1, we would start with a paid "trail task" to see how we work together on a small scale.
1. Complete a (paid) trial task that will be provided to you.
### Phase 3:
If we both like working together, we would then kick off the main project!
1. Onboarding & project kickoff!
Posted On: March 23, 2024 08:32 UTC Category: Data Engineering Skills:ETL Pipeline, Python, Apache Spark, SQL, Data Management, Data Mining, Data Cleaning, Data Quality Assessment
Skills: ETL Pipeline, Python, Apache Spark, SQL, Data Management, Data Mining, Data Cleaning, Data Quality Assessment Country: United States
click to apply
Project ID:
3377258
Project category:
ETL Pipeline, Python, Apache Spark, SQL, Data Management, Data Mining, Data Cleaning, Data Quality Assessment
