Overview We are seeking a technically strong developer/data engineer to build systems for large-scale collection, analysis, classification, and monitoring of publicly available data. The role involves designing tools that can identify channels, extract metadata, analyse trends, detect behavioural patterns, and organise data into searchable datasets for commercial intelligence and analytics purposes. This is a hands-on engineering role requiring strong API, scraping, data processing, and automation experience. ________________________________________ Core Responsibilities • Build systems to collect and process video metadata at scale • Integrate with the Data API and alternative public data sources • Develop automated workflows for: o Platform discovery o keyword/topic mining o influencer identification o trend analysis o engagement analysis o comment extraction o metadata classification • Build pipelines to clean, normalise, and structure large datasets • Create systems to classify channels by: o niche o geography o language o engagement quality o audience patterns • Develop databases and indexing systems for fast querying • Implement anti-duplication and entity matching systems • Build monitoring tools for ongoing tracking of channels/videos • Create dashboards, exports, and reporting tools • Optimise collection systems for reliability and scale ________________________________________ Required Technical Skills Strong Programming Ability Candidate must be highly competent in: • Python (preferred) • Node.js / TypeScript • Go Python stack experience is strongly preferred. ________________________________________ Required API & Data Skills Strong experience with: • API v3 • quota management • pagination • channel/video/comment endpoints • search optimisation • rate limit handling Must understand: • API authentication • batching • retry logic • parallelisation ________________________________________ Web Scraping & Automation Experience with: • Playwright • Puppeteer • Selenium • BeautifulSoup • Scrapy Must understand: • dynamic content extraction • browser automation • proxy handling • anti-bot limitations • resilient scraping architectures ________________________________________ Data Engineering Skills Required experience with: • PostgreSQL • MySQL • MongoDB • Elasticsearch / OpenSearch Must be able to: • design schemas • optimise indexing • handle large datasets • create efficient query structures ________________________________________ Data Processing & Analytics Candidate should understand: • NLP basics • keyword extraction • sentiment analysis • topic clustering • tagging/classification systems • duplicate detection • statistical analysis Preferred: • experience using LLM APIs or AI classification systems ________________________________________ Infrastructure & DevOps Useful skills include: • Docker • Linux server administration • cloud infrastructure (AWS/GCP/Azure) • task queues • cron automation • distributed processing Preferred: • Airflow • Celery • Kafka • Redis ________________________________________ Frontend / Dashboard Skills (Preferred) Useful but not mandatory: • React • Next.js • dashboard development • charting/data visualisation ________________________________________ Candidate Profile Ideal candidate: • has built large-scale scraping or intelligence systems before • understands data reliability issues • can work independently • writes clean maintainable code • understands scaling and automation • can think analytically about datasets and patterns ________________________________________ Nice-to-Have Experience • Social media analytics • Influencer discovery systems • OSINT tools • Ad-tech or martech systems • Search/indexing platforms • AI-assisted classification systems • Large-scale crawler development ________________________________________ Deliverables Candidate should be capable of building: • automated collection systems • structured databases • monitoring pipelines • analytics dashboards • export/reporting tools • scalable infrastructure for ongoing data ingestion ________________________________________ Important Notes • System must comply with applicable laws and platform policies. • Focus is on analysis of publicly accessible information. • Reliability, scalability, and data quality are critical.
China Employment Verification Category: Communications, Compliance, Data Protection, Employment Law, English Translation, Human Resources, Local Job, Photography, Research, Time Management Budget: ₹12500 - ₹37500 INR
Social Media Assistance Instagram / Facebook Category: Brand Management, Content Creation, Facebook Marketing, Google Plus, Instagram, Social Media Copy, Social Media Management, Social Media Marketing, Social Media Post Design, Twitter Budget: $10 - $30 CAD
07-Jun-2026 03:56 GMT
Google Form Auto Email Setup Category: API Integration, Automation, Data Processing, Documentation, Google Sheets, HTML, JavaScript, PHP, Scripting, Software Architecture Budget: $10 - $30 USD
Home Care & E-Commerce Platform Development Category: ECommerce, Graphic Design, Payment Gateway Integration, PHP, UI / User Interface, User Experience Research, Web Application, Web Design, Web Development Budget: $30 - $250 USD
07-Jun-2026 03:52 GMT
Social Media Video Edit Category: Adobe Premiere Pro, After Effects, DaVinci Resolve, Final Cut Pro, Motion Graphics, Video Editing, Video Production, Video Services Budget: ₹12500 - ₹37500 INR
07-Jun-2026 03:52 GMT
Elegant Fashion Retail WordPress Site Category: Article Rewriting, Copy Typing, Copy.AI, Copywriting, CV Design, English (UK) Translator, Translation, Web Design, WordPress Budget: $15 - $25 USD
