1001 Freelance Projects
Latest Projects from Freelance Marketplaces
Today is: 28-Mar-2024 18:43 GMT
View Project
View this project in detail (Note: you will be redirected to external marketplace)
Project title: Junior Symfony developer for website crawler
Posted by: External project from Upwork
Started: 02-Jun-2021 12:38 GMT
Description: Looking for a junior developer which wants to learn and improve their web scraping and back-end development skills. The developer will have full autonomy on the project, but at the same time will receive support and guidance from a senior developer.


This project is a content aggregator, which receives parsing requests from a queue (Amazon SQS). The requests can have three forms.

-  a sitemap (XML), in which case the app has to crawl all the links in the sitemap and gather their content and metadata;

- an RSS feed, in which case the app also crawls all the links in the feed as in the sitemap case;

- a specific URL, in which case the app only crawls that specific page and no other linked pages.


From the crawled pages, the app needs to gather some information, like the page title, the final location in case of redirects, the content of meta tags, the schema.org json data, a representative image (will be picked based on specific criteria), the age of the content (we'll discuss some techniques for determining the age of the content) and a few other data points that we'll discuss in private.


Also, from some of the crawled pages, based on certain criteria, the app will need to gather extra information, some from the page itself, other from sources like Lighthouse, grab a screenshot with Selenium (also an API) and fetch data from other external APIs.


The data collected needs to be stored in a database and in some instances, compared with the data already available.


The same project also needs a frontend interface (we already have a coded design for it). It will allow content from the database to be browsed and searched, based on specific criteria.


The required skills to complete this project are:

- PHP with (at least basic) OOP knowledge;

- SQL databases knowledge (MySQL, MariaDB, PostgreSQL);

- good communicator and self-motivated.


Skills that are not mandatory but would be nice to have (you'll learn them on the job):

- Symfony 5 experience;

- PostgreSQL;

- Docker;

- API integration;

- API development;

- data scraping;

- cURL and similar tools;

- Selenium;

- AWS;

- git and git workflows;


The tech stack is PostgreSQL for database, PHP8 with Symfony 5.3 for backend, Docker for development and testing, Sentry for error management, GitHub for versioning and AWS for deployment. It's not negotiable, but as mentioned above, if you want to learn this stack, don't be afraid to apply without prior experience with it.


Although the project is complex and some tasks will be over a junior's capabilities, we're looking specifically for juniors that want to improve their skills and gain experience on real life projects. The deadlines and work hours are flexible and you'll be able to ask for help from a senior at any point.


Posted On: June 02, 2021 12:38 UTC
Category: Back-End Development
Skills:PostgreSQL, Laravel, CodeIgniter, PHP, Symfony, MySQL, HTTP, Web Crawling, Web Scraper, Web Crawler, Selenium, API, Yii2

Skills: PostgreSQL, Laravel, CodeIgniter, PHP, Symfony, MySQL, HTTP, Web Crawling, Web Scraper, Web Crawler, Selenium, API, Yii2
Country: Romania

click to apply
Project ID: 3176020
Project category: PostgreSQL, Laravel, CodeIgniter, PHP, Symfony, MySQL, HTTP, Web Crawling, Web Scraper, Web Crawler, Selenium, API, Yii2
Project budget:
View this project in detail (Note: you will be redirected to external marketplace)
Last Projects / Browse Projects
  Project Started
3D Promotion Video for LED Tennis Court Lights
Category: 3D Animation, 3D Modelling, 3D Rendering, After Effects, Video Services
Budget: $10 - $50 USD
28-Mar-2024
17:04 GMT
Creative, Minimalist Pictorial Logo Design
Category: Graphic Design, Illustration, Logo Design, Photoshop
Budget: $30 - $250 USD
28-Mar-2024
17:02 GMT
Netsuite Order Management system Tester
Category: NetSuite, Oracle
Budget: $15 - $25 USD
28-Mar-2024
17:02 GMT
Web-Based Bitcoin & Ethereum Forecasting Tool
Category: HTML, JavaScript, PHP, Software Architecture, Web Design
Budget: $30 - $250 USD
28-Mar-2024
17:02 GMT
Lifestyle Models for Reels Production For A Digital Marketing Company
Category: Fashion Modeling, Instagram, Video Production, Video Services, Videography
Budget: $8 - $15 USD
28-Mar-2024
17:02 GMT
WordPress landing page for leads required
Category: CSS, HTML, PHP, Web Design, WordPress
Budget: ₹100 - ₹400 INR
28-Mar-2024
17:02 GMT
Android App Dev & Google Console
Category: Android, Android Studio, Flutter, Mobile App Development
Budget: ₹1500 - ₹12500 INR
28-Mar-2024
17:00 GMT
Firmware Architect for Consumer Electronics
Category: Arduino, Circuit Design, Electrical Engineering, Electronics, PCB Layout
Budget: £18 - £36 GBP
28-Mar-2024
16:58 GMT
Shopify Website Customization: Fonts & Layout
Category: CSS, HTML, Shopify, Shopify Templates, Web Design
Budget: $250 - $750 USD
28-Mar-2024
16:58 GMT
Hostile Reconnaissance Behavioural Analysis
Category: Article Writing, Report Writing, Research, Research Writing, Technical Writing
Budget: £20 - £250 GBP
28-Mar-2024
16:57 GMT
E-commerce Website Migration: Webflow to Wix
Category: ECommerce, Graphic Design, HTML, Shopping Carts, Web Design
Budget: $250 - $750 USD
28-Mar-2024
16:56 GMT
Bold, Eye-Catching Website Banner Design
Category: Banner Design, Graphic Design, Logo Design, Photoshop, Photoshop Design
Budget: $8 - $15 AUD
28-Mar-2024
16:56 GMT
Infidelity Discovery: Social Media and GPS
Category: Certified Ethical Hacking, Mobile App Development, Research
Budget: $250 - $750 USD
28-Mar-2024
16:53 GMT
Mobile App & Admin Panel Enhancements/Modifications
Category: Android, Drupal, IOS Development, Mobile App Development, PHP
Budget: $250 - $750 USD
28-Mar-2024
16:52 GMT
EXtremely talented custom CSS coder needed
Category: CSS, HTML, JavaScript, JQuery / Prototype, PHP
Budget: £10 - £20 GBP
28-Mar-2024
16:51 GMT
Browse All Projects
Projects by Skills ...
Projects for 'android'
Projects for 'ajax'
Projects for 'asp'
Projects for 'aspnet'
Projects for 'cms'
Projects for 'cpp'
Projects for 'csharp'
Projects for 'css'
Projects for 'delphi'
Projects for 'design'
Projects for 'drupal'
Projects for 'excel'
Projects for 'facebook'
Projects for 'flash'
Projects for 'html'
Projects for 'java'
Projects for 'javascript'
Projects for 'joomla'
Projects for 'iphone'
Projects for 'mysql'
Projects for 'photoshop'
Projects for 'php'
Projects for 'python'
Projects for 'ruby'
Projects for 'seo'
Projects for 'sql'
Projects for 'sysadm'
Projects for 'translate'
Projects for 'typing'
Projects for 'twitter'
Projects for 'vbnet'
Projects for 'xml'
Projects for 'wordpress'
Projects for 'writing'
Read RSS feeds ... New!
RSS feed for 'android'
RSS feed for 'ajax'
RSS feed for 'asp'
RSS feed for 'aspnet'
RSS feed for 'cms'
RSS feed for 'cpp'
RSS feed for 'csharp'
RSS feed for 'css'
RSS feed for 'delphi'
RSS feed for 'design'
RSS feed for 'drupal'
RSS feed for 'excel'
RSS feed for 'facebook'
RSS feed for 'flash'
RSS feed for 'html'
RSS feed for 'java'
RSS feed for 'javascript'
RSS feed for 'joomla'
RSS feed for 'iphone'
RSS feed for 'mysql'
RSS feed for 'photoshop'
RSS feed for 'php'
RSS feed for 'python'
RSS feed for 'ruby'
RSS feed for 'seo'
RSS feed for 'sql'
RSS feed for 'sysadm'
RSS feed for 'translate'
RSS feed for 'typing'
RSS feed for 'twitter'
RSS feed for 'vbnet'
RSS feed for 'xml'
RSS feed for 'wordpress'
RSS feed for 'writing'
New!
Проекты на русском
(Projects in Russian)

Short URL:
1001fp.com
Mobile version:
m.1001freelanceprojects.com
Copyright © 2005-2022 1001 Freelance Projects