Here's a refined assignment outline for your software freelancer project, including necessary clarifications and added details for crawling medication data:
---
### Assignment: Web Crawling of Medication Data for Inhaled Drugs (ATC Code R03)
#### Project Overview Develop a web scraping tool to extract data on inhaled medications (ATC code R03) from the official medical and pharmaceutical websites of Spain, Argentina, and Chile. The extracted data should be formatted into a CSV file with specific columns as defined in the attached Excel file.
#### Target Websites - **Spain**: [CIMA](https://cima.aemps.es/cima/publico/listaclinica.html) (AEMPS - Agencia Española de Medicamentos y Productos Sanitarios) - **Argentina**: Relevant national pharmaceutical website(s) - **Chile**: Relevant national pharmaceutical website(s)
**Note**: The freelancer must identify and ensure access to reliable sources for Argentina and Chile.
#### Objectives - Extract all inhaled medication with ATC code R03. - Populate a CSV file with the specified structure (refer to the attached Excel sheet for column specifications).
#### Data Structure Each entry in the CSV should include, but is not limited to, the following columns: - `Name` - `Drug Format` - `Dosage` - `Drug Class` - `Drug Type` - `ATC Code` - `Total Puffs per Inhaler` - `Puffs a Day` - `Link` to the source page - `Image URL` of the product - `Video URL` (if available) - Additional data fields based on information structure on the source websites
Refer to the attached Excel file for the exact layout and any additional required fields.
#### Example of Data Format ```json { "dosage": 322, "drugFormat": "HFA", "drugClass": "ICS", "drugType": "ACLIDINIO BROMURO", "id": "1a478549-31ca-d61f-a9da-35cdc4616230", "image": "https://cima.aemps.es/cima/fotos/thumbnails/formafarmac/12778002/12778002_formafarmac.jpg", "link": "https://cima.aemps.es/cima/publico/detalle.html?nregistro=12778002", "name": "EKLIRA GENUAIR 322 MICROGRAMOS POLVO PARA INHALACION", "puffsADay": 0, "shortName": "ACLIDINIO BROMURO", "totalPuffsPerInhaler": 60, "video": "https://zorgatlasweb.nl/inhalatie/Films_Nederlands/Dosisaerosol.mp4", "atc_code": "R03BB05" } ```
#### Requirements 1. **Web Scraping Framework**: Use Python libraries such as `BeautifulSoup` and `Selenium` for scraping. Alternatively, any proven web scraping tool is acceptable. 2. **Data Cleaning**: Ensure consistency in data formatting and structure. 3. **Output File**: The final output must be a CSV file with columns as defined in the attached Excel template. 4. **Documentation**: Provide a clear README file explaining: - How to run the scraping tool. - Requirements for execution (e.g., Python version, dependencies). - Any challenges or limitations encountered during the project.
#### Important Notes - The tool should handle potential changes in website structure with minimal adjustments. - Data accuracy and integrity are paramount; cross-verify results to ensure reliability. - Respect website terms of service and avoid overloading sites with requests (implement proper rate limiting). - Ensure proper handling of errors and unexpected issues (e.g., CAPTCHA).
#### Deadline Specify the project completion date and any milestones for progress updates.
#### Deliverables 1. A Python script or package for web crawling. 2. A CSV file with structured data as per the given specifications. 3. README file with instructions for running the script and an overview of the project.
#### Contact Reach out for clarification or further questions related to project details.
---
Make sure the assignment template is aligned with your needs and modify as needed.
