The Job Collection Pipeline is a comprehensive system designed to crawl job postings from LinkedIn and Indeed, process the data in real-time, and store it in MongoDB. The system utilizes Docker, Kafka, Puppeteer, Spring Boot, and LLM (Large Language Model) integration to achieve its goals.
The Job Collection Pipeline is a comprehensive system designed to crawl job postings from LinkedIn and Indeed, process the data in real-time, and store it in MongoDB. The system utilizes Docker, Kafka, Puppeteer, Spring Boot, and LLM (Large Language Model) integration to achieve its goals.