# @lg-tools/crawler

## 0.0.2

### Patch Changes

- ## Added

  - Implemented prune command in CLI
  - Added prune script in package.json
  - Created CrawlerDocument interface for better type safety
  - Implemented robots.txt checking functionality
  - Added new utility function newURL for enhanced URL processing

  ## Changed

  - Updated SOURCES in constants.ts to include additional URLs and collections
  - Changed log color to green in processSingleUrl for better visibility
  - Refactored crawler logic to improve URL processing
  - Enhanced logging in recursive crawling functionality
  - Improved URL processing with better logging

  ## Updated

  - Refactored crawler constants for better organization
  - Updated various log formats and display