TeeqScraper WordPress Plugin

A comprehensive WordPress plugin for automated domain monitoring, URL discovery, and AI-powered content generation.

Features

• 🔍 Automatic Sitemap Detection - Finds and monitors XML sitemaps automatically
• 📈 URL Monitoring - Tracks domains for new URLs and content changes
• ⏰ Smart Scheduling - Flexible monitoring intervals from minutes to daily
• 🤖 AI Auto-Posting - Generate content using OpenAI GPT integration
• 📊 Comprehensive Logging - Detailed activity tracking and reporting
• 🎛️ Admin Interface - Clean WordPress dashboard integration

Quick Start

1. Install & Activate the plugin
2. Configure Settings - Go to TeeqScraper → Settings
3. Add Domains - Go to TeeqScraper → Domains → Add New
4. Start Monitoring - Set domain status to "Active"

The plugin handles all setup automatically upon activation.

System Requirements

• WordPress 5.0 or higher
• PHP 7.4 or higher
• MySQL 5.6 or higher
• OpenAI API key (optional, for AI features)

Main Features

Domain Management

• Auto-detect sitemaps from robots.txt and common paths
• Support for custom sitemap URLs
• Manual URL list management
• Domain-specific monitoring settings

Monitoring & Discovery

• Customizable intervals (2 minutes to daily)
• Automatic new URL detection
• Meta data fetching (titles, descriptions)
• Status tracking and bulk operations

AI Content Generation

• OpenAI GPT integration
• Automated post creation
• Customizable content templates
• Scheduled publishing

Administration

• Real-time monitoring dashboard
• Comprehensive activity logs
• System diagnostics and health checks
• Manual execution controls

Documentation

• INSTALLATION.md - Complete setup guide
• CRON-SETUP.md - Advanced scheduling configuration
• Built-in help system in WordPress admin

Advanced Features

Command Line Tools

php cron-runner.php status    # Check system status
php cron-runner.php monitor   # Run monitoring manually
php cron-runner.php test      # Test functionality

Production Ready

• System cron support for better reliability
• Performance optimization for high-traffic sites
• Automated cleanup and maintenance
• Comprehensive error handling

Support

The plugin includes built-in diagnostic tools and comprehensive logging. Check the Settings page for system status and troubleshooting options.

---

Developed by TechTeeq - Professional WordPress development and AI integration services.

Features

🔍 Automatic Sitemap Detection

• Auto-detects sitemaps from robots.txt
• Checks common sitemap paths (/sitemap.xml, /sitemap_index.xml, etc.)
• Validates sitemap accessibility and format
• Supports both sitemap indexes and URL sets

🎯 Flexible URL Sources

• Auto-detection: Automatically find and use domain sitemaps
• Custom Sitemap: Specify a custom sitemap URL
• Custom URLs: Manually provide a list of URLs to monitor

⏰ Automated Scheduling

• Customizable monitoring intervals (5 minutes to daily)
• Built-in WordPress cron integration
• Manual monitoring trigger option
• Automatic cleanup of old processed URLs

📊 Comprehensive Logging

• Detailed activity logs with different severity levels
• Domain-specific log filtering
• Real-time monitoring status
• System health checks

🎛️ Admin Interface

• User-friendly domain management
• URL status tracking (pending/processed)
• Bulk URL operations
• Statistics and monitoring dashboard

Installation

1. Upload the teeqscraper folder to /wp-content/plugins/
2. Activate the plugin through the 'Plugins' menu in WordPress
3. Go to the TeeqScraper menu to start adding domains

Usage

Adding a Domain

1. Navigate to TeeqScraper > Domains
2. Click Add New
3. Enter the domain name (e.g., example.com or https://example.com)
4. Choose sitemap type:
   • Auto-detect: Plugin will find the sitemap automatically
   • Custom Sitemap: Provide a specific sitemap URL
   • Custom URLs: Enter URLs manually (one per line)
5. Click Auto-detect Sitemap to automatically find the sitemap
6. Set status to Active to enable monitoring
7. Save the domain

Monitoring URLs

The plugin automatically monitors active domains based on your configured schedule. You can:

• View discovered URLs in TeeqScraper > URLs
• Filter URLs by domain and status
• Manually update URL status (pending/processed)
• Use bulk operations for multiple URLs

Viewing Logs

Check TeeqScraper > Logs to see:

• Monitoring activity
• Error messages
• System events
• Domain-specific logs

Configuration

Go to TeeqScraper > Settings to configure:

Monitoring Settings

• Enable Monitoring: Turn automatic monitoring on/off
• Monitoring Interval: How often to check for new URLs
• Fetch Meta Data: Optionally retrieve page titles and descriptions

Cleanup Settings

• Enable Cleanup: Automatically remove old processed URLs
• Cleanup After Days: How long to keep processed URLs

Database Tables

The plugin creates three tables:

wp_teeqscraper_domains

Stores domain configurations and monitoring settings.

wp_teeqscraper_urls

Stores discovered URLs with their status and metadata.

wp_teeqscraper_logs

Stores activity logs and system messages.

API Reference

Classes

TeeqScraper_Database

Handles all database operations including domain, URL, and log management.

TeeqScraper_Sitemap_Detector

Provides sitemap detection and parsing functionality.

TeeqScraper_URL_Monitor

Manages URL monitoring and discovery processes.

TeeqScraper_Scheduler

Handles WordPress cron scheduling and task management.

TeeqScraper_Admin

Manages the admin interface and AJAX operations.

Key Functions

// Detect sitemap for a domain
$sitemap_url = TeeqScraper_Sitemap_Detector::detect_sitemap('example.com');

// Monitor a specific domain
$new_urls_count = TeeqScraper_URL_Monitor::monitor_domain($domain_id);

// Get monitoring statistics
$stats = TeeqScraper_URL_Monitor::get_monitoring_stats();

// Add a new domain
$domain_id = TeeqScraper_Database::save_domain($domain_data);

Scheduling

The plugin uses WordPress cron to schedule monitoring tasks. Available intervals:

• Every 2 minutes ⚡ (High frequency - use with caution)
• Every 5 minutes
• Every 15 minutes
• Every 30 minutes
• Hourly
• Every 2 hours
• Every 6 hours
• Twice daily
• Daily

Requirements

• WordPress 5.0 or higher
• PHP 7.4 or higher
• MySQL 5.6 or higher
• wp_remote_get() functionality enabled
• WordPress cron enabled (recommended)

Troubleshooting

Common Issues

Monitoring not working?

• Check if WordPress cron is enabled
• Verify domain accessibility
• Check error logs for specific issues

Sitemap not detected?

• Ensure the domain is accessible
• Check if robots.txt exists and contains sitemap directive
• Try manual sitemap URL if auto-detection fails

High memory usage?

• Reduce monitoring frequency
• Disable meta data fetching
• Enable cleanup for old URLs

Debug Information

Go to TeeqScraper > Settings to view:

• Plugin version
• WordPress version
• PHP version
• WP-Cron status
• Database version

Security

• All AJAX requests are nonce-protected
• User capability checks for admin operations
• Input sanitization and validation
• SQL injection prevention with prepared statements

Performance

• Batch processing for large URL sets
• Configurable monitoring intervals
• Automatic cleanup of old data
• Efficient database queries with proper indexing

Support

For support and feature requests, please visit TechTeeq.

License

This plugin is proprietary software developed by TechTeeq.

Version History

1.0.0

• Initial release
• Automatic sitemap detection
• Custom URL monitoring
• Flexible scheduling
• Comprehensive admin interface
• Detailed logging system