Property Scrapper for WP Residence

Property Scrapper for WP Residence (Prague)

Overview This plugin automates importing real-estate listings into the WP Residence theme. It supports a scraping mode (primary) with configurable selectors and an optional feed mode, assigns City/Area taxonomy terms, downloads media, and provides widgets/shortcodes for frontend display. It is built for Prague listings but can be adapted.

Requirements

• WP Residence theme active
• WordPress Cron enabled (for Auto Sync)
• PHP extensions: cURL, JSON, SimpleXML (standard WP stack)

Installation

1. Upload the plugin folder to wp-content/plugins/property-scrapper
2. Activate “Property Scrapper for WP Residence”
3. Go to Admin → Property Scrapper to configure settings

Quick Start

1. Geocoding tab: (optional but recommended)
   • Enter your Mapy.cz API key (if you use Mapy.cz services elsewhere).
   • Upload “Areas GeoJSON” (shapes per neighborhood) and “Street→Area CSV” mapping. If you don’t upload, bundled defaults under config/ are used.
2. Scraping tab:
   • Paste 1+ Start URLs from the target site (e.g. ceskereality.cz list pages). Legacy ceskereality paths are normalized automatically.
   • Adjust Selectors JSON to match list/detail pages.
   • Set rate limits and HTTP timeouts.
   • Use “Preview Scrape (list only)” from the tab header area to verify item discovery.
3. Import tab:
   • Choose Mode = Scraping (or Feed if implemented for your source).
   • Set Max Properties per Run, Max Images, timeouts.
   • Enable Auto Sync (WP‑Cron) and set the Cron Interval.
4. Tools/Import:
   • Click “Run Now” to trigger an on-demand import. Check the “Recent Log” at the bottom of the page.

Admin Tabs

• Import Controls import runtime behavior:

  • Enable Auto Sync (WP‑Cron): schedule recurring imports.
  • Cron Interval (hours): how often to run.
  • Max Run Time (seconds): hard cap per run.
  • Image Timeout (seconds) and Max Images per Post.
  • Max Properties per Run: cap number of new items per run.
  • Mode: Scraping (default) or Feed (if you have a compatible feed).
  • Debug mode: skip strict area matching during assignment (useful for testing).

• Geocoding Location data helpers and uploads:

  • Mapy.cz API Key: stored for use by features that integrate with Mapy.cz.
  • Areas GeoJSON: upload a file with polygons for neighborhoods/areas. Stored under uploads/property-scrapper/geo/areas.geojson (falls back to config/areas.geojson if not uploaded).
  • Street→Area CSV: upload mapping of street names to area slugs. Stored under uploads/property-scrapper/geo/street_map.csv (falls back to config/street_map.csv if not uploaded).

• Scraping Configure how the crawler finds and parses listings:

  • Rate Limit (req/min), HTTP Timeout, HTTP Retries
  • Selectors Config: JSON (see “Selectors JSON” below). Saved to config/selectors.json on save.
  • Start URLs: listing pages to crawl, one per line.
  • Preview section (after using “Preview Scrape” action): shows sample list items discovered by the current selectors.

• Logs Shows the internal logs directory path: uploads/property-scrapper/logs. A “Recent Log” tail is also shown on Import/Tools tab after actions.

• Docs Displays this README content.

• Tools Convenience actions:

  • Reassign City/Area: run full location reassignment over existing properties.
  • Seed Sample Locations (10): creates a few Prague City/Area terms.
  • Generate Listing Pages for Locations: creates WordPress pages with the shortcode for each Area.
  • Run Now: trigger an import immediately (also shown on Import tab).
  • Preview Scrape (list only): run list-page discovery and store samples for the Scraping → Preview section.
  • Download Latest CSV: retrieve the last generated export if available.
  • Delete All Property Terms: remove all property taxonomies’ terms (irreversible).

• Areas Read-only listing of current Area terms with counts.

Selectors JSON The selectors JSON controls how list pages and detail pages are parsed. It is edited in Admin → Property Scrapper → Scraping and saved to config/selectors.json when you click Save.

High-level shape (example only): { "list": { "item": "CSS selector for each result card", "url": "CSS selector (relative to item) for detail URL", "title": "CSS selector (relative to item)" }, "detail": { "title": "CSS selector for the listing title", "price": "CSS selector or XPath", "gallery": "selector for images", "address": "selector", "lat": "selector or extraction rule", "lng": "selector or extraction rule" } }

Notes

• Use Preview Scrape to validate the list.item and list.url rules first.
• Detail selectors should produce fields compatible with WP Residence meta: price, currency, address, size, condition, floor, gallery, etc.
• The importer also assigns taxonomy terms (City/Area) using GeoJSON shapes and/or the Street→Area CSV; Debug mode can skip strict validation.

Shortcodes

• [realt_ps_properties] Displays a grid of properties with pagination.

  Attributes:

  • per_page (int, default 30)
  • limit (int, deprecated; kept for compatibility)
  • title (string, optional section title)
  • city (slug of taxonomy property_city)
  • area (slug of taxonomy property_area)
  • orderby (WP_Query orderby, default date)
  • order (ASC|DESC, default DESC)

  Example (area page): [realt_ps_properties title="Byty – Karlín" area="karlin" per_page="12"]

  Pagination:

  • Uses the query arg realt_ps_page for internal navigation.

Widgets

• Mapy.cz Map (Location) Shows a Leaflet map with markers:
  • On Area/City taxonomy pages: centers on term meta gps_lat/gps_lng and loads markers from the same term.
  • On single property pages: centers on the property coords; shows nearby properties from the same Area (or City).
  • Uses OpenStreetMap tiles via Leaflet; currency symbols are derived from property_currency.

Frontend Enhancements

• On single property pages, the plugin appends a concise specs list (Condition, Size m², Floor, Price) if available.
• Admin-only debug on single property: append ?realt_ps_debug=1 to the URL to render raw meta/taxonomies as JSON (requires manage_options).

Cron & Scheduling

• Auto Sync schedules a WP-Cron event (realt_ps/sync) at your chosen interval.
• Manual run is available via “Run Now”.
• Last run time is stored in the option realt_ps_last_run and shown on the Import/Tools view.

Files & Storage

• Uploads base: uploads/property-scrapper/
  • logs/ — text logs
  • exports/ — generated CSVs
  • geo/ — uploaded areas.geojson and street_map.csv
• Config (within plugin):
  • config/selectors.json — overwritten from the Scraping tab on save
  • config/areas.geojson — default Areas GeoJSON
  • config/street_map.csv — default Street→Area CSV

Troubleshooting

• “No properties found” in shortcode: ensure imported posts exist and are published, check taxonomy slugs for city/area filters.
• Scrape preview empty: verify list.item and list.url selectors; try increasing HTTP Timeout or Rate Limit; confirm Start URLs are reachable.
• Media not downloading: raise Image Timeout and Max Images; verify remote server allows fetching.
• City/Area not assigned: upload better Areas GeoJSON or Street→Area CSV; disable Debug mode for strict matching.
• Cron not running: ensure WP-Cron is enabled on your site; try “Run Now” to test importer.

Uninstall / Reset

• Use “Delete All Property Terms” to remove property taxonomies’ terms.
• Manually delete generated pages if needed (from Tools → Generate Listing Pages).
• Remove uploads/property-scrapper/ to delete logs/exports/geo uploads.

Credits

• Author: Programmer Ikram — https://facebook.com/Programmerikram
• Leaflet map and OpenStreetMap tiles are used for the Map widget.