Church Directory Data Extraction
Overview
Structured data for thousands of churches, pulled from two directories that had no usable export. Neither could simply be listed: masstimes.org has no site search, so coverage is driven by sweeping US ZIP codes, and directory.charlestondiocese.org is walked in full. Because the ZIP sweep overlaps — the same church surfaces under several codes — every record is deduplicated so each church appears exactly once in the output.
From each listing the scraper takes the full set of details the client needed: church profile, mass timings, clergy assignments, and the church's images, downloaded and renamed to the client's own convention. Both sources run in parallel behind rotating proxies to stay ahead of rate limits at that volume, on a schedule from an Ubuntu server, writing JSON per church that syncs to Google Drive ready for the client's directory product to consume.
Outcome
One clean, deduplicated dataset and image library covering thousands of churches across both directories — nationwide mass times from masstimes.org and the full Charleston diocesan listing — delivered ready for the client's directory product.
Screens & Demos
Architecture & Pipeline
-
01
US ZIP Code Sweep
Search substitute
-
02
masstimes.org
Nationwide directory
-
03
directory.charlestondiocese.org
Diocesan directory
-
04
Parallel Scrapers
Python · Requests · BeautifulSoup
- BeautifulSoup
- Python
- Requests
-
05
Rotating Proxies
Rate-limit safety
-
06
Extract Listing
Profile · mass times · clergy · images
-
07
Deduplicate
One record per church
-
08
Image Pipeline
Renamed to the agreed convention
-
09
JSON per Church
Structured records
- JSON
-
10
Google Drive Sync
Delivery
- Google Drive
End-to-end flow, in order. Scroll sideways on a narrow screen.
Key Features
- Thousands of churches covered across both directories
- Nationwide coverage of masstimes.org via a ZIP-code sweep, standing in for the missing site search
- Deduplication so each church appears once despite overlapping ZIP-code results
- Full extraction of the Charleston diocesan directory
- Church profiles, mass schedules and clergy assignments in one structured output
- Church images downloaded and renamed to the client's naming convention
- Parallel processing for speed across directories of this size
- Rotating proxies to avoid rate-limiting
- Scheduled runs from an Ubuntu server
- JSON per church, synced to Google Drive