Idle
Overview
Companies
Crawl Jobs
HubSpot Sync
Settings
Lead Gen
Staging Bay
Shopping Scan
Compliance
UK Trends
Total Companies
—
in database
Crawled
—
ever checked
Successful
—
status = ok
Synced to HubSpot
—
pushed successfully
Unsynced
—
ok but not yet pushed
Product Count Bandings
Avg Price Bandings
Status Breakdown
Recent Crawl Runs
Run IDModeTriggered ByStartedOKFailedStatus
DomainPlatformProductsVariantsAvail. VariantsAvg PriceStatusLast CheckedHubSpot
Start Crawl
Run History
IDModeTriggeredStartedDurationTotalOKFailedStatus

Import from HubSpot

Pulls all companies from HubSpot into local DB for duplicate detection. Runs hourly automatically.

Last import—
Total fetched—
New companies—

Push to HubSpot

Pushes all ok records not yet synced. Happens automatically after every crawl.

Unsynced records—
Total synced—
Import Log
WhenTriggered ByFetchedNewUpdated
Webhook Setup (BDM Real-Time Updates)

Set this up once in HubSpot to get product counts within ~60 seconds of a BDM adding a new company.

Your webhook URL:

Loading…
  1. In HubSpot, go to Settings → Integrations → Private Apps and open your app
  2. Go to the Webhooks tab and click Create subscription
  3. Set Event type: Company created and also Company property changed → web_technologies
  4. Paste the webhook URL above as the Target URL
  5. Copy the webhook secret and add it to your Railway environment as HUBSPOT_WEBHOOK_SECRET
  6. Save and activate — new companies will be crawled within ~60 seconds
Enrichment Thresholds

These thresholds are enforced on every /api/companies response — including calls from your colleague's enrichment tool. Any company below these values will never be returned, regardless of query parameters passed.

Min Product Count Companies with fewer products than this are excluded
products
Min Variant Count Total variant count across all products
variants
Min Average Price Average price across all variants
£
Max Average Price Set to 0 for no upper limit
£
Max Products for Re-crawl Stores above this are crawled once only — skipped on future runs. Set to 0 for no limit.
Min Monthly GB Visits (SimilarWeb) Minimum monthly UK visits to qualify. Set to 0 to disable traffic filter.
✓ Saved
SearchAPI Credit Usage (this session)
—
RECENT CALLS (last 50)

Fixes truncated domains in HubSpot caused by the www-stripping bug (e.g. estbankwholesale.com → westbankwholesale.com). Checks DNS resolution to confirm corrections.

Created after:
Before:
Loading…
YoY Trends CSV

Upload the monthly Google Ads trending products CSV. The pipeline state resets automatically so the next run starts from row 1 of the new file.

Current file —
Rows —
Size —
Last uploaded —
📂
Drop CSV here or click to browse
Must have a "Product Title" column
Pipeline Status
Terms Processed
—
Domains Found
—
Yes Domains
—

Running the pipeline will: clean terms → find retailer domains → check Google Ads Transparency → push Yes domains to HubSpot → crawl Shopify product counts.

Run Options
Terms per run
How many terms to pull from the CSV each run. The pipeline resumes from where it left off, so you can run in smaller batches if needed.
All Domains Found
Yes Domains (Ads Confirmed)
Run Log
Time Terms URLs Found Yes No HS Created HS Updated HS Failed
No runs yet
Total Staged
—
Pending Checks
—
Qualified
—
Ready to Push
—
Pushed to HubSpot
—
Pipeline Actions
Step 1 — Checks

Run Ads Transparency + platform crawl on pending domains. Only domains passing both checks proceed.

Step 2 — SimilarWeb Traffic

Submit batch job to SimilarWeb. Automatically polls every 30s and collects when ready.

Step 3 — Qualify & Push

Evaluate all checks, mark qualified leads, then push to HubSpot.

Exclude BDMs from assignment (optional):
Hold Ctrl/Cmd to select multiple
Staging Domains
Domain Ads Platform Products SW Visits/mo Status
No domains in staging yet — run the pipeline to populate
Test Crawler — Quick Platform Check

Test platform detection and product counting on specific domains. No credits used — runs the crawler directly.

Verify runs_google_shopping

Queries Google Ads Transparency Center for each HubSpot company and sets runs_google_shopping to YES, NO or UNKNOWN. Uses accurate client-side format + recency detection. 1 credit per domain.

Test Specific Domains

Enter domains to test (one per line) and see the raw API response to debug detection.

CTPS & Blacklist Check

Checks all HubSpot companies and contacts against the CTPS register and domain blacklist. Runs automatically at 6am daily and after every HubSpot push.

SFTP + HubSpot credentials —
Required Environment Variables

Add these in Railway → Variables:

VariableDescription
CTPS_HUBSPOT_TOKENHubSpot token for compliance updates
SFTP_HOSTSFTP server hostname
SFTP_USERSFTP username
SFTP_PASSSFTP password
SFTP_REMOTE_DIRRemote directory (default: files)