Operator guide

Why Bot Blocking And 403s Stop AI Agents

Evidence-backed guide to access denial, HTTP 403, rate limits, bot blocking, and human verification outcomes in CrawlDex rows.

Read flagship report Dispute a figure

Publication gate

Indexable

This report has at least 25 qualifying evidence-backed rows and is eligible for search indexing.

1,262: ranked rows
1,377 observations: evidence
16%: need a person nearby
Jul 8, 2026: last verified

Operator guide

Current through Jul 8, 2026, 1,262 CrawlDex rows match this report's evidence criteria.

Which websites stop measurement with access denied, HTTP 403, bot-blocked, rate-limited, or human-verification outcomes?

Rows qualify when CrawlDex evidence records site-access-denied, HTTP 403, bot-blocked, CAPTCHA, rate-limit, HTTP 429, or unavailable bot-defense blockers.

What site owners can make clearer

Publish a supported automation or API path for delegated public tasks.
Return explicit error states instead of ambiguous blank or blocked pages.
Document how legitimate operators should request access for measurement or delegated work.

What agent builders should plan for

Treat access-denied, 403, rate-limit, and human-verification outcomes as blocked measurements.
Avoid claiming task completion when the browser never reached the task surface.
Use official APIs or site-provided channels when they exist.

Live rows

Ranked from current CrawlDex evidence

Rank	Site task	Verdict	AES	Evidence	Main blocker
1	Travelexinsurance insurance.travel_claim_status travelexinsurance.com	Blocked	34	3	login walls
2	Joinroot insurance.file_or_track_claim joinroot.com	Blocked	44	3	login walls
3	Molinahealthcare insurance.health_claim_or_appeal_lookup molinahealthcare.com	Blocked	44	3	login walls
4	Dominos food_local.track_delivery dominos.com	Blocked	45	4	unstable page controls
5	Dmv Ca gov_civic.renew_vehicle_registration dmv.ca.gov	Blocked	51	3	CAPTCHA checks
6	Consumerfinance gov_civic.file_complaint consumerfinance.gov	Blocked	52	3	CAPTCHA checks
7	Thrivemarket food_local.manage_grocery_delivery thrivemarket.com	Blocked	39	3	unstable page controls
8	Uspto gov_civic.search_public_records uspto.gov	Blocked	46	4	CAPTCHA checks
9	Recreation gov_civic.apply_for_permit recreation.gov	Blocked	50	3	login walls
10	My Clevelandclinic healthcare_admin.book_appointment my.clevelandclinic.org	Blocked	56	4	login walls
11	Va healthcare_admin.book_appointment va.gov	Blocked	56	4	login walls
12	Xfinity finance_bills.pay_utility_bill xfinity.com	Blocked	61	3	login walls
13	Reportfraud Ftc gov_civic.file_complaint reportfraud.ftc.gov	Blocked	39	3	CAPTCHA checks
14	Westjet travel.manage_flight_reservation westjet.com	Blocked	47	3	login walls
15	Booking customer_support.reservation_help booking.com	Blocked	53	3	login walls
16	Allianztravelinsurance insurance.travel_claim_status allianztravelinsurance.com	Blocked	60	3	login walls
17	Equifax finance_bills.credit_report_freeze_or_dispute equifax.com	Blocked	60	3	two-factor checks that need the user present
18	Experian finance_bills.credit_report_freeze_or_dispute experian.com	Blocked	60	3	two-factor checks that need the user present
19	Travelguard insurance.travel_claim_status travelguard.com	Blocked	60	3	login walls
20	Coned finance_bills.pay_utility_bill coned.com	Blocked	61	3	login walls
21	Verizon finance_bills.pay_utility_bill verizon.com	Blocked	61	3	login walls
22	Fpl finance_bills.pay_utility_bill fpl.com	Blocked	62	3	login walls
23	Sce finance_bills.pay_utility_bill sce.com	Blocked	62	3	login walls
24	Netflix subscriptions.cancel netflix.com	Blocked	51	3	login walls
25	Snapchat customer_support.account_access_help snapchat.com	Blocked	51	3	login walls
26	Zoom subscriptions.request_refund zoom.us	Blocked	55	2	login walls
27	Worldnomads insurance.travel_claim_status worldnomads.com	Blocked	60	3	login walls
28	Pennymac real_estate_moving.estimate_mortgage pennymac.com	Blocked	47	3	forced account creation
29	Irs gov_civic.check_tax_refund irs.gov	Blocked	58	3	CAPTCHA checks
30	Login legal_admin.understand_gov_identity login.gov	Blocked	59	4	two-factor checks that need the user present
31	Internationalvanlines real_estate_moving.compare_moving_quote internationalvanlines.com	Blocked	46	4	forced account creation
32	Budget travel.modify_car_rental budget.com	Blocked	48	3	login walls
33	Disneyplus customer_support.streaming_account_help disneyplus.com	Blocked	67	3	login walls
34	Blazepizza food_local.order_pickup_delivery blazepizza.com	Needs user presence	37	3	unstable page controls
35	Cvs food_local.order_pickup_delivery cvs.com	Needs user presence	39	3	unstable page controls
36	Gopuff food_local.order_pickup_delivery gopuff.com	Needs user presence	40	3	unstable page controls
37	Grubhub food_local.order_pickup_delivery grubhub.com	Needs user presence	45	3	login walls
38	Dmv Ny gov_civic.book_dmv_appointment dmv.ny.gov	Blocked	60	3	CAPTCHA checks
39	Chewy commerce.check_return_policy chewy.com	Needs user presence	46	3	login walls
40	Doordash food_local.cancel_order doordash.com	Needs user presence	56	4	login walls
41	Travel State gov_civic.schedule_passport_appointment travel.state.gov	Blocked	63	3	CAPTCHA checks
42	Apify future_agent_tasks.monitor_prices apify.com	Blocked	0	1	public url verified
43	Booking account_support.find_login_help_guidance booking.com	Blocked	0	1	public url verified
44	Booking finance_bills.find_refund_guidance booking.com	Blocked	0	1	public url verified
45	Booking travel_support.find_customer_service_guidance booking.com	Blocked	0	1	public url verified
46	Booking travel.find_cancel_booking_guidance booking.com	Blocked	0	1	public url verified
47	Booking travel.find_cancel_reservation_guidance booking.com	Blocked	0	1	public url verified
48	Booking travel.find_contact_property_guidance booking.com	Blocked	0	1	public url verified
49	Booking travel.find_refund_policy_guidance booking.com	Blocked	0	1	public url verified
50	Budget account_support.find_forgot_password_guidance budget.com	Blocked	0	1	public url verified
51	Budget customer_support.find_contact_numbers_guidance budget.com	Blocked	0	1	public url verified
52	Budget customer_support.find_rental_car_faq_guidance budget.com	Blocked	0	1	public url verified
53	Budget insurance.find_rental_car_accident_claim_guidance budget.com	Blocked	0	1	public url verified
54	Budget travel.find_modify_cancel_reservation_guidance budget.com	Blocked	0	1	public url verified
55	Budget travel.find_receipt_guidance budget.com	Blocked	0	1	public url verified
56	Capitalone account_support.find_security_center_guidance capitalone.com	Blocked	0	1	public url verified
57	Clerk dev_saas_api.read_docs clerk.com	Blocked	0	1	public url verified
58	Cloudflare account_support.find_two_factor_authentication_guidance cloudflare.com	Blocked	0	1	public url verified
59	Cloudflare business_ops.compare_vendors cloudflare.com	Blocked	0	1	public url verified
60	Cloudflare commerce.manage_account_billing_guidance cloudflare.com	Blocked	0	1	public url verified
61	Cloudflare customer_support.find_status_page cloudflare.com	Blocked	0	1	public url verified
62	Cloudflare dev_saas_api.find_pages_docs cloudflare.com	Blocked	0	1	public url verified
63	Cloudflare dev_saas_api.find_pages_docs_current cloudflare.com	Blocked	0	1	public url verified
64	Cloudflare dev_saas_api.find_pages_functions_docs cloudflare.com	Blocked	0	1	public url verified
65	Cloudflare dev_saas_api.find_turnstile_docs cloudflare.com	Blocked	0	1	public url verified
66	Cloudflare dev_saas_api.find_turnstile_docs_expansion14 cloudflare.com	Blocked	0	1	public url verified
67	Cloudflare dev_saas_api.read_docs cloudflare.com	Blocked	0	1	public url verified
68	Cloudflare dev_saas_api.read_pages_hosting_docs cloudflare.com	Blocked	0	1	public url verified
69	Cloudflare future_agent_tasks.monitor_prices cloudflare.com	Blocked	0	1	public url verified
70	Developers Cloudflare dev_saas_api.read_cloudflare_auth_docs developers.cloudflare.com	Blocked	0	1	public url verified
71	Developers Cloudflare dev_saas_api.read_pages_docs developers.cloudflare.com	Blocked	0	1	public url verified
72	Developers Cloudflare dev_saas_api.read_turnstile_docs developers.cloudflare.com	Blocked	0	1	public url verified
73	Disneyplus subscriptions.find_billing_frequency_help disneyplus.com	Blocked	0	1	bot detection blocks
74	Docs Firecrawl dev_saas_api.find_firecrawl_api_docs docs.firecrawl.dev	Blocked	0	1	public url verified
75	Docs Joinmastodon account_support.find_signup_guidance docs.joinmastodon.org	Blocked	0	1	public url verified
76	Docs Joinmastodon creator_social.find_profile_guidance docs.joinmastodon.org	Blocked	0	1	public url verified
77	Docs Joinmastodon creator_social.schedule_post docs.joinmastodon.org	Blocked	0	1	public url verified
78	Docs Joinmastodon dev_saas_api.find_api_docs docs.joinmastodon.org	Blocked	0	1	public url verified
79	Docs Joinmastodon privacy.find_account_export_guidance docs.joinmastodon.org	Blocked	0	1	public url verified
80	Docs Stripe dev_saas_api.read_stripe_error_docs docs.stripe.com	Blocked	0	1	public url verified
81	Docs Vercel dev_saas_api.read_documentation docs.vercel.com	Blocked	0	1	public url verified
82	Dominos account_support.find_rewards_help_guidance dominos.com	Blocked	0	1	public url verified
83	Google business_ops.find_cloud_status_page google.com	Blocked	0	1	public url verified
84	Harborfreight commerce.evaluate_returns_or_checkout harborfreight.com	Blocked	0	1	bot detection blocks
85	Help Google customer_support.find_public_help_center_guidance help.google.com	Blocked	0	1	public url verified
86	Help Snapchat privacy.find_data_download_guidance help.snapchat.com	Blocked	0	1	public url verified
87	Hm commerce.evaluate_returns_or_checkout hm.com	Blocked	0	1	bot detection blocks
88	Klarna finance_bills.find_refund_payment_help klarna.com	Blocked	0	1	public url verified
89	Netflix subscriptions.cancel_subscription_guidance netflix.com	Blocked	0	1	public url verified
90	Newrelic dev_saas_api.find_apm_docs newrelic.com	Blocked	0	1	public url verified
91	Newrelic dev_saas_api.read_observability_docs newrelic.com	Blocked	0	1	public url verified
92	Pinecone business_ops.find_billing_docs pinecone.io	Blocked	0	1	public url verified
93	Pypi account_support.find_two_factor_help pypi.org	Blocked	0	1	public url verified
94	Pypi customer_support.find_public_help_guidance pypi.org	Blocked	0	1	public url verified
95	Sendgrid business_ops.find_sender_authentication_docs sendgrid.com	Blocked	0	1	public url verified
96	Snapchat customer_support.find_support_bot_guidance snapchat.com	Blocked	0	1	public url verified
97	Snapchat privacy.find_data_download_guidance snapchat.com	Blocked	0	1	public url verified
98	Spotify account_support.find_password_reset_guidance spotify.com	Blocked	0	1	public url verified
99	Support Google customer_support.find_gmail_missing_email_guidance support.google.com	Blocked	0	1	public url verified
100	Support Google customer_support.find_public_support_guidance support.google.com	Blocked	0	1	public url verified