In the US, mostly yes, if the pages are public and you aren’t logged in. Courts have ruled that reading a public page isn’t hacking, even with a bot and even after the site tells you to stop. Scrapers who lose get caught on something else, and the best-known case shows how.

hiQ, the startup that won the “public pages aren’t hacking” ruling twice, still agreed to a $500,000 judgment and a permanent ban. It had accepted LinkedIn’s terms when it bought ads. Bright Data had Facebook and Instagram accounts for its own ads too, and beat Meta, because it scraped logged out.

The new risk, since July 2026, is bot checks. Four rulings let platforms and creators sue companies accused of getting past them, under the DMCA, a copyright law. That law protects posts, photos and videos, not facts: a follower count isn’t copyrighted, a video is. None of this is legal advice.

Public pages, logged out: likely not hacking

The federal anti-hacking law, the CFAA, covers accessing a computer “without authorization”. The Supreme Court read that in 2021 as a question of gates: did you go somewhere that was closed to you? Applying it to hiQ, which scraped public LinkedIn profiles, the Ninth Circuit found that a public website “has erected no gates to lift or lower in the first place”. A cease-and-desist letter didn’t change that.

Logging in is different. Power Ventures kept pulling users’ data from behind Facebook’s login after Facebook told it to stop, and the Ninth Circuit called that access without authorization in 2016. Even then, a company suing under the CFAA has to show $5,000 of loss in a year. Ryanair won a jury verdict against Booking.com and had it set aside in 2025, because it could prove only $2,457.72.

Having accounts doesn’t put you behind the login either. Bright Data scraped Meta logged out, and the judge wrote that “a visitor is not a ‘user’” bound by Meta’s terms. X’s suit against Bright Data was dismissed the same year: rate limits and CAPTCHAs didn’t make public posts “any less public”.

What courts have said so far, by what the scraper did.

Terms you accepted: hiQ won twice and ended with a $500,000 judgment

hiQ won the hacking question twice, then lost on the terms it had accepted: $500,000 and a ban.

Most articles call hiQ a win for scrapers. It ended in December 2022 with hiQ agreeing to a $500,000 judgment and a permanent ban on scraping LinkedIn “whether logged in to a LinkedIn account or not”.

The hacking rulings never touched the contract. hiQ had accepted LinkedIn’s User Agreement when it bought ads and subscriptions, and that agreement bans scraping. The trial judge found hiQ in breach, and found that contractors it hired had breached it too by making fake accounts. A month later hiQ settled.

The same judge read Meta’s terms the other way in the Bright Data case: they reached only scraping done while logged in. So whether paying a platform ties your scraping to its terms comes down to their wording.

Terms can bind you even if you never clicked Accept. Reddit’s sit behind a link saying that using the site means accepting them, and in March 2026 a federal judge said that could still bind Anthropic: Reddit alleges Anthropic’s bots kept visiting more than 100,000 times after Anthropic said it had stopped.

Reddit, X, LinkedIn and Meta all ban scraping without permission in their terms. X’s add $15,000 in liquidated damages for requesting, viewing or accessing more than 1,000,000 posts in 24 hours.

In 2026, the fight moved to bot checks

Hacking-law suits have gone the scrapers’ way since 2019. Suits over bot checks only started with four rulings, July to October 2026.

With public pages mostly out of the hacking law’s reach, platforms and creators turned to the DMCA. Its section 1201 bars getting around a measure that “effectively controls access” to a copyrighted work. Between July and October 2026, four rulings let claims like that go ahead against companies accused of getting past bot checks.

• Reddit v. SerpApi: a New York judge said Google’s bot check, which lets people through and keeps automated users out, plausibly controls access to the Reddit posts in Google’s results. Reddit’s claims go on against SerpApi and Perplexity.
• Ted Entertainment v. Snap, ByteDance and Apple: YouTube creators say the companies got past YouTube’s protections to scrape their videos. Apple argued that public videos have no access control at all, and the judge in its case wrote that this “defies logic and Ninth Circuit precedent.”

In 2024, X v. Bright Data said CAPTCHAs didn’t make posts any less public, but X had sued over its terms. Under the DMCA, these courts treat the check itself as a possible access control.

The DMCA has a built-in limit: it protects copyrighted works only. Google’s own suit against SerpApi failed for search results with no copyrighted content in them. Facts such as prices or follower counts aren’t copyrighted. Posts, photos and videos are, so most social media content is covered.

All four are rulings on motions to dismiss. The judges took the complaints’ facts as true and let the cases go on; none found that anyone broke the law.

Six answers to have before you scrape

A lawyer will ask you these:

• Are you logged in, with any account, while you collect? LurkAPI, which publishes this blog, reads public pages and doesn’t log in to anyone’s account.
• Has your company accepted the platform’s terms, and do they mention scraping?
• Has the site told you to stop?
• Is there a bot check between you and the page?
• Are you copying posts, photos or videos, or only facts about them?
• Is any of it personal data about people in the EU or UK?

In Europe, public personal data is still personal data

California’s privacy law leaves “publicly available information” out. The GDPR and the UK’s version of it have no such exception: data about a person is personal data wherever it was posted, and a company that collects it owes those people notice, unless that would take disproportionate effort. France’s regulator fined KASPR €240,000 in 2024 for a Chrome extension that pulled LinkedIn users’ contact details into a database of about 160 million contacts, some of them from people who had shown their details only to their connections.

Questions

Is web scraping legal?

In the US, scraping public pages without logging in is likely not hacking under the CFAA, the Ninth Circuit said in hiQ v. LinkedIn (2019 and 2022). Scrapers can still be sued over terms they accepted, fake accounts, getting past bot checks (under the DMCA) and personal data.

Is it legal to scrape social media?

Courts have sided with scrapers of public, logged-out pages: judges ruled against Meta and X in their 2024 suits against Bright Data. Companies that had accepted the terms or were accused of using fake accounts, hiQ in 2022 and ProAPIs in 2026, agreed to bans.

Is scraping LinkedIn legal?

Scraping public LinkedIn profiles is likely not hacking, the Ninth Circuit said in 2022. But hiQ, which had accepted LinkedIn’s User Agreement, was found in breach and agreed to a $500,000 judgment and a permanent ban in December 2022.

Can you get sued for web scraping?

Yes. Platforms and creators have sued scrapers for breach of contract, trespass, hacking under the CFAA, trade-secret theft and DMCA circumvention. Between July and October 2026, four rulings let DMCA claims go ahead against companies accused of getting past bot checks.

Is web scraping legal in the EU?

Public personal data is still personal data under the GDPR, so its rules apply to scraping it. France’s CNIL fined KASPR €240,000 in 2024 for collecting LinkedIn users’ contact details.

When youtube-transcript-api works on your laptop and raises RequestBlocked on AWS, a VPS, Colab or a CI runner, your code is fine. YouTube is refusing your server’s IP: it answers requests from cloud and datacenter addresses with “Sign in to confirm you’re not a bot” instead of captions, and the library gives up in under a second.

From a fresh cloud server, 1 of 10 popular videos comes back; the other nine raise RequestBlocked. Two fixes still work: send the library through rotating residential proxies, or let a hosted transcript API make the request so your server never talks to YouTube. The third fix the error message offers, cookies, is switched off in the current release.

The block hides inside an HTTP 200

Each fetch is three requests. The library loads the video’s watch page to read an API key, posts to YouTube’s internal player endpoint as the Android app, then downloads the caption track the player lists. From a datacenter IP the first request goes through: about 1.3 MB of HTML, key included, no captcha. The second also returns 200, but its body says LOGIN_REQUIRED, with the bot check as the reason. The library raises RequestBlocked and never sends the third.

Both requests return 200; the second one’s body is the bot check.

So curling youtube.com from your server proves nothing: the page loads and every status code is 200. Without a proxy config the library doesn’t retry, and retrying yourself from the same IP gets the same answer.

IpBlocked is the other flavour: an HTTP 429, or a reCAPTCHA page where the watch page should be. It subclasses RequestBlocked, so one except RequestBlocked catches both. Versions before 1.0.0 reported the same block as TranscriptsDisabled (“Subtitles are disabled for this video”), so if an old pinned version swears a popular video has no subtitles, it’s probably being blocked too.

A green smoke test can hide a blocked server

On that blocked server, one video still worked: Rick Astley’s Never Gonna Give You Up, dQw4w9WgXcQ, the go-to test video. It came back all three times. Nine other popular videos raised RequestBlocked. A deploy check that fetches only that id goes green while the jobs behind it fail.

Check a server with the videos your job needs. If most of them raise RequestBlocked, it’s the IP.

Check your server · Python

from youtube_transcript_api import YouTubeTranscriptApi

api = YouTubeTranscriptApi()
ids = [  # use your own
    "dQw4w9WgXcQ",
    "kJQP7kiw5Fk",
    "arj7oStGLkU",
]
for vid in ids:
    try:
        t = api.fetch(vid)
        print(vid, "OK", len(t))
    except Exception as e:
        print(vid, type(e).__name__)

# From a cloud server:
# dQw4w9WgXcQ OK 61
# kJQP7kiw5Fk RequestBlocked
# arj7oStGLkU RequestBlocked

Cookies are off in 1.2.4, and proxies need upkeep

The error message offers cookies as its second fix, with a warning that YouTube will eventually ban the account you log in with. In 1.2.4, the current release, cookie auth is switched off (“Cookie auth has been temporarily disabled”), CLI flag included.

Its first fix is a proxy. The README recommends rotating residential proxies, because YouTube bans static ones after extended use. There’s a built-in config for one residential provider, and GenericProxyConfig for any other:

Any proxy · adapted from the README

from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api.proxies import GenericProxyConfig

proxy = GenericProxyConfig(
    http_url="http://user:pass@my-custom-proxy.org:port",
    https_url="https://user:pass@my-custom-proxy.org:port",
)
ytt_api = YouTubeTranscriptApi(proxy_config=proxy)
ytt_api.fetch(video_id)

Know how retries behave before you buy traffic. The built-in provider config retries a blocked video up to 10 times; GenericProxyConfig tries once unless you subclass it and raise retries_when_blocked. Every attempt starts over at the watch page, so each refusal still costs a full page of traffic, and residential traffic is billed by the gigabyte.

If the videos are your own, there’s an official route. The YouTube Data API’s captions.download needs OAuth and permission to edit the video, and each call costs 200 of your 10,000 daily quota units, so 50 downloads a day. It can’t read anyone else’s videos.

Or take YouTube off your server: one GET

With a hosted API, your server makes one HTTPS call and the YouTube side happens elsewhere, so it stops mattering where your code runs. LurkAPI’s transcript endpoint fetches the captions for you, and a cached repeat comes back in 20 ms.

Same ten videos: 1 from the library, 10 from LurkAPI.

Video transcript: What's said in a video or Short: its captions as timed segments and as one block of text.
GET https://api.lurkapi.com/v1/youtube/video/transcript · 1 credit per call · Read the docs

Get a transcript · 1 credit

curl "https://api.lurkapi.com/v1/youtube/video/transcript?url=https://www.youtube.com/watch?v=arj7oStGLkU" \
  -H "x-api-key: $LURK_KEY"

The JSON, trimmed to 2 segments

Response

{
  "success": true,
  "credits_charged": 1,
  "videoId": "arj7oStGLkU",
  "url": "https://www.youtube.com/watch?v=arj7oStGLkU",
  "language": "English",
  "languageCode": "en",
  "isGenerated": false,
  "transcript": [
    { "text": "So in college,", "startMs": 12645, "endMs": 14015, "startTimeText": "0:12" },
    { "text": "I was a government major,", "startMs": 15349, "endMs": 16913, "startTimeText": "0:15" }
  ],
  "transcript_only_text": "So in college, I was a government major, …",
  "availableLanguages": [
    { "code": "ar", "name": "Arabic", "isGenerated": false },
    { "code": "hy", "name": "Armenian", "isGenerated": false },
    { "code": "bg", "name": "Bulgarian", "isGenerated": false }
  ]
}

Each call is 1 credit: $0.45 to $0.90 per 1,000 transcripts depending on the pack, and a new account starts with 100 free credits.

Switching from the library takes a few lines. Segments come back as text, startMs and endMs; to_snippet converts one to the library’s text, start and duration:

Python · 1 credit

import requests

API = ("https://api.lurkapi.com"
       "/v1/youtube/video/transcript")
KEY = "lk_live_..."

def transcript(video_id):
    r = requests.get(
        API,
        params={"url": video_id},
        headers={"x-api-key": KEY},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["transcript"]

def to_snippet(seg):
    s = seg["startMs"]
    e = seg["endMs"]
    return {
        "text": seg["text"],
        "start": s / 1000,
        "duration": (e - s) / 1000,
    }

segs = transcript("arj7oStGLkU")
print(len(segs), to_snippet(segs[0]))
# 315 {'text': 'So in college,',
#  'start': 12.645, 'duration': 1.37}

One default differs. The library asks for English only and raises NoTranscriptFound otherwise; the endpoint returns English when the video has it and falls back to the video’s main language when it doesn’t. Pass language=es (or pt-BR, and so on) for a specific track, and check languageCode if you need English. A video with no captions returns 404 not_found.

For a whole channel, list its uploads with channel videos and make one transcript call per video. For one transcript without code, the free YouTube transcript tool runs the same endpoint in your browser, or try it here:

Run GET /v1/youtube/video/transcript in the live playground

Questions

What does RequestBlocked mean in youtube-transcript-api?

YouTube answered the library’s player request with “Sign in to confirm you’re not a bot” instead of captions, almost always because the request came from a cloud or datacenter IP. IpBlocked, a subclass, means an HTTP 429 or a reCAPTCHA page; except RequestBlocked catches both.

Why does youtube-transcript-api work locally but not on a server?

Same code, different IP. YouTube blocks most IPs that belong to cloud providers such as AWS, Google Cloud and Azure, and a home IP can be blocked too after too many requests.

Do cookies work with youtube-transcript-api?

Not in 1.2.4. Cookie auth is switched off in the code, CLI flag included, even though the RequestBlocked message still suggests it.

How do I use a proxy with youtube-transcript-api?

Pass a proxy_config to YouTubeTranscriptApi: the built-in config for one residential provider, or GenericProxyConfig(http_url=..., https_url=...) for any other. The README recommends rotating residential proxies, since static ones get banned after extended use.

Is there an official YouTube transcript API?

Only for videos you can edit. The YouTube Data API’s captions.download needs OAuth and edit permission, and costs 200 of the default 10,000 daily quota units per call. For anyone else’s videos, LurkAPI’s transcript endpoint returns the captions for 1 credit a call.

If Chrome opens a page and your Python script gets 403 Forbidden, it’s usually a bot wall in front of the site that refused you: Cloudflare, DataDome, Akamai, PerimeterX or AWS WAF. Walls judge how your client connects (the TLS handshake, the HTTP version and its settings, the order of your headers) and compare that with the name in your User-Agent. requests gets every one of those wrong for a browser, which is why pasting in Chrome’s User-Agent so rarely helps.

The 403 also says which wall sent it and whether it’s a block or a challenge, and each has a different fix.

Send Chrome’s handshake, not just its User-Agent

requests introduces itself as python-requests/ plus a version number, so swapping the User-Agent feels like the obvious fix. Put Chrome’s User-Agent on a plain HTTP client, though, and Kickstarter and Priceline still answer 403 with the same block page as before. The rest of the request still says “script”: requests negotiates TLS through OpenSSL with its own ciphers and extensions, and it only speaks HTTP/1.1, where Chrome talks to these sites over HTTP/2. Cloudflare calls the result a JA3 or JA4 fingerprint, and a Chrome name over a non-Chrome fingerprint is an easy catch.

The same request three ways. Chrome’s User-Agent alone still gets a 403 from Kickstarter and Priceline; Chrome’s handshake gets both pages and six more, and IMDb answers it with a JavaScript challenge (a 202).

Send the same request with Chrome’s handshake, HTTP/2 settings and header order, and both pages load, and so do six more walled sites. In Python, curl_cffi gives you that handshake with one argument:

curl_cffi · Python

from curl_cffi import requests

r = requests.get(url,
                 impersonate="chrome")

Read the 403: each wall signs its response

Each vendor marks its response:

Check the headers first, then the cookies, then the page. The first match names the wall.

Don’t read server: cloudflare as a Cloudflare block: Priceline’s 403 travels through Cloudflare, but the page is PerimeterX’s. And Akamai’s _abck and bm_sz cookies show up on pages that load fine; they mean Akamai is there, not that you’re blocked.

To check in code, pass the status, headers and body to this function. It looks for the challenge headers and Akamai’s page before it looks at the status, because those can arrive with a 202 or even a 200.

which_wall() · Python

import re
import requests

AKAMAI_PAGE = re.compile(
    r"errors\.edgesuite\.net|Reference #18")

def which_wall(status, headers, body):
    h = {k.lower(): v for k, v in headers.items()}
    via = h.get("via", "").lower()
    if h.get("cf-mitigated") == "challenge":
        return "Cloudflare challenge"
    if h.get("x-amzn-waf-action"):
        return f"AWS WAF {h['x-amzn-waf-action']}"
    if AKAMAI_PAGE.search(body):
        return "Akamai block"
    if status not in (401, 403, 429, 503):
        return None
    if "x-datadome-cid" in h or "captcha-delivery.com" in body:
        return "DataDome block"
    if "px-captcha" in body or "_pxAppId" in body:
        return "PerimeterX block"
    if h.get("server", "").lower() == "akamaighost":
        return "Akamai block"
    if "cf-ray" in h:
        return f"Cloudflare {status}"
    if "Request blocked" in body and "cloudfront" in via:
        return "AWS WAF block"
    return f"Unbranded {status}"

r = requests.get(url)  # your request
print(r.status_code, which_wall(r.status_code, r.headers, r.text))

In axios, a 403 throws AxiosError: Request failed with status code 403, and the same three inputs are err.response.status, err.response.headers and err.response.data.

What each wall’s block page looks like

kickstarter.com · Cloudflare challenge

HTTP/2 403
server: cloudflare
cf-mitigated: challenge
cf-ray: a480d8d37e259aac-IAD
set-cookie: __cf_bm=…

<title>Just a moment...</title>
<noscript>… Enable JavaScript and cookies to continue …</noscript>
<script>(function(){window._cf_chl_opt = {… cType: 'managed', …};
… a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/…'; …

marketwatch.com · DataDome, as a 401

HTTP/2 401
server: CloudFront
x-datadome: protected
x-datadome-cid: …
x-dd-b: 2
set-cookie: datadome=…; Max-Age=31536000; Domain=.marketwatch.com; …

<title>marketwatch.com</title>
<p id="cmsg">Please enable JS and disable any ad blocker</p>
<script>var dd={'rt':'c','cid':'…','hsh':'…','t':'bv',…,
'host':'geo.captcha-delivery.com','cookie':'…'}</script>
<script src="https://ct.captcha-delivery.com/c.js"></script>

sephora.com · Akamai

HTTP/2 403
server: AkamaiGHost
akamai-grn: 0.c424c317.1791585316.3a9f823

<TITLE>Access Denied</TITLE>
<H1>Access Denied</H1>
You don't have permission to access "http://www.sephora.com/"
on this server.<P>
Reference #18.c424c317.1791585316.3a9f823
<P>https://errors.edgesuite.net/18.c424c317.1791585316.3a9f823</P>

priceline.com · PerimeterX through Cloudflare

HTTP/2 403
server: cloudflare
cf-cache-status: DYNAMIC
set-cookie: _pxhd=…

<meta name="description" content="px-captcha">
<title>Access to this page has been denied</title>
window._pxAppId = 'PX9aTjSd0n';
script.src = 'https://captcha.px-cloud.net/PX9aTjSd0n/captcha.js?…';
Press & Hold to confirm you are<br>a human (and not a bot).

imdb.com · AWS WAF challenge, as a 202

HTTP/2 202
x-amzn-waf-action: challenge
via: 1.1 ….cloudfront.net (CloudFront)

<title></title>
window.gokuProps = { "key":"…", "iv":"…", "context":"…" };
<script src="https://….token.awswaf.com/…/challenge.js"></script>
AwsWafIntegration.getToken().then(() => { window.location.reload(true); });

Some blocks don’t look like errors

raise_for_status() only fires on a 4xx or 5xx, and axios only throws outside 2xx. Walls don’t always answer that way:

• Lowe’s sends Akamai’s Access Denied page with a 200. A scraper that trusts the status saves the block page as data.
• IMDb’s AWS WAF challenge, which it sends once your handshake looks like Chrome’s, comes back as a 202. Both libraries treat that as success.
• MarketWatch’s DataDome block is a 401, which reads like a login problem.
• Best Buy doesn’t answer at all. It drops the connection, so you get a connection error instead of a 403.
• Kickstarter gives Chrome’s handshake a 200, but its homepage is a JavaScript app: the content appears only once a browser runs it.

So run the check on every response before you parse it, 200s included, and always set a timeout.

A block needs a better client, a challenge needs a browser

A block refuses your client or your IP. Fix the client first with Chrome’s handshake. If the block survives that, the wall is judging your IP (DataDome scores IP reputation alongside the client), and the fix is a different network.

A challenge is a page that runs JavaScript to check that a browser is there. A good handshake often avoids it: Kickstarter challenges a plain client and serves Chrome’s handshake the page. When the challenge comes anyway, as IMDb’s does, only a browser that runs the script gets through, which means Playwright or Puppeteer and a Chrome to keep running.

Or let one call climb for you

LurkAPI’s scrape endpoint climbs that ladder for you: a quick direct request, then Chrome’s handshake from a static ISP IP, then a browser, then other networks for walls that judge the IP. It stops at the first step that gets the page and charges for that step: 1 credit up to the browser, and more only for the other networks. A call that every step fails costs nothing.

Three URLs, one endpoint. Sephora opens on the first, cheapest step and Priceline on Chrome’s handshake (1 credit each); Chrome’s handshake gets only Kickstarter’s JavaScript shell, so it goes on to the browser (1 credit). 3 credits for the three pages.

Priceline needed one step up:

Scrape Priceline’s homepage · 1 credit

curl -G "https://api.lurkapi.com/v1/web/scrape" \
  --data-urlencode "url=https://www.priceline.com/" \
  -d format=markdown \
  -H "x-api-key: $LURK_KEY"

Response · trimmed

{
  "success": true,
  "credits_charged": 1,
  "status": 200,
  "url": "https://www.priceline.com/",
  "content": "Skip to main content\n…\n- [Hotels](https://www.priceline.com/hotels)\n- [Cars](…",
  "via": "static",
  "attempts": [
    { "via": "direct", "ms": 116, "status": 403,
      "wall": "perimeterx challenge" },
    { "via": "static", "ms": 565, "status": 200,
      "wall": null }
  ],
  "billing": { "mode": "auto", "credits": 1,
    "breakdown": { "base": 1 } }
}

attempts lists every step and the wall it met, in plain labels like perimeterx challenge or datadome block, so you can see why a page cost what it did.

Kickstarter went one step further: Chrome’s handshake got a 200, the endpoint saw a page that JavaScript still had to build, and the browser came back with the full page in about 7 seconds for 1 credit. Most pages take 1 to 3 seconds and browser renders 5 to 15, so give the call a 90-second timeout for the slow ones. The web scraping guide lists every step and its price.

Paste the URL that’s giving you the 403 and see which step gets it:

Run GET /v1/web/scrape in the live playground

Questions

Why do I get 403 Forbidden with Python requests but not in the browser?

A bot wall in front of the site saw a client that isn’t a browser. The User-Agent says python-requests, and the TLS handshake and HTTP version don’t match Chrome’s. The response’s headers, cookies or page say which wall it was.

AxiosError: Request failed with status code 403

Axios throws on any status outside 2xx. Read err.response.headers and err.response.data to see who sent it. On a page Chrome opens fine, it’s usually a bot wall reacting to the axios/ User-Agent and Node’s non-browser handshake; on an API URL, usually a missing or wrong token.

Does changing the User-Agent fix a 403?

Rarely, on a site behind a bot wall. Kickstarter and Priceline still answer 403 to a plain client sending Chrome’s User-Agent, and serve the page once the request also has Chrome’s TLS handshake and HTTP/2 settings.

What is cf-mitigated: challenge?

A Cloudflare response header that marks a challenge page. The page clears only in a browser that runs Cloudflare’s JavaScript, which then gets a cf_clearance cookie.

How do I tell if a site uses PerimeterX?

Look for a _pxhd or _px3 cookie, window._pxAppId in the HTML, or a “Press & Hold” page that loads from px-cloud.net. PerimeterX is now called HUMAN.