Block scrapers posing as browsers on nginx or Apache
No CDN in front of your site? Sumly gives each scraper it finds a rule to copy into nginx or Apache: the network's address ranges to deny, or a pace limit on the part of the site proxy scrapers crowd onto.
-
AstroChart Google Play · in the app 🇪🇸 +€9.99
-
Humanize Stripe · on the site · from Facebook 🇿🇦 +€11.04
- TimeLoop App Store · in the app · via RevenueCat 🇺🇸 +€29.99
-
The live demo: our own apps, with example numbers.
Blocking in three steps
Nothing to connect: Sumly never reaches your server.
-
1
In Sumly, open Projects and payments, the project, Bots & blocking and choose Own server.
-
2
Under Insights › Bots › Disguised scrapers, open a scraper's row and Show the rule: pick nginx or Apache and copy it.
-
3
Paste it where it says (the site's server block, or .htaccess) and reload the server. Copy a network's rule again now and then, as its ranges change.
What you see
Your own server next to every other project and provider, on the same pages.
Bots
Disguised scrapers, each with its nginx and Apache rule under Show the rule.
Overview
One card while a project is being scraped, with Copy the nginx rule.
Alerts
One Bot attack notice per attack, in the bell and the app.
Visitors and Live
How many page views the Smart bot filter kept out.
Counts toward your plan
Nothing: the Smart bot filter and its rules are on every plan. Plans →
What you don't get
Sumly never connects to your server and keeps no key for it.
The details
What the rules say
- For a network on nginx: deny <range>; for every address range it holds, from the same table Sumly looks visits up in, with touching ranges joined.
- For a network on Apache 2.4: Require not ip with the same ranges, inside RequireAll.
- For proxy scrapers on one part of the site, on nginx: visits there with no referrer share one pace (12 a minute, then 429), while people who follow a link keep theirs.
- On Apache, which has no pace limit built in: a rewrite rule that turns away visits there with no referrer, to take out when the scraping stops.
Keeping it current
- Each rule carries the date it was made and the network it is for.
- A network's ranges change now and then: copy its rule again every few months.
- Behind a proxy or load balancer, set real_ip (nginx) or mod_remoteip (Apache) first, so the server sees the visitor's address.
Good to know
- No one-click block, autopilot or undo: Sumly does not reach your server, so you add and remove the rules.
- A network's rule spares no one on it, verified bots included.
- The Apache path rule also turns away people who type the address or open a bookmark.
Your own server questions
Where do the address ranges come from?
From iptoasn.com's public table of who holds each address range, read again each day: the same table Sumly looks every visit up in.
Is my site on Cloudflare, Bunny or Vercel instead?
Then choose that one under Bots & blocking: Sumly adds and takes back the rule itself.
Can I use the rules with another server?
They are plain address ranges, so most firewalls take them. Sumly writes them out for nginx and Apache.
Other integrations
Shopify
Try it with your own projects
Sign in with Google or a code by e-mail and connect your first project. The first 14 days of a paid plan are free, with no card.
Start free trialOr look around first: open the live demo.