Artifacts Gallery Guilds Search Wiki Login Register

Welcome, Guest. Please login or register. - Thinking of joining?
a Summer night - @876.40
Activity rating: Four Stars Posts & Arts: 69/1k.beats Random | Recent Posts | Guild Recents
News: It's only life! :dog: Guild Events: Summer Projects

+  MelonLand Forum
|-+  Life & The Web
| |-+  ✁ ∙ Web Crafting
| | |-+  ☔︎ ∙ I need Help!
| | | |-+  Stop my blog from Google search result?


« previous next »
Pages: [1] Print Embed
Author Topic: Stop my blog from Google search result?  (Read 333 times)
chilblands
Jr. Member ⚓︎
**
View Profile WWW


make it happen or regret
⛺︎ My Room

Guild Memberships:
Artifacts:
Visited on Melon's 10th Anniversary!I got robbed by Dan Q on Melonland!Joined 2026!
« on: a Spring day » Embed

One of my blog on BearBlog went trending, when I check my referrers, I can see Google and Facebook. I am guessing my posts ended up on the Google search result. Is there a way like robot.txt to hide me from Google and other big search engines like Bing and Yahoo?

edit: this is my robot.txt
Code
User-agent: *
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /

edit:edit: I just added this <meta name="robots" content="noindex, nofollow">
via: this, I hope no more Google jumpscare!!

« Last Edit: a Spring day by chilblands » Logged

https://chilblands.neocities.org/files/horseicon.gif it's gettin cold in here...

Artifact Swap: Cursor!The poop hooliganShocked SharaDevil HeartHungry MouseI met Dan Q on Melonland!Batman Spiffocup (for tea lovers)PurbyMinecraft Veteranmanstarmargiesimp
shouldseethis
Newbie ⚓︎
*
View Profile WWW

⛺︎ My Room

Artifacts:
Joined 2026!
« Reply #1 on: a Spring day » Embed

how else would we find you?  :melon:

Logged
mrparker
Casual Poster ⚓︎
*
View Profile WWW

⛺︎ My Room

Guild Memberships:
Artifacts:
Visited on Melon's 10th Anniversary!Joined 2026!
« Reply #2 on: a Spring day » Embed

You'll want to add the bots listed here if you want to block every google bot:

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

The only entry you have (Google-Extended) relates to Google's AI training crawler. To block all usage of AI from Google - you'll have to add some of these (or all):

https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers

However, the most reliable way to block these types of scrapers is to block IP ranges - depending on your host, you'll have to add firewall rules to block those.

Logged
chilblands
Jr. Member ⚓︎
**
View Profile WWW


make it happen or regret
⛺︎ My Room

Guild Memberships:
Artifacts:
Visited on Melon's 10th Anniversary!I got robbed by Dan Q on Melonland!Joined 2026!
« Reply #3 on: a Summer day » Embed

how else would we find you?  :melon:

hope and dreams :sleep:

You'll want to add the bots listed here if you want to block every google bot:

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

The only entry you have (Google-Extended) relates to Google's AI training crawler. To block all usage of AI from Google - you'll have to add some of these (or all):

https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers

However, the most reliable way to block these types of scrapers is to block IP ranges - depending on your host, you'll have to add firewall rules to block those.
I did not know this! I don't think my host allow firewall rules but the Google bots doc is very useful. Thank you so much for sharing  :transport:


Logged

https://chilblands.neocities.org/files/horseicon.gif it's gettin cold in here...

Artifact Swap: Cursor!The poop hooliganShocked SharaDevil HeartHungry MouseI met Dan Q on Melonland!Batman Spiffocup (for tea lovers)PurbyMinecraft Veteranmanstarmargiesimp
Pages: [1] Print Embed 
« previous next »
 

Melonking.Net © Always and ever was! SMF 2.0.19 | SMF © 2021 | Privacy Notice | Send Feedback | Supporters ♥ Forum Guide | Rules | RSS | WAP | Mobile


MelonLand Badges and Other Melon Sites!

MelonLand Project! Visit the MelonLand Forum! Support the Forum
Visit Melonking.Net! Visit the Gif Gallery! Pixel Sea TamaNOTchi
@000 MelonLand Zap! Want to Login or Join ? Forum Art Hub Chat Webring Editor Recent Edits Tenement Arcade Web Guides Graphics Catalogue Wiki Newsletters Image Stream
Melon's Sites TamaNotchi Textures PixelSea GifyPet MoMG Ozwomp Online Loom Videos Leaky Webring Melonking
Tools Melon Software ArtHub Embed Maker ML Passports
Outlinks Webrings Internet Phone Book HTML Energy Declarations Hackers & Designers Frutiger Aero Forum m15o's Web Services 32Bit Cafe iMood
Minecraft: Online
Join: craft.melonking.net