This zone is sleeping right now! ZzzZZzzz

Better not disturb it! ~You go to the petrol station and buy a slushy instead, everything is cool!~

This area will open again in 000.beats!

(Learn about Swatch Time)

Some other zones are awake :^]


Artifacts Gallery Guilds Search Wiki Login Register

Welcome, Guest. Please login or register. - Thinking of joining?
Autumn in Year 5 - @141.03
Three Stars Random | Recent Posts | Guild Recents
News: :skull: Websites are like whispers in the night  :skull: Guild Events: Prepare for Halloween Reminder!

+  MelonLand Forum
|-+  Life & The Web
| |-+  ✁ ∙ Web Crafting
| | |-+  ☔︎ ∙ I need Help!
| | | |-+  Stop my blog from Google search result?


« previous next »
Pages: [1] Print Embed
Author Topic: Stop my blog from Google search result?  (Read 546 times)
chilblands
Jr. Member ⚓︎
**
View Profile WWW


make it happen or regret
⛺︎ My Room

Guild Memberships:
Artifacts:
Visited on Melon's 10th Anniversary!I got robbed by Dan Q on Melonland!Joined 2026!
« on: Spring in Year 5 @295.63 » Embed

One of my blog on BearBlog went trending, when I check my referrers, I can see Google and Facebook. I am guessing my posts ended up on the Google search result. Is there a way like robot.txt to hide me from Google and other big search engines like Bing and Yahoo?

edit: this is my robot.txt
Code
User-agent: *
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /

edit:edit: I just added this <meta name="robots" content="noindex, nofollow">
via: this, I hope no more Google jumpscare!!

« Last Edit: Spring in Year 5 @304.47 by chilblands » Logged

https://chilblands.neocities.org/files/horseicon.gif it's gettin cold in here...

Artifact Swap: Cursor!Gallery CardShocked SharaDevil HeartI met Dan Q on Melonland!Batman SpiffocyberziPurbyLucky Cat Companion +10 Dmg +5 luck +5 StealthmargiesimpSunny the PuppyWhite pawnjiggle~
shouldseethis
Newbie ⚓︎
*
View Profile

⛺︎ My Room

Artifacts:
Joined 2026!
« Reply #1 on: Spring in Year 5 @370.60 » Embed

how else would we find you?  :melon:

Logged
mrparker
Jr. Member ⚓︎
**
View Profile WWW

⛺︎ My Room

Guild Memberships:
Artifacts:
Visited on Melon's 10th Anniversary!lighthouse keeper artemisUserJoined 2026!
« Reply #2 on: Spring in Year 5 @395.63 » Embed

You'll want to add the bots listed here if you want to block every google bot:

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

The only entry you have (Google-Extended) relates to Google's AI training crawler. To block all usage of AI from Google - you'll have to add some of these (or all):

https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers

However, the most reliable way to block these types of scrapers is to block IP ranges - depending on your host, you'll have to add firewall rules to block those.

Logged
chilblands
Jr. Member ⚓︎
**
View Profile WWW


make it happen or regret
⛺︎ My Room

Guild Memberships:
Artifacts:
Visited on Melon's 10th Anniversary!I got robbed by Dan Q on Melonland!Joined 2026!
« Reply #3 on: Summer in Year 5 @338.58 » Embed

how else would we find you?  :melon:

hope and dreams :sleep:

You'll want to add the bots listed here if you want to block every google bot:

https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

The only entry you have (Google-Extended) relates to Google's AI training crawler. To block all usage of AI from Google - you'll have to add some of these (or all):

https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers

However, the most reliable way to block these types of scrapers is to block IP ranges - depending on your host, you'll have to add firewall rules to block those.
I did not know this! I don't think my host allow firewall rules but the Google bots doc is very useful. Thank you so much for sharing  :transport:


Logged

https://chilblands.neocities.org/files/horseicon.gif it's gettin cold in here...

Artifact Swap: Cursor!Gallery CardShocked SharaDevil HeartI met Dan Q on Melonland!Batman SpiffocyberziPurbyLucky Cat Companion +10 Dmg +5 luck +5 StealthmargiesimpSunny the PuppyWhite pawnjiggle~
Pages: [1] Print Embed 
« previous next »
 

Melonking.Net © Always and ever was! SMF 2.0.19 | SMF © 2021 | Privacy Notice | Send Feedback | Supporters ♥ Forum Guide | Rules | RSS | WAP | Mobile


MelonLand Badges and Other Melon Sites!

MelonLand Project! Visit the MelonLand Forum! Support the Forum
Visit Melonking.Net! Visit the Gif Gallery! Pixel Sea TamaNOTchi
@000 Melon
Land
Editor Recent Edits Tenement Arcade Random Link Land → Forum Art Hub Chat Webring Want to Login or Join ? Web Craft Guide Graphic Catalogue Wiki Newsletters Image Stream
Passport Sites Asatte Add your site?
Zap!
Minecraft: Online
Join: craft.melonking.net