Is there a way to obfuscate email in xml?

[poesu]

Building upon advice by @arcus from a recent spam discussion thread, is there a way to obfuscate email from LLMs and other crawlers not only in HTML but in other types of web documents such as XML? I offer a few web feeds on my website and each of them have my email like so:

Code

<author>
	<name>El Poesu</name>
	<email>email@example.com</email>
</author>

My feeds are styled, so you don't see the email upon visiting feeds. But the email is there in the code. XML is all prissy, especially Atom, and it's meant to be machine-readable. I wonder if there's a way to protect oneself from spam still.

Also, do bots scrape XML documents on the regular basis?

Any thoughts?

[Dan Q]

Yes, scrapers target non-HTML content including XML files. This almost certainly includes those that feed LLMs.

If you're looking to make your email address usable by humans but not exploitable by bots, the simplest solution is any kind of obfuscation that'll make sense to a human but is less-likely to be parseable by a machine. For LLMs, that's tough of course, but I'm yet to see any evidence that anybody's specifically using LLMs to extract email addresses for spam purposes (the regular scraper bots find plenty!). So you might consider this approach (which basically works the same in HTML and XML):

Code

<email>email my shirt @ my pants example.com (don't forget to remove my shirt and my pants before you reach out!)</email>

The real-life approach I take with email scrapers is more complex, but here's the essence of it: I publish a different email address on every page on my website, that all come to the same inbox. Anybody that sends fundamentally the same email to 5+ different addresses in a short space of time is blocklisted as a spammer. If just one address starts attracting spam, I block all email that comes to that address and change the address on the page it's on.

I also have hidden email addresses buried invisibly on many pages to act as honeypots: anybody who emails any of those is immediately flagged as a spammer, too. This approach lets me get away with publishing email addresses "in the open" on my site and I get relatively little unwanted email (most of it's very targetted, e.g. people wanting to offer to pay me to put sponsored content onto a particular page, and they're emailing on that page's email address, but only about two or three of those get through a week).

Oh, and see this article, which was linked from another thread. If you're happy to put a HTTP(S) URL into the <email>...</email> and provide your email address via a redirect, that seems to be pretty bulletproof right now.

[Rubbereon]

AI cannot parse javascript and to my knowledge css, so all you'd need to do is to hide your email address into either a css stylesheet or a javascript file.

Here's a working example in css:

Code

.class::after {
  content: "Your email address here";
}

or I guess this could work too:

Code

email::after {
  content: "Your email address here";
}

Putting your email in email tags is the quickest way to get spotted by spammers because it's not just AI, traditional crawlers that gather public email addresses will find this just as easily and on a silver platter. Before AI came around, people would usually leave 2 spaces around "@" to make it harder for spambots to see their email, but that might not work anymore. Example "myemail @ gmail.com"

[Dan Q]

AI cannot parse javascript and to my knowledge css

CSS will often fool (AI) scrapers, as will anything else "visual", e.g. you could do something like this and it'd be readable to most humans (accessibility might be an issue) but to anybody without CSS support, or that can't process the result, they won't get the right answer. With some creative work you could even make a honeypot that CSS-transforms into the real address:

Code

<p>
  Email me on:
  <span class="obfuscated-email">
    <span class="obfuscated-email-part-d">exam</span>
    <span class="obfuscated-email-part-b">il</span>
    <span class="obfuscated-email-part-f">om</span>
    <span class="obfuscated-email-part-a">ema</span>
    <span class="obfuscated-email-part-c">@</span>
    <span class="obfuscated-email-part-e">ple.c</span>
  </span>
</p>

<style>
  .obfuscated-email {
    user-select: none;
    display: inline-grid;
    grid-template: 'a b c d e f';

    .obfuscated-email-part-a {
      grid-area: a;
    }

    .obfuscated-email-part-b {
      grid-area: b;
    }

    .obfuscated-email-part-c {
      grid-area: c;
    }

    .obfuscated-email-part-d {
      grid-area: d;
    }

    .obfuscated-email-part-e {
      grid-area: e;
    }

    .obfuscated-email-part-f {
      grid-area: f;
    }
  }
</style>

Of course there are other approaches like putting your email address in an SVG (which can still be copy-pasted from in most browsers but are more-likely to dodge scraping).

AI scrapers definitely can and do execute JS though: they don't need to parse it: they can just run it! You can see that they do this by asking an LLM about the contents of a web page that requires JS.

[Rubbereon]

AI scrapers definitely can and do execute JS though: they don't need to parse it: they can just run it! You can see that they do this by asking an LLM about the contents of a web page that requires JS.

If I had to guess that's because they're pre-rendered/cached from a previous crawl where the webpage had a lot less javascript (or maybe they use archive.org because it uses a classic crawler that renders javascript) unless they recently added that functionality to ai bots because there was an amazon page that would appear completely blank if you had noscript on and somebody tested and found that the ai couldn't access the website.

Keep in mind ai is not always honest, I heard of a trick from somebody irl that turned out to be completely wrong where to bypass an ai chatbot's image limit you could just link it the image, but as it turns out if the image linked to has an odd name or just a bunch of numbers, it will actually be honest and tell you that no it cannot parse external images. so anytime it "does", it's just hallucinating, being dishonest and using the file name to give you an answer. In short no that's false, I don't believe it.

Several more articles on the topic say the exact same thing
https://mindgruve.com/artificial-intelligence/javascript-seo-ai-search/
https://www.linkedin.com/posts/seobility-seo-tool_why-javascript-can-hurt-your-visibility-in-activity-7330864934047662081-RBdz
https://visively.com/kb/ai/ai-crawlers-javascript-rendering
https://kairosphere.co/blog/your-website-may-be-invisible-to-ai-the-javascript-problem (This one was made by an AIbro you can easily tell the text is AI-generated and they'd obviously know better on this topic)
https://www.facebook.com/groups/localseowarriors/posts/1311791267621756/
https://edgecomet.com/javascript-rendering-for-seo/
https://medium.com/@mthoufeek709/javascript-seo-problems-developers-ignore-dfe871adfd22
and all of them are recent.

[Dan Q]

I just did an experiment using Claude and a completely new web page that had never been accessed before. First, I created this page:

index.html:

Code

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>AI-JS</title>
</head>
<body>
	<p>
		This page requires JavaScript.
	</p>
	<script src="test.js"></script>
</body>
</html>

test.js:

Code

const p = document.querySelector('p');
p.innerText = 'My email address is asecret@example.net!';

That web page looks fine to a human (using a browser with JavaScript enabled), but a basic scraper will just see the message about JS being required.

I hosted that page at a brand new subdomain that had never been used for anything ever before.

Then I asked Claude "What email address is shown on https://mora51.fox.q-t-a.uk/?"



The AI quickly determined that it needed to run JS, spun up a (presumably Puppeteer-managed or similar) copy of Chromium to access it (which ran the JS), and then provided the correct answer.

I can conclude that while those articles might be correct that AI scrapers are not currently (or at least, not in a widespread way) executing JS, they are - as I've just demonstrated - absolutely capable of doing do. And, depending on their scraping strategy, they might well be able to do it retroactively: i.e. if you're relying on JS as protection from them today, then that protection might already have been breached but you just don't know it yet.

[arcus]

The reason why crawlers don't usually run scripts is because it's more overhead. It's much faster and lighter on resources to grab the source code.

I recommend against using JavaScript or CSS obfuscation tricks, since they don't just effect crawlers. Not all feed readers support JavaScript or CSS, so some readers wouldn't be able to display it. Dan's method of asking strangers to strip you is the ideal solution.

[Dan Q]

Dan's method of asking strangers to strip you is the ideal solution.

Be advised that it's much more successful at blocking spam than it is as a chat-up line.

(Also note of course that LLMs can remove my shirt and my pants pretty well too, but it takes time, effort and money for a harvester to try to do that, and I imagine most don't bother.)

[Rubbereon]

(Also note of course that LLMs can remove my shirt and my pants pretty well too, but it takes time, effort and money for a harvester to try to do that, and I imagine most don't bother.)

Effortlessly too in fact:


This is why I said messing around with an email address in html is outdated.

Before AI came around, people would usually leave 2 spaces around "@" to make it harder for spambots to see their email, but that might not work anymore. Example "myemail @ gmail.com"

At least hiding the email into a javascript file or a css stylesheet will induce it to incorrectly assume there is no email address to find if somebody makes a simple request like "Look at site A, B and C and find me every email they made public".

The biggest problem I see here is that poesu is trying to be semantically correct with <email> (and even if they were working with html instead of xml and using <label for="email"> or <div class="email"> my opinion wouldn't change, don't do that!) when the email shouldn't be shown like that in plaintext. How about uh I don't know how xml works but whatever's xml's div equivalent is, but a 'riddle' sentence:

Code

<div equivalent here>If you wanna email me use my username and my carrier is gmail.</div equivalent here>

to further obfuscate the email from ai, do the trick I showed earlier:

Code

div::after {
  content: "If you wanna email me use my username and my carrier is gmail.";
}

and if you're concerned about accessibility, add an aria label or whatever's the equivalent for xml (that said I have no idea if the ai will look at that or not)

Code

<label aria-label="If you wanna email me use my username and my carrier is gmail."></label>

But I think I'm overcomplicating this. The point I'm making is that there will never be a 100% foolproof method, but hiding an email address in a way that causes it to appear completely invisible to AI agents while still being accessible to people is the only way to be at least relatively safe meanwhile having plaintext on the source html/xml is significantly more likely to not only get spotted but also deciphered. It doesn't matter how you wanna do it, the AI can see it plainly and then assess how the text should be interpreted as shown earlier.

Furthermore, the end user needs to know that Claude and specifically Claude can load up a browser to generate the required css/javascript, I didn't (well to be fair I touch AI at all, but I can see someone not really using AI all that often also not thinking about this feature). So this is your best bet to hide your email from AI, nothing else is nearly as safe.

[Rubbereon]

I'm worried folks might still be skeptical of what I'm saying, so I'm not gonna take the last word. I've decided to make a point by asking a clanker to find me poesu's email on their website to demonstrate how much plaintext is ineffective no matter what you do.



I cannot test on my site at nekoweb because it blocks all clankers and AI garbage (BASED! No ideas why neocities doesn't do the same)

Thing I don't understand is why Arcus didn't link this article they posted to another thread since I think some of the ideas it offers might have a chance to be ignored by AI crawlers.

https://spencermortensen.com/articles/email-obfuscation/

This statement is especially true if you're gonna have the email address appear in plaintext.

Ideally, you would be using more than one technique in combination.

See this for example would more likely have an impact especially if we assume AI loads pages like this.

Code

&#x6d;&#89;&#x65;&#109;&#65;&#x69;&#76;<!--&#64;&#x65;&#x78;&#x61;&#x6d;&#x70;&#x6c;&#x65;&#x2e;&#x63;&#x6f;&#x6d;-->&#20;&#64;&#20;&#x67;&#77;&#x41;&#73;&#x6c;&#20;&#x2e;&#20;&#x43;&#x4f;&#x4d;

^ "myemail @ gmail . com" with a html comment that reads "@example.com" as a form of obfuscation <- Also it swaps between uppercase and lowercase at random which I know can cause problems for automated spam if it gets saved like that, but not people. Email providers nowadays just default to all lowercase normally.

I'll add that since Arcus made a comment about accessibility, but if CSS content is not accessible then this should be, the website said javascript was safe for screen readers. Anything that is unsafe is marked as "breaks usability"

Code

<script>document.write('a'+'i'+'@'+'e'+'m'+'a'+'i'+'l'+
'.'+'s'+'p'+'e'+'n'+'c'+'e'+'r'+'m'+'o'+'r'+'t'+'e'+'n'+'s'+'e'+'n'+
'.'+'c'+'o'+'m');</script>

I know DanQ is very knowledgeable, but in this case his tip would only work on classic crawlers not modern ones powered by AI, but if you don't want problems with spambots and ai agents in the first place, you should block them via htaccess and php as discussed here and there.

But anyway we could be talking about this all day, but I'll let you decide how you're gonna obfuscate your email address @poesu I think this thread has run its course. 

[poesu]

Thing I don't understand is why Arcus didn't link this article they posted to another thread since I think some of the ideas it offers might have a chance to be ignored by AI crawlers.

That's my bad actually Before creating this thread, I already read this article. I think I wasn't entirely clear that I only wanted to know if the same methods can be applied to XML as CSS and JS tricks won't work there.

I've decided to make a point by asking a clanker to find me poesu's email on their website to demonstrate how much plaintext is ineffective no matter what you do.

That's interesting that it pulled the email from the feed actually, because at the time of your post I already updated the feeds to be obfuscated. I did the obfuscation on every other page somewhat earlier, and it didn't pull the data from there — my email address is literally one of the first things you see on the index page

Omg it didn't actually pull the data from my site, it pulled it from MelonLand! It shows the source on the screenshot. I put it in a post where I copypasted my Atom feed code to be an example for someone who might want to style their feed. That's awkward! Removed email from that post now.

Well, success? I dunno. I appreciate you folks running tests, but I don't want to interact with clankers myself to confirm stuff.

I cannot test on my site at nekoweb because it blocks all clankers and AI garbage

How does it do that by the way? I thought it was just robots.txt (which isn't that effective generally), but I can't find info anywhere.

[Noah_S]

I can confirm from my experimenting earlier this week that they ignore robots.txt

[Rubbereon]

How does it do that by the way? I thought it was just robots.txt (which isn't that effective generally), but I can't find info anywhere.

No idea, probably htaccess or php. I discovered that when I was doing some investigating for this because I wanted to make sure googlebot could load the correct webpage and not the noscript page because then the anti-ai canonical link would send it out of the website and that'd not be good obviously.

Omg it didn't actually pull the data from my site, it pulled it from MelonLand! It shows the source on the screenshot. I put it in a post where I copypasted my Atom feed code to be an example for someone who might want to style their feed. That's awkward! Removed email from that post now.

Turns out I was wrong about something oops, neocities does protect their sites from bots (I wouldn't entirely trust chatgpt when it claims it's being blocked by robots.txt, might have been a banal 403 error)

That's why it pulled from melonland. I saw the links too but didn't really know what they meant because I don't use AI. I thought the bot posted a generic citation, no idea it would pull info from melonland of all things.
I literally called it earlier too, never trust the clankers they can't stop lying.

[arcus]

Thing I don't understand is why Arcus didn't link this article they posted to another thread since I think some of the ideas it offers might have a chance to be ignored by AI crawlers.

https://spencermortensen.com/articles/email-obfuscation/

It was linked (indirectly) in the original post.

I'll add that since Arcus made a comment about accessibility, but if CSS content is not accessible then this should be, the website said javascript was safe for screen readers. Anything that is unsafe is marked as "breaks usability"

Code

<script>document.write('a'+'i'+'@'+'e'+'m'+'a'+'i'+'l'+
'.'+'s'+'p'+'e'+'n'+'c'+'e'+'r'+'m'+'o'+'r'+'t'+'e'+'n'+'s'+'e'+'n'+
'.'+'c'+'o'+'m');</script>

Accessibility is not strictly used in reference to disability. Not all feed readers render JavaScript, so that email would not display at all. My feed reader doesn't render it, nor does it render CSS. If the goal here is to block most bots, me, and some other random users of minimalist feed readers, then that's fine.

I know DanQ is very knowledgeable, but in this case his tip would only work on classic crawlers not modern ones powered by AI, but if you don't want problems with spambots and ai agents in the first place, you should block them via htaccess and php as discussed here and there.

This is a misunderstanding of how LMMs work.

LMMs do not harvest their own training data. Training data is harvested through other means such as curl and other lightweight tools that don't run JavaScript or CSS. Rendering JavaScript and CSS uses more electricity and takes more time to run, which adds up when you're aiming to scrape billions of pages. The goal here is quantity, not quality. This is also true for anyone scraping the web to harvest email addresses. The goal with harvesters is to grab as many emails as possible to spam, and hope for a hit.

There ARE crawlers that run JavaScript and CSS, but they're more costly to run, so they're less common. But they absolutely are out there, and are being used to harvest training data for LMMs.

Robots.txt doesn't actually prevent bots. It's an optional set of etiquette rules that can be ignored. Blocking unwanted bots requires various tools, and often effects wanted visitors in the process. It does help to a degree, since a site with robots.txt will be less likely to appear in search engines that do respect it.
 
The datasets LMMs are trained on are not guaranteed to follow robots.txt. Even if a corporation's LMM respects robots.txt, the data they're trained on is not guaranteed to have followed robots.txt.

Concern about Claude or any other similar large scale LMMs being able to grab email addresses is pointless. Anyone that can use a commercial LMM can instead go to a site manually, and find addresses the old fashion way to phish.

[Dan Q]

I cannot test on my site at nekoweb because it blocks all clankers and AI garbage

How does it do that by the way?

No idea, probably htaccess or php.

Point of clarification: "htaccess" isn't something magical: it's just a way to add extra configuration to the Apache HTTPD webserver on a directory-by-directory basis. Given that Apache isn't even the most-popular webserver any more, attaching value to "htaccess" specifically seems a bit backwards. What we're actually talking about here is: configuring the webserver to block (or e.g. 403) connections that appear to be from bad robots, based on some criteria or another. The examples given in the other thread related to their User-Agent: string; other approaches involve blocking IP address blocks, or even attempting to dynamically block based on "suspicious behaviour" (there are webserver plugins that can help with this).

In the other thread I provided examples of configuration for blocking by user-agent: header in Apache and Caddy, Nginx is a more-popular choice than both and has a slightly different syntax still (only Apache does .htaccess though).

As to how Nekoweb is doing their blocking? They're outsourcing it to Cloudflare, who're well-resourced for this kind of activity: because they reverse-proxy a lot of different sites, they've got a big dataset from which to stop badly-behaved bots and identify their IP ranges etc.; and where they're not-quite-so suspicious, they can inject an (annoying) CAPTCHA. Of course, monetising this knowledge they've gained is part of CloudFlare's business model so you can't just replicate it wholesale, but the essence of it will be a mixture of user-agent, IP range, and dynamic behavioural blocks, backed-up with a CAPTCHA for the edge cases.

Turns out I was wrong about something oops, neocities does protect their sites from bots (I wouldn't entirely trust chatgpt when it claims it's being blocked by robots.txt, might have been a banal 403 error)

Yeah, it's mistaken, lying/hallucinating, or something's changed: Neocities' robots.txt doesn't disallow any bots; individual users' sites can include their own robots.txt, like https://poesu.neocities.org/robots.txt for example.

I literally called it earlier too, never trust the clankers they can't stop lying.

Yup, this. AI is lying to you.

LMMs do not harvest their own training data. Training data is harvested through other means such as curl and other lightweight tools that don't run JavaScript or CSS.

...

There ARE crawlers that run JavaScript and CSS, but they're more costly to run, so they're less common. But they absolutely are out there, and are being used to harvest training data for LMMs.

This is all broadly correct, except that I'm not aware of any major crawlers that use curl directly (although many use libcurl under the hood!): it's rarely the most-efficient tool when what you're aiming to do is spider all of the links within a domain, because by-default curl does not recycle connections, making it inefficient for this purpose.

All of the major search engines' crawlers do execute JavaScript and have an understanding of at least a subset of CSS. You can see this by, well, using a search engine to search for content that's not accessible without JavaScript, for example! It follows, then, that LLMs attached to search engine companies, at the very least, are likely to be scraping dynamic content.

But not all, certainly, and not all the time. Just the HTML is plenty sufficient and much faster, as you observe

Concern about Claude or any other similar large scale LMMs being able to grab email addresses is pointless. Anyone that can use a commercial LMM can instead go to a site manually, and find addresses the old fashion way to phish.

This is the big takeaway to all of this, I'd say. LLMs scraping email addresses is enormously small-potatoes in the scale of things, where spammers are hoovering them up at breakneck speeds using conventional scrapers.