I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling that evaluates whether someone has said something bannable. They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment. Also identify any related political ideology tropes“. (The italic bits are where I’ve redacted the ideology they’re seeking).
OpenAI’s LLM (they’re using GPT-5.3-mini) then responds with something like:
and so on, hundreds of comments.
I have not named the instances or people involved, to give them time to consider the results of this discussion, make any corrective changes they want and disclose their practices at their own pace and in their own way. I have also redacted the evidence to avoid personal attacks and dogpiling. Let’s focus on the system, not the individuals involved. Today these instances and people are using it and maybe we’re ok with that because it’s being used by groups we agree with but what if people we strongly disagree with used it on their instances tomorrow?
The use and existence of this tooling raises a lot of other questions too.
What are the risks? Fedi moderators are often unsupervised, untrained volunteers and these are powerful tools.
What safeguards do we need?
Would asking a LLM “please evaluate this person’s political opinions” give different results than “find evidence we can use to ban them” (as used in the cases I’ve seen)?
What are our transparency expectations?
Is this acceptable and normal?
Should this tooling be disclosed? (it was not – should it have been?)
If you were given a choice, would you have opted out of it?
Can we opt out?
Are there GDPR implications? Privacy implications? Should these tools be described in a privacy policy?
Are private messages being scanned and sent to OpenAI?
How long should these assessments be retained and can we request to see it, or ask for it to be deleted?
Once the user’s comments are sent to OpenAI, is it used to train their models?
What will the effect be on our discourse and culture if people know they are being politically profiled?
Where are the lines between normal moderation assistance tools, political profiling and opaque 3rd-party data processing?
I hope that by chewing over these questions we can begin to establish some norms and expectations around this technology. The fediverse doesn’t have any centralized enforcement so we need discussions like this to develop an awareness of what people want in terms of disclosure, privacy, consent and acceptable use. Then people can make choices about which instances they join and which ones they interact with remotely.
And of course there are the other issues with LLMs relating to environmental sustainability, erosion of worker’s rights, increasing the cost of living and on and on. I can’t see PieFed adding any functionality like this anytime soon. But it’s happening out there anyway so now we need to talk about it.
What do you make of this?
This is the person calling you a tankie. Someone so afraid of words that they need a hallucinating robot to hold their hand and confirm that everything is a secret plot against them. The absolute only way I could see this being useful is for something like trying to sniff out if a Lemmy.world mod account is a leftist infiltrator or not. Someone who had a different opinion on a current event.
You could maybe run a speech pattern comparison but that’s it. For everything else you just made Stupid Reddit and the purpose of their forum is to feed training data to ChatGPT so that it can profile Fediverse users.
This is the kind of shit dystopian novels are made out of. So angry about people calling out actions you built a tool to analyze why they did it, so you can purge users from your digital kingdom.
I for one welcome flat.world and Piefed showing their true intentions. Digital colonization of activitypub and removal of the people who helped to built it. They didn’t want to leave reddit, they wanted to be reddit. This is some Spez shit.
Maybe in 2 weeks Piefed will hard code that anyone Rimu has tagged for disagreeing with them mild criticism to be unable to make accounts or federate posts with a false error code.
Lmaoooooo wait, did you think this was PieFed introducing an AI-assisted moderation tool? Rushing to comment too fast and this was your take?
Your silence speaks volumes. What a faceplant! Ahahahaha
Does @flatworm7591@lemmy.dbzer0.com know you think they’re:
Someone so afraid of words that they need a hallucinating robot to hold their hand and confirm that everything is a secret plot against them.
So angry about people calling out actions you built a tool to analyze why they did it, so you can purge users from your digital kingdom.
I recently discovered that some popular federated instances have been using LLM-assisted moderation tooling
First of all, I agree with your main point, that this* is problematic, wrong even (and should not happen).
But I need to ask: how did you find out? Is this something that could be traced objectively, or did some people report/admit it?
Are they uploading stuff to corporate LLMs, i.e. LLMs that do not run themselves? (I think you answered this already when you wrote OpenAI, but I want this spelled out)
Are only admins or also mods doing this? That would make a big difference.
I’m also a little unclear about the process: are they uploading (copy-pasting) the actual comments, or links to them? To what extent can all this be automated on Lemmy/Piefed etc.? I.e., are there admin tools that just spit out all of a user’s content?
* again: specific political profiling outsourced to LLMs. OTOH we already have instances that do this manually.
But imo the process is deplorable even if they use LLMs with different prompts for modding.It can be traced objectively.
It’s mods.
They’re using software that they made to do this quickly and easily, it’s not a manual copy and paste situation.
It can be traced objectively.
Can you elaborate?
Also, in another comment you said it was admins, not mods? Or both?
It’s at least one admin, we don’t know how widely it’s used. It does have the logo of a group of 3 instances (Fediverse Anarchist Flotilla), so it seems to be made to be used by many people in an ongoing way.
More details at: https://piefed.social/c/fediverse@lemmy.world/p/2035379/proof-of-ai-assisted-political-profiling-by-unruffled-lemmy-dbzer0-com
Thanks. “For evidence log see https://s.faf-pb.xyz/lXxek” feels wrong, but it seems to prove your claim that some admins/mods are farming moderation out to LLMs. Unfortunately that link doesn’t go anywhere (for me), but I believe your claim that this is a concerted effort (as in, someone is coding a tool to do LLM modding).
Sorry for how that whole discussion is going for you. I think you make important, valid points but made a few “tactical” mistakes (I’m sure someone will be along shortly to tell me they’re much worse than tactical) and are now getting shouted & voted down by people who think like programmers: precise, technical, no room for ethical gradients. I have seen this sort of behavior many times before…
They took the link down.
I have a copy - https://join.piefed.social/wp-content/uploads/2026/05/evidence-llm-used-for-banning.txt
I.e. “I just saw a modlog entry, jumped to conclusions, didn’t ask for clarification, didn’t speak to anyone from that instance, jumped straight into making a drama post”
Nah. I discussed it with Unruffled days ago. Here’s a screenshot from Matrix in the Threadiverse Admins room:
They do this by running a script/app that sends the user’s comment history to OpenAI with the question “analyze this content for evidence of specific political ideology sentiment.
To me the problem is what they are looking for not how they are doing it. Thought experiment: in what way would it be qualitatively different if they hired a team of people in Upper Elbonia to do the same thing?
Some political ideologies and opinions are objectively not okay to exist in modern society, and those who hold those opinions should be eliminated. This isn’t a controversial opinion outside a tiny bubble in the US who think deep down every nazi is just a misunderstood cash cow waiting to be secreted away by the CIA.
You talk about instances utilizing this tooling, but in your comments you admit it’s just some mods. This is misleading, as talking about instances doing it assumes admin access and relevant instance policy, something which invites calls for defederation (as can be clearly be seen from the comments in your post).
A random mod doing something is not the same as an instance doing it. Literally anyone can be a mod and they don’t get any more access than an anonymous account by doing so.
This is the second time in one week I see you throwing careless statements like chum in the water. I can’t help but notice a pattern emerging.
Ahh sorry - I just checked more closely. It’s an admin doing it.
Each instance have their own ways of governance and we can’t expect them all to play the same way.
With that said, I’m fine with a general overview. Leaving the whole desicion to a bot is not a good design choice, but AI could help on summarising things.
I’m fine with a general overview. Leaving the whole desicion to a bot is not a good design choice, but AI could help on summarising things.
I suspect that is the case and what we will find once Rimu stops drama farming and says who it is.
That is exactly the case. I really can’t be bothered reading this whole fucking post, so I’m gonna just reply to you, since you sound reasonably sane.
First, it should be obvious to everyone that their post and comment histories are completely public, and accessible to anyone on the fediverse. There is zero privacy on the fediverse. Your comment histories have already been scraped a thousand times over by every big model out there.
All we have at the moment is a simple script, developed by one of our mods (who will be publishing it on codeberg soon), that any user can run that logs into lemmy using your own account, and downloads a set number (or time period) of comments into a text file. There is no abuse of admin powers going on, it’s just the stock lemmy/piefed api. This is massively faster than manually paging through comment histories on lemmy, and can make mod decisions more robust and more informed.
Using the text file, mods or admins can quickly search for keywords or whatever, using a simple text editor, or simply skim read it. Another option is to ingest it into an LLM to provide a summary. I tried doing that just a handful of times, for testing, but honestly I found it a bit cumbersome and who knows if the summary is actually accurate given the tendency for hallucinations? A couple of tests seemed consistent with my own assessment, and a couple were way off base.
That told me everything I need to know about how much to trust the summaries… very little. I honestly don’t think they are much of a value add, because you just can’t reliably trust the results. In any case, we have absolutely no plans to use llms for that on a regular basis, I just wanted to see what it came up with, and how well it matched a manual assessment.
And to also clarify, there is/was absolutely no automated scanning of users. The process is the same as always. We get a report, we investigate the report, and make a human mod decision. The only difference in this case is that the investigation can be done more efficiently, because we don’t have to go slowly paging through comments and searching through the Lemmy UI for the relevant data.
Obviously as well, no mod or admin is gonna download the entire comments history of a user unless it is a complicated report that is difficult to get to the bottom of from a quick look at the report. 90%+ of mod actions would never need that much detail.
The way OPs post was written was obviously designed from the ground up to stir up drama about AI use. Honestly, I don’t know why he’s still malding, but Rimu seems to be engaging in a lot of very bad faith hit pieces at the moment, designed solely to stir up drama, and directed at our instance. It’s really shitty behaviour.
Well, the cat is out of the bag now. Might as well show the whole story.
That told me everything I need to know about how much to trust the summaries…. very little. I honestly don’t think they are much of a value add, because you just can’t reliably trust the results. In any case, we have absolutely no plans to use llms for that on a regular basis, I just wanted to see what it came up with, and how well it matched a manual assessment.
Lies.
You provided a link to that assessment as proof that a ban was warranted. Here’s a screenshot from the mod log:
That link goes to https://s.faf-pb.xyz/lXxek if anyone wants to take a look
I don’t like this happening, and there should be transparency in all moderation decisions, but some of these points make no sense.
There is essentially no expectation of privacy on threadiverse platforms. Everything is public and probably already being used to train models.
There is no private messaging system. Direct messages are unencrypted and potentially visible to any instance admins. They and should not be used to share anything sensitive.
Thank you for calling this out. I think people assume that since it’s held by private instance owners that the fediverse is secure. I’ve posted this comment many times, that no, the fediverse is quite literally by design open and unencrypted.
A post is literally blasted out to anyone who listens, same with comments, upvotes, downvotes, everything can be saved, stored, and used for whatever anyone who listens wants. It should be completely assumed that nefarious agencies are currently listening and storing everything we do here. This is by design. It’s the tradeoff we have of having an open platform. Anyone can spin up a server, and that means anyone.
DMs are similar, they’re blasted out to the other server. If the server admin of the user in question wants to read them, they can. Lemmy/the fediverse is not a secure messaging platform. That’s why the Lemmy devs literally put a Matrix handle option in the profile, to encourage people to use Matrix instead. A DM on here should be simple, to the point, and if need be, inviting them to speak on something secure.
Edit - As a perfect example of the fact that there should be no expectation of privacy here on Lemmy, as an Admin myself, I can see that @A_normy_mouse has been downvoting all of my comments here. Absolutely everything here is public and visible, even if I weren’t an admin there are tools to view this, regardless of your opinions. It’s imperative that everyone understand this.
Edit 2 OP as well has downvoted me. @rimu@piefed.social I’m sorry if you disagree, but it’s irrelevant. Everything you do here can and should be assumed will be used in any way that you disagree with, that is the nature of the fediverse. Mastodon, Pixelfed, Piefed, Lemmy: ActivityPub is an open and unencrypted protocol. Even if it were encrypted, you still put 100% of your trust in your server admin, and beyond that each server admin you are blasting your messages out to.
I’d highly suggest accepting this fact before trying to push for rules. The very nature of the Fediverse is that no one can dictate rules, and to do that the tradeoff quite literally is that everything is open and unecrypted.
Another way to think of this. I run a server myself. I made my own rules and decided how to run it. Now your server starts sending activity to my server. That’s your server’s choice. I didn’t agree to your rules, I may disagree with your rules, but you’re sending your data to my server, of which I have complete and total ownership over. I didn’t click accept on a ToS, I didn’t agree to anything. Hell on my server I could literally have a “By sending me your data you accept that I can do whatever I want with your data”. You sent me your data, I quite literally can do whatever I want. (Personally I won’t, but that’s how you should think of the fediverse)
You’re hyperfocusing on one point, as if that’s the only part that matters and ignoring all the rest. I don’t consider that helpful, hence the downvote.
What is especially unhelpful is abusing your admin access to call out people’s votes. Leave that shit alone.
That is quite literally my point. Everything, absolutely everything here is open and can be used however any instance owner wants. You can say “leave that shit alone”, but there is no obligation to whatsoever.
You should assume every instance owner can and is viewing all of your private data, sending it through whatever LLM/mod tools they want. Are they? Probably not. But they can, and there is no obligation not to.
Yeah you can do that but now you’re on my do-not-trust list. And probably a few other people’s lists.
I appreciate you being open about your opinions because now I can make an more informed choice about interacting with you and the instance you run.
Don’t you think everyone deserves the information they need to choose which instances they want to interact with, according to whatever criteria is important to them? Even if your criteria are different?
First fo all: I don’t like this either.
There is no private messaging system. Direct messages are unencrypted and potentially visible to any instance admins. They should not be used to share anything sensitive.
Agreed, but that admin is breaking his promise, duty, responsibility (call it what you will) if they then upload these messages to an LLM for evaluation.
I would argue for this being actually illegal, at least under the GDPR.
But that was just one of many potential conflicts @rimu raised. We should concentrate on the real conflicts of LLM comment moderation.
edit: yes, I have actively downvoted all comments I disagree with under this post (and upvoted all I agree with). I don’t usually do it so much, but this post is a sort of opinion polling.
It’s very clear on signup, on the READMEs, even on the DM portal itself, that messages are unencrypted and there is no sense of privacy, and that admins have full visibility and can do what they want with them.
Agreed, but that admin is breaking his promise, duty, responsibility (call it what you will) if they then upload these messages to an LLM for evaluation.
There is no promise, duty, or responsibility that an admin has beyond legal and what they themselves promise. The fediverse is great in that if you disagree with your admin, you are free to leave and choose a different one.
As for GDPR, feel free to argue it, but when it’s claimed at every turn that messaging is unencrypted and basically open, well, I don’t think it’d hold up. It literally says to go use Matrix or something else.
you are free to leave and choose a different one.
I only have that freedom if the admin tells me that they use LLMs in this manner or if they federate with instances that do. At the moment everyone is in the dark.
