so there’s a new search engine for the fediverse:

most people probably don’t expect their public posts to be searchable, especially if they use mastodon, so here’s how to opt out (for mastodon): open your settings and go to Preferences > Other, then check “Opt out of search engine indexing”.

if you’re an instance admin, you can opt your entire instance through robots.txt. you can set this up in nginx:

@tao Why would most people not expect public to mean searchable?

@jaywink mastodon implements a limited search feature -- in order to prevent abuse, you can only search for statuses that you've interacted with in some way (e.g. favorited, re-posted, etc.)

this is a mastodon feature so people may not expect a third-party search service to index their posts

(though i agree, we need to be better at reminding people that public means public)

@tao != though 😏 Public in this case I assume to mean "as:Public" -

> Activities addressed to this special URI shall be accessible to all users, without authentication.

Well, I hope the search engine is not indexing anything else..

@jaywink @tao it's also good to note that unlisted posts are also public in this way

@lain @jaywink @tao that sounds like a bug with "unlisted" posts, if they are public 🙄

@jaywink @tao well, they can be seen without access restrictions, so they are public in that sense.

@lain @jaywink @tao but then it works as expected I would think? If an unlisted post gets exposed to a public search engine, it stops being unlisted. There isn't really anything that one can do to have both "unrestricted" and "private" at the same time.

It's like sharing a secret URL on the Internet. That content is available to anyone who happens to get the URL.

Maybe the "unlisted" feature makes too strong a promise to users?


@lain @jaywink @tao my implementation for example doesn't have this "unlisted" at all. Anything with "as:Public" gets attached a "Public" visibility which makes things available in the public streams, which don't require authentication 😉

@jaywink out of curiosity, and off topic to the main thread of discussion: why do you have "public" and "non-public" streams? Isn't one stream with different content based on being authenticated or not enough, or am I missing something?

@mariusor Currently there are various provided streams, like "Public", "Followed people", "Followed tags", "My content", "Local", for example. Non-public stream isn't so useful in its own, as you say.

What I've planned is for the user to be able to build custom streams using different criteria. So one could for example have "Followed people + Followed tags + all posts with images + any posts in Japanese" :) Or just a combo of the basic streams to provide the kind of base stream the user wants.

Sign in to participate in the conversation

Instance majoritairement francophone. Nous ne régulons la fédération. Aucun contenu du fédiverse n'est filtré par une décision d'administrateur ou de modérateur.