Merge remote-tracking branch 'origin/develop' into merge-ogp-twitter-parsers

This commit is contained in:
Egor Kislitsyn 2020-06-12 18:25:29 +04:00
commit 697cf92024
No known key found for this signature in database
GPG key ID: 1B49CB15B71E7805
20 changed files with 344 additions and 22 deletions

View file

@ -6,6 +6,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## [unreleased]
### Changed
- MFR policy to set global expiration for all local Create activities
- OGP rich media parser merged with TwitterCard
<details>
<summary>API Changes</summary>
@ -27,6 +28,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Configuration: `filename_display_max_length` option to set filename truncate limit, if filename display enabled (0 = no limit).
- New HTTP adapter [gun](https://github.com/ninenines/gun). Gun adapter requires minimum OTP version of 22.2 otherwise Pleroma wont start. For hackney OTP update is not required.
- Mix task to create trusted OAuth App.
- Mix task to reset MFA for user accounts
- Notifications: Added `follow_request` notification type.
- Added `:reject_deletes` group to SimplePolicy
- MRF (`EmojiStealPolicy`): New MRF Policy which allows to automatically download emojis from remote instances
@ -38,6 +40,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Mastodon API: Add support for filtering replies in public and home timelines
- Admin API: endpoints for create/update/delete OAuth Apps.
- Admin API: endpoint for status view.
- OTP: Add command to reload emoji packs
</details>
### Fixed

View file

@ -371,6 +371,8 @@
config :pleroma, :mrf_subchain, match_actor: %{}
config :pleroma, :mrf_activity_expiration, days: 365
config :pleroma, :mrf_vocabulary,
accept: [],
reject: []

View file

@ -1471,6 +1471,21 @@
}
]
},
%{
group: :pleroma,
key: :mrf_activity_expiration,
label: "MRF Activity Expiration Policy",
type: :group,
description: "Adds expiration to all local Create Note activities",
children: [
%{
key: :days,
type: :integer,
description: "Default global expiration time for all local Create activities (in days)",
suggestions: [90, 365]
}
]
},
%{
group: :pleroma,
key: :mrf_subchain,

View file

@ -44,3 +44,11 @@ Currently, only .zip archives are recognized as remote pack files and packs are
The manifest entry will either be written to a newly created `pack_name.json` file (pack name is asked in questions) or appended to the existing one, *replacing* the old pack with the same name if it was in the file previously.
The file list will be written to the file specified previously, *replacing* that file. You _should_ check that the file list doesn't contain anything you don't need in the pack, that is, anything that is not an emoji (the whole pack is downloaded, but only emoji files are extracted).
## Reload emoji packs
```sh tab="OTP"
./bin/pleroma_ctl emoji reload
```
This command only works with OTP releases.

View file

@ -135,6 +135,16 @@ mix pleroma.user reset_password <nickname>
```
## Disable Multi Factor Authentication (MFA/2FA) for a user
```sh tab="OTP"
./bin/pleroma_ctl user reset_mfa <nickname>
```
```sh tab="From Source"
mix pleroma.user reset_mfa <nickname>
```
## Set the value of the given user's settings
```sh tab="OTP"
./bin/pleroma_ctl user set <nickname> [option ...]

View file

@ -39,7 +39,7 @@ To add configuration to your config file, you can copy it from the base config.
* `rewrite_policy`: Message Rewrite Policy, either one or a list. Here are the ones available by default:
* `Pleroma.Web.ActivityPub.MRF.NoOpPolicy`: Doesnt modify activities (default).
* `Pleroma.Web.ActivityPub.MRF.DropPolicy`: Drops all activities. It generally doesnt makes sense to use in production.
* `Pleroma.Web.ActivityPub.MRF.SimplePolicy`: Restrict the visibility of activities from certains instances (See [`:mrf_simple`](#mrf_simple)).
* `Pleroma.Web.ActivityPub.MRF.SimplePolicy`: Restrict the visibility of activities from certain instances (See [`:mrf_simple`](#mrf_simple)).
* `Pleroma.Web.ActivityPub.MRF.TagPolicy`: Applies policies to individual users based on tags, which can be set using pleroma-fe/admin-fe/any other app that supports Pleroma Admin API. For example it allows marking posts from individual users nsfw (sensitive).
* `Pleroma.Web.ActivityPub.MRF.SubchainPolicy`: Selectively runs other MRF policies when messages match (See [`:mrf_subchain`](#mrf_subchain)).
* `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`: Drops posts with non-public visibility settings (See [`:mrf_rejectnonpublic`](#mrf_rejectnonpublic)).
@ -49,7 +49,8 @@ To add configuration to your config file, you can copy it from the base config.
* `Pleroma.Web.ActivityPub.MRF.MentionPolicy`: Drops posts mentioning configurable users. (See [`:mrf_mention`](#mrf_mention)).
* `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (See [`:mrf_vocabulary`](#mrf_vocabulary)).
* `Pleroma.Web.ActivityPub.MRF.ObjectAgePolicy`: Rejects or delists posts based on their age when received. (See [`:mrf_object_age`](#mrf_object_age)).
* `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
* `Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy`: Adds expiration to all local Create activities (see [`:mrf_activity_expiration`](#mrf_activity_expiration)).
* `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
* `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
* `managed_config`: Whenether the config for pleroma-fe is configured in [:frontend_configurations](#frontend_configurations) or in ``static/config.json``.
* `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML).
@ -154,6 +155,10 @@ config :pleroma, :mrf_user_allowlist,
* `rejected_shortcodes`: Regex-list of shortcodes to reject
* `size_limit`: File size limit (in bytes), checked before an emoji is saved to the disk
#### :mrf_activity_expiration
* `days`: Default global expiration time for all local Create activities (in days)
### :activitypub
* `unfollow_blocked`: Whether blocks result in people getting unfollowed
* `outgoing_blocks`: Whether to federate blocks to other instances

View file

@ -237,6 +237,12 @@ def run(["gen-pack" | args]) do
end
end
def run(["reload"]) do
start_pleroma()
Pleroma.Emoji.reload()
IO.puts("Emoji packs have been reloaded.")
end
defp fetch_and_decode(from) do
with {:ok, json} <- fetch(from) do
Jason.decode!(json)

View file

@ -144,6 +144,18 @@ def run(["reset_password", nickname]) do
end
end
def run(["reset_mfa", nickname]) do
start_pleroma()
with %User{local: true} = user <- User.get_cached_by_nickname(nickname),
{:ok, _token} <- Pleroma.MFA.disable(user) do
shell_info("Multi-Factor Authentication disabled for #{user.nickname}")
else
_ ->
shell_error("No local user #{nickname}")
end
end
def run(["deactivate", nickname]) do
start_pleroma()

View file

@ -113,6 +113,10 @@ defp get_proxy_and_attachment_sources do
add_source(acc, host)
end)
media_proxy_base_url =
if Config.get([:media_proxy, :base_url]),
do: URI.parse(Config.get([:media_proxy, :base_url])).host
upload_base_url =
if Config.get([Pleroma.Upload, :base_url]),
do: URI.parse(Config.get([Pleroma.Upload, :base_url])).host
@ -122,6 +126,7 @@ defp get_proxy_and_attachment_sources do
do: URI.parse(Config.get([Pleroma.Uploaders.S3, :public_endpoint])).host
[]
|> add_source(media_proxy_base_url)
|> add_source(upload_base_url)
|> add_source(s3_endpoint)
|> add_source(media_proxy_whitelist)

View file

@ -5,6 +5,7 @@
defmodule Pleroma.Web.ActivityPub.ActivityPub do
alias Pleroma.Activity
alias Pleroma.Activity.Ir.Topics
alias Pleroma.ActivityExpiration
alias Pleroma.Config
alias Pleroma.Constants
alias Pleroma.Conversation
@ -146,12 +147,14 @@ def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when
{:containment, :ok} <- {:containment, Containment.contain_child(map)},
{:ok, map, object} <- insert_full_object(map) do
{:ok, activity} =
Repo.insert(%Activity{
%Activity{
data: map,
local: local,
actor: map["actor"],
recipients: recipients
})
}
|> Repo.insert()
|> maybe_create_activity_expiration()
# Splice in the child object if we have one.
activity = Maps.put_if_present(activity, :object, object)
@ -189,6 +192,14 @@ def notify_and_stream(activity) do
stream_out_participations(participations)
end
defp maybe_create_activity_expiration({:ok, %{data: %{"expires_at" => expires_at}} = activity}) do
with {:ok, _} <- ActivityExpiration.create(activity, expires_at) do
{:ok, activity}
end
end
defp maybe_create_activity_expiration(result), do: result
defp create_or_bump_conversation(activity, actor) do
with {:ok, conversation} <- Conversation.create_or_bump_for(activity),
%User{} = user <- User.get_cached_by_ap_id(actor) do

View file

@ -8,11 +8,8 @@ defmodule Pleroma.Web.ActivityPub.MRF do
def filter(policies, %{} = object) do
policies
|> Enum.reduce({:ok, object}, fn
policy, {:ok, object} ->
policy.filter(object)
_, error ->
error
policy, {:ok, object} -> policy.filter(object)
_, error -> error
end)
end

View file

@ -0,0 +1,43 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy do
@moduledoc "Adds expiration to all local Create activities"
@behaviour Pleroma.Web.ActivityPub.MRF
@impl true
def filter(activity) do
activity =
if note?(activity) and local?(activity) do
maybe_add_expiration(activity)
else
activity
end
{:ok, activity}
end
@impl true
def describe, do: {:ok, %{}}
defp local?(%{"id" => id}) do
String.starts_with?(id, Pleroma.Web.Endpoint.url())
end
defp note?(activity) do
match?(%{"type" => "Create", "object" => %{"type" => "Note"}}, activity)
end
defp maybe_add_expiration(activity) do
days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
expires_at = NaiveDateTime.utc_now() |> Timex.shift(days: days)
with %{"expires_at" => existing_expires_at} <- activity,
:lt <- NaiveDateTime.compare(existing_expires_at, expires_at) do
activity
else
_ -> Map.put(activity, "expires_at", expires_at)
end
end
end

View file

@ -197,6 +197,13 @@ defp preview?(draft) do
defp changes(draft) do
direct? = draft.visibility == "direct"
additional = %{"cc" => draft.cc, "directMessage" => direct?}
additional =
case draft.expires_at do
%NaiveDateTime{} = expires_at -> Map.put(additional, "expires_at", expires_at)
_ -> additional
end
changes =
%{
@ -204,7 +211,7 @@ defp changes(draft) do
actor: draft.user,
context: draft.context,
object: draft.object,
additional: %{"cc" => draft.cc, "directMessage" => direct?}
additional: additional
}
|> Utils.maybe_add_list_data(draft.user, draft.visibility)

View file

@ -423,20 +423,10 @@ def listen(user, data) do
def post(user, %{status: _} = data) do
with {:ok, draft} <- Pleroma.Web.CommonAPI.ActivityDraft.create(user, data) do
draft.changes
|> ActivityPub.create(draft.preview?)
|> maybe_create_activity_expiration(draft.expires_at)
ActivityPub.create(draft.changes, draft.preview?)
end
end
defp maybe_create_activity_expiration({:ok, activity}, %NaiveDateTime{} = expires_at) do
with {:ok, _} <- ActivityExpiration.create(activity, expires_at) do
{:ok, activity}
end
end
defp maybe_create_activity_expiration(result, _), do: result
def pin(id, %{ap_id: user_ap_id} = user) do
with %Activity{
actor: ^user_ap_id,

View file

@ -124,6 +124,7 @@ defp resource_search(:v1, "hashtags", query, _options) do
defp prepare_tags(query, add_joined_tag \\ true) do
tags =
query
|> preprocess_uri_query()
|> String.split(~r/[^#\w]+/u, trim: true)
|> Enum.uniq_by(&String.downcase/1)
@ -147,6 +148,20 @@ defp prepare_tags(query, add_joined_tag \\ true) do
end
end
# If `query` is a URI, returns last component of its path, otherwise returns `query`
defp preprocess_uri_query(query) do
if query =~ ~r/https?:\/\// do
query
|> String.trim_trailing("/")
|> URI.parse()
|> Map.get(:path)
|> String.split("/")
|> Enum.at(-1)
else
query
end
end
defp joined_tag(tags) do
tags
|> Enum.map(fn tag -> String.capitalize(tag) end)

View file

@ -4,6 +4,7 @@
defmodule Mix.Tasks.Pleroma.UserTest do
alias Pleroma.Activity
alias Pleroma.MFA
alias Pleroma.Object
alias Pleroma.Repo
alias Pleroma.Tests.ObanHelpers
@ -278,6 +279,35 @@ test "no user to reset password" do
end
end
describe "running reset_mfa" do
test "disables MFA" do
user =
insert(:user,
multi_factor_authentication_settings: %MFA.Settings{
enabled: true,
totp: %MFA.Settings.TOTP{secret: "xx", confirmed: true}
}
)
Mix.Tasks.Pleroma.User.run(["reset_mfa", user.nickname])
assert_received {:mix_shell, :info, [message]}
assert message == "Multi-Factor Authentication disabled for #{user.nickname}"
assert %{enabled: false, totp: false} ==
user.nickname
|> User.get_cached_by_nickname()
|> MFA.mfa_settings()
end
test "no user to reset MFA" do
Mix.Tasks.Pleroma.User.run(["reset_password", "nonexistent"])
assert_received {:mix_shell, :error, [message]}
assert message =~ "No local user"
end
end
describe "running invite" do
test "invite token is generated" do
assert capture_io(fn ->

View file

@ -1986,4 +1986,20 @@ test "it just returns the input if the user has no following/follower addresses"
end) =~ "Follower/Following counter update for #{user.ap_id} failed"
end
end
describe "global activity expiration" do
setup do: clear_config([:instance, :rewrite_policy])
test "creates an activity expiration for local Create activities" do
Pleroma.Config.put(
[:instance, :rewrite_policy],
Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy
)
{:ok, %{id: id_create}} = ActivityBuilder.insert(%{"type" => "Create", "context" => "3hu"})
{:ok, _follow} = ActivityBuilder.insert(%{"type" => "Follow", "context" => "3hu"})
assert [%{activity_id: ^id_create}] = Pleroma.ActivityExpiration |> Repo.all()
end
end
end

View file

@ -0,0 +1,77 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicyTest do
use ExUnit.Case, async: true
alias Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy
@id Pleroma.Web.Endpoint.url() <> "/activities/cofe"
test "adds `expires_at` property" do
assert {:ok, %{"type" => "Create", "expires_at" => expires_at}} =
ActivityExpirationPolicy.filter(%{
"id" => @id,
"type" => "Create",
"object" => %{"type" => "Note"}
})
assert Timex.diff(expires_at, NaiveDateTime.utc_now(), :days) == 364
end
test "keeps existing `expires_at` if it less than the config setting" do
expires_at = NaiveDateTime.utc_now() |> Timex.shift(days: 1)
assert {:ok, %{"type" => "Create", "expires_at" => ^expires_at}} =
ActivityExpirationPolicy.filter(%{
"id" => @id,
"type" => "Create",
"expires_at" => expires_at,
"object" => %{"type" => "Note"}
})
end
test "overwrites existing `expires_at` if it greater than the config setting" do
too_distant_future = NaiveDateTime.utc_now() |> Timex.shift(years: 2)
assert {:ok, %{"type" => "Create", "expires_at" => expires_at}} =
ActivityExpirationPolicy.filter(%{
"id" => @id,
"type" => "Create",
"expires_at" => too_distant_future,
"object" => %{"type" => "Note"}
})
assert Timex.diff(expires_at, NaiveDateTime.utc_now(), :days) == 364
end
test "ignores remote activities" do
assert {:ok, activity} =
ActivityExpirationPolicy.filter(%{
"id" => "https://example.com/123",
"type" => "Create",
"object" => %{"type" => "Note"}
})
refute Map.has_key?(activity, "expires_at")
end
test "ignores non-Create/Note activities" do
assert {:ok, activity} =
ActivityExpirationPolicy.filter(%{
"id" => "https://example.com/123",
"type" => "Follow"
})
refute Map.has_key?(activity, "expires_at")
assert {:ok, activity} =
ActivityExpirationPolicy.filter(%{
"id" => "https://example.com/123",
"type" => "Create",
"object" => %{"type" => "Cofe"}
})
refute Map.has_key?(activity, "expires_at")
end
end

View file

@ -111,6 +111,44 @@ test "constructs hashtags from search query", %{conn: conn} do
%{"name" => "prone", "url" => "#{Web.base_url()}/tag/prone"},
%{"name" => "AccidentProne", "url" => "#{Web.base_url()}/tag/AccidentProne"}
]
results =
conn
|> get("/api/v2/search?#{URI.encode_query(%{q: "https://shpposter.club/users/shpuld"})}")
|> json_response_and_validate_schema(200)
assert results["hashtags"] == [
%{"name" => "shpuld", "url" => "#{Web.base_url()}/tag/shpuld"}
]
results =
conn
|> get(
"/api/v2/search?#{
URI.encode_query(%{
q:
"https://www.washingtonpost.com/sports/2020/06/10/" <>
"nascar-ban-display-confederate-flag-all-events-properties/"
})
}"
)
|> json_response_and_validate_schema(200)
assert results["hashtags"] == [
%{"name" => "nascar", "url" => "#{Web.base_url()}/tag/nascar"},
%{"name" => "ban", "url" => "#{Web.base_url()}/tag/ban"},
%{"name" => "display", "url" => "#{Web.base_url()}/tag/display"},
%{"name" => "confederate", "url" => "#{Web.base_url()}/tag/confederate"},
%{"name" => "flag", "url" => "#{Web.base_url()}/tag/flag"},
%{"name" => "all", "url" => "#{Web.base_url()}/tag/all"},
%{"name" => "events", "url" => "#{Web.base_url()}/tag/events"},
%{"name" => "properties", "url" => "#{Web.base_url()}/tag/properties"},
%{
"name" => "NascarBanDisplayConfederateFlagAllEventsProperties",
"url" =>
"#{Web.base_url()}/tag/NascarBanDisplayConfederateFlagAllEventsProperties"
}
]
end
test "excludes a blocked users from search results", %{conn: conn} do

View file

@ -11,7 +11,10 @@ defmodule Pleroma.Workers.Cron.PurgeExpiredActivitiesWorkerTest do
import Pleroma.Factory
import ExUnit.CaptureLog
setup do: clear_config([ActivityExpiration, :enabled])
setup do
clear_config([ActivityExpiration, :enabled])
clear_config([:instance, :rewrite_policy])
end
test "deletes an expiration activity" do
Pleroma.Config.put([ActivityExpiration, :enabled], true)
@ -36,6 +39,35 @@ test "deletes an expiration activity" do
refute Pleroma.Repo.get(Pleroma.ActivityExpiration, expiration.id)
end
test "works with ActivityExpirationPolicy" do
Pleroma.Config.put([ActivityExpiration, :enabled], true)
Pleroma.Config.put(
[:instance, :rewrite_policy],
Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy
)
user = insert(:user)
days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
{:ok, %{id: id} = activity} = Pleroma.Web.CommonAPI.post(user, %{status: "cofe"})
past_date =
NaiveDateTime.utc_now() |> Timex.shift(days: -days) |> NaiveDateTime.truncate(:second)
activity
|> Repo.preload(:expiration)
|> Map.get(:expiration)
|> Ecto.Changeset.change(%{scheduled_at: past_date})
|> Repo.update!()
Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker.perform(:ops, :pid)
assert [%{data: %{"type" => "Delete", "deleted_activity_id" => ^id}}] =
Pleroma.Repo.all(Pleroma.Activity)
end
describe "delete_activity/1" do
test "adds log message if activity isn't find" do
assert capture_log([level: :error], fn ->