sam/pleroma
sam/pleroma
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Merge branch 'ffmpeg-imagemagick-dependencies-documentation' into 'develop'

Browse source 
`ffmpeg` / `ImageMagick` handling as optional dependencies

See merge request pleroma/pleroma!3031
This commit is contained in:
lain 2020-09-29 14:48:30 +00:00
commit 2a11068a11
17 changed files with 235 additions and 33 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.gitlab-ci.yml
View file

@ -59,7 +59,7 @@ unit-testing:
alias: postgres
command: ["postgres", "-c", "fsync=off", "-c", "synchronous_commit=off", "-c", "full_page_writes=off"]
script:
- apt-get update && apt-get install -y libimage-exiftool-perl
- apt-get update && apt-get install -y libimage-exiftool-perl ffmpeg
- mix deps.get
- mix ecto.create
- mix ecto.migrate
@ -93,7 +93,7 @@ unit-testing-rum:
<<: *global_variables
RUM_ENABLED: "true"
script:
- apt-get update && apt-get install -y libimage-exiftool-perl
- apt-get update && apt-get install -y libimage-exiftool-perl ffmpeg
- mix deps.get
- mix ecto.create
- mix ecto.migrate

3
CHANGELOG.md
View file

@ -18,9 +18,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- The `discoverable` field in the `User` struct will now add a NOINDEX metatag to profile pages when false.
- Users with the `discoverable` field set to false will not show up in searches.
- Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).
- Introduced optional dependencies on `ffmpeg`, `ImageMagick`, `exiftool` software packages. Please refer to `docs/installation/optional/media_graphics_packages.md`.
### Added
- Media preview proxy (requires media proxy be enabled; see `:media_preview_proxy` config for more details).
- Media preview proxy (requires `ffmpeg` and `ImageMagick` to be installed and media proxy to be enabled; see `:media_preview_proxy` config for more details).
- Pleroma API: Importing the mutes users from CSV files.
- Experimental websocket-based federation between Pleroma instances.
- Admin API: Importing emoji from a zip file

11
docs/installation/alpine_linux_en.md
View file

@ -20,6 +20,9 @@ It assumes that you have administrative rights, either as root or a user with [s
* `nginx` (preferred, example configs for other reverse proxies can be found in the repo)
* `certbot` (or any other ACME client for Let’s Encrypt certificates)
* `ImageMagick`
* `ffmpeg`
* `exiftool`
### Prepare the system
@ -29,7 +32,6 @@ It assumes that you have administrative rights, either as root or a user with [s
awk 'NR==2' /etc/apk/repositories | sed 's/main/community/' | tee -a /etc/apk/repositories
```
* Then update the system, if not already done:
```shell
@ -56,6 +58,7 @@ sudo apk add erlang erlang-runtime-tools erlang-xmerl elixir
```shell
sudo apk add erlang-eldap
```
### Install PostgreSQL
* Install Postgresql server:
@ -76,6 +79,12 @@ sudo /etc/init.d/postgresql start
sudo rc-update add postgresql
```
### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md))
```shell
sudo apk add ffmpeg imagemagick exiftool
```
### Install PleromaBE
* Add a new system user for the Pleroma service:

9
docs/installation/arch_linux_en.md
View file

@ -15,6 +15,9 @@ This guide will assume that you have administrative rights, either as root or a
* `nginx` (preferred, example configs for other reverse proxies can be found in the repo)
* `certbot` (or any other ACME client for Let’s Encrypt certificates)
* `ImageMagick`
* `ffmpeg`
* `exiftool`
### Prepare the system
@ -52,6 +55,12 @@ sudo -iu postgres initdb -D /var/lib/postgres/data
sudo systemctl enable --now postgresql.service
```
### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md))
```shell
sudo pacman -S ffmpeg imagemagick perl-image-exiftool
```
### Install PleromaBE
* Add a new system user for the Pleroma service:

9
docs/installation/debian_based_en.md
View file

@ -18,6 +18,9 @@ This guide will assume you are on Debian Stretch. This guide should also work wi
* `nginx` (preferred, example configs for other reverse proxies can be found in the repo)
* `certbot` (or any other ACME client for Let’s Encrypt certificates)
* `ImageMagick`
* `ffmpeg`
* `exiftool`
### Prepare the system
@ -50,6 +53,12 @@ sudo apt update
sudo apt install elixir erlang-dev erlang-nox
```
### Optional packages: [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md)
```shell
sudo apt install imagemagick ffmpeg libimage-exiftool-perl
```
### Install PleromaBE
* Add a new system user for the Pleroma service:

12
docs/installation/debian_based_jp.md
View file

@ -22,6 +22,9 @@
- `nginx` (おすすめです。他のリバースプロキシを使う場合は、参考となる設定をこのリポジトリから探してください)
- `certbot` (または何らかのLet's Encrypt向けACMEクライアント)
- `ImageMagick`
- `ffmpeg`
- `exiftool`
### システムを準備する
@ -33,10 +36,9 @@ sudo apt full-upgrade
* 上記に挙げたパッケージをインストールしておきます。
```
sudo apt install git build-essential postgresql postgresql-contrib cmake
sudo apt install git build-essential postgresql postgresql-contrib cmake ffmpeg imagemagick
```
### ElixirとErlangをインストールします
* Erlangのリポジトリをダウンロードおよびインストールします。
@ -51,6 +53,12 @@ sudo apt update
sudo apt install elixir erlang-dev erlang-nox
```
### オプションパッケージ: [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md)
```shell
sudo apt install imagemagick ffmpeg libimage-exiftool-perl
```
### Pleroma BE (バックエンド) をインストールします
* Pleroma用に新しいユーザーを作ります。

6
docs/installation/freebsd_en.md
View file

@ -26,6 +26,12 @@ Setup the required services to automatically start at boot, using `sysrc(8)`.
# service postgresql start
```
### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md))
```shell
# pkg install imagemagick ffmpeg p5-Image-ExifTool
```
## Configuring Pleroma
Create a user for Pleroma:

9
docs/installation/gentoo_en.md
View file

@ -35,6 +35,9 @@ Gentoo quite pointedly does not come with a cron daemon installed, and as such i
* `www-servers/nginx` (preferred, example configs for other reverse proxies can be found in the repo)
* `app-crypt/certbot` (or any other ACME client for Let’s Encrypt certificates)
* `app-crypt/certbot-nginx` (nginx certbot plugin that allows use of the all-powerful `--nginx` flag on certbot)
* `media-gfx/imagemagick`
* `media-video/ffmpeg`
* `media-libs/exiftool`
### Prepare the system
@ -87,6 +90,12 @@ If you do not plan to make any modifications to your Pleroma instance, cloning d
Not only does this make it much easier to deploy changes you make, as you can commit and pull from upstream and all that good stuff from the comfort of your local machine then simply `git pull` on your instance server when you're ready to deploy, it also ensures you are compliant with the Affero General Public Licence that Pleroma is licenced under, which stipulates that all network services provided with modified AGPL code must publish their changes on a publicly available internet service and for free. It also makes it much easier to ask for help from and provide help to your fellow Pleroma admins if your public repo always reflects what you are running because it is part of your deployment procedure.
### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md))
```shell
# emerge --ask media-video/ffmpeg media-gfx/imagemagick media-libs/exiftool
```
### Install PleromaBE
* Add a new system user for the Pleroma service and set up default directories:

6
docs/installation/netbsd_en.md
View file

@ -10,7 +10,7 @@ Pleroma uses.
The `mksh` shell is needed to run the Elixir `mix` script.
`# pkgin install acmesh elixir git-base git-docs mksh nginx postgresql11-server postgresql11-client postgresql11-contrib sudo`
`# pkgin install acmesh elixir git-base git-docs mksh nginx postgresql11-server postgresql11-client postgresql11-contrib sudo ffmpeg4 ImageMagick`
You can also build these packages using pkgsrc:
```
@ -44,6 +44,10 @@ pgsql=YES
First, run `# /etc/rc.d/pgsql start`. Then, `$ sudo -Hu pgsql -g pgsql createdb`.
### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md))
`# pkgin install ImageMagick ffmpeg4 p5-Image-ExifTool`
## Configuring Pleroma
Create a user for Pleroma:

18
docs/installation/openbsd_en.md
View file

@ -10,20 +10,34 @@ The following packages need to be installed:
* elixir
* gmake
* ImageMagick
* git
* postgresql-server
* postgresql-contrib
* cmake
* ffmpeg
* ImageMagick
To install them, run the following command (with doas or as root):
```
pkg_add elixir gmake ImageMagick git postgresql-server postgresql-contrib cmake
pkg_add elixir gmake git postgresql-server postgresql-contrib cmake ffmpeg ImageMagick
```
Pleroma requires a reverse proxy, OpenBSD has relayd in base (and is used in this guide) and packages/ports are available for nginx (www/nginx) and apache (www/apache-httpd). Independently of the reverse proxy, [acme-client(1)](https://man.openbsd.org/acme-client) can be used to get a certificate from Let's Encrypt.
#### Optional software
Per [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md):
* ImageMagick
* ffmpeg
* exiftool
To install the above:
```
pkg_add ImageMagick ffmpeg p5-Image-ExifTool
```
#### Creating the pleroma user
Pleroma will be run by a dedicated user, \_pleroma. Before creating it, insert the following lines in login.conf:
```

13
docs/installation/openbsd_fi.md
View file

@ -16,7 +16,18 @@ Matrix-kanava #freenode_#pleroma:matrix.org ovat hyviä paikkoja löytää apua
Asenna tarvittava ohjelmisto:
`# pkg_add git elixir gmake postgresql-server-10.3 postgresql-contrib-10.3 cmake`
`# pkg_add git elixir gmake postgresql-server-10.3 postgresql-contrib-10.3 cmake ffmpeg ImageMagick`
#### Optional software
[`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md):
* ImageMagick
* ffmpeg
* exiftool
Asenna tarvittava ohjelmisto:
`# pkg_add ImageMagick ffmpeg p5-Image-ExifTool`
Luo postgresql-tietokanta:

32
docs/installation/optional/media_graphics_packages.md Normal file
View file

@ -0,0 +1,32 @@
# Optional software packages needed for specific functionality
For specific Pleroma functionality (which is disabled by default) some or all of the below packages are required:
* `ImageMagic`
* `ffmpeg`
* `exiftool`
Please refer to documentation in `docs/installation` on how to install them on specific OS.
Note: the packages are not required with the current default settings of Pleroma.
## `ImageMagick`
`ImageMagick` is a set of tools to create, edit, compose, or convert bitmap images.
It is required for the following Pleroma features:
* `Pleroma.Upload.Filters.Mogrify`, `Pleroma.Upload.Filters.Mogrifun` upload filters (related config: `Plaroma.Upload/filters` in `config/config.exs`)
* Media preview proxy for still images (related config: `media_preview_proxy/enabled` in `config/config.exs`)
## `ffmpeg`
`ffmpeg` is software to record, convert and stream audio and video.
It is required for the following Pleroma features:
* Media preview proxy for videos (related config: `media_preview_proxy/enabled` in `config/config.exs`)
## `exiftool`
`exiftool` is media files metadata reader/writer.
It is required for the following Pleroma features:
* `Pleroma.Upload.Filters.Exiftool` upload filter (related config: `Plaroma.Upload/filters` in `config/config.exs`)

19
docs/installation/otp_en.md
View file

@ -40,6 +40,25 @@ Other than things bundled in the OTP release Pleroma depends on:
apt install curl unzip libncurses5 postgresql postgresql-contrib nginx certbot
```
### Installing optional packages
Per [`docs/installation/optional/media_graphics_packages.md`](docs/installation/optional/media_graphics_packages.md):
* ImageMagick
* ffmpeg
* exiftool
=== "Alpine"
```
echo "http://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories
apk update
apk add imagemagick ffmpeg exiftool
```
=== "Debian/Ubuntu"
```
apt install imagemagick ffmpeg libimage-exiftool-perl
```
## Setup
### Configuring PostgreSQL
#### (Optional) Installing RUM indexes

18
lib/pleroma/application.ex
View file

@ -56,7 +56,6 @@ def start(_type, _args) do
Pleroma.ApplicationRequirements.verify!()
setup_instrumenters()
load_custom_modules()
check_system_commands()
Pleroma.Docs.JSON.compile()
adapter = Application.get_env(:tesla, :adapter)
@ -260,21 +259,4 @@ defp http_children(Tesla.Adapter.Gun, _) do
end
defp http_children(_, _), do: []
defp check_system_commands do
filters = Config.get([Pleroma.Upload, :filters])
check_filter = fn filter, command_required ->
with true <- filter in filters,
false <- Pleroma.Utils.command_available?(command_required) do
Logger.error(
"#{filter} is specified in list of Pleroma.Upload filters, but the #{command_required} command is not found"
)
end
end
check_filter.(Pleroma.Upload.Filters.Exiftool, "exiftool")
check_filter.(Pleroma.Upload.Filters.Mogrify, "mogrify")
check_filter.(Pleroma.Upload.Filters.Mogrifun, "mogrify")
end
end

72
lib/pleroma/application_requirements.ex
View file

@ -9,6 +9,9 @@ defmodule Pleroma.ApplicationRequirements do
defmodule VerifyError, do: defexception([:message])
alias Pleroma.Config
alias Pleroma.Helpers.MediaHelper
import Ecto.Query
require Logger
@ -16,7 +19,8 @@ defmodule VerifyError, do: defexception([:message])
@spec verify!() :: :ok | VerifyError.t()
def verify! do
:ok
|> check_confirmation_accounts!
|> check_system_commands!()
|> check_confirmation_accounts!()
|> check_migrations_applied!()
|> check_welcome_message_config!()
|> check_rum!()
@ -48,7 +52,9 @@ def check_confirmation_accounts!(:ok) do
if Pleroma.Config.get([:instance, :account_activation_required]) &&
not Pleroma.Config.get([Pleroma.Emails.Mailer, :enabled]) do
Logger.error(
"Account activation enabled, but no Mailer settings enabled.\nPlease set config :pleroma, :instance, account_activation_required: false\nOtherwise setup and enable Mailer."
"Account activation enabled, but no Mailer settings enabled.\n" <>
"Please set config :pleroma, :instance, account_activation_required: false\n" <>
"Otherwise setup and enable Mailer."
)
{:error,
@ -81,7 +87,9 @@ def check_migrations_applied!(:ok) do
Enum.map(down_migrations, fn {:down, id, name} -> "- #{name} (#{id})\n" end)
Logger.error(
"The following migrations were not applied:\n#{down_migrations_text}If you want to start Pleroma anyway, set\nconfig :pleroma, :i_am_aware_this_may_cause_data_loss, disable_migration_check: true"
"The following migrations were not applied:\n#{down_migrations_text}" <>
"If you want to start Pleroma anyway, set\n" <>
"config :pleroma, :i_am_aware_this_may_cause_data_loss, disable_migration_check: true"
)
{:error, "Unapplied Migrations detected"}
@ -124,14 +132,22 @@ defp do_check_rum!(setting, migrate) do
case {setting, migrate} do
{true, false} ->
Logger.error(
"Use `RUM` index is enabled, but were not applied migrations for it.\nIf you want to start Pleroma anyway, set\nconfig :pleroma, :database, rum_enabled: false\nOtherwise apply the following migrations:\n`mix ecto.migrate --migrations-path priv/repo/optional_migrations/rum_indexing/`"
"Use `RUM` index is enabled, but were not applied migrations for it.\n" <>
"If you want to start Pleroma anyway, set\n" <>
"config :pleroma, :database, rum_enabled: false\n" <>
"Otherwise apply the following migrations:\n" <>
"`mix ecto.migrate --migrations-path priv/repo/optional_migrations/rum_indexing/`"
)
{:error, "Unapplied RUM Migrations detected"}
{false, true} ->
Logger.error(
"Detected applied migrations to use `RUM` index, but `RUM` isn't enable in settings.\nIf you want to use `RUM`, set\nconfig :pleroma, :database, rum_enabled: true\nOtherwise roll `RUM` migrations back.\n`mix ecto.rollback --migrations-path priv/repo/optional_migrations/rum_indexing/`"
"Detected applied migrations to use `RUM` index, but `RUM` isn't enable in settings.\n" <>
"If you want to use `RUM`, set\n" <>
"config :pleroma, :database, rum_enabled: true\n" <>
"Otherwise roll `RUM` migrations back.\n" <>
"`mix ecto.rollback --migrations-path priv/repo/optional_migrations/rum_indexing/`"
)
{:error, "RUM Migrations detected"}
@ -140,4 +156,50 @@ defp do_check_rum!(setting, migrate) do
:ok
end
end
defp check_system_commands!(:ok) do
filter_commands_statuses = [
check_filter(Pleroma.Upload.Filters.Exiftool, "exiftool"),
check_filter(Pleroma.Upload.Filters.Mogrify, "mogrify"),
check_filter(Pleroma.Upload.Filters.Mogrifun, "mogrify")
]
preview_proxy_commands_status =
if !Config.get([:media_preview_proxy, :enabled]) or
MediaHelper.missing_dependencies() == [] do
true
else
Logger.error(
"The following dependencies required by Media preview proxy " <>
"(which is currently enabled) are not installed: " <>
inspect(MediaHelper.missing_dependencies())
)
false
end
if Enum.all?([preview_proxy_commands_status | filter_commands_statuses], & &1) do
:ok
else
{:error,
"System commands missing. Check logs and see `docs/installation` for more details."}
end
end
defp check_system_commands!(result), do: result
defp check_filter(filter, command_required) do
filters = Config.get([Pleroma.Upload, :filters])
if filter in filters and not Pleroma.Utils.command_available?(command_required) do
Logger.error(
"#{filter} is specified in list of Pleroma.Upload filters, but the " <>
"#{command_required} command is not found"
)
false
else
true
end
end
end

12
lib/pleroma/helpers/media_helper.ex
View file

@ -9,6 +9,18 @@ defmodule Pleroma.Helpers.MediaHelper do
alias Pleroma.HTTP
require Logger
def missing_dependencies do
Enum.reduce([imagemagick: "convert", ffmpeg: "ffmpeg"], [], fn {sym, executable}, acc ->
if Pleroma.Utils.command_available?(executable) do
acc
else
[sym | acc]
end
end)
end
def image_resize(url, options) do
with executable when is_binary(executable) <- System.find_executable("convert"),
{:ok, args} <- prepare_image_resize_args(options),

15
test/web/media_proxy/media_proxy_controller_test.exs
View file

@ -81,6 +81,15 @@ test "it returns 404 when url is in banned_urls cache", %{conn: conn, url: url}
end
describe "Media Preview Proxy" do
def assert_dependencies_installed do
missing_dependencies = Pleroma.Helpers.MediaHelper.missing_dependencies()
assert missing_dependencies == [],
"Error: missing dependencies (please refer to `docs/installation`): #{
inspect(missing_dependencies)
}"
end
setup do
clear_config([:media_proxy, :enabled], true)
clear_config([:media_preview_proxy, :enabled], true)
@ -184,6 +193,8 @@ test "with `static=true` and GIF image preview requested, responds with JPEG ima
url: url,
media_proxy_url: media_proxy_url
} do
assert_dependencies_installed()
# Setting a high :min_content_length to ensure this scenario is not affected by its logic
clear_config([:media_preview_proxy, :min_content_length], 1_000_000_000)
@ -270,6 +281,8 @@ test "thumbnails PNG images into PNG", %{
url: url,
media_proxy_url: media_proxy_url
} do
assert_dependencies_installed()
Tesla.Mock.mock(fn
%{method: "head", url: ^media_proxy_url} ->
%Tesla.Env{status: 200, body: "", headers: [{"content-type", "image/png"}]}
@ -290,6 +303,8 @@ test "thumbnails JPEG images into JPEG", %{
url: url,
media_proxy_url: media_proxy_url
} do
assert_dependencies_installed()
Tesla.Mock.mock(fn
%{method: "head", url: ^media_proxy_url} ->
%Tesla.Env{status: 200, body: "", headers: [{"content-type", "image/jpeg"}]}