Merge branch 'master' of https://github.com/NicolasConstant/BirdsiteLive
continuous-integration/drone/push Build is passing Details

pull/42/head
Sam Therapy 3 weeks ago
commit 1037449e18
Signed by: sam
GPG Key ID: 4D8B07C18F31ACBD

@ -0,0 +1,59 @@
kind: pipeline
name: testing
type: docker
steps:
- name: Install Dependencies
image: mcr.microsoft.com/dotnet/sdk:6.0
commands:
- dotnet restore ./src
- name: Build
image: mcr.microsoft.com/dotnet/sdk:6.0
commands:
- dotnet build --configuration Release ./src
- name: Test
image: mcr.microsoft.com/dotnet/sdk:6.0
commands:
- sed -i "s/127\.0\.0\.1/database/g" ./src/Tests/BirdsiteLive.DAL.Postgres.Tests/DataAccessLayers/Base/PostgresTestingBase.cs
- dotnet test --verbosity minimal ./src
services:
- name: database
image: postgres:15
environment:
POSTGRES_USER: birdtest
POSTGRES_PASSWORD: mysecretpassword
POSTGRES_DB: birdsitetest
---
kind: pipeline
name: docker-publish
type: docker
depends_on:
- testing
steps:
- name: Build & Publish
privileged: true
image: quay.io/thegeeklab/drone-docker-buildx
settings:
auto_tag: true
repo: git.froth.zone/sam/birdsitelive
registry: git.froth.zone
username: sam
password:
from_secret: password
platforms:
- linux/amd64
- linux/arm64
when:
branch:
- master
event:
- push
depends_on:
- "clone"

2
.gitignore vendored

@ -352,3 +352,5 @@ MigrationBackup/
# Ionide (cross platform F# VS Code tools) working folder
.ionide/
/src/BSLManager/Properties/launchSettings.json
.dccache

@ -1,14 +1,15 @@
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.
FROM mcr.microsoft.com/dotnet/aspnet:3.1-buster-slim AS base
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/sdk:3.1-buster AS publish
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS publish
WORKDIR /
COPY ./src/ ./src/
RUN dotnet publish "/src/BirdsiteLive/BirdsiteLive.csproj" -c Release -o /app/publish
RUN dotnet publish "/src/BSLManager/BSLManager.csproj" -r linux-x64 --self-contained true -p:PublishSingleFile=true -p:IncludeAllContentForSelfExtract=true -c Release -o /app/publish
RUN dotnet publish "/src/BirdsiteLive/BirdsiteLive.csproj" -c Release -o /app/publish \
&& dotnet publish "/src/BSLManager/BSLManager.csproj" -r linux-x64 --self-contained true -p:PublishSingleFile=true -p:IncludeAllContentForSelfExtract=true -c Release -o /app/publish
FROM base AS final
WORKDIR /app

@ -4,6 +4,9 @@
You will need a Twitter API key to make BirdsiteLIVE working. First create an **Standalone App** in the [Twitter developer portal](https://developer.twitter.com/en/portal/projects-and-apps) and retrieve the API Key and API Secret Key.
Please make sure you are using a **Standalone App** API Key and not a **Project App** API Key (that will NOT work with BirdsiteLIVE), if you don't see the **Standalone App** section, you might need to [apply for Elevated Access](https://developer.twitter.com/en/portal/products/elevated) as described in the [API documentation](https://developer.twitter.com/en/support/twitter-api/developer-account).
## Server prerequisites
Your instance will need [docker](https://docs.docker.com/engine/install/) and [docker-compose](https://docs.docker.com/compose/install/) installed and working.
@ -116,7 +119,7 @@ sudo certbot --nginx -d {your-domain-name.com}
Make sure you're redirecting all traffic to https when asked.
Finally check that the auto-revewal will work as espected:
Finally check that the auto-renewal will work as espected:
```
sudo certbot renew --dry-run

@ -24,15 +24,15 @@ BirdsiteLIVE is an ActivityPub bridge from Twitter, it's mostly a pet project/pl
## State of development
The code is pretty messy and far from a good state, since it's a playground for me the aim was to understand some AP concepts, not provide a good state-of-the-art codebase. But I might refactor it to make it cleaner.
The code is pretty messy and far from a good state, since it's a playground for me the aim was to understand some AP concepts, not to provide a good state-of-the-art codebase. But I might refactor it to make it cleaner.
## Official instance
You can find an official (and temporary) instance here: [beta.birdsite.live](https://beta.birdsite.live). This instance can disapear at any time, if you want a long term instance you should install your own or use another one.
There's none! Please read [here why I've stopped it](https://write.as/nicolas-constant/closing-the-official-bsl-instance).
## Installation
I'm providing a [docker build](https://hub.docker.com/r/nicolasconstant/birdsitelive). To install it on your own server, please follow [those instructions](https://github.com/NicolasConstant/BirdsiteLive/blob/master/INSTALLATION.md). More [options](https://github.com/NicolasConstant/BirdsiteLive/blob/master/VARIABLES.md) are also available.
I'm providing a [docker build](https://hub.docker.com/r/nicolasconstant/birdsitelive) (linux/amd64 only). To install it on your own server, please follow [those instructions](https://github.com/NicolasConstant/BirdsiteLive/blob/master/INSTALLATION.md). More [options](https://github.com/NicolasConstant/BirdsiteLive/blob/master/VARIABLES.md) are also available.
Also a [CLI](https://github.com/NicolasConstant/BirdsiteLive/blob/master/BSLManager.md) is available for adminitrative tasks.

@ -58,6 +58,8 @@ If both whitelisting and blacklisting are set, only the whitelisting will be act
* `Instance:FailingTwitterUserCleanUpThreshold` (default: 700) set the max allowed errors (due to a banned/deleted/private account) from a Twitter Account retrieval before auto-removal. (by default an account is called every 15 mins)
* `Instance:MaxStatusFetchAge` (default: 0 - no limit) statuses with a Snowflake older than this age in days will not be fetched by the service and will instead return 410 Gone
* `Instance:EnableQuoteRT` (default: false) enable Soapbox-style quote-RTs
* `Instance:FailingFollowerCleanUpThreshold` (default: 30000) set the max allowed errors from a Follower (Fediverse) Account before auto-removal. (often due to account suppression, instance issues, etc)
* `Instance:UserCacheCapacity` (default: 10000) set the caching limit of the Twitter User retrieval. Must be higher than the number of synchronized accounts on the instance.
# Docker Compose full example

@ -6,7 +6,7 @@ networks:
services:
server:
image: pasture/birdsitelive:latest
image: git.froth.zone/birdsitelive:latest
restart: always
container_name: birdsitelive
environment:
@ -27,7 +27,7 @@ services:
- db
db:
image: postgres:13
image: postgres:15
restart: always
environment:
- POSTGRES_USER=birdsitelive

@ -2,7 +2,7 @@
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.1</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
@ -10,7 +10,7 @@
<PackageReference Include="Microsoft.Extensions.Configuration" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="5.0.0" />
<PackageReference Include="Terminal.Gui" Version="1.0.0-beta.11" />
<PackageReference Include="Terminal.Gui" Version="1.8.0" />
</ItemGroup>
<ItemGroup>

@ -1,4 +1,5 @@
using System;
using BirdsiteLive.ActivityPub.Models;
using Newtonsoft.Json;
namespace BirdsiteLive.ActivityPub
@ -19,6 +20,8 @@ namespace BirdsiteLive.ActivityPub
if(a.apObject.type == "Follow")
return JsonConvert.DeserializeObject<ActivityUndoFollow>(json);
break;
case "Delete":
return JsonConvert.DeserializeObject<ActivityDelete>(json);
case "Accept":
var accept = JsonConvert.DeserializeObject<ActivityAccept>(json);
//var acceptType = JsonConvert.DeserializeObject<Activity>(accept.apObject);

@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
<ItemGroup>

@ -0,0 +1,11 @@
using Newtonsoft.Json;
namespace BirdsiteLive.ActivityPub.Models
{
public class ActivityDelete : Activity
{
public string[] to { get; set; }
[JsonProperty("object")]
public object apObject { get; set; }
}
}

@ -20,6 +20,7 @@ namespace BirdsiteLive.ActivityPub
public string name { get; set; }
public string summary { get; set; }
public string url { get; set; }
public string movedTo { get; set; }
public bool manuallyApprovesFollowers { get; set; }
public string inbox { get; set; }
public bool? discoverable { get; set; } = true;

@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
</Project>

@ -5,5 +5,7 @@ namespace BirdsiteLive.Common.Regexes
public class UrlRegexes
{
public static readonly Regex Url = new Regex(@"(.?)(((http|ftp|https):\/\/)[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&amp;:/~\+#]*[\w\-\@?^=%&amp;/~\+#])?)");
public static readonly Regex Domain = new Regex(@"^[a-zA-Z0-9\-_]+(\.[a-zA-Z0-9\-_]+)+$");
}
}

@ -30,5 +30,8 @@
public int MaxStatusFetchAge { get; set; }
public bool EnableQuoteRT { get; set; }
public int FailingFollowerCleanUpThreshold { get; set; } = -1;
public int UserCacheCapacity { get; set; }
}
}

@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
<ItemGroup>

@ -11,17 +11,24 @@ using BirdsiteLive.ActivityPub.Models;
using BirdsiteLive.Common.Settings;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Org.BouncyCastle.Bcpg;
namespace BirdsiteLive.Domain
{
public interface IActivityPubService
{
Task<string> GetUserIdAsync(string acct);
Task<Actor> GetUser(string objectId);
Task<HttpStatusCode> PostDataAsync<T>(T data, string targetHost, string actorUrl, string inbox = null);
Task PostNewNoteActivity(Note note, string username, string noteId, string targetHost,
string targetInbox);
Task<WebFingerData> WebFinger(string account);
Task DeleteUserAsync(string username, string targetHost, string targetInbox);
Task<WebFingerData> WebFinger(string account);
}
public class WebFinger
{
public string subject { get; set; }
public string[] aliases { get; set; }
}
public class ActivityPubService : IActivityPubService
@ -41,11 +48,35 @@ namespace BirdsiteLive.Domain
}
#endregion
public async Task<string> GetUserIdAsync(string acct)
{
var splittedAcct = acct.Trim('@').Split('@');
var url = $"https://{splittedAcct[1]}/.well-known/webfinger?resource=acct:{splittedAcct[0]}@{splittedAcct[1]}";
var httpClient = _httpClientFactory.CreateClient();
httpClient.DefaultRequestHeaders.Add("Accept", "application/json");
var result = await httpClient.GetAsync(url);
result.EnsureSuccessStatusCode();
var content = await result.Content.ReadAsStringAsync();
var actor = JsonConvert.DeserializeObject<WebFinger>(content);
return actor.aliases.FirstOrDefault();
}
public async Task<Actor> GetUser(string objectId)
{
var httpClient = _httpClientFactory.CreateClient();
httpClient.DefaultRequestHeaders.Add("Accept", "application/activity+json");
var result = await httpClient.GetAsync(objectId);
if (result.StatusCode == HttpStatusCode.Gone)
throw new FollowerIsGoneException();
result.EnsureSuccessStatusCode();
var content = await result.Content.ReadAsStringAsync();
var actor = JsonConvert.DeserializeObject<Actor>(content);
@ -53,6 +84,31 @@ namespace BirdsiteLive.Domain
return actor;
}
public async Task DeleteUserAsync(string username, string targetHost, string targetInbox)
{
try
{
var actor = UrlFactory.GetActorUrl(_instanceSettings.Domain, username);
var deleteUser = new ActivityDelete
{
context = "https://www.w3.org/ns/activitystreams",
id = $"{actor}#delete",
type = "Delete",
actor = actor,
to = new [] { "https://www.w3.org/ns/activitystreams#Public" },
apObject = actor
};
await PostDataAsync(deleteUser, targetHost, actor, targetInbox);
}
catch (Exception e)
{
_logger.LogError(e, "Error deleting {Username} to {Host}{Inbox}", username, targetHost, targetInbox);
throw;
}
}
public async Task PostNewNoteActivity(Note note, string username, string noteId, string targetHost, string targetInbox)
{
try

@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
@ -15,4 +15,8 @@
<ProjectReference Include="..\DataAccessLayers\BirdsiteLive.DAL\BirdsiteLive.DAL.csproj" />
</ItemGroup>
<ItemGroup>
<Folder Include="Enum\" />
</ItemGroup>
</Project>

@ -0,0 +1,51 @@
using System.Linq;
using System.Threading.Tasks;
using BirdsiteLive.DAL.Contracts;
using BirdsiteLive.DAL.Models;
namespace BirdsiteLive.Domain.BusinessUseCases
{
public interface IProcessDeleteUser
{
Task ExecuteAsync(Follower follower);
Task ExecuteAsync(string followerUsername, string followerDomain);
}
public class ProcessDeleteUser : IProcessDeleteUser
{
private readonly IFollowersDal _followersDal;
private readonly ITwitterUserDal _twitterUserDal;
#region Ctor
public ProcessDeleteUser(IFollowersDal followersDal, ITwitterUserDal twitterUserDal)
{
_followersDal = followersDal;
_twitterUserDal = twitterUserDal;
}
#endregion
public async Task ExecuteAsync(string followerUsername, string followerDomain)
{
// Get Follower and Twitter Users
var follower = await _followersDal.GetFollowerAsync(followerUsername, followerDomain);
if (follower == null) return;
await ExecuteAsync(follower);
}
public async Task ExecuteAsync(Follower follower)
{
// Remove twitter users if no more followers
var followings = follower.Followings;
foreach (var following in followings)
{
var followers = await _followersDal.GetFollowersAsync(following);
if (followers.Length == 1 && followers.First().Id == follower.Id)
await _twitterUserDal.DeleteTwitterUserAsync(following);
}
// Remove follower from DB
await _followersDal.DeleteFollowerAsync(follower.Id);
}
}
}

@ -0,0 +1,9 @@
namespace BirdsiteLive.Domain.Enum
{
public enum MigrationTypeEnum
{
Unknown = 0,
Migration = 1,
Deletion = 2
}
}

@ -0,0 +1,8 @@
using System;
namespace BirdsiteLive.Domain
{
public class FollowerIsGoneException : Exception
{
}
}

@ -0,0 +1,352 @@
using System;
using System.Collections.Generic;
using System.Linq;
using BirdsiteLive.Twitter;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Tasks;
using BirdsiteLive.ActivityPub;
using BirdsiteLive.ActivityPub.Models;
using BirdsiteLive.DAL.Contracts;
using BirdsiteLive.ActivityPub.Converters;
using BirdsiteLive.Common.Settings;
using BirdsiteLive.DAL.Models;
using BirdsiteLive.Domain.Enum;
using System.Net.Http;
using BirdsiteLive.Common.Regexes;
using Microsoft.Extensions.Logging;
namespace BirdsiteLive.Domain
{
public class MigrationService
{
private readonly InstanceSettings _instanceSettings;
private readonly ITheFedInfoService _theFedInfoService;
private readonly ITwitterTweetsService _twitterTweetsService;
private readonly IActivityPubService _activityPubService;
private readonly ITwitterUserDal _twitterUserDal;
private readonly IFollowersDal _followersDal;
private readonly IHttpClientFactory _httpClientFactory;
private readonly ILogger<MigrationService> _logger;
#region Ctor
public MigrationService(ITwitterTweetsService twitterTweetsService, IActivityPubService activityPubService, ITwitterUserDal twitterUserDal, IFollowersDal followersDal, InstanceSettings instanceSettings, ITheFedInfoService theFedInfoService, IHttpClientFactory httpClientFactory, ILogger<MigrationService> logger)
{
_twitterTweetsService = twitterTweetsService;
_activityPubService = activityPubService;
_twitterUserDal = twitterUserDal;
_followersDal = followersDal;
_instanceSettings = instanceSettings;
_theFedInfoService = theFedInfoService;
_httpClientFactory = httpClientFactory;
_logger = logger;
}
#endregion
public string GetMigrationCode(string acct)
{
var hash = GetHashString(acct);
return $"[[BirdsiteLIVE-MigrationCode|{hash.Substring(0, 10)}]]";
}
public string GetDeletionCode(string acct)
{
var hash = GetHashString(acct);
return $"[[BirdsiteLIVE-DeletionCode|{hash.Substring(0, 10)}]]";
}
public bool ValidateTweet(string acct, string tweetId, MigrationTypeEnum type)
{
string code;
if (type == MigrationTypeEnum.Migration)
code = GetMigrationCode(acct);
else if (type == MigrationTypeEnum.Deletion)
code = GetDeletionCode(acct);
else
throw new NotImplementedException();
var castedTweetId = ExtractedTweetId(tweetId);
var tweet = _twitterTweetsService.GetTweet(castedTweetId);
if (tweet == null)
throw new Exception("Tweet not found");
if (tweet.CreatorName.Trim().ToLowerInvariant() != acct.Trim().ToLowerInvariant())
throw new Exception($"Tweet not published by @{acct}");
if (!tweet.MessageContent.Contains(code))
{
var message = "Tweet don't have migration code";
if (type == MigrationTypeEnum.Deletion)
message = "Tweet don't have deletion code";
throw new Exception(message);
}
return true;
}
private long ExtractedTweetId(string tweetId)
{
if (string.IsNullOrWhiteSpace(tweetId))
throw new ArgumentException("No provided Tweet ID");
long castedId;
if (long.TryParse(tweetId, out castedId))
return castedId;
var urlPart = tweetId.Split('/').LastOrDefault();
if (long.TryParse(urlPart, out castedId))
return castedId;
throw new ArgumentException("Unvalid Tweet ID");
}
public async Task<ValidatedFediverseUser> ValidateFediverseAcctAsync(string fediverseAcct)
{
if (string.IsNullOrWhiteSpace(fediverseAcct))
throw new ArgumentException("Please provide Fediverse account");
if (!fediverseAcct.Contains('@') || !fediverseAcct.StartsWith("@") || fediverseAcct.Trim('@').Split('@').Length != 2)
throw new ArgumentException("Please provide valid Fediverse handle");
var objectId = await _activityPubService.GetUserIdAsync(fediverseAcct);
var user = await _activityPubService.GetUser(objectId);
var result = new ValidatedFediverseUser
{
FediverseAcct = fediverseAcct,
ObjectId = objectId,
User = user,
IsValid = user != null
};
return result;
}
public async Task MigrateAccountAsync(ValidatedFediverseUser validatedUser, string acct)
{
// Apply moved to
var twitterAccount = await _twitterUserDal.GetTwitterUserAsync(acct);
if (twitterAccount == null)
{
await _twitterUserDal.CreateTwitterUserAsync(acct, -1, validatedUser.ObjectId, validatedUser.FediverseAcct);
twitterAccount = await _twitterUserDal.GetTwitterUserAsync(acct);
}
twitterAccount.MovedTo = validatedUser.User.id;
twitterAccount.MovedToAcct = validatedUser.FediverseAcct;
twitterAccount.LastSync = DateTime.UtcNow;
await _twitterUserDal.UpdateTwitterUserAsync(twitterAccount);
// Notify Followers
var message = $@"<p>[BSL MIRROR SERVICE NOTIFICATION]<br/>This bot has been disabled by its original owner.<br/>It has been redirected to {validatedUser.FediverseAcct}.</p>";
NotifyFollowers(acct, twitterAccount, message);
}
private void NotifyFollowers(string acct, SyncTwitterUser twitterAccount, string message)
{
var t = Task.Run(async () =>
{
var followers = await _followersDal.GetFollowersAsync(twitterAccount.Id);
foreach (var follower in followers)
{
try
{
var noteId = Guid.NewGuid().ToString();
var actorUrl = UrlFactory.GetActorUrl(_instanceSettings.Domain, acct);
var noteUrl = UrlFactory.GetNoteUrl(_instanceSettings.Domain, acct, noteId);
//var to = validatedUser.ObjectId;
var to = follower.ActorId;
var cc = new string[0];
var note = new Note
{
id = noteUrl,
published = DateTime.UtcNow.ToString("s") + "Z",
url = noteUrl,
attributedTo = actorUrl,
to = new[] { to },
cc = cc,
content = message,
tag = new Tag[]{
new Tag()
{
type = "Mention",
href = follower.ActorId,
name = $"@{follower.Acct}@{follower.Host}"
}
},
};
if (!string.IsNullOrWhiteSpace(follower.SharedInboxRoute))
await _activityPubService.PostNewNoteActivity(note, acct, noteId, follower.Host, follower.SharedInboxRoute);
else
await _activityPubService.PostNewNoteActivity(note, acct, noteId, follower.Host, follower.InboxRoute);
}
catch (Exception e)
{
_logger.LogError(e, e.Message);
}
}
});
}
public async Task DeleteAccountAsync(string acct)
{
// Apply deleted state
var twitterAccount = await _twitterUserDal.GetTwitterUserAsync(acct);
if (twitterAccount == null)
{
await _twitterUserDal.CreateTwitterUserAsync(acct, -1);
twitterAccount = await _twitterUserDal.GetTwitterUserAsync(acct);
}
twitterAccount.Deleted = true;
twitterAccount.LastSync = DateTime.UtcNow;
await _twitterUserDal.UpdateTwitterUserAsync(twitterAccount);
// Notify Followers
var message = $@"<p>[BSL MIRROR SERVICE NOTIFICATION]<br/>This bot has been deleted by its original owner.<br/></p>";
NotifyFollowers(acct, twitterAccount, message);
// Delete remote accounts
DeleteRemoteAccounts(acct);
}
private void DeleteRemoteAccounts(string acct)
{
var t = Task.Run(async () =>
{
var allUsers = await _followersDal.GetAllFollowersAsync();
var followersWtSharedInbox = allUsers
.Where(x => !string.IsNullOrWhiteSpace(x.SharedInboxRoute))
.GroupBy(x => x.Host)
.ToList();
foreach (var followerGroup in followersWtSharedInbox)
{
var host = followerGroup.First().Host;
var sharedInbox = followerGroup.First().SharedInboxRoute;
var t1 = Task.Run(async () =>
{
try
{
await _activityPubService.DeleteUserAsync(acct, host, sharedInbox);
}
catch (Exception e)
{
_logger.LogError(e, e.Message);
}
});
}
var followerWtInbox = allUsers
.Where(x => !string.IsNullOrWhiteSpace(x.SharedInboxRoute))
.ToList();
foreach (var followerGroup in followerWtInbox)
{
var host = followerGroup.Host;
var sharedInbox = followerGroup.InboxRoute;
var t1 = Task.Run(async () =>
{
try
{
await _activityPubService.DeleteUserAsync(acct, host, sharedInbox);
}
catch (Exception e)
{
_logger.LogError(e, e.Message);
}
});
}
});
}
public async Task TriggerRemoteMigrationAsync(string id, string tweetIdStg, string handle)
{
var url = $"https://{{0}}/migration/move/{{1}}/{{2}}/{handle}";
await ProcessRemoteMigrationAsync(id, tweetIdStg, url);
}
public async Task TriggerRemoteDeleteAsync(string id, string tweetIdStg)
{
var url = $"https://{{0}}/migration/delete/{{1}}/{{2}}";
await ProcessRemoteMigrationAsync(id, tweetIdStg, url);
}
private async Task ProcessRemoteMigrationAsync(string id, string tweetIdStg, string urlPattern)
{
try
{
var instances = await RetrieveCompatibleBslInstancesAsync();
var tweetId = ExtractedTweetId(tweetIdStg);
foreach (var instance in instances)
{
try
{
var host = instance.Host;
if(!UrlRegexes.Domain.IsMatch(host)) continue;
var url = string.Format(urlPattern, host, id, tweetId);
var client = _httpClientFactory.CreateClient();
var result = await client.PostAsync(url, null);
result.EnsureSuccessStatusCode();
}
catch (Exception e)
{
_logger.LogError(e, e.Message);
}
}
}
catch (Exception e)
{
_logger.LogError(e, e.Message);
}
}
private async Task<List<BslInstanceInfo>> RetrieveCompatibleBslInstancesAsync()
{
var instances = await _theFedInfoService.GetBslInstanceListAsync();
var filteredInstances = instances
.Where(x => x.Version >= new Version(0, 21, 0))
.Where(x => string.Compare(x.Host,
_instanceSettings.Domain,
StringComparison.InvariantCultureIgnoreCase) != 0)
.ToList();
return filteredInstances;
}
private byte[] GetHash(string inputString)
{
using (HashAlgorithm algorithm = SHA256.Create())
return algorithm.ComputeHash(Encoding.UTF8.GetBytes(inputString));
}
private string GetHashString(string inputString)
{
StringBuilder sb = new StringBuilder();
foreach (byte b in GetHash(inputString))
sb.Append(b.ToString("X2"));
return sb.ToString();
}
}
public class ValidatedFediverseUser
{
public string FediverseAcct { get; set; }
public string ObjectId { get; set; }
public Actor User { get; set; }
public bool IsValid { get; set; }
}
}

@ -0,0 +1,162 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Newtonsoft.Json;
namespace BirdsiteLive.Domain
{
public interface ITheFedInfoService
{
Task<List<BslInstanceInfo>> GetBslInstanceListAsync();
}
public class TheFedInfoService : ITheFedInfoService
{
private readonly IHttpClientFactory _httpClientFactory;
#region Ctor
public TheFedInfoService(IHttpClientFactory httpClientFactory)
{
_httpClientFactory = httpClientFactory;
}
#endregion
public async Task<List<BslInstanceInfo>> GetBslInstanceListAsync()
{
var cancellationToken = CancellationToken.None;
var result = await CallGraphQlAsync<MyResponseData>(
new Uri("https://the-federation.info/graphql"),
HttpMethod.Get,
"query ($platform: String!) { nodes(platform: $platform) { host, version } }",
new
{
platform = "birdsitelive",
},
cancellationToken);
var convertedResults = ConvertResults(result);
return convertedResults;
}
private List<BslInstanceInfo> ConvertResults(GraphQLResponse<MyResponseData> qlData)
{
var results = new List<BslInstanceInfo>();
foreach (var instanceInfo in qlData.Data.Nodes)
{
try
{
var rawVersion = instanceInfo.Version.Split('+').First();
if (string.IsNullOrWhiteSpace(rawVersion)) continue;
var version = Version.Parse(rawVersion);
if(version <= new Version(0,1,0)) continue;
var instance = new BslInstanceInfo
{
Host = instanceInfo.Host,
Version = version
};
results.Add(instance);
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
return results;
}
private async Task<GraphQLResponse<TResponse>> CallGraphQlAsync<TResponse>(Uri endpoint, HttpMethod method, string query, object variables, CancellationToken cancellationToken)
{
var content = new StringContent(SerializeGraphQlCall(query, variables), Encoding.UTF8, "application/json");
var httpRequestMessage = new HttpRequestMessage
{
Method = method,
Content = content,
RequestUri = endpoint,
};
//add authorization headers if necessary here
httpRequestMessage.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var httpClient = _httpClientFactory.CreateClient();
using (var response = await httpClient.SendAsync(httpRequestMessage, cancellationToken))
{
//if (response.IsSuccessStatusCode)
if (response?.Content.Headers.ContentType?.MediaType == "application/json")
{
var responseString = await response.Content.ReadAsStringAsync(); //cancellationToken supported for .NET 5/6
return DeserializeGraphQlCall<TResponse>(responseString);
}
else
{
throw new ApplicationException($"Unable to contact '{endpoint}': {response.StatusCode} - {response.ReasonPhrase}");
}
}
}
private string SerializeGraphQlCall(string query, object variables)
{
var sb = new StringBuilder();
var textWriter = new StringWriter(sb);
var serializer = new JsonSerializer();
serializer.Serialize(textWriter, new
{
query = query,
variables = variables,
});
return sb.ToString();
}
private GraphQLResponse<TResponse> DeserializeGraphQlCall<TResponse>(string response)
{
var serializer = new JsonSerializer();
var stringReader = new StringReader(response);
var jsonReader = new JsonTextReader(stringReader);
var result = serializer.Deserialize<GraphQLResponse<TResponse>>(jsonReader);
return result;
}
private class GraphQLResponse<TResponse>
{
public List<GraphQLError> Errors { get; set; }
public TResponse Data { get; set; }
}
private class GraphQLError
{
public string Message { get; set; }
public List<GraphQLErrorLocation> Locations { get; set; }
public List<object> Path { get; set; } //either int or string
}
private class GraphQLErrorLocation
{
public int Line { get; set; }
public int Column { get; set; }
}
private class MyResponseData
{
public Node[] Nodes { get; set; }
}
private class Node
{
public string Host { get; set; }
public string Version { get; set; }
}
}
public class BslInstanceInfo
{
public string Host { get; set; }
public Version Version { get; set; }
}
}

@ -0,0 +1,22 @@
using System.Linq;
namespace BirdsiteLive.Domain.Tools
{
public class SigValidationResultExtractor
{
public static string GetUserName(SignatureValidationResult result)
{
return result.User.preferredUsername.ToLowerInvariant().Trim();
}
public static string GetHost(SignatureValidationResult result)
{
return result.User.url.Replace("https://", string.Empty).Split('/').First();
}
public static string GetSharedInbox(SignatureValidationResult result)
{
return result.User?.endpoints?.sharedInbox;
}
}
}

@ -12,6 +12,7 @@ using BirdsiteLive.Common.Regexes;
using BirdsiteLive.Common.Settings;
using BirdsiteLive.Cryptography;
using BirdsiteLive.DAL.Contracts;
using BirdsiteLive.DAL.Models;
using BirdsiteLive.Domain.BusinessUseCases;
using BirdsiteLive.Domain.Repository;
using BirdsiteLive.Domain.Statistics;
@ -25,15 +26,17 @@ namespace BirdsiteLive.Domain
{
public interface IUserService
{
Actor GetUser(TwitterUser twitterUser);
Actor GetUser(TwitterUser twitterUser, SyncTwitterUser dbTwitterUser);
Task<bool> FollowRequestedAsync(string signature, string method, string path, string queryString, Dictionary<string, string> requestHeaders, ActivityFollow activity, string body);
Task<bool> UndoFollowRequestedAsync(string signature, string method, string path, string queryString, Dictionary<string, string> requestHeaders, ActivityUndoFollow activity, string body);
Task<bool> SendRejectFollowAsync(ActivityFollow activity, string followerHost);
Task<bool> DeleteRequestedAsync(string signature, string method, string path, string queryString, Dictionary<string, string> requestHeaders, ActivityDelete activity, string body);
}
public class UserService : IUserService
{
private readonly IProcessDeleteUser _processDeleteUser;
private readonly IProcessFollowUser _processFollowUser;
private readonly IProcessUndoFollowUser _processUndoFollowUser;
@ -50,7 +53,7 @@ namespace BirdsiteLive.Domain
private readonly IFollowersDal _followerDal;
#region Ctor
public UserService(InstanceSettings instanceSettings, ICryptoService cryptoService, IActivityPubService activityPubService, IProcessFollowUser processFollowUser, IProcessUndoFollowUser processUndoFollowUser, IStatusExtractor statusExtractor, IExtractionStatisticsHandler statisticsHandler, ITwitterUserService twitterUserService, IModerationRepository moderationRepository, IFollowersDal followerDal)
public UserService(InstanceSettings instanceSettings, ICryptoService cryptoService, IActivityPubService activityPubService, IProcessFollowUser processFollowUser, IProcessUndoFollowUser processUndoFollowUser, IStatusExtractor statusExtractor, IExtractionStatisticsHandler statisticsHandler, ITwitterUserService twitterUserService, IModerationRepository moderationRepository, IFollowersDal followerDal, IProcessDeleteUser processDeleteUser)
{
_instanceSettings = instanceSettings;
_cryptoService = cryptoService;
@ -62,10 +65,11 @@ namespace BirdsiteLive.Domain
_twitterUserService = twitterUserService;
_moderationRepository = moderationRepository;
_followerDal = followerDal;
_processDeleteUser = processDeleteUser;
}
#endregion
public Actor GetUser(TwitterUser twitterUser)
public Actor GetUser(TwitterUser twitterUser, SyncTwitterUser dbTwitterUser)
{
var actorUrl = UrlFactory.GetActorUrl(_instanceSettings.Domain, twitterUser.Acct);
var acct = twitterUser.Acct.ToLowerInvariant();
@ -116,9 +120,10 @@ namespace BirdsiteLive.Domain
preferredUsername = acct,
name = twitterUser.Name,
inbox = $"{actorUrl}/inbox",
summary = description,
summary = "[UNOFFICIAL MIRROR: This is a view of Twitter using ActivityPub]<br/><br/>" + description,
url = actorUrl,
manuallyApprovesFollowers = twitterUser.Protected,
discoverable = false,
publicKey = new PublicKey()
{
id = $"{actorUrl}#main-key",
@ -135,11 +140,32 @@ namespace BirdsiteLive.Domain
mediaType = "image/jpeg",
url = twitterUser.ProfileBannerURL
},
attachment = attachments.ToArray(),
attachment = new []
{
new UserAttachment
{
type = "PropertyValue",
name = "Official Account",
value = $"<a href=\"https://twitter.com/{acct}\" rel=\"me nofollow noopener noreferrer\" target=\"_blank\"><span class=\"invisible\">https://</span><span class=\"ellipsis\">twitter.com/{acct}</span></a>"
},
new UserAttachment
{
type = "PropertyValue",
name = "Disclaimer",
value = "This is an automatically created and managed mirror profile from Twitter. While it reflects exactly the content of the original account, it doesn't provide support for interactions and replies. It is an equivalent view from other 3rd party Twitter client apps and uses the same technical means to provide it."
},
new UserAttachment
{
type = "PropertyValue",
name = "Take control of this account",
value = $"<a href=\"https://{_instanceSettings.Domain}/migration/move/{acct}\" rel=\"me nofollow noopener noreferrer\" target=\"_blank\">MANAGE</a>"
}
},
endpoints = new EndPoints
{
sharedInbox = $"https://{_instanceSettings.Domain}/inbox"
}
},
movedTo = dbTwitterUser?.MovedTo
};
if (twitterUser.Verified)
@ -172,10 +198,10 @@ namespace BirdsiteLive.Domain
if (!sigValidation.SignatureIsValidated) return false;
// Prepare data
var followerUserName = sigValidation.User.preferredUsername.ToLowerInvariant().Trim();
var followerHost = sigValidation.User.url.Replace("https://", string.Empty).Split('/').First();
var followerUserName = SigValidationResultExtractor.GetUserName(sigValidation);
var followerHost = SigValidationResultExtractor.GetHost(sigValidation);
var followerInbox = sigValidation.User.inbox;
var followerSharedInbox = sigValidation.User?.endpoints?.sharedInbox;
var followerSharedInbox = SigValidationResultExtractor.GetSharedInbox(sigValidation);
var twitterUser = activity.apObject.Split('/').Last().Replace("@", string.Empty).ToLowerInvariant().Trim();
// Make sure to only keep routes
@ -269,7 +295,7 @@ namespace BirdsiteLive.Domain
return result == HttpStatusCode.Accepted ||
result == HttpStatusCode.OK; //TODO: revamp this for better error handling
}
private string OnlyKeepRoute(string inbox, string host)
{
if (string.IsNullOrWhiteSpace(inbox))
@ -314,6 +340,22 @@ namespace BirdsiteLive.Domain
return result == HttpStatusCode.Accepted || result == HttpStatusCode.OK; //TODO: revamp this for better error handling
}
public async Task<bool> DeleteRequestedAsync(string signature, string method, string path, string queryString, Dictionary<string, string> requestHeaders,
ActivityDelete activity, string body)
{
// Validate
var sigValidation = await ValidateSignature(activity.actor, signature, method, path, queryString, requestHeaders, body);
if (!sigValidation.SignatureIsValidated) return false;
// Remove user and followings
var followerUserName = SigValidationResultExtractor.GetUserName(sigValidation);
var followerHost = SigValidationResultExtractor.GetHost(sigValidation);
await _processDeleteUser.ExecuteAsync(followerUserName, followerHost);
return true;
}
private async Task<SignatureValidationResult> ValidateSignature(string actor, string rawSig, string method, string path, string queryString, Dictionary<string, string> requestHeaders, string body)
{
//Check Date Validity

@ -1,12 +1,6 @@
using System;
using System.Linq;
using System.Threading.Tasks;
using BirdsiteLive.ActivityPub;
using BirdsiteLive.ActivityPub.Converters;
using BirdsiteLive.Common.Settings;
using BirdsiteLive.DAL.Contracts;
using System.Threading.Tasks;
using BirdsiteLive.DAL.Models;
using BirdsiteLive.Domain;
using BirdsiteLive.Domain.BusinessUseCases;
namespace BirdsiteLive.Moderation.Actions
{
@ -17,16 +11,14 @@ namespace BirdsiteLive.Moderation.Actions
public class RemoveFollowerAction : IRemoveFollowerAction
{
private readonly IFollowersDal _followersDal;
private readonly ITwitterUserDal _twitterUserDal;
private readonly IRejectAllFollowingsAction _rejectAllFollowingsAction;
private readonly IProcessDeleteUser _processDeleteUser;
#region Ctor
public RemoveFollowerAction(IFollowersDal followersDal, ITwitterUserDal twitterUserDal, IRejectAllFollowingsAction rejectAllFollowingsAction)
public RemoveFollowerAction(IRejectAllFollowingsAction rejectAllFollowingsAction, IProcessDeleteUser processDeleteUser)
{
_followersDal = followersDal;
_twitterUserDal = twitterUserDal;
_rejectAllFollowingsAction = rejectAllFollowingsAction;
_processDeleteUser = processDeleteUser;
}
#endregion
@ -36,16 +28,7 @@ namespace BirdsiteLive.Moderation.Actions
await _rejectAllFollowingsAction.ProcessAsync(follower);
// Remove twitter users if no more followers
var followings = follower.Followings;
foreach (var following in followings)
{
var followers = await _followersDal.GetFollowersAsync(following);
if (followers.Length == 1 && followers.First().Id == follower.Id)
await _twitterUserDal.DeleteTwitterUserAsync(following);
}
// Remove follower from DB
await _followersDal.DeleteFollowerAsync(follower.Id);
await _processDeleteUser.ExecuteAsync(follower);
}
}
}

@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
<ItemGroup>

@ -29,7 +29,7 @@ namespace BirdsiteLive.Moderation.Processors
{
if (type == ModerationTypeEnum.None) return;
var twitterUsers = await _twitterUserDal.GetAllTwitterUsersAsync();
var twitterUsers = await _twitterUserDal.GetAllTwitterUsersAsync(false);
foreach (var user in twitterUsers)
{

@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard2.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
<LangVersion>latest</LangVersion>
</PropertyGroup>

@ -9,6 +9,7 @@ using BirdsiteLive.Moderation.Actions;
using BirdsiteLive.Pipeline.Contracts;
using BirdsiteLive.Pipeline.Models;
using BirdsiteLive.Twitter;
using BirdsiteLive.Twitter.Models;
namespace BirdsiteLive.Pipeline.Processors
{
@ -35,26 +36,61 @@ namespace BirdsiteLive.Pipeline.Processors
foreach (var user in syncTwitterUsers)
{
var userView = _twitterUserService.GetUser(user.Acct);
if (userView == null)
TwitterUser userView = null;
try
{
await AnalyseFailingUserAsync(user);
userView = _twitterUserService.GetUser(user.Acct);
}
else if (!userView.Protected)
catch (UserNotFoundException)
{
user.FetchingErrorCount = 0;
var userWtData = new UserWithDataToSync
{
User = user
};
usersWtData.Add(userWtData);
await ProcessNotFoundUserAsync(user);
continue;
}
catch (UserHasBeenSuspendedException)
{
await ProcessNotFoundUserAsync(user);
continue;
}
catch (RateLimitExceededException)
{
await ProcessRateLimitExceededAsync(user);
continue;
}
catch (Exception)
{
// ignored
}
if (userView == null || userView.Protected)
{
await ProcessFailingUserAsync(user);
continue;
}
}
user.FetchingErrorCount = 0;
var userWtData = new UserWithDataToSync
{
User = user
};
usersWtData.Add(userWtData);
}
return usersWtData.ToArray();
}
private async Task AnalyseFailingUserAsync(SyncTwitterUser user)
private async Task ProcessRateLimitExceededAsync(SyncTwitterUser user)
{
var dbUser = await _twitterUserDal.GetTwitterUserAsync(user.Acct);
dbUser.LastSync = DateTime.UtcNow;
await _twitterUserDal.UpdateTwitterUserAsync(dbUser);
}
private async Task ProcessNotFoundUserAsync(SyncTwitterUser user)
{
await _removeTwitterAccountAction.ProcessAsync(user);
}
private async Task ProcessFailingUserAsync(SyncTwitterUser user)
{
var dbUser = await _twitterUserDal.GetTwitterUserAsync(user.Acct);
dbUser.FetchingErrorCount++;
@ -68,9 +104,6 @@ namespace BirdsiteLive.Pipeline.Processors
{
await _twitterUserDal.UpdateTwitterUserAsync(dbUser);
}
// Purge
_twitterUserService.PurgeUser(user.Acct);
}
}
}

@ -49,12 +49,12 @@ namespace BirdsiteLive.Pipeline.Processors
{
var tweetId = tweets.Last().Id;
var now = DateTime.UtcNow;
await _twitterUserDal.UpdateTwitterUserAsync(user.Id, tweetId, tweetId, user.FetchingErrorCount, now);
await _twitterUserDal.UpdateTwitterUserAsync(user.Id, tweetId, tweetId, user.FetchingErrorCount, now, user.MovedTo, user.MovedToAcct, user.Deleted);
}
else
{
var now = DateTime.UtcNow;
await _twitterUserDal.UpdateTwitterUserAsync(user.Id, user.LastTweetPostedId, user.LastTweetSynchronizedForAllFollowersId, user.FetchingErrorCount, now);
await _twitterUserDal.UpdateTwitterUserAsync(user.Id, user.LastTweetPostedId, user.LastTweetSynchronizedForAllFollowersId, user.FetchingErrorCount, now, user.MovedTo, user.MovedToAcct, user.Deleted);
}
}

@ -39,7 +39,7 @@ namespace BirdsiteLive.Pipeline.Processors
try
{
var maxUsersNumber = await _maxUsersNumberProvider.GetMaxUsersNumberAsync();
var users = await _twitterUserDal.GetAllTwitterUsersAsync(maxUsersNumber);
var users = await _twitterUserDal.GetAllTwitterUsersAsync(maxUsersNumber, false);
var userCount = users.Any() ? users.Length : 1;
var splitNumber = (int) Math.Ceiling(userCount / 15d);