Commit graph

110 commits

Author SHA1 Message Date
Vincent Cloutier
000214043c RetrieveTwitterUsersProcessor tweaks 2 2023-04-02 11:10:14 -04:00
Vincent Cloutier
46f7594e43 RetrieveTwitterUsersProcessor tweaks 2023-04-02 10:23:45 -04:00
Vincent Cloutier
3346b7b5e8 sharding support 2023-04-01 19:55:20 -04:00
Vincent Cloutier
2a15a3cae6 twitter cache and auth tweaks 2023-03-29 19:03:22 -04:00
Vincent Cloutier
f631e922bc tweak logging 2023-03-25 15:44:42 -04:00
Vincent Cloutier
46be9552e9 pipeline refactoring 2023-03-25 15:26:11 -04:00
Vincent Cloutier
9551c735ea moved followers retrieval 2 2023-03-25 13:53:07 -04:00
Vincent Cloutier
8d6851c639 moved followers retrieval 2023-03-25 13:24:11 -04:00
Vincent Cloutier
62caf7e956 don't send full backlog of tweets on first sync 2023-03-19 11:35:30 -04:00
Vincent Cloutier
1daec5577d tweak logging 2023-03-19 11:04:03 -04:00
Vincent Cloutier
0bc8b96ea5 made retwrieve followers more parallel 2023-03-18 20:48:23 -04:00
Vincent Cloutier
71dfe4b019 tweak checkpointing of twitter fetches 2 2023-03-17 22:02:34 -04:00
Vincent Cloutier
66e2ba9b06 tweak checkpointing of twitter fetches 2023-03-17 19:45:51 -04:00
Vincent Cloutier
5dcb1199c7 limit parallel postgres requests 2023-03-17 16:31:52 -04:00
Vincent Cloutier
240dfd1902 pipeline tweaks 2023-03-17 16:14:30 -04:00
Vincent Cloutier
160ef97626 pipeline simplifications 2023-03-17 15:51:11 -04:00
Vincent Cloutier
4dd071abe2 cache tweaks 2023-03-17 15:10:53 -04:00
Vincent Cloutier
12273abdd1 magic numbers update & some cleanups 2023-03-03 10:37:42 -05:00
Vincent Cloutier
2674041a22 made posting to fediverse servers parallel 2023-02-22 11:54:03 -05:00
Vincent Cloutier
3a47655671 made SaveProgression a SubTask 2023-02-22 11:30:02 -05:00
Vincent Cloutier
da8092cfd5 activity generation refactoring 2023-02-10 11:54:33 -05:00
Vincent Cloutier
e9f3631985 various simplifications 2023-02-03 10:24:50 -05:00
Vincent Cloutier
edec988e05 update some magic numbers 2023-01-27 13:55:19 -05:00
Vincent Cloutier
bd46afa350 fix crash 2023-01-20 14:17:32 -05:00
Vincent Cloutier
c702357cc1 magic tweaks for perf 2023-01-20 13:25:52 -05:00
Vincent Cloutier
714e66e284 added parameter for twitter request parallelism 2023-01-20 12:53:30 -05:00
Vincent Cloutier
4a7373ec07 removed max users 2023-01-20 11:13:00 -05:00
Vincent Cloutier
48d521b757 removed unused nbrTweets 2023-01-20 09:28:32 -05:00
Vincent Cloutier
08f5aef7fc tweak twitter queries parallelism 2 2023-01-15 12:53:04 -05:00
Vincent Cloutier
014dee23a3 tweak twitter queries parallelism 2023-01-15 09:24:30 -05:00
Vincent Cloutier
9026273f45 made tweet fetching concurrent instead 2023-01-10 21:00:21 -05:00
Vincent Cloutier
d1018881ec now fetching twitter feed in parallel 2023-01-10 20:30:07 -05:00
Vincent Cloutier
55b8244d13 magic numbers update 2023-01-05 15:02:24 -05:00
Vincent Cloutier
5c75e79abc boundedcapacity change 2023-01-03 14:50:05 -05:00
Vincent Cloutier
43fb42727e bigger user batch 2023-01-01 15:55:05 -05:00
Vincent Cloutier
3e5b01a923 remove dead code 2023-01-01 15:18:54 -05:00
Vincent Cloutier
97d40b21fb switched to vanilla npgsql for more queries 4 2023-01-01 11:58:36 -05:00
Vincent Cloutier
8551763f77 small changes all over the place 2023-01-01 11:27:00 -05:00
Vincent Cloutier
f2c0d55916 version updates 2023-01-01 11:09:40 -05:00
Vincent Cloutier
b1b8b676b9 changed some magic 2022-12-29 13:02:38 -05:00
Vincent Cloutier
f8c3b5cac7 added twitter user tests 2022-12-28 14:49:49 -05:00
Vincent Cloutier
5fafb1f568 delay tweaking 2022-12-28 13:32:08 -05:00
Vincent Cloutier
ea47f2c058 removed pipeline stage 2022-12-28 11:35:06 -05:00
Vincent Cloutier
d46efe8812 fix 2022-12-28 11:17:54 -05:00
Vincent Cloutier
d6cf46f0c6 removed unneccesary delay 2 2022-12-28 11:05:03 -05:00
Vincent Cloutier
8bc044eeba removed unneccesary delay 2022-12-28 11:00:24 -05:00
Vincent Cloutier
6b6a943294 made RetrieveFollowersProcessor more parallel 2022-12-28 10:59:15 -05:00
Vincent Cloutier
a9b3bc8da9 made inbox processing more async 2022-12-28 10:30:58 -05:00
Vincent Cloutier
e7197f3054 made twitteruser more async 2022-12-28 10:23:46 -05:00
Vincent Cloutier
dbabd61418 increase pipeline depth 2022-12-27 12:37:06 -05:00