Votes
0
Replies
11
Participants
4
Messages
12
Rank 3: Container
When building a python function on self hosted 1.9.6 version, CPU is +200% and build never complete. Server is KO. Need to stop it...
Hosted on CPX21 Hetzner: 3 VCPUs 4Gb Ram
Rank 3: Container
Seems an issue with "appwrite-embedding" which consume a lot of RAM
Rank 3: Container
Workaround:
docker-compose.override.yml file with this:services:
appwrite-embedding:
image: alpine:3.22
entrypoint: ["sleep", "infinity"]
docker compose up -dAppwrite embedding is now swapped with a sleepy dummy container 🙂
To restore it, just:
docker-compose.override.ymldocker compose up -dRank 3: Container
Thanks @Chirag Aggarwal @Arnab Chatterjee for taking the issue 🙂
Rank 3: Container
I think this can explain all user complying about 1.9.5+ upgrade functions timeout, building issues etc.
@iStornZ hey few questions, did you notice the cpu spike when the embedding container is idle? or was it doing it something?
Rank 3: Container
@Arnab Chatterjee For my understanding, all was fine before upgrading to 1.9.5-1.9.6 (was coming from 1.8). Since theses versions the server was going out of ram frequently, making it unusable at all, needed to log in with ssh and do a docker compose stop and start again.
The fix I deployed (dummy embedding container with a sleepy alpine) seems perfect, CPU never goes + 100% and ram usage is perfect.
Can’t tell you why the embedding was buggy… seems that container was too hungry with RAM :/
heyy, so far with my conversation on this with arnab we were not able to reproduce the specific scenario, else we would have caught it in our own production deployment
maybe its a specific bug we fixed since the 1.9.6 release
anyway @Arnab Chatterjee has worked on adding option for disabling the embeddings service entirely for users who don't really want it 💪
Rank 3: Container
Thanks a lot guys!
Rank 3: Container
Hello @Chirag Aggarwal and @Arnab Chatterjee , I investigate more and I think I found something, if you can give me some advice it will be great!
Another time the server was running out of memory and give me some 500 internal error. It seems my server do not have any swap:
root@server-prod:~/appwrite# free -h
total used free shared buff/cache available
Mem: 3.7Gi 3.0Gi 120Mi 61Mi 635Mi 427Mi
Swap: 0B 0B 0B
And seems when running out of memory, the OEM killer killed mysqld...
root@server-prod:~/appwrite# journalctl -k --since "2026-09-01" | grep -iE "oom|killed process"
Sep 01 20:01:42 server-prod kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-d6b3ea27f8876c50cdf9e5a9865a575a09a51b93e113f27f284b8573257a57dc.scope,task=mysqld,pid=816289,uid=999
Sep 01 20:01:42 radion-prod kernel: Out of memory: Killed process 816289 (mysqld) total-vm:2104424kB, anon-rss:245648kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:772kB oom_score_adj:0
He also killed the process headless-shell (do not have any sites with Appwrite, pretty strange...). So my guess is I need to create a 2Gb Swap and it will improve my stack. Evolving from 4Gb to 6Gb or more can also be a solution but I see Appwrite recommendation is 4Gb + 2 Gb Swap.
Can you confirm my investigation?
Thanks a lot
@iStornZ thanks for the detailed breakdown, i think i found one potential fix
currently the embeddings service seems to be using the host memory to calculated how many pools to allocate. it should be using the cgroup(docker) memory limit instead. the fix here should fix the OOM issue