Compare commits
5 Commits
merge/miss
...
cursor/sta
| Author | SHA1 | Date | |
|---|---|---|---|
|
77d10a2b21 | ||
|
|
f77de3e6d8 | ||
|
|
163df59cb8 | ||
|
|
d92d5f4dca | ||
|
|
658c63712d |
121
RAPPORT_FINAL_DP_10MARS2026.md
Normal file
121
RAPPORT_FINAL_DP_10MARS2026.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# RAPPORT FINAL DP — 10 mars 2026
|
||||
|
||||
Branche: cursor/stabilit-serveur-bleu-multi-instance-cb93
|
||||
Repo: Yacineutt/wevads-gpu
|
||||
|
||||
## 1. Probleme initial
|
||||
|
||||
Multi-install bloque sur "Installing In Progress" pour 5 serveurs (190-194).
|
||||
Le spinner tourne indefiniment, l'interface ne se debloque jamais.
|
||||
|
||||
## 2. Cause racine identifiee
|
||||
|
||||
Le processus Java (`nohup java -jar iresponse_services.jar ... &`) lance par
|
||||
`beginInstallation()` dans `/opt/fmgapp/app/webservices/Servers.php` crashe ou
|
||||
timeout sans jamais ecrire "Installation completed !" ou "Installation interrupted !"
|
||||
dans le fichier `inst_{id}_proc.log`.
|
||||
|
||||
Le polling JavaScript (`getInstallationLogs`, toutes les 2 secondes) lit ce fichier
|
||||
et ne voit jamais de statut final => boucle infinie "Installing In Progress".
|
||||
|
||||
## 3. Correctif applique EN PRODUCTION
|
||||
|
||||
### Patch getInstallationLogs (applique sur S89 via Sentinel)
|
||||
|
||||
DEUX apps patchees (meme serveur, apps distinctes) :
|
||||
- `/opt/wevads/app/webservices/Servers.php` (port 5821 = ADX multi-install)
|
||||
- `/opt/fmgapp/app/webservices/Servers.php` (port 5822 = FMG)
|
||||
Backups crees avant chaque modif. PHP syntax OK sur les deux.
|
||||
|
||||
Le patch ajoute une detection automatique :
|
||||
1. Verifie si le processus Java est toujours vivant (`ps aux`)
|
||||
2. Verifie si le log n'a pas bouge depuis 15 minutes
|
||||
3. Si le process est mort OU timeout => ecrit "Installation interrupted !" dans le proc file
|
||||
4. Le polling JS detecte le changement et debloque l'interface
|
||||
|
||||
Zero impact sur PMTA, SSH, JAR, multiInstall.js (lecture seule + ecriture proc file).
|
||||
|
||||
Fix additionnel: `str_replace()` null error (PHP 8.x) -- quand le fichier log
|
||||
n'existe pas encore, `shell_exec("cat ...")` retourne `null` au lieu de `string`.
|
||||
Le patch utilise `(string)($logs ?? '')` et `(string)($procc ?? '')` pour eviter ce crash.
|
||||
Mise a jour 10 mars: le patch a ete renforce avec cast explicite sur toutes les variables
|
||||
pouvant etre null (logs, procc) pour eliminer "Internal server installation error !" au clic bulk install.
|
||||
|
||||
Nettoyage: 154 fichiers proc stale d'anciennes installations orphelines nettoyes.
|
||||
|
||||
### Scripts dans le repo (pour deblocage manuel si necessaire)
|
||||
|
||||
- `debloquer-multiinstall.sh` : deblocage rapide en 5 etapes
|
||||
- `fix-stuck-multiinstall.sh` : diagnostic (`--diagnose`), reset (`--reset`), retry (`--retry`)
|
||||
- `multiinstall-safe-preflight.sh` : verification pre-install
|
||||
- `patches/` : correctifs PHP/JS avec script d'application automatique
|
||||
|
||||
## 4. Etat systeme verifie
|
||||
|
||||
| Composant | Status | Verification |
|
||||
|-----------|--------|-------------|
|
||||
| PMTA S89 | v5.0r3 actif | `pmta --version` + `systemctl is-active pmta` |
|
||||
| Apache S89 | actif | 12 workers |
|
||||
| PostgreSQL S89 | actif | 5 serveurs MTA actives en DB |
|
||||
| Tracking S151 | HTTP 200 | `curl http://151.80.235.110/` |
|
||||
| culturellemejean.charity | HTTP 200 | `curl https://culturellemejean.charity/` |
|
||||
| Ethica DB | 18,596 HCPs | `SELECT count(*) FROM ethica.medecins_real` |
|
||||
| Ethica crons | 3 actifs | `crontab -l | grep ethica` |
|
||||
| Serveurs 190-194 | TCP/22 OK (5/5) | Verifie depuis S89 |
|
||||
| Serveurs 190-194 DB | NOT_INSTALLED | Prets pour multi-install |
|
||||
| Patch Servers.php | APPLIQUE | `grep -c 'AUTO-DETECT'` = 1 |
|
||||
| weval-consulting.com | 13/13 pages 200 | Toutes les pages produits |
|
||||
| WEVIA greeting | < 3s | Teste live |
|
||||
| DeliverScore API | OK | Teste live |
|
||||
| MedReach API | OK (MA/DZ/TN) | Teste live |
|
||||
|
||||
## 5. Front-end — issues restantes (pages servies par S89)
|
||||
|
||||
| Page | Issue | Severite |
|
||||
|------|-------|----------|
|
||||
| workspace.html | 183 emojis HTML entities (icons sections) | P2 (cosmetique) |
|
||||
| deliverscore.html | 11x checkmark Unicode (U+2713) | P3 (acceptable) |
|
||||
| products-index.html | 6 symboles Unicode | P3 (acceptable) |
|
||||
| workspace.html | 2x "Confidentialite" sans accent | P3 (mineur) |
|
||||
|
||||
Ces issues sont sur les fichiers servis par S89 (pas dans ce repo).
|
||||
Les corrections precedentes (600+) par les agents sur cursor/ethica-saas-chantiers-a789
|
||||
couvrent la majorite des problemes d'accents et confidentialite.
|
||||
|
||||
## 6. Regles historiques (NE PAS TOUCHER)
|
||||
|
||||
Depuis les incidents de janvier 2026 :
|
||||
- PMTA config/routes : installation cassee
|
||||
- SSH config : connection reset, acces perdu
|
||||
- Java/JAR : processus bloques indefiniment
|
||||
- multiInstall.js : race conditions, loading infini
|
||||
- Timeouts globaux : installations sans fin
|
||||
|
||||
Batch max recommande : 3-5 serveurs (5 min/serveur).
|
||||
|
||||
## 7. Prochaines etapes
|
||||
|
||||
| # | Action | Priorite |
|
||||
|---|--------|----------|
|
||||
| 1 | Relancer multi-install 190-194 depuis l'interface (le patch va auto-detecter les crashes) | P0 |
|
||||
| 2 | Configurer SSH auth pour les 5 serveurs (credentials DB a verifier) | P0 |
|
||||
| 3 | Remplacer emojis HTML entities par SVG sur workspace.html | P2 |
|
||||
| 4 | PMTA NAT Huawei (serveurs 180-189) | P1 (session dediee) |
|
||||
| 5 | Deploiement WEVADS v2 (/opt/wevads-v2/ sur S88) | P1 (session dediee) |
|
||||
|
||||
## 8. Verdict
|
||||
|
||||
**GO CONDITIONNEL** pour multi-install.
|
||||
|
||||
Le patch est en production. Les serveurs 190-194 sont prets (TCP/22 OK, DB = NOT_INSTALLED).
|
||||
Si le process Java crashe pendant l'installation, le polling detectera automatiquement
|
||||
le crash et debloquera l'interface en 15 minutes max (au lieu de jamais).
|
||||
|
||||
Pour installer les 5 serveurs :
|
||||
1. Verifier les credentials SSH en DB (match avec serveurs reels)
|
||||
2. Aller sur `http://89.167.40.150:5821/mta-servers/multi-install/194-193-192-191-190`
|
||||
3. Cocher Install Services + Install PowerMTA 4.5r8 + IPv4
|
||||
4. Lancer par batch de 3 max
|
||||
5. Si blocage => le patch auto-detecte et debloque
|
||||
|
||||
Git: 0 dirty, tout pousse.
|
||||
98
README.md
98
README.md
@@ -5,3 +5,101 @@
|
||||
- **Disk**: 1.7TB NVMe
|
||||
- **Ollama**: localhost:11434
|
||||
- **Models**: deepseek-r1:8b, deepseek-r1:32b, llama3.1:8b
|
||||
|
||||
## Multi-Install Serveur Bleu — Deblocage
|
||||
|
||||
Scripts pour diagnostiquer et debloquer le multi-install MTA quand les serveurs restent bloques a "Installing In Progress".
|
||||
|
||||
### Serveurs concernes
|
||||
|
||||
| ID | Nom | IP | Domaine |
|
||||
|-----|----------|-----------------|-----------------|
|
||||
| 194 | SERVER_5 | 176.52.138.42 | mailpipe.net |
|
||||
| 192 | SERVER_3 | 110.238.65.222 | mailforge.io |
|
||||
| 191 | SERVER_2 | 176.52.129.86 | postengine.net |
|
||||
| 193 | SERVER_4 | 110.238.69.46 | relaycore.com |
|
||||
| 190 | SERVER_1 | 176.52.132.94 | mailforge.io |
|
||||
|
||||
### Scripts
|
||||
|
||||
| Script | Description |
|
||||
|--------|-------------|
|
||||
| `debloquer-multiinstall.sh` | Deblocage rapide en 5 etapes (tuer processus, reset DB, nettoyer locks, relancer workers, verifier) |
|
||||
| `fix-stuck-multiinstall.sh` | Diagnostic detaille (`--diagnose`), reset (`--reset`), ou retry complet (`--retry`) |
|
||||
| `multiinstall-safe-preflight.sh` | Verification pre-install (SSH, disk, RAM, dpkg, apt) avant de lancer un batch |
|
||||
|
||||
### Utilisation rapide (sur le serveur WEVADS 89.167.40.150)
|
||||
|
||||
```bash
|
||||
# 1. Debloquer les serveurs coinces
|
||||
./debloquer-multiinstall.sh
|
||||
|
||||
# 2. Verifier que les serveurs sont prets
|
||||
SSH_PASS="xxx" ./multiinstall-safe-preflight.sh servers-bleu-multiinstall.csv
|
||||
|
||||
# 3. Relancer le multi-install depuis l'interface
|
||||
# http://89.167.40.150:5821/mta-servers/multi-install/194-193-192-191-190
|
||||
```
|
||||
|
||||
### Correctif code (patches/)
|
||||
|
||||
Le bug racine : quand `beginInstallation` lance `nohup java -jar iresponse_services.jar ... &`,
|
||||
le processus Java peut crasher/timeout sans jamais ecrire "Installation completed !" ou
|
||||
"Installation interrupted !" dans le fichier `inst_{id}_proc.log`. Le polling JS reste bloque
|
||||
sur "Installing In Progress" indefiniment.
|
||||
|
||||
**Patches fournis :**
|
||||
|
||||
| Fichier | Cible | Correctif |
|
||||
|---------|-------|-----------|
|
||||
| `getInstallationLogs_fix.php` | `app/webservices/Servers.php` | Detection auto du process mort + timeout 15min |
|
||||
| `beginInstallation_fix.php` | `app/webservices/Servers.php` | PID tracking + blocage doublons |
|
||||
| `installation_js_fix.js` | `public/scripts/pages/servers/installation.js` | Poll async + detection stale cote client |
|
||||
| `multiInstall_controller.php` | `app/controllers/MtaServers.php` | Methode multiInstall pour route multi-serveurs |
|
||||
| `apply-patches.sh` | Script d'application | Applique tous les patches + nettoie les proc bloques |
|
||||
|
||||
```bash
|
||||
# Appliquer les patches sur le serveur
|
||||
scp -r patches/ root@89.167.40.150:/tmp/patches/
|
||||
ssh root@89.167.40.150 'bash /tmp/patches/apply-patches.sh'
|
||||
```
|
||||
|
||||
### Patch applique en production (10 mars 2026)
|
||||
|
||||
Le patch `getInstallationLogs` a ete applique directement sur S89 via Sentinel :
|
||||
- Fichier : `/opt/fmgapp/app/webservices/Servers.php`
|
||||
- Backup : `/opt/fmgapp/app/webservices/Servers.php.bak-20260310_0113`
|
||||
- PHP syntax : OK (`php -l` = no errors)
|
||||
- Verification : `grep -c 'AUTO-DETECT' Servers.php` = 1
|
||||
|
||||
Le patch detecte automatiquement les installations bloquees :
|
||||
- Si le process Java est mort (pas dans `ps aux`) => auto-interrupt
|
||||
- Si le log n'a pas bouge depuis 15 minutes => auto-interrupt
|
||||
- Le statut passe de "Installing In Progress" a "Installation interrupted !"
|
||||
- L'utilisateur peut relancer proprement depuis l'interface
|
||||
|
||||
## Etat systeme verifie (10 mars 2026)
|
||||
|
||||
| Composant | Status |
|
||||
|-----------|--------|
|
||||
| PMTA S89 | v5.0r3 actif |
|
||||
| Apache S89 | actif, 12 workers |
|
||||
| PostgreSQL S89 | actif |
|
||||
| Tracking S151 | HTTP 200 |
|
||||
| culturellemejean.charity | HTTP 200 |
|
||||
| Ethica DB | 18,596 HCPs |
|
||||
| Ethica crons | 3 actifs |
|
||||
| Serveurs 190-194 TCP/22 | 5/5 OK depuis S89 |
|
||||
| Serveurs 190-194 DB | NOT_INSTALLED (prets) |
|
||||
|
||||
## Regles NE PAS TOUCHER (historique incidents jan 2026)
|
||||
|
||||
| Composant | Risque | Incident |
|
||||
|-----------|--------|----------|
|
||||
| PMTA config/routes | Installation cassee | 21 jan |
|
||||
| SSH config serveurs | Connection reset | 20-21 jan |
|
||||
| Java/JAR (iresponse_services.jar) | Process bloques | 20 jan |
|
||||
| multiInstall.js | Race conditions | 20 jan |
|
||||
| Timeouts globaux | Installations sans fin | 20 jan |
|
||||
|
||||
Batch max recommande : 3-5 serveurs (5 min/serveur = 15-25 min/batch).
|
||||
|
||||
210
debloquer-multiinstall.sh
Executable file
210
debloquer-multiinstall.sh
Executable file
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# debloquer-multiinstall.sh
|
||||
#
|
||||
# Script de deblocage rapide du multi-install serveur bleu.
|
||||
# Executer sur le serveur applicatif WEVADS (89.167.40.150).
|
||||
#
|
||||
# Actions:
|
||||
# 1. Tue les processus d'installation bloques
|
||||
# 2. Remet le statut DB a "Not Installed"
|
||||
# 3. Libere les locks dpkg sur les serveurs cibles (via SSH)
|
||||
# 4. Relance les workers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STUCK_IDS="${STUCK_IDS:-194,193,192,191,190}"
|
||||
DB_NAME="${DB_NAME:-wevads}"
|
||||
DB_USER="${DB_USER:-root}"
|
||||
DB_PASS="${DB_PASS:-}"
|
||||
SSH_USER="${SSH_USER:-root}"
|
||||
SSH_PASS="${SSH_PASS:-}"
|
||||
APP_ROOT="${APP_ROOT:-/var/www/html}"
|
||||
|
||||
SERVERS=(
|
||||
"194:176.52.138.42"
|
||||
"192:110.238.65.222"
|
||||
"191:176.52.129.86"
|
||||
"193:110.238.69.46"
|
||||
"190:176.52.132.94"
|
||||
)
|
||||
|
||||
RED='\033[0;31m'
|
||||
GRN='\033[0;32m'
|
||||
YLW='\033[1;33m'
|
||||
CYN='\033[0;36m'
|
||||
NC='\033[0m'
|
||||
|
||||
ts() { date "+%H:%M:%S"; }
|
||||
info() { echo -e "[$(ts)] ${CYN}▶${NC} $*"; }
|
||||
ok() { echo -e "[$(ts)] ${GRN}✓${NC} $*"; }
|
||||
warn() { echo -e "[$(ts)] ${YLW}⚠${NC} $*"; }
|
||||
fail() { echo -e "[$(ts)] ${RED}✗${NC} $*"; }
|
||||
|
||||
mysql_q() {
|
||||
if [[ -n "${DB_PASS}" ]]; then
|
||||
mysql -u"${DB_USER}" -p"${DB_PASS}" -D"${DB_NAME}" -N -e "$1" 2>/dev/null
|
||||
else
|
||||
mysql -u"${DB_USER}" -D"${DB_NAME}" -N -e "$1" 2>/dev/null
|
||||
fi
|
||||
}
|
||||
|
||||
run_ssh() {
|
||||
local ip="$1" cmd="$2"
|
||||
if [[ -n "${SSH_PASS}" ]] && command -v sshpass &>/dev/null; then
|
||||
sshpass -p "${SSH_PASS}" ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
-o ConnectTimeout=5 "${SSH_USER}@${ip}" "${cmd}" 2>/dev/null
|
||||
else
|
||||
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
-o ConnectTimeout=5 "${SSH_USER}@${ip}" "${cmd}" 2>/dev/null
|
||||
fi
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo -e "${CYN}═══════════════════════════════════════════════${NC}"
|
||||
echo -e "${CYN} DEBLOCAGE MULTI-INSTALL SERVEUR BLEU${NC}"
|
||||
echo -e "${CYN} Serveurs: ${STUCK_IDS}${NC}"
|
||||
echo -e "${CYN}═══════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
|
||||
# --- ETAPE 1: Tuer les processus bloques ---
|
||||
info "ETAPE 1/5 — Arret des processus d'installation bloques"
|
||||
for pattern in 'multiInstall' 'multi.install' 'install.*server' 'pmta.*install'; do
|
||||
pids=$(pgrep -f "${pattern}" 2>/dev/null || true)
|
||||
if [[ -n "${pids}" ]]; then
|
||||
echo "${pids}" | xargs kill -9 2>/dev/null || true
|
||||
ok "Processus '${pattern}' tues: ${pids}"
|
||||
fi
|
||||
done
|
||||
ok "Processus bloques nettoyes"
|
||||
echo ""
|
||||
|
||||
# --- ETAPE 2: Reset DB ---
|
||||
info "ETAPE 2/5 — Reset statut base de donnees"
|
||||
|
||||
TABLES_TO_TRY=("servers" "mta_servers" "server" "mta_server")
|
||||
COLS_TO_TRY=("installation_status" "install_status" "installStatus" "installation_state")
|
||||
RESET_DONE=0
|
||||
|
||||
for tbl in "${TABLES_TO_TRY[@]}"; do
|
||||
for col in "${COLS_TO_TRY[@]}"; do
|
||||
result=$(mysql_q "
|
||||
UPDATE ${tbl}
|
||||
SET ${col} = 'Not Installed', updated_at = NOW()
|
||||
WHERE id IN (${STUCK_IDS})
|
||||
AND (${col} LIKE '%Installing%' OR ${col} LIKE '%Progress%' OR ${col} LIKE '%progress%');
|
||||
SELECT ROW_COUNT();
|
||||
" 2>/dev/null || true)
|
||||
if [[ -n "${result}" && "${result}" -gt 0 ]]; then
|
||||
ok "Table '${tbl}', colonne '${col}': ${result} serveur(s) reinitialise(s)"
|
||||
RESET_DONE=1
|
||||
break 2
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
if [[ "${RESET_DONE}" -eq 0 ]]; then
|
||||
warn "Aucune ligne mise a jour en DB — verifier DB_NAME/DB_USER/DB_PASS"
|
||||
info "Tentative avec requete generique..."
|
||||
mysql_q "SHOW TABLES;" 2>/dev/null | head -20 || warn "Impossible d'acceder a la base"
|
||||
fi
|
||||
|
||||
# Clear stuck jobs
|
||||
mysql_q "
|
||||
DELETE FROM failed_jobs
|
||||
WHERE payload LIKE '%194%' OR payload LIKE '%193%'
|
||||
OR payload LIKE '%192%' OR payload LIKE '%191%' OR payload LIKE '%190%';
|
||||
" 2>/dev/null || true
|
||||
|
||||
mysql_q "
|
||||
UPDATE jobs SET reserved_at = NULL, attempts = 0
|
||||
WHERE reserved_at IS NOT NULL AND reserved_at < DATE_SUB(NOW(), INTERVAL 10 MINUTE);
|
||||
" 2>/dev/null || true
|
||||
ok "Nettoyage jobs queue termine"
|
||||
echo ""
|
||||
|
||||
# --- ETAPE 3: Nettoyer les locks sur les serveurs cibles ---
|
||||
info "ETAPE 3/5 — Nettoyage locks dpkg sur les serveurs cibles"
|
||||
for entry in "${SERVERS[@]}"; do
|
||||
sid="${entry%%:*}"
|
||||
ip="${entry##*:}"
|
||||
printf " SERVER_%s (%s): " "${sid}" "${ip}"
|
||||
|
||||
if ! timeout 3 bash -c "exec 3<>/dev/tcp/${ip}/22" 2>/dev/null; then
|
||||
echo -e "${RED}TCP unreachable${NC}"
|
||||
continue
|
||||
fi
|
||||
|
||||
if run_ssh "${ip}" "
|
||||
rm -f /var/lib/dpkg/lock /var/lib/dpkg/lock-frontend /var/cache/apt/archives/lock 2>/dev/null
|
||||
dpkg --configure -a 2>/dev/null
|
||||
echo OK
|
||||
" 2>/dev/null | grep -q OK; then
|
||||
echo -e "${GRN}locks nettoyes${NC}"
|
||||
else
|
||||
echo -e "${YLW}SSH auth fail — nettoyage manuel requis${NC}"
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
|
||||
# --- ETAPE 4: Relancer les workers ---
|
||||
info "ETAPE 4/5 — Redemarrage des workers"
|
||||
|
||||
if [[ -f "${APP_ROOT}/artisan" ]]; then
|
||||
cd "${APP_ROOT}"
|
||||
php artisan queue:restart 2>/dev/null && ok "Laravel queue restart" || warn "queue:restart echoue"
|
||||
fi
|
||||
|
||||
if command -v pm2 &>/dev/null; then
|
||||
pm2 restart all 2>/dev/null && ok "PM2 redemarre" || warn "PM2 restart echoue"
|
||||
fi
|
||||
|
||||
if command -v supervisorctl &>/dev/null; then
|
||||
supervisorctl restart all 2>/dev/null && ok "Supervisor redemarre" || warn "Supervisor restart echoue"
|
||||
fi
|
||||
|
||||
systemctl restart apache2 2>/dev/null && ok "Apache redemarre" || true
|
||||
systemctl restart php*-fpm 2>/dev/null && ok "PHP-FPM redemarre" || true
|
||||
echo ""
|
||||
|
||||
# --- ETAPE 5: Verification ---
|
||||
info "ETAPE 5/5 — Verification post-deblocage"
|
||||
|
||||
for entry in "${SERVERS[@]}"; do
|
||||
sid="${entry%%:*}"
|
||||
ip="${entry##*:}"
|
||||
printf " SERVER_%s (%s): " "${sid}" "${ip}"
|
||||
if timeout 3 bash -c "exec 3<>/dev/tcp/${ip}/22" 2>/dev/null; then
|
||||
echo -e "${GRN}OK${NC}"
|
||||
else
|
||||
echo -e "${RED}KO${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
db_check=$(mysql_q "
|
||||
SELECT id, installation_status FROM servers WHERE id IN (${STUCK_IDS});
|
||||
" 2>/dev/null || true)
|
||||
|
||||
if [[ -n "${db_check}" ]]; then
|
||||
info "Statut DB apres reset:"
|
||||
echo "${db_check}" | while IFS=$'\t' read -r id status; do
|
||||
if [[ "${status}" == *"Installing"* ]]; then
|
||||
echo -e " ID ${id}: ${RED}${status}${NC} (encore bloque!)"
|
||||
else
|
||||
echo -e " ID ${id}: ${GRN}${status}${NC}"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${CYN}═══════════════════════════════════════════════${NC}"
|
||||
echo -e "${GRN} DEBLOCAGE TERMINE${NC}"
|
||||
echo -e "${CYN}═══════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
echo " Prochaines etapes:"
|
||||
echo " 1. Verifier le statut dans l'interface: http://89.167.40.150:5821/mta-servers/list"
|
||||
echo " 2. Relancer le multi-install: http://89.167.40.150:5821/mta-servers/multi-install/194-193-192-191-190"
|
||||
echo " 3. Surveiller les logs: tail -f ${APP_ROOT}/storage/logs/laravel.log"
|
||||
echo ""
|
||||
383
fix-stuck-multiinstall.sh
Executable file
383
fix-stuck-multiinstall.sh
Executable file
@@ -0,0 +1,383 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# fix-stuck-multiinstall.sh
|
||||
#
|
||||
# Diagnose and fix multi-install stuck at "Installing In Progress".
|
||||
# Run this ON the WEVADS application server (89.167.40.150).
|
||||
#
|
||||
# Usage:
|
||||
# ./fix-stuck-multiinstall.sh [--diagnose|--reset|--retry]
|
||||
#
|
||||
# Modes:
|
||||
# --diagnose Check processes, DB state, logs (default)
|
||||
# --reset Reset stuck "Installing In Progress" to "Not Installed"
|
||||
# --retry Reset + re-launch multi-install for the stuck batch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MODE="${1:---diagnose}"
|
||||
APP_ROOT="${APP_ROOT:-/var/www/html}"
|
||||
NODE_APP="${NODE_APP:-/opt/wevads}"
|
||||
DB_NAME="${DB_NAME:-wevads}"
|
||||
DB_USER="${DB_USER:-root}"
|
||||
DB_PASS="${DB_PASS:-}"
|
||||
STUCK_IDS="${STUCK_IDS:-194,193,192,191,190}"
|
||||
LOG_DIR="${LOG_DIR:-/var/log/wevads}"
|
||||
|
||||
RED='\033[0;31m'
|
||||
GRN='\033[0;32m'
|
||||
YLW='\033[1;33m'
|
||||
CYN='\033[0;36m'
|
||||
NC='\033[0m'
|
||||
|
||||
ts() { date "+%Y-%m-%d %H:%M:%S"; }
|
||||
info() { echo -e "[$(ts)] ${CYN}INFO${NC} $*"; }
|
||||
ok() { echo -e "[$(ts)] ${GRN}OK${NC} $*"; }
|
||||
warn() { echo -e "[$(ts)] ${YLW}WARN${NC} $*"; }
|
||||
fail() { echo -e "[$(ts)] ${RED}FAIL${NC} $*"; }
|
||||
|
||||
mysql_q() {
|
||||
local query="$1"
|
||||
if [[ -n "${DB_PASS}" ]]; then
|
||||
mysql -u"${DB_USER}" -p"${DB_PASS}" -D"${DB_NAME}" -N -e "${query}" 2>/dev/null
|
||||
else
|
||||
mysql -u"${DB_USER}" -D"${DB_NAME}" -N -e "${query}" 2>/dev/null
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auto-detect application paths
|
||||
# ---------------------------------------------------------------------------
|
||||
detect_paths() {
|
||||
info "Auto-detecting application paths..."
|
||||
|
||||
for candidate in /var/www/html /opt/wevads /var/www/wevads /home/wevads /srv/wevads; do
|
||||
if [[ -d "${candidate}" ]]; then
|
||||
if [[ -f "${candidate}/app/Http/Controllers" || -f "${candidate}/index.php" || -d "${candidate}/app" ]]; then
|
||||
APP_ROOT="${candidate}"
|
||||
info "APP_ROOT detected: ${APP_ROOT}"
|
||||
break
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
for candidate in /opt/wevads /var/www/html /srv/wevads; do
|
||||
if [[ -d "${candidate}" && -f "${candidate}/package.json" ]]; then
|
||||
NODE_APP="${candidate}"
|
||||
info "NODE_APP detected: ${NODE_APP}"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
for candidate in wevads wevadsup wevads_app mta_app; do
|
||||
if mysql -u"${DB_USER}" ${DB_PASS:+-p"${DB_PASS}"} -e "USE ${candidate}" 2>/dev/null; then
|
||||
DB_NAME="${candidate}"
|
||||
info "DB_NAME detected: ${DB_NAME}"
|
||||
break
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. DIAGNOSE
|
||||
# ---------------------------------------------------------------------------
|
||||
diagnose() {
|
||||
info "=== MULTI-INSTALL DIAGNOSTIC ==="
|
||||
echo ""
|
||||
|
||||
# 1a. Check for running multiInstall/install processes
|
||||
info "--- Running install processes ---"
|
||||
local procs
|
||||
procs=$(ps aux | grep -iE 'multiInstall|multi.install|install.*mta|pmta.*install|node.*install' | grep -v grep || true)
|
||||
if [[ -n "${procs}" ]]; then
|
||||
warn "Active install processes found:"
|
||||
echo "${procs}"
|
||||
else
|
||||
fail "No active install processes running — installations likely crashed or timed out"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 1b. Check Node.js processes
|
||||
info "--- Node.js processes ---"
|
||||
local node_procs
|
||||
node_procs=$(ps aux | grep -E 'node|pm2|forever' | grep -v grep || true)
|
||||
if [[ -n "${node_procs}" ]]; then
|
||||
echo "${node_procs}"
|
||||
else
|
||||
warn "No Node.js processes running"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 1c. Check PHP-FPM / Apache workers
|
||||
info "--- Web server workers ---"
|
||||
local web_procs
|
||||
web_procs=$(ps aux | grep -iE 'apache|httpd|php-fpm|nginx' | grep -v grep | wc -l || true)
|
||||
info "Web server workers: ${web_procs}"
|
||||
echo ""
|
||||
|
||||
# 1d. Check queue workers (Laravel/Artisan)
|
||||
info "--- Queue workers ---"
|
||||
local queue_procs
|
||||
queue_procs=$(ps aux | grep -iE 'queue:work|queue:listen|artisan|supervisor' | grep -v grep || true)
|
||||
if [[ -n "${queue_procs}" ]]; then
|
||||
echo "${queue_procs}"
|
||||
else
|
||||
warn "No queue workers found — jobs may not be processing"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 1e. Check PM2 status (if available)
|
||||
if command -v pm2 &>/dev/null; then
|
||||
info "--- PM2 status ---"
|
||||
pm2 list 2>/dev/null || true
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# 1f. Check supervisor status (if available)
|
||||
if command -v supervisorctl &>/dev/null; then
|
||||
info "--- Supervisor status ---"
|
||||
supervisorctl status 2>/dev/null || true
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# 1g. Database state of stuck servers
|
||||
info "--- Database: installation status for servers ${STUCK_IDS} ---"
|
||||
local db_result
|
||||
db_result=$(mysql_q "
|
||||
SELECT id, name, main_ip, status, installation_status, updated_at
|
||||
FROM servers
|
||||
WHERE id IN (${STUCK_IDS})
|
||||
ORDER BY id;
|
||||
" 2>/dev/null || true)
|
||||
|
||||
if [[ -n "${db_result}" ]]; then
|
||||
printf "%-6s %-15s %-18s %-10s %-25s %-20s\n" "ID" "Name" "IP" "Status" "InstallStatus" "Updated"
|
||||
echo "-----------------------------------------------------------------------------------------------"
|
||||
echo "${db_result}" | while IFS=$'\t' read -r id name ip status inst_status updated; do
|
||||
printf "%-6s %-15s %-18s %-10s %-25s %-20s\n" "${id}" "${name}" "${ip}" "${status}" "${inst_status}" "${updated}"
|
||||
done
|
||||
else
|
||||
warn "Could not query database — check DB_USER/DB_PASS/DB_NAME"
|
||||
info "Trying alternative table names..."
|
||||
for tbl in mta_servers servers server mta_server; do
|
||||
local alt
|
||||
alt=$(mysql_q "SELECT COUNT(*) FROM ${tbl} WHERE 1=1 LIMIT 1;" 2>/dev/null || true)
|
||||
if [[ -n "${alt}" && "${alt}" -gt 0 ]]; then
|
||||
info "Found table: ${tbl} (${alt} rows)"
|
||||
mysql_q "SELECT * FROM ${tbl} WHERE id IN (${STUCK_IDS}) LIMIT 10;" 2>/dev/null || true
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 1h. Check installation jobs/queue table
|
||||
info "--- Database: pending/stuck jobs ---"
|
||||
mysql_q "
|
||||
SELECT id, queue, payload, attempts, reserved_at, available_at
|
||||
FROM jobs
|
||||
WHERE reserved_at IS NOT NULL
|
||||
ORDER BY reserved_at ASC
|
||||
LIMIT 20;
|
||||
" 2>/dev/null || warn "No jobs table found or query failed"
|
||||
|
||||
mysql_q "
|
||||
SELECT id, connection, queue, exception
|
||||
FROM failed_jobs
|
||||
ORDER BY id DESC
|
||||
LIMIT 10;
|
||||
" 2>/dev/null || warn "No failed_jobs table found"
|
||||
echo ""
|
||||
|
||||
# 1i. Check recent logs
|
||||
info "--- Recent install-related logs ---"
|
||||
for logfile in \
|
||||
"${LOG_DIR}/multiinstall.log" \
|
||||
"${APP_ROOT}/storage/logs/laravel.log" \
|
||||
"${NODE_APP}/logs/install.log" \
|
||||
"/var/log/syslog" \
|
||||
"/var/log/messages"; do
|
||||
if [[ -f "${logfile}" ]]; then
|
||||
info "Last 20 lines of ${logfile}:"
|
||||
tail -20 "${logfile}" 2>/dev/null || true
|
||||
echo ""
|
||||
fi
|
||||
done
|
||||
|
||||
# 1j. SSH connectivity to stuck servers
|
||||
info "--- SSH connectivity to target servers ---"
|
||||
for ip in 176.52.138.42 110.238.65.222 176.52.129.86 110.238.69.46 176.52.132.94; do
|
||||
if timeout 3 bash -c "exec 3<>/dev/tcp/${ip}/22" 2>/dev/null; then
|
||||
ok " ${ip}:22 reachable"
|
||||
else
|
||||
fail " ${ip}:22 unreachable"
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
|
||||
# 1k. System resources
|
||||
info "--- System resources on this server ---"
|
||||
echo " Load: $(cat /proc/loadavg)"
|
||||
echo " Memory: $(free -h | awk 'NR==2{print $3"/"$2}')"
|
||||
echo " Disk: $(df -h / | awk 'NR==2{print $3"/"$2" ("$5" used)"}')"
|
||||
echo ""
|
||||
|
||||
info "=== DIAGNOSTIC COMPLETE ==="
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. RESET stuck installations
|
||||
# ---------------------------------------------------------------------------
|
||||
reset_stuck() {
|
||||
info "=== RESETTING STUCK INSTALLATIONS ==="
|
||||
|
||||
# Kill any hanging install processes
|
||||
info "Killing any hanging install processes..."
|
||||
local pids
|
||||
pids=$(ps aux | grep -iE 'multiInstall|multi.install|install.*mta' | grep -v grep | awk '{print $2}' || true)
|
||||
if [[ -n "${pids}" ]]; then
|
||||
echo "${pids}" | xargs kill -9 2>/dev/null || true
|
||||
ok "Killed hanging processes: ${pids}"
|
||||
else
|
||||
info "No hanging install processes found"
|
||||
fi
|
||||
|
||||
# Reset DB status
|
||||
info "Resetting installation status in database..."
|
||||
|
||||
local updated
|
||||
updated=$(mysql_q "
|
||||
UPDATE servers
|
||||
SET installation_status = 'Not Installed',
|
||||
updated_at = NOW()
|
||||
WHERE id IN (${STUCK_IDS})
|
||||
AND installation_status LIKE '%Installing%';
|
||||
SELECT ROW_COUNT();
|
||||
" 2>/dev/null || true)
|
||||
|
||||
if [[ -n "${updated}" && "${updated}" -gt 0 ]]; then
|
||||
ok "Reset ${updated} server(s) from 'Installing In Progress' to 'Not Installed'"
|
||||
else
|
||||
warn "No rows updated — trying alternative column names..."
|
||||
for col in install_status installStatus installation_state; do
|
||||
mysql_q "
|
||||
UPDATE servers
|
||||
SET ${col} = 'Not Installed', updated_at = NOW()
|
||||
WHERE id IN (${STUCK_IDS})
|
||||
AND ${col} LIKE '%Installing%';
|
||||
" 2>/dev/null && ok "Updated using column: ${col}" && break || true
|
||||
done
|
||||
fi
|
||||
|
||||
# Clear failed jobs for these servers
|
||||
info "Clearing failed jobs for stuck servers..."
|
||||
mysql_q "
|
||||
DELETE FROM failed_jobs
|
||||
WHERE payload LIKE '%194%'
|
||||
OR payload LIKE '%193%'
|
||||
OR payload LIKE '%192%'
|
||||
OR payload LIKE '%191%'
|
||||
OR payload LIKE '%190%';
|
||||
" 2>/dev/null || true
|
||||
|
||||
# Clear reserved/stuck jobs
|
||||
mysql_q "
|
||||
UPDATE jobs
|
||||
SET reserved_at = NULL, attempts = 0
|
||||
WHERE reserved_at IS NOT NULL
|
||||
AND reserved_at < DATE_SUB(NOW(), INTERVAL 30 MINUTE);
|
||||
" 2>/dev/null || true
|
||||
|
||||
ok "Reset complete. Verify with: $0 --diagnose"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. RETRY multi-install
|
||||
# ---------------------------------------------------------------------------
|
||||
retry_install() {
|
||||
info "=== RETRY MULTI-INSTALL ==="
|
||||
|
||||
reset_stuck
|
||||
|
||||
info "Waiting 5 seconds for cleanup..."
|
||||
sleep 5
|
||||
|
||||
# Restart queue workers
|
||||
info "Restarting queue workers..."
|
||||
if command -v php &>/dev/null && [[ -f "${APP_ROOT}/artisan" ]]; then
|
||||
cd "${APP_ROOT}"
|
||||
php artisan queue:restart 2>/dev/null && ok "Queue workers restarted" || warn "queue:restart failed"
|
||||
fi
|
||||
|
||||
# Restart PM2 processes if applicable
|
||||
if command -v pm2 &>/dev/null; then
|
||||
pm2 restart all 2>/dev/null && ok "PM2 processes restarted" || warn "PM2 restart failed"
|
||||
fi
|
||||
|
||||
# Restart supervisor if applicable
|
||||
if command -v supervisorctl &>/dev/null; then
|
||||
supervisorctl restart all 2>/dev/null && ok "Supervisor processes restarted" || warn "Supervisor restart failed"
|
||||
fi
|
||||
|
||||
info "Triggering multi-install via web endpoint..."
|
||||
local ids_path
|
||||
ids_path=$(echo "${STUCK_IDS}" | tr ',' '-')
|
||||
|
||||
local session_cookie
|
||||
session_cookie=$(curl -s -c - "http://localhost:5821/auth/login.html" 2>/dev/null | grep wevupsession | awk '{print $NF}' || true)
|
||||
|
||||
if [[ -n "${session_cookie}" ]]; then
|
||||
info "Attempting to trigger multi-install at /mta-servers/multi-install/${ids_path}"
|
||||
curl -s -b "wevupsession=${session_cookie}" \
|
||||
"http://localhost:5821/mta-servers/multi-install/${ids_path}" \
|
||||
-o /dev/null -w "HTTP %{http_code}\n" 2>/dev/null || true
|
||||
else
|
||||
warn "Could not get session cookie — trigger manually from UI"
|
||||
info "URL: http://89.167.40.150:5821/mta-servers/multi-install/${ids_path}"
|
||||
fi
|
||||
|
||||
# Alternative: trigger via artisan command if available
|
||||
if [[ -f "${APP_ROOT}/artisan" ]]; then
|
||||
info "Trying artisan install command..."
|
||||
for sid in ${STUCK_IDS//,/ }; do
|
||||
php "${APP_ROOT}/artisan" mta:install "${sid}" 2>/dev/null && ok "Triggered install for ${sid}" || true
|
||||
done
|
||||
fi
|
||||
|
||||
info "=== RETRY COMPLETE — Monitor with: $0 --diagnose ==="
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MAIN
|
||||
# ---------------------------------------------------------------------------
|
||||
detect_paths
|
||||
|
||||
case "${MODE}" in
|
||||
--diagnose|-d)
|
||||
diagnose
|
||||
;;
|
||||
--reset|-r)
|
||||
reset_stuck
|
||||
;;
|
||||
--retry|-R)
|
||||
retry_install
|
||||
;;
|
||||
*)
|
||||
echo "Usage: $0 [--diagnose|--reset|--retry]"
|
||||
echo ""
|
||||
echo " --diagnose Check processes, DB state, logs (default)"
|
||||
echo " --reset Reset stuck installations to 'Not Installed'"
|
||||
echo " --retry Reset + restart workers + re-trigger install"
|
||||
echo ""
|
||||
echo "Environment variables:"
|
||||
echo " APP_ROOT PHP application root (default: /var/www/html)"
|
||||
echo " NODE_APP Node.js app path (default: /opt/wevads)"
|
||||
echo " DB_NAME Database name (default: wevads)"
|
||||
echo " DB_USER Database user (default: root)"
|
||||
echo " DB_PASS Database password"
|
||||
echo " STUCK_IDS Comma-separated server IDs (default: 194,193,192,191,190)"
|
||||
echo " LOG_DIR Log directory (default: /var/log/wevads)"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
191
multiinstall-safe-preflight.sh
Executable file
191
multiinstall-safe-preflight.sh
Executable file
@@ -0,0 +1,191 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Multi-install SAFE preflight v2
|
||||
#
|
||||
# Pre-check all target servers before launching multi-install.
|
||||
# Verifies: TCP/22, SSH auth, disk, RAM, dpkg locks, apt health,
|
||||
# PMTA port (25), hostname resolution.
|
||||
#
|
||||
# Input: CSV file with server_id,server_name,ip,domain
|
||||
# or server_id,ip,username,password
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
INPUT_FILE="${1:-servers-bleu-multiinstall.csv}"
|
||||
SSH_USER="${SSH_USER:-root}"
|
||||
SSH_PASS="${SSH_PASS:-}"
|
||||
CONNECT_TIMEOUT="${CONNECT_TIMEOUT:-5}"
|
||||
SSH_BIN="${SSH_BIN:-ssh}"
|
||||
SSHPASS_BIN="${SSHPASS_BIN:-sshpass}"
|
||||
OUT_DIR="${OUT_DIR:-./reports}"
|
||||
RUN_ID="$(date +%Y%m%d_%H%M%S)"
|
||||
OUT_CSV="${OUT_DIR}/preflight_bleu_${RUN_ID}.csv"
|
||||
MIN_DISK_GB="${MIN_DISK_GB:-5}"
|
||||
MIN_RAM_GB="${MIN_RAM_GB:-1}"
|
||||
|
||||
RED='\033[0;31m'
|
||||
GRN='\033[0;32m'
|
||||
YLW='\033[1;33m'
|
||||
CYN='\033[0;36m'
|
||||
NC='\033[0m'
|
||||
|
||||
if [[ ! -f "${INPUT_FILE}" ]]; then
|
||||
echo "Usage: $0 <servers.csv>"
|
||||
echo "File not found: ${INPUT_FILE}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "${OUT_DIR}"
|
||||
echo "server_id,ip,tcp22,ssh_auth,disk,ram,dpkg_lock,apt,smtp25,ready,notes" > "${OUT_CSV}"
|
||||
|
||||
TOTAL=0
|
||||
READY=0
|
||||
BLOCKED=0
|
||||
|
||||
check_tcp() {
|
||||
local ip="$1" port="$2"
|
||||
timeout "${CONNECT_TIMEOUT}" bash -c "exec 3<>/dev/tcp/${ip}/${port}" 2>/dev/null
|
||||
}
|
||||
|
||||
run_ssh() {
|
||||
local ip="$1" cmd="$2"
|
||||
if [[ -n "${SSH_PASS}" ]] && command -v "${SSHPASS_BIN}" &>/dev/null; then
|
||||
"${SSHPASS_BIN}" -p "${SSH_PASS}" "${SSH_BIN}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
-o UserKnownHostsFile=/dev/null \
|
||||
-o ConnectTimeout="${CONNECT_TIMEOUT}" \
|
||||
-o BatchMode=no \
|
||||
"${SSH_USER}@${ip}" "${cmd}" 2>/dev/null
|
||||
else
|
||||
"${SSH_BIN}" \
|
||||
-o StrictHostKeyChecking=no \
|
||||
-o UserKnownHostsFile=/dev/null \
|
||||
-o ConnectTimeout="${CONNECT_TIMEOUT}" \
|
||||
-o BatchMode=yes \
|
||||
"${SSH_USER}@${ip}" "${cmd}" 2>/dev/null
|
||||
fi
|
||||
}
|
||||
|
||||
while IFS=',' read -r c1 c2 c3 c4; do
|
||||
[[ -z "${c1}" ]] && continue
|
||||
[[ "${c1}" =~ ^# ]] && continue
|
||||
[[ "${c1}" == "server_id" ]] && continue
|
||||
|
||||
if [[ "${c2}" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
|
||||
server_id="${c1}"
|
||||
ip="${c2}"
|
||||
elif [[ "${c3}" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
|
||||
server_id="${c1}"
|
||||
ip="${c3}"
|
||||
else
|
||||
continue
|
||||
fi
|
||||
|
||||
TOTAL=$((TOTAL + 1))
|
||||
tcp22="FAIL"; ssh_auth="FAIL"; disk="FAIL"; ram="FAIL"
|
||||
dpkg_lock="SKIP"; apt="SKIP"; smtp25="FAIL"
|
||||
ready="NO"; notes=""
|
||||
|
||||
printf "${CYN}[%s]${NC} Checking server %s (%s)... " "$(date +%H:%M:%S)" "${server_id}" "${ip}"
|
||||
|
||||
# TCP 22
|
||||
if check_tcp "${ip}" 22; then
|
||||
tcp22="PASS"
|
||||
else
|
||||
notes="port22_unreachable"
|
||||
printf "${RED}TCP22 FAIL${NC}\n"
|
||||
echo "${server_id},${ip},${tcp22},${ssh_auth},${disk},${ram},${dpkg_lock},${apt},${smtp25},${ready},${notes}" >> "${OUT_CSV}"
|
||||
BLOCKED=$((BLOCKED + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
# SSH auth
|
||||
if run_ssh "${ip}" "echo ok" >/dev/null 2>&1; then
|
||||
ssh_auth="PASS"
|
||||
else
|
||||
notes="ssh_auth_failed"
|
||||
printf "${RED}SSH AUTH FAIL${NC}\n"
|
||||
echo "${server_id},${ip},${tcp22},${ssh_auth},${disk},${ram},${dpkg_lock},${apt},${smtp25},${ready},${notes}" >> "${OUT_CSV}"
|
||||
BLOCKED=$((BLOCKED + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
# Disk check
|
||||
if run_ssh "${ip}" \
|
||||
"avail=\$(df -BG / | awk 'NR==2 {gsub(\"G\",\"\",\$4); print \$4}'); [ \"\${avail:-0}\" -ge ${MIN_DISK_GB} ]"; then
|
||||
disk="PASS"
|
||||
else
|
||||
notes="${notes:+$notes|}low_disk"
|
||||
fi
|
||||
|
||||
# RAM check
|
||||
if run_ssh "${ip}" \
|
||||
"mem=\$(awk '/MemTotal/ {print int(\$2/1024/1024)}' /proc/meminfo); [ \"\${mem:-0}\" -ge ${MIN_RAM_GB} ]"; then
|
||||
ram="PASS"
|
||||
else
|
||||
notes="${notes:+$notes|}low_ram"
|
||||
fi
|
||||
|
||||
# dpkg lock check
|
||||
if run_ssh "${ip}" \
|
||||
"! fuser /var/lib/dpkg/lock /var/lib/dpkg/lock-frontend 2>/dev/null | grep -q ."; then
|
||||
dpkg_lock="PASS"
|
||||
else
|
||||
dpkg_lock="FAIL"
|
||||
notes="${notes:+$notes|}dpkg_locked"
|
||||
fi
|
||||
|
||||
# apt health check
|
||||
if run_ssh "${ip}" "apt-cache policy >/dev/null 2>&1"; then
|
||||
apt="PASS"
|
||||
else
|
||||
apt="FAIL"
|
||||
notes="${notes:+$notes|}apt_broken"
|
||||
fi
|
||||
|
||||
# SMTP port 25 check (for PMTA)
|
||||
if check_tcp "${ip}" 25; then
|
||||
smtp25="PASS"
|
||||
else
|
||||
smtp25="SKIP"
|
||||
notes="${notes:+$notes|}smtp25_closed(normal_pre_install)"
|
||||
fi
|
||||
|
||||
# Overall readiness
|
||||
if [[ "${tcp22}" == "PASS" && "${ssh_auth}" == "PASS" && "${disk}" == "PASS" && "${ram}" == "PASS" && "${dpkg_lock}" == "PASS" ]]; then
|
||||
ready="YES"
|
||||
READY=$((READY + 1))
|
||||
printf "${GRN}READY${NC}\n"
|
||||
else
|
||||
BLOCKED=$((BLOCKED + 1))
|
||||
printf "${YLW}BLOCKED${NC} (${notes})\n"
|
||||
fi
|
||||
|
||||
echo "${server_id},${ip},${tcp22},${ssh_auth},${disk},${ram},${dpkg_lock},${apt},${smtp25},${ready},${notes}" >> "${OUT_CSV}"
|
||||
|
||||
done < "${INPUT_FILE}"
|
||||
|
||||
echo ""
|
||||
echo "=============================="
|
||||
echo " PREFLIGHT SUMMARY"
|
||||
echo "=============================="
|
||||
echo " Total: ${TOTAL}"
|
||||
echo " Ready: ${READY}"
|
||||
echo " Blocked: ${BLOCKED}"
|
||||
echo ""
|
||||
echo " Report: ${OUT_CSV}"
|
||||
echo ""
|
||||
|
||||
if [[ "${READY}" -eq "${TOTAL}" && "${TOTAL}" -gt 0 ]]; then
|
||||
echo -e " ${GRN}GO — All servers ready for multi-install${NC}"
|
||||
exit 0
|
||||
elif [[ "${READY}" -gt 0 ]]; then
|
||||
echo -e " ${YLW}PARTIAL — ${READY}/${TOTAL} servers ready${NC}"
|
||||
echo " Ready servers:"
|
||||
awk -F',' 'NR>1 && $10=="YES" {print " - " $1 " (" $2 ")"}' "${OUT_CSV}"
|
||||
exit 0
|
||||
else
|
||||
echo -e " ${RED}NO-GO — No servers ready${NC}"
|
||||
exit 1
|
||||
fi
|
||||
22
patches/SENTINEL_VERIFY_CMD.txt
Normal file
22
patches/SENTINEL_VERIFY_CMD.txt
Normal file
@@ -0,0 +1,22 @@
|
||||
# Commande a envoyer a Sentinel pour verifier le patch multi-install
|
||||
# POST http://89.167.40.150:5890/api/sentinel-brain.php
|
||||
# action=exec
|
||||
# cmd=<contenu ci-dessous>
|
||||
|
||||
echo "=== PATCH MULTI-INSTALL VERIF ==="
|
||||
for app in /opt/wevads /opt/fmgapp; do
|
||||
f="${app}/app/webservices/Servers.php"
|
||||
[[ -f "$f" ]] || continue
|
||||
echo "--- $app ---"
|
||||
grep -c INSTALL_TIMEOUT_MINUTES "$f" 2>/dev/null && echo " timeout: OK" || echo " timeout: MANQUANT"
|
||||
grep -q 'string).*logs' "$f" 2>/dev/null && echo " null-safe: OK" || echo " null-safe: A VERIFIER"
|
||||
grep -c AUTO-DETECT "$f" 2>/dev/null && echo " auto-detect: OK" || echo " auto-detect: MANQUANT"
|
||||
php -l "$f" 2>&1 | tail -1
|
||||
done
|
||||
echo "--- Proc files stale ---"
|
||||
find /opt/wevads/logs/installations /opt/fmgapp/logs/installations -name 'inst_*_proc.log' 2>/dev/null | while read f; do
|
||||
s=$(cat "$f" 2>/dev/null)
|
||||
[[ "$s" == *"completed"* || "$s" == *"interrupted"* ]] && continue
|
||||
echo "STALE: $f"
|
||||
done
|
||||
echo "=== FIN ==="
|
||||
198
patches/apply-patches.sh
Executable file
198
patches/apply-patches.sh
Executable file
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# apply-patches.sh
|
||||
#
|
||||
# Apply multi-install fixes to WEVADS/iResponse applications.
|
||||
# IMPORTANT: Multi-install (port 5821) uses /opt/wevads, NOT /opt/fmgapp.
|
||||
# Run ON the application server (89.167.40.150).
|
||||
#
|
||||
# Usage:
|
||||
# scp -r patches/ root@89.167.40.150:/tmp/patches/
|
||||
# ssh root@89.167.40.150 'bash /tmp/patches/apply-patches.sh'
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
BACKUP_DIR="/tmp/iresponse_backup_$(date +%Y%m%d_%H%M%S)"
|
||||
PATCHES_DIR="${PATCHES_DIR:-/tmp/patches}"
|
||||
|
||||
RED='\033[0;31m'
|
||||
GRN='\033[0;32m'
|
||||
YLW='\033[1;33m'
|
||||
CYN='\033[0;36m'
|
||||
NC='\033[0m'
|
||||
|
||||
info() { echo -e "${CYN}[INFO]${NC} $*"; }
|
||||
ok() { echo -e "${GRN}[OK]${NC} $*"; }
|
||||
warn() { echo -e "${YLW}[WARN]${NC} $*"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $*"; }
|
||||
|
||||
# Both apps must be patched: wevads (port 5821) + fmgapp (port 5822)
|
||||
APP_ROOTS=()
|
||||
for candidate in /opt/wevads /opt/fmgapp /var/www/html /opt/iresponse /var/www/iresponse /srv/iresponse; do
|
||||
if [[ -f "${candidate}/app/webservices/Servers.php" ]]; then
|
||||
APP_ROOTS+=("${candidate}")
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ ${#APP_ROOTS[@]} -eq 0 ]]; then
|
||||
fail "No Servers.php found. Set APP_ROOT or install iResponse."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
info "Will patch ${#APP_ROOTS[@]} app(s): ${APP_ROOTS[*]}"
|
||||
|
||||
mkdir -p "${BACKUP_DIR}"
|
||||
|
||||
for APP_ROOT in "${APP_ROOTS[@]}"; do
|
||||
WS_FILE="${APP_ROOT}/app/webservices/Servers.php"
|
||||
CTRL_FILE="${APP_ROOT}/app/controllers/MtaServers.php"
|
||||
JS_FILE="${APP_ROOT}/public/scripts/pages/servers/installation.js"
|
||||
|
||||
info "=== Patching ${APP_ROOT} ==="
|
||||
|
||||
# --- Backup ---
|
||||
cp "${WS_FILE}" "${BACKUP_DIR}/Servers_$(basename ${APP_ROOT}).php.bak"
|
||||
cp "${CTRL_FILE}" "${BACKUP_DIR}/MtaServers_$(basename ${APP_ROOT}).php.bak" 2>/dev/null || true
|
||||
cp "${JS_FILE}" "${BACKUP_DIR}/installation_$(basename ${APP_ROOT}).js.bak" 2>/dev/null || true
|
||||
|
||||
# --- Patch 1: getInstallationLogs (timeout + liveness + PHP 8.x null-safe) ---
|
||||
info "Patch 1/3: getInstallationLogs — timeout + null-safe"
|
||||
|
||||
if grep -q 'INSTALL_TIMEOUT_MINUTES' "${WS_FILE}" 2>/dev/null; then
|
||||
if grep -q '(string)($logs' "${WS_FILE}" 2>/dev/null || grep -q '(string)\$logs' "${WS_FILE}" 2>/dev/null; then
|
||||
warn "Patch 1 already applied (null-safe present)"
|
||||
else
|
||||
warn "Patch 1 partial — adding null-safe str_replace for PHP 8.x..."
|
||||
php -r "
|
||||
\$f = '${WS_FILE}';
|
||||
\$c = file_get_contents(\$f);
|
||||
\$c = str_replace(\"str_replace(PHP_EOL,'<br/>',\$logs)\", \"str_replace(PHP_EOL,'<br/>', (string)(\$logs ?? ''))\", \$c);
|
||||
file_put_contents(\$f, \$c);
|
||||
echo 'OK\n';
|
||||
" 2>/dev/null && ok "Null-safe added" || warn "Could not add null-safe (manual check)"
|
||||
fi
|
||||
else
|
||||
php -r "
|
||||
\$content = file_get_contents('${WS_FILE}');
|
||||
\$patchPath = '${PATCHES_DIR}/getInstallationLogs_fix.php';
|
||||
if(!file_exists(\$patchPath)) { \$patchPath = '/tmp/patches/getInstallationLogs_fix.php'; }
|
||||
\$patch = file_get_contents(\$patchPath);
|
||||
|
||||
\$old = 'public function getInstallationLogs(\$parameters = [])';
|
||||
\$pos = strpos(\$content, \$old);
|
||||
if(\$pos === false) { echo 'ERROR: Cannot find getInstallationLogs\n'; exit(1); }
|
||||
|
||||
\$braceCount = 0; \$methodStart = \$pos; \$foundOpen = false; \$methodEnd = \$pos;
|
||||
for(\$i = \$pos; \$i < strlen(\$content); \$i++) {
|
||||
if(\$content[\$i] === '{') { \$braceCount++; \$foundOpen = true; }
|
||||
if(\$content[\$i] === '}') { \$braceCount--; }
|
||||
if(\$foundOpen && \$braceCount === 0) { \$methodEnd = \$i + 1; break; }
|
||||
}
|
||||
\$searchBack = substr(\$content, 0, \$methodStart);
|
||||
\$docStart = strrpos(\$searchBack, '/**');
|
||||
if(\$docStart !== false && \$docStart > \$methodStart - 200) { \$methodStart = \$docStart; }
|
||||
|
||||
\$patch = preg_replace('/^<\?php\s*/', '', \$patch);
|
||||
\$patch = preg_replace('/^\/\*\*.*?\*\//s', '', \$patch, 1);
|
||||
\$newContent = substr(\$content, 0, \$methodStart) . trim(\$patch) . \"\\n\" . substr(\$content, \$methodEnd);
|
||||
file_put_contents('${WS_FILE}', \$newContent);
|
||||
echo 'OK\n';
|
||||
" && ok "getInstallationLogs patched" || fail "Could not patch getInstallationLogs"
|
||||
fi
|
||||
|
||||
# --- Patch 2: beginInstallation (PID tracking) ---
|
||||
info "Patch 2/3: beginInstallation — PID tracking"
|
||||
|
||||
if grep -q 'pidFile' "${WS_FILE}" 2>/dev/null; then
|
||||
warn "Patch 2 already applied (pidFile found)"
|
||||
else
|
||||
php -r "
|
||||
\$content = file_get_contents('${WS_FILE}');
|
||||
\$patchPath = '${PATCHES_DIR}/beginInstallation_fix.php';
|
||||
if(!file_exists(\$patchPath)) { \$patchPath = '/tmp/patches/beginInstallation_fix.php'; }
|
||||
\$patch = file_get_contents(\$patchPath);
|
||||
|
||||
\$old = 'public function beginInstallation(\$parameters = [])';
|
||||
\$pos = strpos(\$content, \$old);
|
||||
if(\$pos === false) { echo 'ERROR: Cannot find beginInstallation\n'; exit(1); }
|
||||
|
||||
\$braceCount = 0; \$methodStart = \$pos; \$foundOpen = false; \$methodEnd = \$pos;
|
||||
for(\$i = \$pos; \$i < strlen(\$content); \$i++) {
|
||||
if(\$content[\$i] === '{') { \$braceCount++; \$foundOpen = true; }
|
||||
if(\$content[\$i] === '}') { \$braceCount--; }
|
||||
if(\$foundOpen && \$braceCount === 0) { \$methodEnd = \$i + 1; break; }
|
||||
}
|
||||
\$searchBack = substr(\$content, 0, \$methodStart);
|
||||
\$docStart = strrpos(\$searchBack, '/**');
|
||||
if(\$docStart !== false && \$docStart > \$methodStart - 200) { \$methodStart = \$docStart; }
|
||||
|
||||
\$patch = preg_replace('/^<\?php\s*/', '', \$patch);
|
||||
\$patch = preg_replace('/^\/\*\*.*?\*\//s', '', \$patch, 1);
|
||||
\$newContent = substr(\$content, 0, \$methodStart) . trim(\$patch) . \"\\n\" . substr(\$content, \$methodEnd);
|
||||
file_put_contents('${WS_FILE}', \$newContent);
|
||||
echo 'OK\n';
|
||||
" && ok "beginInstallation patched" || fail "Could not patch beginInstallation"
|
||||
fi
|
||||
|
||||
# --- Patch 3: Ensure log directory exists ---
|
||||
LOGS_DIR=$(php -r "
|
||||
\$files = glob('${APP_ROOT}/app/config/*.php');
|
||||
foreach(\$files as \$f) {
|
||||
\$c = @file_get_contents(\$f);
|
||||
if(\$c && preg_match(\"/define.*LOGS_PATH.*?['\\\"](.+?)['\\\"]/\", \$c, \$m)) { echo \$m[1]; exit; }
|
||||
}
|
||||
echo '${APP_ROOT}/logs';
|
||||
" 2>/dev/null || echo "${APP_ROOT}/logs")
|
||||
|
||||
mkdir -p "${LOGS_DIR}/installations" 2>/dev/null
|
||||
chmod 777 "${LOGS_DIR}/installations" 2>/dev/null
|
||||
ok "Logs ready: ${LOGS_DIR}/installations"
|
||||
|
||||
done
|
||||
|
||||
# --- Clear stuck proc files (all apps) ---
|
||||
info "Clearing stuck proc files..."
|
||||
for LOGS_DIR in /opt/wevads/logs /opt/fmgapp/logs /var/www/html/logs; do
|
||||
[[ ! -d "${LOGS_DIR}/installations" ]] && continue
|
||||
for f in "${LOGS_DIR}"/installations/inst_*_proc.log; do
|
||||
if [[ -f "$f" ]]; then
|
||||
status=$(cat "$f" 2>/dev/null || true)
|
||||
if [[ "$status" != *"completed"* && "$status" != *"interrupted"* ]]; then
|
||||
mtime=$(stat -c %Y "$f" 2>/dev/null || stat -f %m "$f" 2>/dev/null || echo 0)
|
||||
now=$(date +%s)
|
||||
age=$(( (now - mtime) / 60 ))
|
||||
if [[ $age -gt 10 ]]; then
|
||||
echo "Installation interrupted !" > "$f"
|
||||
ok " Reset stale proc file: $f (${age}min old)"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
# --- Kill orphan Java install processes ---
|
||||
info "Killing orphan install processes older than 30min..."
|
||||
pids=$(ps aux | grep 'iresponse_services.jar' | grep -v grep | awk '{
|
||||
split($10, t, ":");
|
||||
mins = t[1]*60 + t[2];
|
||||
if(mins > 30) print $2
|
||||
}' 2>/dev/null || true)
|
||||
|
||||
if [[ -n "$pids" ]]; then
|
||||
echo "$pids" | xargs kill -9 2>/dev/null || true
|
||||
ok "Killed orphan PIDs: $pids"
|
||||
else
|
||||
info "No orphan processes found"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${GRN}Patches applied successfully!${NC}"
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo " 1. Restart Apache: systemctl restart apache2"
|
||||
echo " 2. Clear browser cache and reload the multi-install page"
|
||||
echo " 3. Relaunch installations from the UI"
|
||||
echo ""
|
||||
echo "Backup location: ${BACKUP_DIR}"
|
||||
echo "To rollback: cp ${BACKUP_DIR}/*.bak back to original locations"
|
||||
133
patches/beginInstallation_fix.php
Normal file
133
patches/beginInstallation_fix.php
Normal file
@@ -0,0 +1,133 @@
|
||||
<?php
|
||||
/**
|
||||
* PATCH: beginInstallation — robust process launch with PID tracking
|
||||
*
|
||||
* Replace the beginInstallation method in:
|
||||
* app/webservices/Servers.php
|
||||
*
|
||||
* Changes:
|
||||
* 1. Stores the Java process PID for liveness tracking
|
||||
* 2. Writes a timestamp to detect stale processes
|
||||
* 3. Prevents duplicate installations on the same server
|
||||
*/
|
||||
|
||||
/**
|
||||
* @name beginInstallation
|
||||
* @description begin server installation
|
||||
* @before init
|
||||
*/
|
||||
public function beginInstallation($parameters = [])
|
||||
{
|
||||
if(!Authentication::isUserAuthenticated())
|
||||
{
|
||||
Page::printApiResults(401,'Only logged-in access allowed !');
|
||||
}
|
||||
|
||||
Authentication::checkUserRoles();
|
||||
|
||||
$access = Permissions::checkForAuthorization(Authentication::getAuthenticatedUser(),'MtaServers','install');
|
||||
|
||||
if($access == false)
|
||||
{
|
||||
Page::printApiResults(403,'Access Denied !');
|
||||
}
|
||||
|
||||
$serverId = intval($this->app->utils->arrays->get($parameters,'server-id'));
|
||||
|
||||
if($serverId > 0)
|
||||
{
|
||||
$server = MtaServer::first(MtaServer::FETCH_ARRAY,['id = ?',$serverId]);
|
||||
|
||||
if(count($server) == 0)
|
||||
{
|
||||
Page::printApiResults(404,'Server not found !');
|
||||
}
|
||||
|
||||
$logFile = LOGS_PATH . '/installations/inst_' . $serverId . '.log';
|
||||
$processFile = LOGS_PATH . '/installations/inst_' . $serverId . '_proc.log';
|
||||
$pidFile = LOGS_PATH . '/installations/inst_' . $serverId . '.pid';
|
||||
|
||||
// Prevent duplicate: if a process is already running for this server, block
|
||||
if(file_exists($pidFile))
|
||||
{
|
||||
$existingPid = trim(file_get_contents($pidFile));
|
||||
if(!empty($existingPid))
|
||||
{
|
||||
$psCheck = shell_exec("ps -p {$existingPid} -o pid= 2>/dev/null");
|
||||
if(!empty(trim($psCheck ?? '')))
|
||||
{
|
||||
Page::printApiResults(409,'Installation already running for this server (PID: ' . $existingPid . ')');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$updateIps = $this->app->utils->arrays->get($parameters,'update-ips');
|
||||
|
||||
if($updateIps == 'enabled')
|
||||
{
|
||||
$domains = [];
|
||||
$mapping = $this->app->utils->arrays->get($parameters,'mapping');
|
||||
|
||||
foreach ($mapping as $map)
|
||||
{
|
||||
if(is_array($map) && count($map))
|
||||
{
|
||||
$domain = (key_exists('domain',$map)) ? $map['domain'] : '';
|
||||
$ips4sum = (key_exists('ips-v4',$map)) ? count($map['ips-v4']) : 0;
|
||||
$ips6sum = (key_exists('ips-v6',$map)) ? count($map['ips-v6']) : 0;
|
||||
|
||||
if($domain == '')
|
||||
{
|
||||
Page::printApiResults(500,'No domain found !');
|
||||
}
|
||||
|
||||
if($ips4sum == 0 && $ips6sum == 0)
|
||||
{
|
||||
Page::printApiResults(500,'Each mapping should have at least one ip ( v4 or v6 ) !');
|
||||
}
|
||||
|
||||
if(in_array($map['domain'], $domains))
|
||||
{
|
||||
Page::printApiResults(500,'Domains should be appearing only once in mapping !');
|
||||
}
|
||||
|
||||
$domains[] = $domain;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Create log directory if needed
|
||||
@mkdir(LOGS_PATH . '/installations', 0755, true);
|
||||
|
||||
// Clear previous logs
|
||||
$this->app->utils->terminal->cmd("> " . $logFile, Terminal::RETURN_NOTHING);
|
||||
$this->app->utils->terminal->cmd('echo "Installation Started !" > ' . $processFile, Terminal::RETURN_NOTHING);
|
||||
|
||||
// Launch with PID capture — use bash wrapper to get PID
|
||||
$api = 'sudo java -Dfile.encoding=UTF8 -jar ' . API_PATH . DS . 'iresponse_services.jar';
|
||||
$userId = intval(Authentication::getAuthenticatedUser()->getId());
|
||||
|
||||
$data = base64_encode(json_encode([
|
||||
'user-id' => $userId,
|
||||
'endpoint' => 'Servers',
|
||||
'action' => 'installServer',
|
||||
'parameters' => $parameters
|
||||
], JSON_UNESCAPED_UNICODE));
|
||||
|
||||
$cmd = "nohup {$api} {$data} > {$logFile} 2>&1 & echo \$!";
|
||||
$pid = trim(shell_exec($cmd) ?? '');
|
||||
|
||||
if(!empty($pid) && is_numeric($pid))
|
||||
{
|
||||
file_put_contents($pidFile, $pid);
|
||||
}
|
||||
|
||||
AuditLog::registerLog($serverId, $server['name'], 'MtaServer', 'Start Mta Servers Installation');
|
||||
|
||||
Page::printApiResults(200, 'Mta server installation started !', ['server-id' => $serverId]);
|
||||
}
|
||||
else
|
||||
{
|
||||
Page::printApiResults(500,'Incorrect server id !');
|
||||
}
|
||||
}
|
||||
104
patches/getInstallationLogs_fix.php
Normal file
104
patches/getInstallationLogs_fix.php
Normal file
@@ -0,0 +1,104 @@
|
||||
<?php
|
||||
/**
|
||||
* PATCH: getInstallationLogs — timeout detection + process liveness check
|
||||
*
|
||||
* Replace the getInstallationLogs method in:
|
||||
* app/webservices/Servers.php
|
||||
*
|
||||
* This fixes the "stuck Installing In Progress" bug by:
|
||||
* 1. Checking if the Java install process is still running (ps aux)
|
||||
* 2. Checking if the log file hasn't been modified for > INSTALL_TIMEOUT_MINUTES
|
||||
* 3. Auto-marking as "Installation interrupted !" when stale
|
||||
*/
|
||||
|
||||
/**
|
||||
* @name getInstallationLogs
|
||||
* @description get installation logs
|
||||
* @before init
|
||||
*/
|
||||
public function getInstallationLogs($parameters = [])
|
||||
{
|
||||
if(!Authentication::isUserAuthenticated())
|
||||
{
|
||||
Page::printApiResults(401,'Only logged-in access allowed !');
|
||||
}
|
||||
|
||||
Authentication::checkUserRoles();
|
||||
|
||||
$access = Permissions::checkForAuthorization(Authentication::getAuthenticatedUser(),'MtaServers','install');
|
||||
|
||||
if($access == false)
|
||||
{
|
||||
Page::printApiResults(403,'Access Denied !');
|
||||
}
|
||||
|
||||
$serverId = intval($this->app->utils->arrays->get($parameters,'server-id'));
|
||||
|
||||
if($serverId > 0)
|
||||
{
|
||||
$logFile = LOGS_PATH . '/installations/inst_' . $serverId . '.log';
|
||||
$processFile = LOGS_PATH . '/installations/inst_' . $serverId . '_proc.log';
|
||||
|
||||
$logs = (string)(file_exists($logFile) ? (shell_exec("cat " . $logFile) ?? '') : '');
|
||||
$procc = (string)(file_exists($processFile) ? trim((string)(shell_exec("cat " . $processFile) ?? '')) : '');
|
||||
|
||||
$timeoutMinutes = defined('INSTALL_TIMEOUT_MINUTES') ? INSTALL_TIMEOUT_MINUTES : 15;
|
||||
|
||||
if($procc !== 'Installation completed !' && $procc !== 'Installation interrupted !')
|
||||
{
|
||||
$isStale = false;
|
||||
$processAlive = false;
|
||||
|
||||
if(file_exists($logFile))
|
||||
{
|
||||
$lastModified = filemtime($logFile);
|
||||
$elapsed = time() - $lastModified;
|
||||
$isStale = ($elapsed > ($timeoutMinutes * 60));
|
||||
}
|
||||
else
|
||||
{
|
||||
if(file_exists($processFile))
|
||||
{
|
||||
$lastModified = filemtime($processFile);
|
||||
$elapsed = time() - $lastModified;
|
||||
$isStale = ($elapsed > ($timeoutMinutes * 60));
|
||||
}
|
||||
}
|
||||
|
||||
$psResult = shell_exec("ps aux | grep 'inst_" . $serverId . "' | grep -v grep 2>/dev/null");
|
||||
if(empty(trim($psResult ?? '')))
|
||||
{
|
||||
$psResult = shell_exec("ps aux | grep 'iresponse_services.jar' | grep -v grep 2>/dev/null");
|
||||
$b64Check = base64_encode(json_encode(['server-id' => $serverId]));
|
||||
if(strpos($psResult ?? '', (string)$serverId) === false)
|
||||
{
|
||||
$processAlive = false;
|
||||
}
|
||||
else
|
||||
{
|
||||
$processAlive = true;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
$processAlive = true;
|
||||
}
|
||||
|
||||
if(!$processAlive || $isStale)
|
||||
{
|
||||
$reason = $isStale ? "timeout ({$timeoutMinutes}min)" : "process not running";
|
||||
$logs .= "\n\n[AUTO-DETECT] Installation stalled — {$reason}. Auto-interrupting.\n";
|
||||
@mkdir(dirname($logFile), 0755, true);
|
||||
file_put_contents($logFile, $logs);
|
||||
file_put_contents($processFile, 'Installation interrupted !');
|
||||
$procc = 'Installation interrupted !';
|
||||
}
|
||||
}
|
||||
|
||||
Page::printApiResults(200,'',['logs' => str_replace(PHP_EOL,'<br/>', (string)$logs) , 'process' => $procc]);
|
||||
}
|
||||
else
|
||||
{
|
||||
Page::printApiResults(500,'Incorrect server id !');
|
||||
}
|
||||
}
|
||||
136
patches/installation_js_fix.js
Normal file
136
patches/installation_js_fix.js
Normal file
@@ -0,0 +1,136 @@
|
||||
/**
|
||||
* PATCH: handleInstallationLogs — add stall detection on the client side
|
||||
*
|
||||
* Replace handleInstallationLogs in:
|
||||
* public/scripts/pages/servers/installation.js
|
||||
*
|
||||
* Changes:
|
||||
* 1. Tracks poll count — if > MAX_POLLS without progress change, auto-interrupt
|
||||
* 2. Shows elapsed time in the status
|
||||
* 3. Better error handling on AJAX failure (don't silently hang)
|
||||
*/
|
||||
|
||||
var _installPollState = {};
|
||||
|
||||
var handleInstallationLogs = function(serverId, button)
|
||||
{
|
||||
if(!_installPollState[serverId])
|
||||
{
|
||||
_installPollState[serverId] = {
|
||||
startTime: Date.now(),
|
||||
lastProcess: '',
|
||||
staleCount: 0,
|
||||
maxStalePolls: 150 // 150 polls * 2s = 5 minutes with no change
|
||||
};
|
||||
}
|
||||
|
||||
var state = _installPollState[serverId];
|
||||
|
||||
var data =
|
||||
{
|
||||
'controller': 'Servers',
|
||||
'action': 'getInstallationLogs',
|
||||
'parameters':
|
||||
{
|
||||
'server-id': serverId
|
||||
}
|
||||
};
|
||||
|
||||
$.ajax({
|
||||
type: 'POST',
|
||||
url: iResponse.getBaseURL() + '/api.json',
|
||||
data: data,
|
||||
dataType: 'JSON',
|
||||
async: true, // Changed from false to true to avoid blocking
|
||||
timeout: 30000, // 30s timeout per poll (was 3600000)
|
||||
success: function(result)
|
||||
{
|
||||
if(result != false)
|
||||
{
|
||||
var status = result['status'];
|
||||
|
||||
if(status == 200)
|
||||
{
|
||||
$('#installation-log').html(result['data']['logs']);
|
||||
document.getElementById("installation-log").scrollTop = document.getElementById("installation-log").scrollHeight;
|
||||
|
||||
var process = result['data']['process'];
|
||||
var elapsed = Math.floor((Date.now() - state.startTime) / 1000);
|
||||
var elapsedStr = Math.floor(elapsed / 60) + 'm ' + (elapsed % 60) + 's';
|
||||
|
||||
if(process == "Installation completed !")
|
||||
{
|
||||
$('#installation-status').html('Installation completed ! (' + elapsedStr + ')');
|
||||
button.html("<i class='fa fa-magic'></i> Start Instalation");
|
||||
button.removeAttr('disabled');
|
||||
iResponse.alertBox({title: "Installation completed !", type: "success", allowOutsideClick: "true", confirmButtonClass: "btn-primary"});
|
||||
clearLogs(serverId);
|
||||
delete _installPollState[serverId];
|
||||
}
|
||||
else if(process == "Installation interrupted !")
|
||||
{
|
||||
$('#installation-status').html('Installation interrupted ! (' + elapsedStr + ')');
|
||||
button.html("<i class='fa fa-magic'></i> Start Instalation");
|
||||
button.removeAttr('disabled');
|
||||
iResponse.alertBox({title: "Installation interrupted !", type: "error", allowOutsideClick: "true", confirmButtonClass: "btn-danger"});
|
||||
clearLogs(serverId);
|
||||
delete _installPollState[serverId];
|
||||
}
|
||||
else
|
||||
{
|
||||
// Track stale progress
|
||||
if(process === state.lastProcess)
|
||||
{
|
||||
state.staleCount++;
|
||||
}
|
||||
else
|
||||
{
|
||||
state.staleCount = 0;
|
||||
state.lastProcess = process;
|
||||
}
|
||||
|
||||
if(state.staleCount >= state.maxStalePolls)
|
||||
{
|
||||
$('#installation-status').html('Installation timed out (no progress for 5min)');
|
||||
button.html("<i class='fa fa-magic'></i> Start Instalation");
|
||||
button.removeAttr('disabled');
|
||||
iResponse.alertBox({title: "Installation timed out — no progress detected for 5 minutes.", type: "error", allowOutsideClick: "true", confirmButtonClass: "btn-danger"});
|
||||
delete _installPollState[serverId];
|
||||
return;
|
||||
}
|
||||
|
||||
$('#installation-status').html(process + ' <i class="fa fa-spinner fa-spin"></i> (' + elapsedStr + ')');
|
||||
|
||||
setTimeout(function(){
|
||||
handleInstallationLogs(serverId, button);
|
||||
}, 2000);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
iResponse.alertBox({title: result['message'], type: "error", allowOutsideClick: "true", confirmButtonClass: "btn-danger"});
|
||||
delete _installPollState[serverId];
|
||||
}
|
||||
}
|
||||
},
|
||||
error: function (jqXHR, textStatus, errorThrown)
|
||||
{
|
||||
// On AJAX error, retry up to 3 times before giving up
|
||||
state.staleCount += 10;
|
||||
if(state.staleCount < state.maxStalePolls)
|
||||
{
|
||||
setTimeout(function(){
|
||||
handleInstallationLogs(serverId, button);
|
||||
}, 5000);
|
||||
}
|
||||
else
|
||||
{
|
||||
iResponse.alertBox({title: textStatus + ' : ' + errorThrown, type: "error", allowOutsideClick: "true", confirmButtonClass: "btn-danger"});
|
||||
button.html("<i class='fa fa-magic'></i> Start Instalation");
|
||||
button.removeAttr('disabled');
|
||||
$('#installation-status').html('Installation connection lost');
|
||||
delete _installPollState[serverId];
|
||||
}
|
||||
}
|
||||
});
|
||||
};
|
||||
61
patches/multiInstall_controller.php
Normal file
61
patches/multiInstall_controller.php
Normal file
@@ -0,0 +1,61 @@
|
||||
<?php
|
||||
/**
|
||||
* PATCH: Add multiInstall method to MtaServers controller
|
||||
*
|
||||
* Add this method in:
|
||||
* app/controllers/MtaServers.php (after the install() method)
|
||||
*
|
||||
* This provides the multi-install page at:
|
||||
* /mta-servers/multi-install/{id1-id2-id3-...}
|
||||
*
|
||||
* It launches installations sequentially (not all at once) to avoid
|
||||
* overloading the master server with too many Java processes.
|
||||
*/
|
||||
|
||||
/**
|
||||
* @name multiInstall
|
||||
* @description the multi install action
|
||||
* @before init
|
||||
* @after closeConnections,checkForMessage
|
||||
*/
|
||||
public function multiInstall()
|
||||
{
|
||||
# check for permissions
|
||||
$access = Permissions::checkForAuthorization($this->authenticatedUser,__CLASS__,'install');
|
||||
|
||||
if($access == false)
|
||||
{
|
||||
throw new PageException('Access Denied !',403);
|
||||
}
|
||||
|
||||
# set menu status
|
||||
$this->masterView->set([
|
||||
'servers_management' => 'true',
|
||||
'mta_servers' => 'true',
|
||||
'mta_servers_install' => 'true'
|
||||
]);
|
||||
|
||||
$arguments = func_get_args();
|
||||
$serverIdsStr = isset($arguments) && count($arguments) > 0 ? $arguments[0] : '';
|
||||
$serverIds = array_filter(array_map('intval', explode('-', $serverIdsStr)));
|
||||
|
||||
$servers = [];
|
||||
if(count($serverIds) > 0)
|
||||
{
|
||||
foreach($serverIds as $sid)
|
||||
{
|
||||
$srv = MtaServer::first(MtaServer::FETCH_ARRAY, ['id = ?', $sid]);
|
||||
if(count($srv) > 0)
|
||||
{
|
||||
$servers[] = $srv;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# set data to the page view
|
||||
$this->pageView->set([
|
||||
'serverIds' => $serverIds,
|
||||
'selectedServers' => $servers,
|
||||
'allServers' => MtaServer::all(MtaServer::FETCH_ARRAY, ['status = ?', 'Activated'], ['id','name','provider_name'], 'id', 'DESC')
|
||||
]);
|
||||
}
|
||||
54
patches/verifier-sentinel.sh
Normal file
54
patches/verifier-sentinel.sh
Normal file
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
# ---------------------------------------------------------------------------
|
||||
# verifier-sentinel.sh — Verification multi-install patch via Sentinel
|
||||
#
|
||||
# A executer via: curl -X POST "http://89.167.40.150:5890/api/sentinel-brain.php" \
|
||||
# --data-urlencode "action=exec" --data-urlencode "cmd=$(cat verifier-sentinel.sh)"
|
||||
#
|
||||
# Ou copier le bloc CMD ci-dessous dans Sentinel.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
echo "=== VERIFICATION PATCH MULTI-INSTALL S89 ==="
|
||||
echo ""
|
||||
|
||||
echo "1. Patch getInstallationLogs (timeout + null-safe):"
|
||||
for app in /opt/wevads /opt/fmgapp; do
|
||||
f="${app}/app/webservices/Servers.php"
|
||||
if [[ -f "$f" ]]; then
|
||||
timeout=$(grep -c 'INSTALL_TIMEOUT_MINUTES' "$f" 2>/dev/null || echo 0)
|
||||
nullsafe=$(grep -c '(string)($logs' "$f" 2>/dev/null || grep -c '(string)\$logs' "$f" 2>/dev/null || echo 0)
|
||||
autodetect=$(grep -c 'AUTO-DETECT' "$f" 2>/dev/null || echo 0)
|
||||
php_ok=$(php -l "$f" 2>&1 | grep -c "No syntax errors" || echo 0)
|
||||
echo " $app: timeout=$timeout nullsafe=$nullsafe autodetect=$autodetect php_ok=$php_ok"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "2. Fichiers proc stale (a nettoyer si > 0):"
|
||||
for logdir in /opt/wevads/logs /opt/fmgapp/logs; do
|
||||
if [[ -d "${logdir}/installations" ]]; then
|
||||
stale=0
|
||||
for f in "${logdir}"/installations/inst_*_proc.log; do
|
||||
[[ -f "$f" ]] || continue
|
||||
status=$(cat "$f" 2>/dev/null || true)
|
||||
if [[ "$status" != *"completed"* && "$status" != *"interrupted"* ]]; then
|
||||
mtime=$(stat -c %Y "$f" 2>/dev/null || echo 0)
|
||||
age=$(( ($(date +%s) - mtime) / 60 ))
|
||||
[[ $age -gt 10 ]] && ((stale++)) || true
|
||||
fi
|
||||
done
|
||||
echo " $logdir: $stale stale"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "3. Processus Java install en cours:"
|
||||
ps aux | grep -E 'iresponse_services|inst_[0-9]' | grep -v grep | wc -l
|
||||
|
||||
echo ""
|
||||
echo "4. Apache + PMTA:"
|
||||
systemctl is-active apache2 2>/dev/null || echo "apache2: unknown"
|
||||
systemctl is-active pmta 2>/dev/null || echo "pmta: unknown"
|
||||
|
||||
echo ""
|
||||
echo "=== FIN VERIFICATION ==="
|
||||
6
servers-bleu-multiinstall.csv
Normal file
6
servers-bleu-multiinstall.csv
Normal file
@@ -0,0 +1,6 @@
|
||||
server_id,server_name,ip,domain
|
||||
194,SERVER_5,176.52.138.42,mailpipe.net
|
||||
192,SERVER_3,110.238.65.222,mailforge.io
|
||||
191,SERVER_2,176.52.129.86,postengine.net
|
||||
193,SERVER_4,110.238.69.46,relaycore.com
|
||||
190,SERVER_1,176.52.132.94,mailforge.io
|
||||
|
Reference in New Issue
Block a user