Hi all,
So as I wrote on another topic our org uses EL8 Linux, previously mainly CentOS, now with the recent move of CentOS to the Stream model, we've opted for diversifying within the EL8 family. So we're using Red Hat Enterprise Linux for some servers, Oracle Linux for others, and we'll be deploying AlmaLinux for the new production web-servers. AlmaLinux is a EL8 distro from CloudLinux, who are very experienced with Linux specifically for web servers.
So we've deployed a new server just for aaPanel testing and I'll be presenting our issues here, in hope the aaPanel team will pick this up and fix so the whole community can enjoy aaPanel on AlmaLinux.
# cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.4 (Electric Cheetah)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.4 (Electric Cheetah)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:almalinux:almalinux:8.4:GA"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"
ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.4"
Before we move to the installation of aaPanel, we enable the PowerTools repository and install epel-release, iirc the installer also installs epel-release but we usually do this before:
# dnf config-manager --set-enabled powertools
# dnf install -y epel-release
AlmaLinux 8 - PowerTools 7.8 kB/s | 4.3 kB 00:00
Dependencies resolved.
=========================================================================================================================================================================
Package Architecture Version Repository Size
=========================================================================================================================================================================
Installing:
epel-release noarch 8-10.el8 extras 22 k
Transaction Summary
=========================================================================================================================================================================
Install 1 Package
Total download size: 22 k
Installed size: 32 k
Downloading Packages:
epel-release-8-10.el8.noarch.rpm 465 kB/s | 22 kB 00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 227 kB/s | 22 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : epel-release-8-10.el8.noarch 1/1
Running scriptlet: epel-release-8-10.el8.noarch 1/1
Verifying : epel-release-8-10.el8.noarch 1/1
Installed:
epel-release-8-10.el8.noarch
Complete!
Next, to the aaPanel installation using the new_install_en.sh. Since this is quite long I pasted the full output on pastebin:
Install output part 1
Install output part 2
Install output part 3
After the installation completes, everything seems fine. I can access the aaPanel, configure it, start installing apps.
So next I'm installing Apache, compiled method which we always use. I would like to put the full output of apache installation here, but unfortunately I can't find the log. The small part I still have access via aapanel GUI shows:
mkdir /www/server/apache/cgi-bin
Installing header files
Installing build system files
Installing man pages and online manual
mkdir /www/server/apache/man
mkdir /www/server/apache/man/man1
mkdir /www/server/apache/man/man8
mkdir /www/server/apache/manual
make[1]: Leaving directory '/www/server/apache/src'
--2021-06-04 13:22:29-- http://128.1.164.196/conf/httpd24.conf
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18385 (18K) [application/octet-stream]
Saving to: ‘/www/server/apache/conf/httpd.conf’
0K .......... ....... 100% 118K=0.2s
2021-06-04 13:22:30 (118 KB/s) - ‘/www/server/apache/conf/httpd.conf’ saved [18385/18385]
--2021-06-04 13:22:30-- http://128.1.164.196/conf/httpd-vhosts.conf
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2352 (2.3K) [application/octet-stream]
Saving to: ‘/www/server/apache/conf/extra/httpd-vhosts.conf’
0K .. 100% 294M=0s
2021-06-04 13:22:30 (294 MB/s) - ‘/www/server/apache/conf/extra/httpd-vhosts.conf’ saved [2352/2352]
--2021-06-04 13:22:30-- http://128.1.164.196/conf/httpd-default.conf
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2482 (2.4K) [application/octet-stream]
Saving to: ‘/www/server/apache/conf/extra/httpd-default.conf’
0K .. 100% 248M=0s
2021-06-04 13:22:30 (248 MB/s) - ‘/www/server/apache/conf/extra/httpd-default.conf’ saved [2482/2482]
--2021-06-04 13:22:30-- http://128.1.164.196/conf/mod_remoteip.conf
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 114 [application/octet-stream]
Saving to: ‘/www/server/apache/conf/extra/mod_remoteip.conf’
0K 100% 27.3M=0s
2021-06-04 13:22:31 (27.3 MB/s) - ‘/www/server/apache/conf/extra/mod_remoteip.conf’ saved [114/114]
--2021-06-04 13:22:31-- http://128.1.164.196/conf/httpd-mpm.conf
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4446 (4.3K) [application/octet-stream]
Saving to: ‘/www/server/apache/conf/extra/httpd-mpm.conf’
0K .... 100% 50.0M=0s
2021-06-04 13:22:31 (50.0 MB/s) - ‘/www/server/apache/conf/extra/httpd-mpm.conf’ saved [4446/4446]
--2021-06-04 13:22:31-- http://128.1.164.196/error/index.html
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1326 (1.3K) [text/html]
Saving to: ‘/www/server/apache/htdocs/index.html’
0K . 100% 256M=0s
2021-06-04 13:22:31 (256 MB/s) - ‘/www/server/apache/htdocs/index.html’ saved [1326/1326]
--2021-06-04 13:22:31-- http://128.1.164.196/conf/apache/en.0.default.conf
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 379 [application/octet-stream]
Saving to: ‘/www/server/panel/vhost/apache/0.default.conf’
0K 100% 53.0M=0s
2021-06-04 13:22:32 (53.0 MB/s) - ‘/www/server/panel/vhost/apache/0.default.conf’ saved [379/379]
--2021-06-04 13:22:32-- http://128.1.164.196/init/init.d.httpd
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [application/octet-stream]
Saving to: ‘/etc/init.d/httpd’
0K . 100% 393M=0s
2021-06-04 13:22:32 (393 MB/s) - ‘/etc/init.d/httpd’ saved [2020/2020]
start apache... done
--2021-06-04 13:22:32-- http://128.1.164.196/tools/check.sh
Connecting to 128.1.164.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 565 [application/octet-stream]
Saving to: ‘/www/server/panel/install/check.sh’
0K 100% 75.3M=0s
2021-06-04 13:22:32 (75.3 MB/s) - ‘/www/server/panel/install/check.sh’ saved [565/565]
And it completes successfully.
Major Issues start
As soon as Apache finishes installation, it killed the distro. If I try to install any other app it will fail, if I reboot this machine it won't come up.
If I run command rpm:
# rpm -qa
rpm: symbol lookup error: /lib64/librpmio.so.8: undefined symbol: EVP_md2, version OPENSSL_1_1_0
If I run the command dnf:
# dnf update
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/libdnf/error.py", line 14, in swig_import_helper
return importlib.import_module(mname)
File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
File "<frozen importlib._bootstrap>", line 571, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 922, in create_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/dnf", line 57, in <module>
from dnf.cli import main
File "/usr/lib/python3.6/site-packages/dnf/__init__.py", line 30, in <module>
import dnf.base
File "/usr/lib/python3.6/site-packages/dnf/base.py", line 29, in <module>
import libdnf.transaction
File "/usr/lib64/python3.6/site-packages/libdnf/__init__.py", line 8, in <module>
from . import error
File "/usr/lib64/python3.6/site-packages/libdnf/error.py", line 17, in <module>
_error = swig_import_helper()
File "/usr/lib64/python3.6/site-packages/libdnf/error.py", line 16, in swig_import_helper
return importlib.import_module('_error')
File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_error'
So basically, the apache installation script simply fucks up the OS.
Since it seems a library issue, let's see what libraries are on path:
# echo $LD_LIBRARY_PATH
#
Comes empty.
as libk5crypto is at /usr/lib64/ I can export that path:
# export LD_LIBRARY_PATH=/usr/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
# echo $LD_LIBRARY_PATH
/usr/lib64
# dnf update
Last metadata expiration check: 1:15:00 ago on Fri 04 Jun 2021 12:28:17 PM WEST.
Dependencies resolved.
Nothing to do.
Complete!
So actually the $LD_LIBRARY_PATH is also empty before installing aaPanel's apache, we just understand that aaPanel changes settings in a way that breaks the distro.
Adding the path seems like a quick fix (being seem a keyword). I added this export to /etc/environment but after reboot the machine never comes up and we're unable to even login via console as it keeps throwing errors. This is a major issue as it completely breaks the linux installation.
So we kindly ask the aaPanel team to take a look into this issue. I would provide the full Apache install log if I could find it, but the steps are quite simple so I'm sure you can recreate this easily. Thank you
EDIT:
Using the "Fast" method also breaks Linux.
Running ldconfig -p BEFORE and AFTER shows several libraries have been overriden with custom outdated versions of themselves, which is breaking this all. The first line is after running any aaPanel installation, the second line is from before
libssl.so.1.1 (libc6,x86-64) => /usr/local/openssl111/lib/libssl.so.1.1
libssl.so.1.1 (libc6,x86-64) => /lib64/libssl.so.1.1
libssl.so (libc6,x86-64) => /usr/local/openssl111/lib/libssl.so
libssl.so (libc6,x86-64) => /usr/local/openssl/lib/libssl.so
libcrypto.so.1.1 (libc6,x86-64) => /usr/local/openssl111/lib/libcrypto.so.1.1
libcrypto.so.1.1 (libc6,x86-64) => /lib64/libcrypto.so.1.1
libcrypto.so (libc6,x86-64) => /usr/local/openssl111/lib/libcrypto.so
libcrypto.so (libc6,x86-64) => /usr/local/openssl/lib/libcrypto.so
You can click on BEFORE and AFTER above to see the full output of ldconfig.
# ls -la /usr/local/openssl111/lib/libcrypto.so
lrwxrwxrwx 1 root root 16 Jun 4 14:02 /usr/local/openssl111/lib/libcrypto.so -> libcrypto.so.1.1
# ls -la /lib64/libcrypto.so.1.1
lrwxrwxrwx. 1 root root 19 Mar 30 18:19 /lib64/libcrypto.so.1.1 -> libcrypto.so.1.1.1g
The system uses newer versions of these libraries. aaPanel shouldn't try to replace these, it should link to the existing libraries.
Installation of anything after fails:
cmake: symbol lookup error: /lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
AlmaLinux release 8.4 (Electric Cheetah)
Bit:64 Mem:3715M Core:2 gcc:8.4.1 cmake:
Linux 4.18.0-305.3.1.el8_4.x86_64 x86_64
ERROR: php-7.4.19 install failed.
