NetDRMS Installation » History » Version 353
Version 352 (Pablo Alingery, 31/01/2017 18:03) → Version 353/369 (Pablo Alingery, 01/02/2017 10:55)
{{toc}}
h1. NetDRMS 8.12 Installation
h2. Preliminary note
In the following sections, when there is a command line :
<pre>#</pre>means that command is executed into _root_ user
<pre>$</pre>means that command is executed into _production_ user
h2. Environment
Server : DELL PowerEdge R730
RAM : 64.00GB
CPU : 32 CPUs Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Storage Configuration :
* 264 GB mounted in / (2 X 278.88 GB SAS 12G disks in RAID1)
* 1,7 TB mounted in /var/lib/postgresql (7 X 278.88 GB SAS 12G disks in RAID5 configuration)
* 1 global spare for both RAID
* DDP storage mounted in /SUM*
Guest OS : Debian 8.4
Kernel : Linux sdo-new 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) x86_64 GNU/Linux
h2. Requirements
h3. Configuring production environment
Adding user production
<pre>
# adduser production
# addgroup ias
# adduser production ias
</pre>
h3. Configure /home/production/.profile
Adding environment variables
<pre>
#Modif Herve.B Pablo.A 12/05/2016
PATH="/usr/local/netdrms_current/bin/linux_x86_64:$PATH"
PATH="/usr/local/netdrms_current/scripts:$PATH"
PATH="/usr/local/jmd/bin:$PATH"
PATH="/usr/local/jmd/scripts:$PATH"
#The following is not needed as installed with 'apt-get install'
#PATH="/usr/local/pgsql/bin:$PATH"
PATH="/usr/local/netdrms-tools/scripts:$PATH"
#2014-12-17 Avec Herve et Pablo on pense que la ligne suivante est intutile as it is f.. install in /usr/local/lib
PATH="/usr/local/cfitsio:$PATH"
#Done
DERBY_HOME=/usr/local/jmd/databases/derby/derbyBD
export DERBY_HOME
export CVSROOT=:pserver:anonymous@solarch.tuc.noao.edu:2401/vtarc1/vso/cvsroot
</pre>
h3. Configure /varl/lib/postgres/.profile
Add to $PATH so postgres can exeute command like initdb
<pre>
#Modif Herve.B Pablo.A 19/05/2016
PATH="/usr/lib/postgresql/9.4/bin:$PATH"
</pre>
h3. Setting privileges for SUMS_MANAGER
Please modify file /etc/sudoers so user 'production' will be able to execute 'sum_chown' that we will locate in the dir : /usr/local/bin . See below
Add to /etc/sudoers the following line :
production host=NOPASSWD:/usr/local/bin/sum_chmown
h3. Set passwords file for db postgres
Please create file .pgpass with the following information for user 'production" so he will be able to connect without password as 'production' or 'postgres' to 2 db that we will create later ias_sdo and ias_sdo_sums.
.pgpass content :
<pre>
#hostname:5432:ias_sdo:production:
#hostname:5434:ias_sdo_sums:production:
#hostname:5432:ias_sdo:postgres:
#hostname:5434:ias_sdo_sums:postgres
</pre>
where #hostname is the name of your server
%{color:red}WARNING : change hostname when migrate sdo-new to sdo%
h3. Install missing libraries
* libreadline-dev (but already installed with postgresql)
* libssl-dev
* libpam0g-dev
* libperl-dev
* libnet-ssh-perl
* libstring-shellquote-perl
* libdbi-perl
* libdbd-pg-perl
NB : Already done in Debian 8.4 Jessy
<pre>
# apt-get install $library_name
</pre>
where
_$library_name_ is the name of library
Then to update path on libraries, please type :
<pre>
# ldconfig
</pre>
h3. Reconfigure locale
<pre>
# dpkg-reconfigure locales
</pre>
select fr_FR.UTF8 UTF8 and unselect en_US.UTF-8
localedef -i en_US -f UTF-8 en_US.UTF-8
(see here : [[http://www.thomas-krenn.com/en/wiki/Perl_warning_Setting_locale_failed_in_Debian]])
h3. Install gfortran packages
<pre>
# apt-get install gfortran swig
</pre>
h3. Installation of cfitsio library
In /usr/local dir
Version 3.39 (currently on sdo : 3.24)
<pre>
# wget ftp://heasarc.gsfc.nasa.gov/software/fitsio/c/cfitsio3390.tar.gz
</pre>
Untar and decompression
<pre>
# tar -xzvf cfitsio3390.tar.gz
</pre>
Go into /usr/local and type
su - root and :
<pre>
# ./configure --prefix=/usr/local
# make
# make install
</pre>
h3. Installation des libraries libtar-dev, libcurl3-dev et libecpg-dev
<pre>
apt-get install libtar libtar-dev
</pre>
Note, selecting 'libcurl4-openssl-dev' instead of 'libcurl3-dev'
libcurl4-openssl-dev is already the newest version
<pre>
apt-get install libcurl3-dev
</pre>
<pre>
apt-get install libecpg-dev
</pre>
h3. Installation of apache2 and lib perl
<pre>
# apt-get install apache2
# apt-get install libjson-perl
# apt-get install libapache2-mod-perl2
</pre>
h3. Installation of postgresql
Into 'root'
<pre>
apt-get install postgresql-9.4 postgresql-client-9.4
</pre>
h3. Installation of python3
Into 'root'
<pre>
apt-get install python3
</pre>
Add some modules psycopg2 (postgres connector) and pySmartDL (Download manager)
<pre>
apt-get install python3-psycopg2
</pre>
<pre>
apt-get install python3-psycopg2
</pre>
Add python3-pip for modules within python3
<pre>
apt-get install python3-pip
</pre>
<pre>
pip3 install pySmartDL
</pre>
Both python2.7 and 3.4 versions are installed on the system.
By default, the python2.7 is used.
In order to change in 3.4 by default, type :
<pre>
update-alternatives --config python
</pre>
h3. Installation of sdo_scripts
Install monitoring scripts for sdo
<pre>
rsync -av scripts production@sdo-new:/home/production/netdrms-tools/
</pre>
h3. lib64 symbolic link
Creating lib64 link (lib64 does not exist on debian 8)
<pre>
# cd /usr
# ln -s lib/ lib64
</pre>
h2. NetDRMS databases
into /var/lib/postgresql (hard mount point separated, 7 disks 15000 tr/min in raid5 in order to distribute read and write)
<pre>
# cd /var/lib/postgresql
# mkdir data data_sums data_monitor
# chown -R postgres:postgres *
</pre>
h3. Initialize 3 servers data, data_sums, data_monitor
Into postgres
<pre>
# su - postgres
$ initdb --locale=C -D data -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data
or
pg_ctl -D data -l logfile start
$ initdb --locale=C -D data_sums -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data_sums
or
pg_ctl -D data_sums -l logfile start
$ initdb --locale=C -D data_monitor -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data_monitor
or
pg_ctl -D data_monitor -l logfile start
</pre>
Edition des fichiers postgresql.conf pour configurer les bons ports d'écoute :
listen_addresses = '*'
data -> port 5432
data_sums -> port 5434
data_monitor -> port 5436
Edit files data*/pg_hba.conf : replace 'md5' by 'trust'
<pre>
#"local"
local all all trust
#"IPv4"
host all all trust
</pre>
h3. Tunning postgres
In file postgres.conf ( according to https://www.qwant.com/?q=Tunning+postgres+server+9.4+&client=firefox )
shared_buffer=4096MB %{color:red}So 1/12 of the total RAM memory available%
work_mem = 24MB
maintenance_work_mem = 4096MB
wal_buffers = 16MB
checkpoint_segments = 32
checkpoint_completion_target = 0.9
random_page_cost = 1.0
effective_cache_size = 16384MB so 1/4 of the max memory
#log
logging_collector = on
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,
log_truncate_on_rotation = on # If on, an existing log file with the
log_rotation_age = 1d # Automatic rotation of logfiles will
client_min_messages = error
log_min_messages = error
log_min_duration_statement = 0
log_connections = on
log_disconnections = on
log_duration = off
log_hostname = on
log_line_prefix = '%t [%p]: %u@%h - %d :'
track_counts = on
autovacuum = on
extra_float_digits = 3
h3. Start the 3 databases :
<pre>
# su - postgres
$ cd /var/lib/postgresql
$ pg_ctl -D data start
$ pg_ctl -D data_sums start
$ pg_ctl -D data_monitor start
</pre>
h3. Create the 3 databases :
Into user 'postgres'
* createdb --locale C -E LATIN1 -T template0 ias_sdo
* createdb --locale C -E LATIN1 -T template0 -p 5434 ias_sdo_sums
* createdb --locale C -E LATIN1 -T template0 -p 5436 ias_sdo_monitor
h3. Automatic startup
Add following lines in the /etc/rc.local file :
<pre>
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data start" postgres
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_sums start" postgres
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_monitor start" postgres
</pre>
h3. Install additional modules
We need dblink for triggers between db so let's install postgresql-contrib-9.4
<pre>
root@sdo-new:~# apt-get install postgresql-contrib-9.4
</pre>
Create extension dblink for each db
ias_sdo
<pre>
root@sdo-new:$ psql ias_sdo -p5432 -U postgres
psql (9.4.6)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
CREATE EXTENSION
</pre>
ias_sdo_sums
<pre>
root@sdo-new:$ psql ias_sdo -p5434 -U postgres
psql (9.4.6)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
CREATE EXTENSION
</pre>
ias_sdo_monitor
<pre>
root@sdo-new:$ psql ias_sdo -p5436 -U postgres
psql (9.4.6)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
CREATE EXTENSION
</pre>
To check the list of installed extensions type :
<pre>
ias_sdo=\dx
List of installed extensions
Name | Version | Schema | Description
---------+---------+------------+--------------------------------------------------------------
dblink | 1.1 | public | connect to other PostgreSQL databases from within a database
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
(2 rows)
</pre>
h2. System tuning
Current configuration :
<pre>
# sysctl -a
</pre>
In the following file :
<pre>
# vi /etc/sysctl.conf
</pre>
we change some values (default values are commented with '#')
(resources links [[Links|here]])
<pre>
# Semaphore is a object that is used to control utilization of a particular process.
# kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI
# SEMMSL maximum number of semaphores per array (min 128)
# SEMMNS maximum semaphores system-wide
# SEMOPM maximum operations per semop call
# SEMMNI maximum arrays
#semop incresed from 32 (default value)operation to 100 (3rd value )
# kernel.sem = 250 32000 32 128
kernel.sem = 250 32000 100 128
In order to apply changes :#This sets the OS receive buffer size for all types of connections (default value 212992)
#net.core.rmem_default = 212992
#net.core.rmem_max = 212992
net.core.rmem_default = 33554432
net.core.rmem_max = 33554432
#This is the OS send buffer size for all types of connections (212992 default value)
#net.core.wmem_default = 212992
#net.core.wmem_max = 212992
net.core.wmem_default = 33554432
net.core.wmem_max = 33554432
#TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always
# allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive
# buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ...
# The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."
#net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_rmem = 10240 87380 33554432
#TCP Autotuning setting. "This variable takes 3 different values which holds information on how much TCP sendbuffer memory space
# each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three
#values are used under different conditions. ... The first value in this variable tells the minimum TCP send buffer space available
# for a single TCP socket. ... The second value in the variable tells us the default buffer space allowed for a single TCP socket to use. ...
#The third value tells the kernel the maximum TCP send buffer space."
#net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_wmem = 10240 87380 33554432
#Disable cache metrics so the initial conditions of the closed connections will not be saved to be used in near future connections
#net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_no_metrics_save = 1
# Increase number of incoming connections backlog
# net.core.netdev_max_backlog = 1000
net.core.netdev_max_backlog = 5000
# The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage. ...
#The first value specified in the tcp_mem variable tells the kernel the low threshold.
#Below this point, the TCP stack do not bother at all about putting any pressure on the memory usage by different TCP sockets. ...
#The second value tells the kernel at which point to start pressuring memory usage down. ... The final value tells the kernel how many
# memory pages it may use maximally
# net.ipv4.tcp_mem = 1543347 2057796 3086694
net.ipv4.tcp_mem = 786432 1048576 26777216
# local port range that is used by TCP and UDP traffic to choose the local port. You will see in the parameters of this file two numbers:
# The first number is the first local port allowed for TCP and UDP traffic on the server, the second is the last local port number.
# net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.ip_local_port_range = 1024 65535
# maximum number of sockets in TIME-WAIT to be held simultaneously.
# net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_max_tw_buckets = 360000
</pre>
Then we run the following command to make our change take effect:
<pre>
# sysctl -p
</pre>
h2. JMD Installation
Download new JMD package build by Niles Oien 2016-04-07 following thoses actions :
<pre>
cd /urs/local
[oien@spsc-nso19-12 ~]$ ftp gong2.nso.edu
Name (gong2.nso.edu:oien): anonymous
Password: <---------------- use your email as the password
ftp> cd outgoing/oien
ftp> get pablo_jmd.tar.gz
ftp> quit
</pre>
Install new JMD package following thoses actions
<pre>
[oien@spsc-nso19-12 ~]$ gunzip -vf pablo_jmd.tar.gz
pablo_jmd.tar.gz: 8.3% -- replaced with pablo_jmd.tar
[oien@spsc-nso19-12 ~]$ tar xf pablo_jmd.tar
[oien@spsc-nso19-12 ~]$ cd pablo_jmd
[oien@spsc-nso19-12 pablo_jmd]$ less README.txt
cd ..
mv jmd jmd_old
mv pablo_jmd jmd
</pre>
url provided by jsoc to download last JMD version http://vso.tuc.noao.edu/VSO/downloads/JMD
h2. Apache Configuration for JMD
Configure webserver to request jsoc_fetch
(following the mail from nilesoien@gmail.com)
%{color:red}WARNING : change hostname when migrate sdo-new to sdo%
config d 'apache2 fichier sdo-new.ias.u-psud.fr
<pre>
<VirtualHost *:80>
ServerName sdo-new.ias.u-psud.fr
ServerAdmin pablo.alingery@ias.u-psud.fr
#ErrorLog /var/log/apache2/error.log
DocumentRoot /var/www
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel debug
#CustomLog /var/log/apache2/VSO/access.log combined
Alias /VSO/DRMS/cgi-bin/ "/home/production/netdrms-tools/scripts/cgi-bin/"
<Location /VSO/DRMS/cgi-bin>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Order allow,deny
Allow from all
</Location>
</VirtualHost>
</pre>
Activate the new virtualhost and disable the default one :
<pre>
# a2ensite sdo-new.ias.u-psud.fr.conf
# a2dissite 000-default.conf
# service apache2 restart
</pre>
Allow symlink for the cgi scripts dierctory in apache2.conf :
<pre>
<Directory /home/production/netdrms-tools/scripts/>
Options FollowSymLinks
AllowOverride None
Require all granted
</Directory>
</pre>
<pre>
# service apache2 restart
</pre>
%{color:red}To be added futher to test cgi%
Execution test cgi with URL :
make sure my $hostname=in the cgi file vso_jsoc_fetch.cgi is correct and test
http://sdo-new.ias.u-psud.fr/VSO/DRMS/cgi-bin/vso_jsoc_fetch.cgi
expected result :
<pre>
{
"wait" : 0,
"requestid" : "",
"data" : {},
"method" : "url_quick",
"size" : 0,
"errormsg" : "Empty query",
"protocol" : "as-is",
"status" : 1,
"space_ratio" : 0.1,
"load_ratio" : 0.066875,
"dir" : "",
"count" : 0
}
</pre>
h2. JMD configuration
Modify the file [[/usr/local/jmd/cfg/JMD.cfg]]
%{color:red}Warning : when migrate production server, change _sdo-new_ by _sdo_%
h2. NetDRMS tool Install
The following installed is based on : [[http://inf-redmine.ias.u-psud.fr/redmine/attachments/download/114/Netdrms_First_Install.odt]]
The sources are in : http://jsoc.stanford.edu/netdrms/dist/
Locally on sdo-new in : /home/production/sources
Symbolic link created 'netdrms_current' that corresponds to the last release
<pre>
ln -s netdrms_8.12/ netdrms_current
</pre>
Edit file config.local
<pre>
cp config.local.template config.local
</pre>
%{color:red}Warning WEB_DBUSER apache we think that it should value www maybe , to be checked testing the cgi%
%{color:red}So check the cgi to figure out which user is used to connect to bd%
Create 2 followings dirs :
<pre>
mkdir /usr/local/netdrms_8.12/logs/SUM -p
mkdir /usr/local/netdrms_8.12/logs/slony -p
</pre>
%{color:red}Warning config.local from line 150 =>222 Remote config to provide data, series to externals%
%{color:red}To be checked for PLATO%
Création d'un alias netdrms dans /root/.bashrc
<pre>
alias netdrms='cd /usr/local/netdrms_current'
</pre>
h3. Prepare compiling
Into _production_ profile
Replace all " $? !=0 " by " $status !=0 " in file configure then tape
<pre>
$ ./configure
</pre>
Modification du fichier /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.12 )
ligne 28 :
case "x86_64":
echo linux_x86_64
breaksw
Overwriting the existing files in the right place (base/sums/libs/pg) & modif in jsoc_machine.csh
=> the command 'make' works well.
h3. Compilation
Dans le fichier make_basic.mk ligne 223
ajouter "-lcrypto"
<pre>
$ make
$ make sums
</pre>
h2. SSH-HPN install
Intallation de openssh 7.2p2 car le dernier patch hpn dispo est 7.2hpn14.V10
web site for open ssh : http://ftp2.fr.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.2p2.tar.gz
patch hpn : https://sourceforge.net/projects/hpnssh/files/HPN-SSH%2014v10%207.2p2/openssh-7_2_P2-hpn-14.10.diff
In the dir /home/production/sources into 'root'
<pre>
# tar -xzvf openssh-7.2p2.tar.gz
# cd openssh-7.2p2/
# cat ../openssh-7_2_P2-hpn-14.10.diff |patch -p1
# ./configure --prefix=/usr/local/hpn-ssh --with-pam --with-md5-passwords --without-zlib-version-check --with-tcp-wrappers
# make
# make install
</pre>
Depuis sdo :
<pre>
# cd /usr/local/hpn-ssh/etc/
# rsync -av *key* production@sdo-new:/home/production/
</pre>
Following the link : http://vso.stanford.edu/netdrms/rmtsums.html
<pre>
$ cd /home/production/
$ ssh-agent -c > ~/.ssh-agent_rs
</pre>
NB : correct file ~/.ssh-agent_rs it should look like :
<pre>
#!/bin/csh
export SSH_AUTH_SOCK=/tmp/ssh-9POrTXobhLR4/agent.74272;
export SSH_AGENT_PID=74273;
echo Agent pid 74273;
</pre>
<pre>
$ source ~/.ssh-agent_rs
$ ssh-add ~/.ssh/id_rsa
</pre>
Sur sdo-new :
<pre>
# mv /home/production/*key* /usr/local/hpn-ssh/etc/
</pre>
We have to configure the port number to 55000
<pre>
# vi ssh_config
</pre>
and add :
<pre>
#Port 22
Port 55000
</pre>
We configure also the server hpn (even we don't use it yet)
<pre>
# vi sshd_config
</pre>
and add :
<pre>
#Port 22
Port 55000
#PidFile /var/run/sshd.pid
PidFile /var/run/sshd.55000.pid
# allow the use of the none cipher
#NoneEnabled no
NoneEnabled yes
</pre>
In order to force the use of SSH-hpn by JMD :
<pre>
$ vi /usr/local/jmd/cfg/JMD.cfg
</pre>
and replace the path of scp binary :
<pre>
SCPCommand=/usr/local/hpn-ssh/bin/scp -o NoneSwitch=yes -o NoneEnabled=yes
</pre>
h2. Private key Public key generation
JSOC need your public key to directly connect to your netdrms server. In order to generate your RSA public/private key pair :
<pre>
$ ssh-keygen -t rsa
</pre>
Not necessary in our case cause we recover the ssh keys of the current server 'sdo'.
However, we have to retrieve the private key of production account on sdo-new
From sdo :
<pre>
rsync -av id_rsa production@sdo-new:/home/production/.ssh/
</pre>
In order to test the connection :
<pre>
$ ssh jsocexp@jsocport.stanford.edu -p55000
</pre>
Note : the IP of the server has to be declared to the jsoc
h2. NetDRMS Replication Config
%{color:red}To be check if we keep that here or not%
To prevent from deleting files
<pre>
cd logs/SUM ; scp production@sdo:/usr/local/netdrms/install-config-files/sum_rm.cfg .
</pre>
Create tmp dir in netdrms working directory (/usr/local/netdrms_current)
<pre>
$ mkdir tmp
</pre>
Create config file from template
<pre>
$ cd base/drms/replication/etc
cp repclient.template.cfg ias.repclient.cfg
</pre>
Define node in file : [[/usr/local/netdrms_current/base/drms/replication/etc/ias.repclient.cfg]]
In our case :
<pre>
node=IASprod
</pre>
There are 4 new fields in the new version of repclient (see Art mail June 11, 2016)
<pre>
#kRSBaseURL=<base URL for all Slony services>
kRSBaseURL=http://jsoc.stanford.edu/cgi-bin/ajax
#kSubService=<URL of the subscription service>
kSubService=${kRSBaseURL}/request-subs.py
#kPubListService=<URL of the publication-list service>
kPubListService=${kRSBaseURL}/publist.py
#kSubXfer=<URL of the file transfer directory>
kSubXfer=http://jsoc.stanford.edu/subscription
</pre>
h2. NetDRMS Subsribe list Config
Create config subscribe file from template
<pre>
$ cd base/drms/replication/etc
cp subscribe_list.template.cfg ias.subscribe_list.cfg
</pre>
h2. TO BE DONE BEFORE MIGRATION
%{color:red} configure sytem taille buffer%
%{color:red} swap désactivé ? discussion Stephane et Gilles%
%{color:red}Test ssh connection to JSOC , NSO CFA before subcription and just after name sdo-new into sdo%
%{color:red}Take old counter and place it in the same dir before migration%
h1. NetDRMS 8.12 Installation
h2. Preliminary note
In the following sections, when there is a command line :
<pre>#</pre>means that command is executed into _root_ user
<pre>$</pre>means that command is executed into _production_ user
h2. Environment
Server : DELL PowerEdge R730
RAM : 64.00GB
CPU : 32 CPUs Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Storage Configuration :
* 264 GB mounted in / (2 X 278.88 GB SAS 12G disks in RAID1)
* 1,7 TB mounted in /var/lib/postgresql (7 X 278.88 GB SAS 12G disks in RAID5 configuration)
* 1 global spare for both RAID
* DDP storage mounted in /SUM*
Guest OS : Debian 8.4
Kernel : Linux sdo-new 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) x86_64 GNU/Linux
h2. Requirements
h3. Configuring production environment
Adding user production
<pre>
# adduser production
# addgroup ias
# adduser production ias
</pre>
h3. Configure /home/production/.profile
Adding environment variables
<pre>
#Modif Herve.B Pablo.A 12/05/2016
PATH="/usr/local/netdrms_current/bin/linux_x86_64:$PATH"
PATH="/usr/local/netdrms_current/scripts:$PATH"
PATH="/usr/local/jmd/bin:$PATH"
PATH="/usr/local/jmd/scripts:$PATH"
#The following is not needed as installed with 'apt-get install'
#PATH="/usr/local/pgsql/bin:$PATH"
PATH="/usr/local/netdrms-tools/scripts:$PATH"
#2014-12-17 Avec Herve et Pablo on pense que la ligne suivante est intutile as it is f.. install in /usr/local/lib
PATH="/usr/local/cfitsio:$PATH"
#Done
DERBY_HOME=/usr/local/jmd/databases/derby/derbyBD
export DERBY_HOME
export CVSROOT=:pserver:anonymous@solarch.tuc.noao.edu:2401/vtarc1/vso/cvsroot
</pre>
h3. Configure /varl/lib/postgres/.profile
Add to $PATH so postgres can exeute command like initdb
<pre>
#Modif Herve.B Pablo.A 19/05/2016
PATH="/usr/lib/postgresql/9.4/bin:$PATH"
</pre>
h3. Setting privileges for SUMS_MANAGER
Please modify file /etc/sudoers so user 'production' will be able to execute 'sum_chown' that we will locate in the dir : /usr/local/bin . See below
Add to /etc/sudoers the following line :
production host=NOPASSWD:/usr/local/bin/sum_chmown
h3. Set passwords file for db postgres
Please create file .pgpass with the following information for user 'production" so he will be able to connect without password as 'production' or 'postgres' to 2 db that we will create later ias_sdo and ias_sdo_sums.
.pgpass content :
<pre>
#hostname:5432:ias_sdo:production:
#hostname:5434:ias_sdo_sums:production:
#hostname:5432:ias_sdo:postgres:
#hostname:5434:ias_sdo_sums:postgres
</pre>
where #hostname is the name of your server
%{color:red}WARNING : change hostname when migrate sdo-new to sdo%
h3. Install missing libraries
* libreadline-dev (but already installed with postgresql)
* libssl-dev
* libpam0g-dev
* libperl-dev
* libnet-ssh-perl
* libstring-shellquote-perl
* libdbi-perl
* libdbd-pg-perl
NB : Already done in Debian 8.4 Jessy
<pre>
# apt-get install $library_name
</pre>
where
_$library_name_ is the name of library
Then to update path on libraries, please type :
<pre>
# ldconfig
</pre>
h3. Reconfigure locale
<pre>
# dpkg-reconfigure locales
</pre>
select fr_FR.UTF8 UTF8 and unselect en_US.UTF-8
localedef -i en_US -f UTF-8 en_US.UTF-8
(see here : [[http://www.thomas-krenn.com/en/wiki/Perl_warning_Setting_locale_failed_in_Debian]])
h3. Install gfortran packages
<pre>
# apt-get install gfortran swig
</pre>
h3. Installation of cfitsio library
In /usr/local dir
Version 3.39 (currently on sdo : 3.24)
<pre>
# wget ftp://heasarc.gsfc.nasa.gov/software/fitsio/c/cfitsio3390.tar.gz
</pre>
Untar and decompression
<pre>
# tar -xzvf cfitsio3390.tar.gz
</pre>
Go into /usr/local and type
su - root and :
<pre>
# ./configure --prefix=/usr/local
# make
# make install
</pre>
h3. Installation des libraries libtar-dev, libcurl3-dev et libecpg-dev
<pre>
apt-get install libtar libtar-dev
</pre>
Note, selecting 'libcurl4-openssl-dev' instead of 'libcurl3-dev'
libcurl4-openssl-dev is already the newest version
<pre>
apt-get install libcurl3-dev
</pre>
<pre>
apt-get install libecpg-dev
</pre>
h3. Installation of apache2 and lib perl
<pre>
# apt-get install apache2
# apt-get install libjson-perl
# apt-get install libapache2-mod-perl2
</pre>
h3. Installation of postgresql
Into 'root'
<pre>
apt-get install postgresql-9.4 postgresql-client-9.4
</pre>
h3. Installation of python3
Into 'root'
<pre>
apt-get install python3
</pre>
Add some modules psycopg2 (postgres connector) and pySmartDL (Download manager)
<pre>
apt-get install python3-psycopg2
</pre>
<pre>
apt-get install python3-psycopg2
</pre>
Add python3-pip for modules within python3
<pre>
apt-get install python3-pip
</pre>
<pre>
pip3 install pySmartDL
</pre>
Both python2.7 and 3.4 versions are installed on the system.
By default, the python2.7 is used.
In order to change in 3.4 by default, type :
<pre>
update-alternatives --config python
</pre>
h3. Installation of sdo_scripts
Install monitoring scripts for sdo
<pre>
rsync -av scripts production@sdo-new:/home/production/netdrms-tools/
</pre>
h3. lib64 symbolic link
Creating lib64 link (lib64 does not exist on debian 8)
<pre>
# cd /usr
# ln -s lib/ lib64
</pre>
h2. NetDRMS databases
into /var/lib/postgresql (hard mount point separated, 7 disks 15000 tr/min in raid5 in order to distribute read and write)
<pre>
# cd /var/lib/postgresql
# mkdir data data_sums data_monitor
# chown -R postgres:postgres *
</pre>
h3. Initialize 3 servers data, data_sums, data_monitor
Into postgres
<pre>
# su - postgres
$ initdb --locale=C -D data -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data
or
pg_ctl -D data -l logfile start
$ initdb --locale=C -D data_sums -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data_sums
or
pg_ctl -D data_sums -l logfile start
$ initdb --locale=C -D data_monitor -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data_monitor
or
pg_ctl -D data_monitor -l logfile start
</pre>
Edition des fichiers postgresql.conf pour configurer les bons ports d'écoute :
listen_addresses = '*'
data -> port 5432
data_sums -> port 5434
data_monitor -> port 5436
Edit files data*/pg_hba.conf : replace 'md5' by 'trust'
<pre>
#"local"
local all all trust
#"IPv4"
host all all trust
</pre>
h3. Tunning postgres
In file postgres.conf ( according to https://www.qwant.com/?q=Tunning+postgres+server+9.4+&client=firefox )
shared_buffer=4096MB %{color:red}So 1/12 of the total RAM memory available%
work_mem = 24MB
maintenance_work_mem = 4096MB
wal_buffers = 16MB
checkpoint_segments = 32
checkpoint_completion_target = 0.9
random_page_cost = 1.0
effective_cache_size = 16384MB so 1/4 of the max memory
#log
logging_collector = on
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,
log_truncate_on_rotation = on # If on, an existing log file with the
log_rotation_age = 1d # Automatic rotation of logfiles will
client_min_messages = error
log_min_messages = error
log_min_duration_statement = 0
log_connections = on
log_disconnections = on
log_duration = off
log_hostname = on
log_line_prefix = '%t [%p]: %u@%h - %d :'
track_counts = on
autovacuum = on
extra_float_digits = 3
h3. Start the 3 databases :
<pre>
# su - postgres
$ cd /var/lib/postgresql
$ pg_ctl -D data start
$ pg_ctl -D data_sums start
$ pg_ctl -D data_monitor start
</pre>
h3. Create the 3 databases :
Into user 'postgres'
* createdb --locale C -E LATIN1 -T template0 ias_sdo
* createdb --locale C -E LATIN1 -T template0 -p 5434 ias_sdo_sums
* createdb --locale C -E LATIN1 -T template0 -p 5436 ias_sdo_monitor
h3. Automatic startup
Add following lines in the /etc/rc.local file :
<pre>
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data start" postgres
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_sums start" postgres
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_monitor start" postgres
</pre>
h3. Install additional modules
We need dblink for triggers between db so let's install postgresql-contrib-9.4
<pre>
root@sdo-new:~# apt-get install postgresql-contrib-9.4
</pre>
Create extension dblink for each db
ias_sdo
<pre>
root@sdo-new:$ psql ias_sdo -p5432 -U postgres
psql (9.4.6)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
CREATE EXTENSION
</pre>
ias_sdo_sums
<pre>
root@sdo-new:$ psql ias_sdo -p5434 -U postgres
psql (9.4.6)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
CREATE EXTENSION
</pre>
ias_sdo_monitor
<pre>
root@sdo-new:$ psql ias_sdo -p5436 -U postgres
psql (9.4.6)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
CREATE EXTENSION
</pre>
To check the list of installed extensions type :
<pre>
ias_sdo=\dx
List of installed extensions
Name | Version | Schema | Description
---------+---------+------------+--------------------------------------------------------------
dblink | 1.1 | public | connect to other PostgreSQL databases from within a database
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
(2 rows)
</pre>
h2. System tuning
Current configuration :
<pre>
# sysctl -a
</pre>
In the following file :
<pre>
# vi /etc/sysctl.conf
</pre>
we change some values (default values are commented with '#')
(resources links [[Links|here]])
<pre>
# Semaphore is a object that is used to control utilization of a particular process.
# kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI
# SEMMSL maximum number of semaphores per array (min 128)
# SEMMNS maximum semaphores system-wide
# SEMOPM maximum operations per semop call
# SEMMNI maximum arrays
#semop incresed from 32 (default value)operation to 100 (3rd value )
# kernel.sem = 250 32000 32 128
kernel.sem = 250 32000 100 128
In order to apply changes :#This sets the OS receive buffer size for all types of connections (default value 212992)
#net.core.rmem_default = 212992
#net.core.rmem_max = 212992
net.core.rmem_default = 33554432
net.core.rmem_max = 33554432
#This is the OS send buffer size for all types of connections (212992 default value)
#net.core.wmem_default = 212992
#net.core.wmem_max = 212992
net.core.wmem_default = 33554432
net.core.wmem_max = 33554432
#TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always
# allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive
# buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ...
# The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."
#net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_rmem = 10240 87380 33554432
#TCP Autotuning setting. "This variable takes 3 different values which holds information on how much TCP sendbuffer memory space
# each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three
#values are used under different conditions. ... The first value in this variable tells the minimum TCP send buffer space available
# for a single TCP socket. ... The second value in the variable tells us the default buffer space allowed for a single TCP socket to use. ...
#The third value tells the kernel the maximum TCP send buffer space."
#net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_wmem = 10240 87380 33554432
#Disable cache metrics so the initial conditions of the closed connections will not be saved to be used in near future connections
#net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_no_metrics_save = 1
# Increase number of incoming connections backlog
# net.core.netdev_max_backlog = 1000
net.core.netdev_max_backlog = 5000
# The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage. ...
#The first value specified in the tcp_mem variable tells the kernel the low threshold.
#Below this point, the TCP stack do not bother at all about putting any pressure on the memory usage by different TCP sockets. ...
#The second value tells the kernel at which point to start pressuring memory usage down. ... The final value tells the kernel how many
# memory pages it may use maximally
# net.ipv4.tcp_mem = 1543347 2057796 3086694
net.ipv4.tcp_mem = 786432 1048576 26777216
# local port range that is used by TCP and UDP traffic to choose the local port. You will see in the parameters of this file two numbers:
# The first number is the first local port allowed for TCP and UDP traffic on the server, the second is the last local port number.
# net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.ip_local_port_range = 1024 65535
# maximum number of sockets in TIME-WAIT to be held simultaneously.
# net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_max_tw_buckets = 360000
</pre>
Then we run the following command to make our change take effect:
<pre>
# sysctl -p
</pre>
h2. JMD Installation
Download new JMD package build by Niles Oien 2016-04-07 following thoses actions :
<pre>
cd /urs/local
[oien@spsc-nso19-12 ~]$ ftp gong2.nso.edu
Name (gong2.nso.edu:oien): anonymous
Password: <---------------- use your email as the password
ftp> cd outgoing/oien
ftp> get pablo_jmd.tar.gz
ftp> quit
</pre>
Install new JMD package following thoses actions
<pre>
[oien@spsc-nso19-12 ~]$ gunzip -vf pablo_jmd.tar.gz
pablo_jmd.tar.gz: 8.3% -- replaced with pablo_jmd.tar
[oien@spsc-nso19-12 ~]$ tar xf pablo_jmd.tar
[oien@spsc-nso19-12 ~]$ cd pablo_jmd
[oien@spsc-nso19-12 pablo_jmd]$ less README.txt
cd ..
mv jmd jmd_old
mv pablo_jmd jmd
</pre>
url provided by jsoc to download last JMD version http://vso.tuc.noao.edu/VSO/downloads/JMD
h2. Apache Configuration for JMD
Configure webserver to request jsoc_fetch
(following the mail from nilesoien@gmail.com)
%{color:red}WARNING : change hostname when migrate sdo-new to sdo%
config d 'apache2 fichier sdo-new.ias.u-psud.fr
<pre>
<VirtualHost *:80>
ServerName sdo-new.ias.u-psud.fr
ServerAdmin pablo.alingery@ias.u-psud.fr
#ErrorLog /var/log/apache2/error.log
DocumentRoot /var/www
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel debug
#CustomLog /var/log/apache2/VSO/access.log combined
Alias /VSO/DRMS/cgi-bin/ "/home/production/netdrms-tools/scripts/cgi-bin/"
<Location /VSO/DRMS/cgi-bin>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Order allow,deny
Allow from all
</Location>
</VirtualHost>
</pre>
Activate the new virtualhost and disable the default one :
<pre>
# a2ensite sdo-new.ias.u-psud.fr.conf
# a2dissite 000-default.conf
# service apache2 restart
</pre>
Allow symlink for the cgi scripts dierctory in apache2.conf :
<pre>
<Directory /home/production/netdrms-tools/scripts/>
Options FollowSymLinks
AllowOverride None
Require all granted
</Directory>
</pre>
<pre>
# service apache2 restart
</pre>
%{color:red}To be added futher to test cgi%
Execution test cgi with URL :
make sure my $hostname=in the cgi file vso_jsoc_fetch.cgi is correct and test
http://sdo-new.ias.u-psud.fr/VSO/DRMS/cgi-bin/vso_jsoc_fetch.cgi
expected result :
<pre>
{
"wait" : 0,
"requestid" : "",
"data" : {},
"method" : "url_quick",
"size" : 0,
"errormsg" : "Empty query",
"protocol" : "as-is",
"status" : 1,
"space_ratio" : 0.1,
"load_ratio" : 0.066875,
"dir" : "",
"count" : 0
}
</pre>
h2. JMD configuration
Modify the file [[/usr/local/jmd/cfg/JMD.cfg]]
%{color:red}Warning : when migrate production server, change _sdo-new_ by _sdo_%
h2. NetDRMS tool Install
The following installed is based on : [[http://inf-redmine.ias.u-psud.fr/redmine/attachments/download/114/Netdrms_First_Install.odt]]
The sources are in : http://jsoc.stanford.edu/netdrms/dist/
Locally on sdo-new in : /home/production/sources
Symbolic link created 'netdrms_current' that corresponds to the last release
<pre>
ln -s netdrms_8.12/ netdrms_current
</pre>
Edit file config.local
<pre>
cp config.local.template config.local
</pre>
%{color:red}Warning WEB_DBUSER apache we think that it should value www maybe , to be checked testing the cgi%
%{color:red}So check the cgi to figure out which user is used to connect to bd%
Create 2 followings dirs :
<pre>
mkdir /usr/local/netdrms_8.12/logs/SUM -p
mkdir /usr/local/netdrms_8.12/logs/slony -p
</pre>
%{color:red}Warning config.local from line 150 =>222 Remote config to provide data, series to externals%
%{color:red}To be checked for PLATO%
Création d'un alias netdrms dans /root/.bashrc
<pre>
alias netdrms='cd /usr/local/netdrms_current'
</pre>
h3. Prepare compiling
Into _production_ profile
Replace all " $? !=0 " by " $status !=0 " in file configure then tape
<pre>
$ ./configure
</pre>
Modification du fichier /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.12 )
ligne 28 :
case "x86_64":
echo linux_x86_64
breaksw
Overwriting the existing files in the right place (base/sums/libs/pg) & modif in jsoc_machine.csh
=> the command 'make' works well.
h3. Compilation
Dans le fichier make_basic.mk ligne 223
ajouter "-lcrypto"
<pre>
$ make
$ make sums
</pre>
h2. SSH-HPN install
Intallation de openssh 7.2p2 car le dernier patch hpn dispo est 7.2hpn14.V10
web site for open ssh : http://ftp2.fr.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.2p2.tar.gz
patch hpn : https://sourceforge.net/projects/hpnssh/files/HPN-SSH%2014v10%207.2p2/openssh-7_2_P2-hpn-14.10.diff
In the dir /home/production/sources into 'root'
<pre>
# tar -xzvf openssh-7.2p2.tar.gz
# cd openssh-7.2p2/
# cat ../openssh-7_2_P2-hpn-14.10.diff |patch -p1
# ./configure --prefix=/usr/local/hpn-ssh --with-pam --with-md5-passwords --without-zlib-version-check --with-tcp-wrappers
# make
# make install
</pre>
Depuis sdo :
<pre>
# cd /usr/local/hpn-ssh/etc/
# rsync -av *key* production@sdo-new:/home/production/
</pre>
Following the link : http://vso.stanford.edu/netdrms/rmtsums.html
<pre>
$ cd /home/production/
$ ssh-agent -c > ~/.ssh-agent_rs
</pre>
NB : correct file ~/.ssh-agent_rs it should look like :
<pre>
#!/bin/csh
export SSH_AUTH_SOCK=/tmp/ssh-9POrTXobhLR4/agent.74272;
export SSH_AGENT_PID=74273;
echo Agent pid 74273;
</pre>
<pre>
$ source ~/.ssh-agent_rs
$ ssh-add ~/.ssh/id_rsa
</pre>
Sur sdo-new :
<pre>
# mv /home/production/*key* /usr/local/hpn-ssh/etc/
</pre>
We have to configure the port number to 55000
<pre>
# vi ssh_config
</pre>
and add :
<pre>
#Port 22
Port 55000
</pre>
We configure also the server hpn (even we don't use it yet)
<pre>
# vi sshd_config
</pre>
and add :
<pre>
#Port 22
Port 55000
#PidFile /var/run/sshd.pid
PidFile /var/run/sshd.55000.pid
# allow the use of the none cipher
#NoneEnabled no
NoneEnabled yes
</pre>
In order to force the use of SSH-hpn by JMD :
<pre>
$ vi /usr/local/jmd/cfg/JMD.cfg
</pre>
and replace the path of scp binary :
<pre>
SCPCommand=/usr/local/hpn-ssh/bin/scp -o NoneSwitch=yes -o NoneEnabled=yes
</pre>
h2. Private key Public key generation
JSOC need your public key to directly connect to your netdrms server. In order to generate your RSA public/private key pair :
<pre>
$ ssh-keygen -t rsa
</pre>
Not necessary in our case cause we recover the ssh keys of the current server 'sdo'.
However, we have to retrieve the private key of production account on sdo-new
From sdo :
<pre>
rsync -av id_rsa production@sdo-new:/home/production/.ssh/
</pre>
In order to test the connection :
<pre>
$ ssh jsocexp@jsocport.stanford.edu -p55000
</pre>
Note : the IP of the server has to be declared to the jsoc
h2. NetDRMS Replication Config
%{color:red}To be check if we keep that here or not%
To prevent from deleting files
<pre>
cd logs/SUM ; scp production@sdo:/usr/local/netdrms/install-config-files/sum_rm.cfg .
</pre>
Create tmp dir in netdrms working directory (/usr/local/netdrms_current)
<pre>
$ mkdir tmp
</pre>
Create config file from template
<pre>
$ cd base/drms/replication/etc
cp repclient.template.cfg ias.repclient.cfg
</pre>
Define node in file : [[/usr/local/netdrms_current/base/drms/replication/etc/ias.repclient.cfg]]
In our case :
<pre>
node=IASprod
</pre>
There are 4 new fields in the new version of repclient (see Art mail June 11, 2016)
<pre>
#kRSBaseURL=<base URL for all Slony services>
kRSBaseURL=http://jsoc.stanford.edu/cgi-bin/ajax
#kSubService=<URL of the subscription service>
kSubService=${kRSBaseURL}/request-subs.py
#kPubListService=<URL of the publication-list service>
kPubListService=${kRSBaseURL}/publist.py
#kSubXfer=<URL of the file transfer directory>
kSubXfer=http://jsoc.stanford.edu/subscription
</pre>
h2. NetDRMS Subsribe list Config
Create config subscribe file from template
<pre>
$ cd base/drms/replication/etc
cp subscribe_list.template.cfg ias.subscribe_list.cfg
</pre>
h2. TO BE DONE BEFORE MIGRATION
%{color:red} configure sytem taille buffer%
%{color:red} swap désactivé ? discussion Stephane et Gilles%
%{color:red}Test ssh connection to JSOC , NSO CFA before subcription and just after name sdo-new into sdo%
%{color:red}Take old counter and place it in the same dir before migration%