NetDRMS Installation » History » Version 106
« Previous -
Version 106/369
(diff) -
Next » -
Current version
Pablo Alingery, 08/12/2015 16:54
- Table of contents
- NetDRMS Installation
- Requirements
- Installation of postgresql
- NetDRMS databases
- NetDRMS tool Install
- Modification du fichier /usr/local/netdrms_current/configure (Not Applied in NetDrms_8.10)
- Compilation problem due to uncorrect variable types
- Workaround to solve make sums problem
- Modification du fichier /usr/local/netdrms_current/build/jsoc_machine.csh
- Copier /usr/local/netdrms_2.4_release/base/drms/apps/serverdefs.h dans /usr/local/netdrms_current/base/drms/apps (not applied in NetDrms_8.10)
- Modification du fichier /usr/local/netdrms_current/configure (not applied in NetDrms_8.10)
- Installation des libraries libtar-dev, libcurl3-dev et libecpg-dev
- #H4ck 2 Malade #PabloR0cks# (Not Applied in NetDrm_8.10)
- Compilation
- make sums KO (Not applied in NetDrms_8.10)
- NetDRMS replication
- Jetty installation
- JMD configuration
- NetDRMS Start script
- Database repair (from Art)
- Private key Public key generation
- Node definition
- show_series and delete_series binaries
- SSH-HPN install
- Correction
- TO BE DONE FURTHER
NetDRMS Installation¶
Requirements¶
Configuring production environment¶
Adding user production
# adduser production # addgroup ias # adduser production ias
Configure /home/production/.profile¶
Adding environment variables
# Added by Elie for netdrms #Modif Pablo 07/06/2010 PATH="/usr/local/netdrms_current/bin/linux_x86_64:$PATH" PATH="/usr/local/netdrms_current/scripts:$PATH" PATH="/usr/local/jmd/jetty/bin:$PATH" PATH="/usr/local/jmd/vso/scripts:$PATH" PATH="/usr/local/pgsql/bin:$PATH" PATH="/usr/local/netdrms/scripts:$PATH" #2014-12-17 Avec Herve et Pablo on pense que la ligne suivante est intutile as it is f.. install in /usr/local/lib PATH="/home/production/cfitsio:$PATH" #Done DERBY_HOME=/usr/local/jmd/derby export DERBY_HOME export CVSROOT=:pserver:anonymous@solarch.tuc.noao.edu:2401/vtarc1/vso/cvsroot #Added by Elie for jetty PATH="/usr/local/jmd/derby/bin:$PATH"
sshagent to JSOC¶
TBD
Mise en place de Privilèges pour le SUMS_MANAGER¶
Veuillez modifier le fichier /etc/sudoers afin que le user 'production' puisse exécuter le programme sum_chown que nous placerons plus loin dans le répertoire :usr/local/bin. See here
Cela se fait en ajoutant au fichier la ligne suivante :
production host=NOPASSWD:/usr/local/bin/sum_chmown
Mise en place du fichier de mot de passe pour la base postgres¶
Veuillez créer le fichier .pgpass contenant les informations nécessaires au user 'production" afin qu'il se connecte sans mot de passe, en tant que 'production' ou 'postgres' aux 2 bases que l'on créera plus loin ias_sdo et ias_sdo_sums.
.pgpass content :
sol-sdo1:5432:ias_sdo:production:
sol-sdo1:5434:ias_sdo_sums:production:
sol-sdo1:5432:ias_sdo:postgres:
sol-sdo1:5434:ias_sdo_sums:postgres
Install missing libraries¶
• libreadline-dev (but already installed with postgresql)
• libssl-dev
• libpam0g-dev
• libperl-dev# apt-get install $library_name
where
$library_name is the name of library
Then to update path on libraries, please type :
# ldconfig
Reconfigure locale¶
# dpkg-reconfigure locales
select fr_FR.UTF8 UTF8 and unselect en_US.UTF-8
localedef -i en_US -f UTF-8 en_US.UTF-8
(see here : [[http://www.thomas-krenn.com/en/wiki/Perl_warning_Setting_locale_failed_in_Debian]])
Install missing packages¶
# apt-get install gfortran swig
Installation of cfitsio library¶
Vesrion 3.37 (currently on sdo : 3.24)
# wget ftp://heasarc.gsfc.nasa.gov/software/fitsio/c/cfitsio3370.tar.gz
Dés-archivage et décompression
# tar -xzvf cfitsio3370.tar.gz
Placez vous dans le répertoire /home/production/cftisio
su - root and :
# ./configure --prefix=/usr/local # make # make install
Installation of postgresql¶
See the section : Test_Environment_-_Virtual_Machine_description
NetDRMS databases¶
on /DATABASES
(Note : section 2.4 dans doc netdrms first install)
# mkdir data # mkdir data_sums # mkdir data_monitor # chown postgres:postgres data # chown postgres:postgres data_sums # chown postgres:postgres data_monitor
Initialisation des 2 serveurs data et data_sums¶
En tant qu'utilisateur postgres
# su - postgres
$ initdb --locale=C -D data -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data
or
pg_ctl -D data -l logfile start
$ initdb --locale=C -D data_sums -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data_sums
or
pg_ctl -D data_sums -l logfile start
$ initdb --locale=C -D data_monitor -A md5 -W
(no password)
Success. You can now start the database server using:
postgres -D data_monitor
or
pg_ctl -D data_monitor -l logfile start
Edition des fichiers postgresql.conf pour configurer les bons ports d'écoute :
listen_addresses = '*'
data -> port 5432
data_sums -> port 5434
data_monitor -> port 5436
Edition des fichiers data*/pg_hba.conf : remplacer md5 par trust
#"local" local all all trust #"IPv4" host all all trust
Démarrage des 3 bases :
# su - postgres $ cd /DATABASES $ pg_ctl -D data -l ./PG_LOGS/data.log start $ pg_ctl -D data_sums -l ./PG_LOGS/data_sums.log start $ pg_ctl -D data_monitor -l ./PG_LOGS/data_monitor.log start note : PG_LOGS is the directory for logscreatedb --locale C -E LATIN1 -T template0 ias_sdo
createdb --locale C -E LATIN1 -T template0 -p 5434 ias_sdo_sums
createdb --locale C -E LATIN1 -T template0 -p 5436 ias_sdo_monitor
NetDRMS tool Install¶
On se base sur la doc dipsonible ici : [[http://inf-redmine.ias.u-psud.fr/redmine/attachments/download/114/Netdrms_First_Install.odt]]
Les sources sont dans : http://jsoc.stanford.edu/netdrms/dist/
/usr/local/netdrms_8.10/
(Pour comparaison avec le serveur en production, on copie également le dossier netdrms_2.4 de sdo)
on créé un lien symbolique netdrms_current qui pointe sur la dernière release
ln -s netdrms_8.10/ netdrms_current
Edition fichier config.local
cp config.local.template config.local
Creation de 2 répertoires :
mkdir /usr/local/netdrms_8.10/logs/SUM mkdir /usr/local/netdrms_8.10/logs/slony
Voir le contenu du le fichier config.local à jour
Création d'un alias netdrms dans /root/.bashrc
alias netdrms='cd /usr/local/netdrms_current'
Lancement configure
./configure
Modification du fichier /usr/local/netdrms_current/configure (Not Applied in NetDrms_8.10)¶
# The system configuration for configure script set PERLBIN = "/usr/bin/perl" set PYBIN = "/usr/bin/python"
Compilation problem due to uncorrect variable types¶
Art (art.amezcua@stanford.edu) purpose us a workaround and send us 2 files with correct declaration :
SUMLIB_RmDo.pgc (diff with original file : diff_SUMLIB_RmDo.pgc)
SUMLIB_RmDoX.pgc (diff with original file : diff_SUMLIB_RmDoX.pgc)
# wget http://jsoc.stanford.edu/~arta/SUMLIB_RmDo.pgc # wget http://jsoc.stanford.edu/~arta/SUMLIB_RmDoX.pgc
Overwriting the existing files in the right place (base/sums/libs/pg), the command 'make' works well.
But there is still problem with 'make sums'
Workaround to solve make sums problem¶
Modify sum_rpc.h in order to match the type of some variables
Modification du fichier /usr/local/netdrms_current/build/jsoc_machine.csh¶
ligne 28 :
case "x86_64":
echo linux_x86_64
breaksw
Copier /usr/local/netdrms_2.4_release/base/drms/apps/serverdefs.h dans /usr/local/netdrms_current/base/drms/apps (not applied in NetDrms_8.10)¶
Modification du fichier /usr/local/netdrms_current/configure (not applied in NetDrms_8.10)¶
au lieu de $? en csh on va mettre plutot $status
ligne 110 :
if ($status==0)
et ligne 123 :
if ($status==0)
Lancement :
./configure
Installation des libraries libtar-dev, libcurl3-dev et libecpg-dev¶
apt-get install libtar libtar-dev
apt-get install libcurl3-dev
apt-get install libecpg-dev
#H4ck 2 Malade #PabloR0cks# (Not Applied in NetDrm_8.10)¶
Modification du fichier /usr/include/stdint.h
ligne 55 : long int remplacé par long long int
#if __WORDSIZE == 64 typedef unsigned long int uint64_t; --> ici long long int #else __extension__ typedef unsigned long long int uint64_t; #endif
"Allo Linus ?!?.."
Compilation¶
# make # make sums # make jsoc_fetch
NB : lib64 does not exist on debian 8. so ln -s /usr/lib lib64
make sums KO (Not applied in NetDrms_8.10)¶
Solution found : Take old lib libcrypto instead libssl
dans le fichier base/sums/apps/Rules.mk ligne 51
replace -lssl by -lcrypto LD flag
NetDRMS replication¶
# cd base/drms/replication/etc # scp production@sdo:/usr/local/netdrms/install-config-files/ias.subscribe_list.cfg . # scp production@sdo:/usr/local/netdrms/install-config-files/ias.repclient.cfg . WRONG !!!!! To be done cp repclient.template.cfg into ias.subscribe_list.cfg because new field in config file can't appear Correct that step in NetDrms doc The existing ias.repclient.cfg can be used tough to fill params
Create tmp dir in netdrms working directory (/usr/local/netdrms_current)
# mkdir tmp
Prevent from deleting files
cd logs/SUM ; scp production@sdo:/usr/local/netdrms/install-config-files/sum_rm.cfg .
Jetty installation¶
# cd /usr/local # wget http://vso.tuc.noao.edu/VSO/slony_proxy/jetty_install_kit_v1.4.2.tar.gz # tar xzvf jetty_install_kit_v1.4.2.tar.gz # cd jetty_install_kit_v1.4.2 # scp production@sdo:/usr/local/netdrms/install/install-jetty.cfg . # ./install_jetty.sh install-jetty.cfg
Cherche la commande ij
rajouter son chemin dans le path
vi ~/.bashrc
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
# Added by Elie for netdrms
#Modif Pablo 07/06/2010
PATH="$HOME/netdrms_current/bin/linux_x86_64:$PATH"
PATH="$HOME/netdrms_current/scripts:$PATH"
PATH="/usr/local/jmd/jetty/bin:$PATH"
PATH="/usr/local/jmd/vso/scripts:$PATH"
PATH="/usr/local/pgsql/bin:$PATH"
PATH="/usr/local/netdrms/scripts:$PATH"
#2014-12-17 Avec Herve et Pablo on pense que la ligne suivante est intutile as it is f.. install in /usr/local/lib
PATH="/home/production/cfitsio:$PATH"
#Done
DERBY_HOME=/usr/local/jmd/derby
export DERBY_HOME
export CVSROOT=:pserver:anonymous@solarch.tuc.noao.edu:2401/vtarc1/vso/cvsroot
#Added by Elie for jetty
PATH="/usr/local/jmd/derby/bin:$PATH"
JMD configuration¶
As user production
# cd /usr/local/jmd/jetty/webapps
# scp production@sdo:/usr/local/netdrms/install/JMD.cfg .
We download the last version of JMD.war
# wget http://vso.tuc.noao.edu/VSO/slony_proxy/JMD_v1.5.7.0.war # mv JMD_v1.5.7.0.war JMD.war
# cp /usr/local/netdrms_current/_linux_x86_64/base/export/apps/jsoc_fetch .
Define an alias jettylogs in .bashrc of production account :
alias jettylogs='cd /usr/local/jmd/jetty/logs'
# jettylogs # cd ../bin # vi jetty.sh
Add the following line :
JAVA_OPTIONS="$JAVA_OPTIONS -Xms256m -Xmx512m -Djetty.home=$JETTY_HOME -Djava.io.tmpdir=$TMP"
copy the script remotesums_master_jmd.pl in the right place
# netdrms # cd base/drms/scripts/ # cp home/production/jetty_install_kit_v1.4/scripts/remotesums_master_jmd.pl .
# netdrms # cd scripts # rm remotesums_master.pl # ln -s ../base/drms/scripts/remotesums_master_jmd.pl remotesums_master.pl
Change exec rights
# netdrms # cd base/drms/scripts # chmod 740 remotesums_master_jmd.pl # chmod 740 remotesums_master.pl
Copy sum_chmown binary to /usr/local/bin/¶
# netdrms # cd bin/linux_x86_64 # cp sum_chmown /usr/local/bin/
Retrieve modified get_slony_logs.pl file¶
# netdrms # cd base/drms/replication # scp production@sol-sdo1:/usr/local/netdrms/install/get_slony_logs.pl .
NetDRMS Start script¶
sum_start.NetDRMS
Database repair (from Art)¶
On ias_sdo
delete from admin.ns where name = 'lm_jps'; delete from admin.ns where name = 'aia_test';
Private key Public key generation¶
ssh-keygen -t rsa
Not necessary cause we recover the ssh keys of the current server.
However, we have to retrieve the private key of production account on sdo3
From sdo :
rsync -av id_rsa production@sdo3:/home/production/.ssh/
Node definition¶
file : /usr/local/netdrms_current/base/drms/replication/etc/ias.repclient.cfg
node=IAStest
show_series and delete_series binaries¶
Remark : these 2 sources are not compiled during installation
Issue : in netdrms_current 8.10 , do :
$ make delete_series
$ make show_series
SSH-HPN install¶
Intallation de openssh 6.3p1 car le dernier patch hpn dispo est 6.3hpn13.V6
web site for open ssh : http://ftp2.fr.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-6.3p1.tar.gz
patch hpn : http://www.psc.edu/index.php/hpn-ssh-patches/hpn-14-kitchen-sink-patches/viewdownload/24-hpn-14-kitchen-sink-patches/102-openssh-6-3p1-hpnssh14v2-kitchen-sink-patch
# tar -xzvf openssh-6.3p1.tar.gz # gunzip openssh-6.3p1-hpnssh14v2.diff.gz # cd openssh-6.3p1 # zcat ../openssh-6.3p1-hpnssh14v2.diff.gz | patch -p1 # ./configure --prefix=/usr/local/hpn-ssh --with-pam --with-md5-passwords --without-zlib-version-check --with-tcp-wrappers # make # make install
Depuis sdo :
# cd /usr/local/hpn-ssh/etc/ # rsync -av *key* netadm@sdo3:/home/netadm/
Sur sdo3 :
rsync -av /home/netadm/*key* /usr/local/hpn-ssh/etc/
We have to configure the port number to 55000
# vi ssh_config
and add :
#Port 22 Port 55000
We configure also the server hpn (even we don't use it yet)
# vi sshd_config
and add :
#Port 22 Port 55000 #PidFile /var/run/sshd.pid PidFile /var/run/sshd.55000.pid # allow the use of the none cipher #NoneEnabled no NoneEnabled yes
Correction¶
Modification du fichier ias.repclient.cfg¶
Copier le template existant repclient.template.cfg dans le répertoire [netdrms_current]/base/drms/replication/etc
production@sdo3:/usr/local/netdrms_current/base/drms/replication/etc$ cp repclient.template.cfg ias.repclient.cfg
Nouveau / netdrms2.4
# Apps kRSPerl=<path to Perl binary to use when Perl scripts are executed>
TO BE DONE FURTHER¶
Modification du fichir config.local pour la souscription au jsoc¶
# NetDRMS users can request subdirectories from the Stanford proj directories # (e.g., JSOC/proj/util). These users will need to contact Stanford and obtain # the source subdirectories desired. They will then place these subdirectories # in the JSOC/proj directory of their NetDRMS release. To properly build # targets in these subdirectories, using the JSOC make system, the user needs # to add entries to this configuration file - one entry for each subdirectory # that contains source code to be compiled. Each entry is a space-separated pair # of strings: the string "PROJDIR" followed by a subdirectory (of the proj # directory). For example: # # PROJDIR util/apps # PROJDIR util/libs # # NOTE: It is not sufficient to list only a parent directory, like util. Source code # in child directories will not automatically be compiled.
Python library for multi-threaded SUMS¶
Installation de la librairie python3.4m