MEDOC - Multi Experiment Data & Operation Center » NetDRMS_8.12

{{toc}}

h1. NetDRMS Installation

h2. Requirements

h3. Configuring production environment

Adding user production (en root)

<pre>

# adduser production

# addgroup ias

# adduser production ias

</pre>

h3. Configure /home/production/.profile

Adding environment variables

<pre>

#Modif Herve.B Pablo.A 12/05/2016

PATH="/usr/local/netdrms_current/bin/linux_x86_64:$PATH"

PATH="/usr/local/netdrms_current/scripts:$PATH"

PATH="/usr/local/jmd/bin:$PATH"

PATH="/usr/local/jmd/scripts:$PATH"

#The following is not needed as installed with 'apt-get install' 

#PATH="/usr/local/pgsql/bin:$PATH"

PATH="/usr/local/netdrms-tools/scripts:$PATH"

#2014-12-17 Avec Herve et Pablo on pense que la ligne suivante est intutile as it is f.. install in /usr/local/lib

PATH="/usr/local/cfitsio:$PATH"

#Done

DERBY_HOME=/usr/local/jmd/databases/derby/derbyBD

export DERBY_HOME

export CVSROOT=:pserver:anonymous@solarch.tuc.noao.edu:2401/vtarc1/vso/cvsroot

</pre>

h3. Configure /varl/lib/postgres/.profile

Add to $PATH so postgres can exeute command like initdb

<pre>

#Modif Herve.B Pablo.A 19/05/2016

PATH="/usr/lib/postgresql/9.4/bin:$PATH"

</pre>

h3. Setting privileges for SUMS_MANAGER

Please modify file /etc/sudoers so user 'production' will be able to execute  'sum_chown' that we will locate in the dir : /usr/local/bin . See below

Add to /etc/sudoers the following line :

	production host=NOPASSWD:/usr/local/bin/sum_chmown

h3. Set passwords file for db postgres 

Please create file .pgpass with the following information for user 'production" so he will be able to connect without password as 'production' or 'postgres' to 2 db that we will create later ias_sdo and ias_sdo_sums.

.pgpass content :

<pre>

	#hostname:5432:ias_sdo:production: 

	#hostname:5434:ias_sdo_sums:production: 

	#hostname:5432:ias_sdo:postgres: 

	#hostname:5434:ias_sdo_sums:postgres

</pre>

where #hostname is the name of your server

%{color:red}WARNING : change hostname when migrate sdo-new to sdo%

h3. Install missing libraries 

• libreadline-dev (but already installed with postgresql)

• libssl-dev 

• libpam0g-dev 

• libperl-dev 

NB : Already done in Debian 8.4 Jessy

<pre>

# apt-get install $library_name

</pre> 

where

	_$library_name_ is the name of library

Then to update path on libraries, please type :

<pre>

# ldconfig

</pre>

h3. Reconfigure locale

<pre>

# dpkg-reconfigure locales

</pre>

select fr_FR.UTF8 UTF8 and unselect en_US.UTF-8

localedef -i en_US -f UTF-8 en_US.UTF-8

(see here : [[http://www.thomas-krenn.com/en/wiki/Perl_warning_Setting_locale_failed_in_Debian]])

h3. Install gfortran packages

<pre>

# apt-get install gfortran swig

</pre>

h3. Installation of cfitsio library

In /usr/local dir

Version 3.39 (currently on sdo : 3.24)

<pre>

# wget ftp://heasarc.gsfc.nasa.gov/software/fitsio/c/cfitsio3390.tar.gz

</pre>

Untar and  decompression

<pre>

# tar -xzvf cfitsio3390.tar.gz

</pre> 

Go into /usr/local and type

su - root and :

<pre>

# ./configure --prefix=/usr/local

# make 

# make install

</pre>

h3. Installation of apache2 and lib perl

<pre>

# apt-get install apache2

# apt-get install libjson-perl

# apt-get install libapache2-mod-perl2

</pre>

h3. Installation of postgresql

Into 'root'

<pre>

apt-get install postgresql-9.4 postgresql-client-9.4 

</pre>

h3. Installation of python3

Into 'root'

<pre>

apt-get install python3 

</pre>

Add some modules psycopg2 (postgres connector) and pySmartDL (Download manager) 

<pre>

apt-get install python3-psycopg2

</pre>

<pre>

apt-get install python3-psycopg2

</pre>

Add python3-pip for modules within python3

<pre>

apt-get install python3-pip

</pre>

<pre>

pip3 install pySmartDL

</pre>

Both python2.7 and 3.4 versions are installed on the system.

By default, the python2.7 is used.

In order to change in 3.4 by default, type :

<pre>

update-alternatives --config python

</pre>

h2. NetDRMS databases

into /var/lib/postgresql  (hard mount point separated,  7 disks 15000 tr/min in raid5 in order to distribute read and write) 

<pre>

# cd /var/lib/postgresql

# mkdir data  data_sums data_monitor

# chown -R postgres:postgres * 

</pre>

h3. Initialize 3 servers data,  data_sums, data_monitor

Into postgres

<pre>

# su - postgres

$ initdb --locale=C -D data -A md5 -W

(no password)

Success. You can now start the database server using:

    postgres -D data

or

    pg_ctl -D data -l logfile start

$ initdb --locale=C -D data_sums -A md5 -W

(no password)

Success. You can now start the database server using:

    postgres -D data_sums

or

    pg_ctl -D data_sums -l logfile start

$ initdb --locale=C -D data_monitor -A md5 -W

(no password)

Success. You can now start the database server using:

    postgres -D data_monitor

or

    pg_ctl -D data_monitor -l logfile start

</pre>

Edition des fichiers postgresql.conf pour configurer les bons ports d'écoute :

listen_addresses = '*'

data -> port 5432

data_sums -> port 5434

data_monitor -> port 5436

Edit files data*/pg_hba.conf : replace 'md5' by 'trust'

<pre>

#"local"

local   all             all                                     trust

#"IPv4"

host   all             all                                     trust

</pre>

h3. Tunning postgres 

In file postgres.conf ( according to https://www.qwant.com/?q=Tunning+postgres+server+9.4+&client=firefox  )

shared_buffer=4096MB %{color:red}So 1/12 of the total RAM memory available%

work_mem = 24MB

maintenance_work_mem = 4096MB

wal_buffers = 16MB

checkpoint_segments = 32

checkpoint_completion_target = 0.9

random_page_cost = 1.0

effective_cache_size = 16384MB so 1/4 of the max memory 

#log

logging_collector = on 

log_directory = 'pg_log' 

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,

log_truncate_on_rotation = on           # If on, an existing log file with the

log_rotation_age = 1d                   # Automatic rotation of logfiles will

client_min_messages = error

log_min_messages = error 

log_min_duration_statement = 0

log_connections = on

log_disconnections = on

log_duration = off

log_hostname = on

log_line_prefix = '%t [%p]: %u@%h - %d :'  

track_counts = on

autovacuum = on 

extra_float_digits = 3

h3. Start the 3 databases :

<pre>

# su - postgres

$ cd /var/lib/postgresql

$ pg_ctl -D data  start

$ pg_ctl -D data_sums start

$ pg_ctl -D data_monitor start

</pre>

h3. Create the 3 databases :

Into user 'postgres'

* createdb --locale C -E LATIN1 -T template0 ias_sdo

* createdb --locale C -E LATIN1 -T template0 -p 5434 ias_sdo_sums

* createdb --locale C -E LATIN1 -T template0 -p 5436 ias_sdo_monitor

h3. Automatic startup

Add following lines in the /etc/rc.local file :

<pre>

su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data start" postgres

su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_sums start" postgres

su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_monitor start" postgres

</pre>

h2. System tuning

Current configuration :

<pre>

# sysctl -a

</pre>

In the following file :

<pre>

# vi /etc/sysctl.conf

</pre>

we change some values (default values are commented with '#')

(resources links [[Links|here]])

<pre>

# Semaphore is a object that is used to control utilization of a particular process.

# kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI

# SEMMSL        maximum number of semaphores per array  (min 128)

# SEMMNS        maximum semaphores system-wide  

# SEMOPM        maximum operations per semop call       

# SEMMNI        maximum arrays 

#semop incresed from 32 (default value)operation to 100 (3rd value )

# kernel.sem = 250      32000   32      128

kernel.sem = 250 32000 100 128

In order to  apply changes :#This sets the  OS receive buffer size for all types of connections (default value 212992)

#net.core.rmem_default = 212992

#net.core.rmem_max = 212992

net.core.rmem_default = 33554432

net.core.rmem_max = 33554432

#This is the OS send buffer size for all types of connections (212992 default value) 

#net.core.wmem_default = 212992

#net.core.wmem_max = 212992

net.core.wmem_default = 33554432

net.core.wmem_max = 33554432

#TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always

# allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive

# buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ...

# The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."

#net.ipv4.tcp_rmem = 4096       87380   6291456

net.ipv4.tcp_rmem = 10240 87380 33554432

#TCP Autotuning setting. "This variable takes 3 different values which holds information on how much TCP sendbuffer memory space

# each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three 

#values are used under different conditions. ... The first value in this variable tells the minimum TCP send buffer space available

# for a single TCP socket. ... The second value in the variable tells us the default buffer space allowed for a single TCP socket to use. ... 

#The third value tells the kernel the maximum TCP send buffer space." 

#net.ipv4.tcp_wmem = 4096       16384   4194304

net.ipv4.tcp_wmem = 10240 87380 33554432

#Disable cache metrics so the initial conditions of the closed connections will not be saved to be used in near future connections

#net.ipv4.tcp_no_metrics_save = 0

net.ipv4.tcp_no_metrics_save = 1

# Increase number of incoming connections backlog

# net.core.netdev_max_backlog = 1000

net.core.netdev_max_backlog = 5000

# The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage. ... 

#The first value specified in the tcp_mem variable tells the kernel the low threshold. 

#Below this point, the TCP stack do not bother at all about putting any pressure on the memory usage by different TCP sockets. ... 

#The second value tells the kernel at which point to start pressuring memory usage down. ... The final value tells the kernel how many

# memory pages it may use maximally

# net.ipv4.tcp_mem = 1543347    2057796 3086694

net.ipv4.tcp_mem = 786432 1048576 26777216

# local port range that is used by TCP and UDP traffic to choose the local port. You will see in the parameters of this file two numbers:

# The first number is the first local port allowed for TCP and UDP traffic on the server, the second is the last local port number.

# net.ipv4.ip_local_port_range = 32768  61000

net.ipv4.ip_local_port_range = 1024 65535

# maximum number of sockets in TIME-WAIT to be held simultaneously.

# net.ipv4.tcp_max_tw_buckets = 262144

net.ipv4.tcp_max_tw_buckets = 360000

</pre>

Then we run the following command to make our change take effect:

<pre>

# sysctl -p

</pre>

h2. JMD Installation 

Download new JMD package build by Niles Oien 2016-04-07 following thoses actions :

<pre>

cd /urs/local

[oien@spsc-nso19-12 ~]$ ftp gong2.nso.edu

Name (gong2.nso.edu:oien): anonymous

Password: <---------------- use your email as the password

ftp> cd outgoing/oien

ftp> get pablo_jmd.tar.gz

ftp> quit

</pre>

Install new JMD package following thoses actions 

<pre>

[oien@spsc-nso19-12 ~]$ gunzip -vf pablo_jmd.tar.gz

pablo_jmd.tar.gz:      8.3% -- replaced with pablo_jmd.tar

[oien@spsc-nso19-12 ~]$ tar xf pablo_jmd.tar

[oien@spsc-nso19-12 ~]$ cd pablo_jmd

[oien@spsc-nso19-12 pablo_jmd]$ less PABLO_README.txt

cd ..

mv jmd jmd_old

mv pablo_jmd jmd 

</pre>

h2. JMD Configuration

Configure webserver  to request jsoc_fetch

(following the mail from nilesoien@gmail.com)

config d 'apache2 fichier sdo3.ias.u-psud.fr 

<pre>

<VirtualHost *:80>

        ServerName sdo3.ias.u-psud.fr

        ServerAdmin pablo.alingery@ias.u-psud.fr

        #ErrorLog /var/log/apache2/error.log

        DocumentRoot /var/www

        # Possible values include: debug, info, notice, warn, error, crit,

        # alert, emerg.

        LogLevel debug

        #CustomLog /var/log/apache2/VSO/access.log combined

        Alias /VSO/DRMS/cgi-bin/ "/home/production/netdrms-tools/scripts/cgi-bin/"

        <Location /VSO/DRMS/cgi-bin>

                SetHandler perl-script

                PerlResponseHandler ModPerl::Registry

                PerlOptions +ParseHeaders

                Options +ExecCGI

                Order allow,deny

                Allow from all

        </Location>

</VirtualHost>

</pre>

Execution test cgi with URL :

http://sdo3.ias.u-psud.fr/ VSO/DRMS/cgi-bin/vso_jsoc_fetch.cgi

expected result  :

<pre>

{

   "wait" : 0,

   "requestid" : "",

   "data" : {},

   "method" : "url_quick",

   "size" : 0,

   "errormsg" : "Empty query",

   "protocol" : "as-is",

   "status" : 1,

   "space_ratio" : 0.1,

   "load_ratio" : 0.066875,

   "dir" : "",

   "count" : 0

}

</pre>

h2. NetDRMS tool Install

The following installed is based on : [[http://inf-redmine.ias.u-psud.fr/redmine/attachments/download/114/Netdrms_First_Install.odt]]

The sources are in : http://jsoc.stanford.edu/netdrms/dist/ 

Locally on sdo-new in  : /home/production/sources

Symbolic link created 'netdrms_current' that corresponds to the last release

ln -s netdrms_8.10/ netdrms_current 

Edit file config.local

<pre>

cp config.local.template config.local

</pre>

%{color:red}Warning WEB_DBUSER      apache  We think that it should value www maybe , to be checked testing the cgi%

%{color:red}So check the cgi to figure out which user is used to connect to bd%

Create 2 followings dirs : 

<pre>

mkdir /usr/local/netdrms_8.10/logs/SUM

mkdir /usr/local/netdrms_8.10/logs/slony

</pre>

%{color:red}Warning config.local from line 150 =>222 Remote config to provide data, series to externals%

%{color:red}To be checked for PLATO%

Création d'un alias netdrms dans /root/.bashrc

<pre>

alias netdrms='cd /usr/local/netdrms_current'

</pre>

%{color:red}------------------------------------------------------------------------------------------------------------------%

h3. Modification du fichier /usr/local/netdrms_current/configure (Not Applied in NetDrms_8.10)

<pre>

# The system configuration for configure script

set PERLBIN = "/usr/bin/perl"

set PYBIN = "/usr/bin/python"

</pre>

h3. Compilation problem due to uncorrect variable types

Art (art.amezcua@stanford.edu) purpose us a workaround and send us 2 files with correct declaration :

SUMLIB_RmDo.pgc (diff with original file : [[diff_SUMLIB_RmDo.pgc]])

SUMLIB_RmDoX.pgc (diff with original file : [[diff_SUMLIB_RmDoX.pgc]])

Dans le répertoire base/sums/libs/pg renommer les anciens en *.orig et placer les nouveau fichiers SUMLIB_RmDo.pgc et SUMLIB_RmDoX.pgc

<pre>

# wget http://jsoc.stanford.edu/~arta/SUMLIB_RmDo.pgc

# wget http://jsoc.stanford.edu/~arta/SUMLIB_RmDoX.pgc

</pre>

h3. Modification du fichier /usr/local/netdrms_current/build/jsoc_machine.csh

ligne 28 :

<pre>

  case "x86_64":

      echo linux_x86_64

      breaksw

</pre>

Overwriting the existing files in the right place (base/sums/libs/pg) & modif in jsoc_machine.csh 

=> the command 'make' works well.

But there is still problem with 'make sums'

h3. Workaround to solve make sums problem

Modify [[sum_rpc.h]] in order to match the type of some variables

h3. Copier /usr/local/netdrms_2.4_release/base/drms/apps/serverdefs.h dans /usr/local/netdrms_current/base/drms/apps (not applied in NetDrms_8.10)

h3. Modification du fichier /usr/local/netdrms_current/configure (not applied in NetDrms_8.10)

au lieu de $? en csh on va mettre plutot $status

ligne 110 :

<pre>

  if ($status==0) 

</pre>

et ligne 123 :

<pre>

  if ($status==0) 

</pre>

Lancement :

<pre>

./configure

</pre>

h3. Installation des libraries libtar-dev, libcurl3-dev et libecpg-dev

<pre>

apt-get install libtar libtar-dev

</pre>

<pre>

apt-get install libcurl3-dev

</pre>

<pre>

apt-get install libecpg-dev

</pre>

h3. #H4ck 2 Malade #PabloR0cks# (Not Applied in NetDrm_8.10)

Modification du fichier /usr/include/stdint.h

ligne 55 : long int remplacé par long long int

<pre>

#if __WORDSIZE == 64

typedef unsigned long int       uint64_t; --> ici long long int

#else

__extension__

typedef unsigned long long int  uint64_t;

#endif

</pre>

"Allo Linus ?!?.."

h3. Compilation

<pre>

# make

# make sums

# make jsoc_fetch

</pre>

NB : lib64 does not exist on debian 8. so ln -s /usr/lib lib64

h2. make sums KO  (Not applied in NetDrms_8.10)

Solution found : Take old lib libcrypto instead libssl

dans le fichier base/sums/apps/Rules.mk ligne 51 

replace -lssl by -lcrypto LD flag

h2. NetDRMS replication

<pre>

# cd base/drms/replication/etc

# scp production@sdo:/usr/local/netdrms/install-config-files/ias.subscribe_list.cfg  .

# scp production@sdo:/usr/local/netdrms/install-config-files/ias.repclient.cfg  .

WRONG !!!!!

To be done cp repclient.template.cfg into ias.subscribe_list.cfg because new field in config file can't appear 

Correct that step in NetDrms doc 

The existing ias.repclient.cfg can be used tough to fill params 

</pre>

Create tmp dir in netdrms working directory (/usr/local/netdrms_current)

<pre>

# mkdir tmp

</pre>

Prevent from deleting files

<pre>

cd logs/SUM ; scp production@sdo:/usr/local/netdrms/install-config-files/sum_rm.cfg .

</pre>

h2. NetDRMS Start script

<pre>

sum_start.NetDRMS

</pre>

h2. Database repair (from Art)

On ias_sdo

<pre>

delete from admin.ns where name = 'lm_jps';

delete from admin.ns where name = 'aia_test';

</pre>

h2. Private key Public key generation 

<pre>

ssh-keygen -t rsa

</pre>

Not necessary cause we recover the ssh keys of the current server.

However, we have to retrieve the private key of production account on sdo3

From sdo :

<pre>

rsync -av id_rsa production@sdo3:/home/production/.ssh/

</pre>

h2. Node definition

file : /usr/local/netdrms_current/base/drms/replication/etc/ias.repclient.cfg

<pre>

node=IAStest

</pre>

h2. show_series and delete_series binaries

Remark : these 2 sources are not compiled during installation

Issue : in netdrms_current 8.10 , do :

$ make delete_series

$ make show_series

h2. SSH-HPN install 

Intallation de openssh 6.3p1 car le dernier patch hpn dispo est 6.3hpn13.V6 

web site for open ssh : http://ftp2.fr.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-6.3p1.tar.gz

patch hpn : http://www.psc.edu/index.php/hpn-ssh-patches/hpn-14-kitchen-sink-patches/viewdownload/24-hpn-14-kitchen-sink-patches/102-openssh-6-3p1-hpnssh14v2-kitchen-sink-patch

<pre>

# tar -xzvf openssh-6.3p1.tar.gz

# gunzip openssh-6.3p1-hpnssh14v2.diff.gz

# cd openssh-6.3p1

# zcat ../openssh-6.3p1-hpnssh14v2.diff.gz | patch -p1

# ./configure --prefix=/usr/local/hpn-ssh --with-pam --with-md5-passwords --without-zlib-version-check --with-tcp-wrappers

# make

# make install

</pre>

Depuis sdo :

<pre>

# cd /usr/local/hpn-ssh/etc/

# rsync -av *key* netadm@sdo3:/home/netadm/

</pre>

Sur sdo3 :

<pre>

rsync -av /home/netadm/*key* /usr/local/hpn-ssh/etc/

</pre>

We have to configure the port number to 55000

<pre>

# vi ssh_config

</pre>

and add :

<pre>

#Port 22

Port 55000

</pre>

We configure also the server hpn (even we don't use it yet)

<pre>

# vi sshd_config

</pre>

and add :

<pre>

#Port 22

Port 55000

#PidFile /var/run/sshd.pid

PidFile /var/run/sshd.55000.pid

# allow the use of the none cipher

#NoneEnabled no

NoneEnabled yes

</pre>

h2. Correction

h3. Modification du fichier ias.repclient.cfg

Copier le template existant repclient.template.cfg dans le répertoire [netdrms_current]/base/drms/replication/etc

<pre>

production@sdo3:/usr/local/netdrms_current/base/drms/replication/etc$ cp repclient.template.cfg ias.repclient.cfg

</pre>

Nouveau / netdrms2.4 

<pre>

# Apps

kRSPerl=<path to Perl binary to use when Perl scripts are executed>

</pre>

h2. Subscription test 

h3. Librairies manquantes

Errors 

<pre>

Failure to apply SQL file 'IAStest.subscribe_series.sql' from server: ABORTING!

Can't locate DBI.pm 

.

.

.

Failure to apply SQL file 'IAStest.subscribe_series.sql' from server: ABORTING!

Can't locate DBD/Pg.pm

</pre>

<pre>

root@sdo3:/usr/local/netdrms_8.10/base/drms/replication/subscribe_series# apt-get install libdbi-perl 

root@sdo3:/usr/local/netdrms_8.10/base/drms/replication/subscribe_series# apt-get install libdbd-pg-perl

</pre>

h3. Series souscrites

hmi.sharp_720s_nrt

hmi.mharp_720s_nrt

hmi.bharp_720s_nrt

<pre>

production@sdo3:~$ show_info -s hmi.sharp_720s_nrt

First Record: hmi.sharp_720s_nrt[476][2012.09.14_02:12:00_TAI], Recnum = 829707

Last Record:  hmi.sharp_720s_nrt[4943][2016.03.22_18:24:00_TAI], Recnum = 2015202

Last Recnum:  2015202

Has shadow table: no

</pre>

h3. Lancement de jetty

<pre>

$ jetty.sh start

</pre>

h2. Test recuperation de meta data 

librairies manquantes 

apt-get install libnet-ssh-perl

apt-get install libstring-shellquote-perl

h2. TO BE DONE FURTHER

%{color:red} configure sytem nbr de fichier ouvert en meme temps%

%{color:red} configure sytem taille  buffer%

%{color:red} swap désactivé ? discussion Stephane et Gilles%

h3. Modification du fichir config.local pour la souscription au jsoc

<pre>

# NetDRMS users can request subdirectories from the Stanford proj directories 

# (e.g., JSOC/proj/util). These users will need to contact Stanford and obtain

# the source subdirectories desired. They will then place these subdirectories

# in the JSOC/proj directory of their NetDRMS release. To properly build 

# targets in these subdirectories, using the JSOC make system, the user needs

# to add entries to this configuration file - one entry for each subdirectory

# that contains source code to be compiled. Each entry is a space-separated pair

# of strings: the string "PROJDIR" followed by a subdirectory (of the proj 

# directory).  For example:

#

# PROJDIR     util/apps

# PROJDIR     util/libs

# 

# NOTE: It is not sufficient to list only a parent directory, like util. Source code

# in child directories will not automatically be compiled.

</pre>

h3. Python library for multi-threaded SUMS

Installation de la librairie python3.4m