MEDOC - Multi Experiment Data & Operation Center » NetDrms_9.0

h1. Installation Netdrms_9.0

h2. Create dir 

Create the directory to untar netdrms_9.0  from  http://jsoc.stanford.edu/~arta/for-pablo/netdrms-9.X.20170530tar.gz

<pre>

mkdir /usr/local/netdrms_9.0

</pre>

Create the directory netdmrs_9.1.1 (location Art's home directory at Stanford )  as root 

<pre>

mkdir /usr/local/netdrms_9.0

</pre>

Change owner for production:ias 

<pre>

chown -R production:ias netdrms_9.1.1 

</pre>

h2. Configuration file 

<pre>

cp config.local.template config.local

</pre>

Config from netdmrs_8.12 

Warning : Config not done for remote sum as jmd is used 

h2. File configure modif

In file /usr/local/netdrms_current/configure

Replace all " $? !=0 " by " $status !=0 " in file configure then tape

h2. File jsoc_machine.csh modif 

In file /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.11 )

ligne 28 :

<pre>

case "x86_64":

      echo linux_x86_64

      breaksw

</pre>

h2. File make_basic.mk 

In file make_basic.mk line 223

look for "GCC_LF_ALL ="

add "-lcrypto"

h2. Execution configure

<pre>

./configure

</pre>

h2. Build 

Type 

<pre>

make

make sums

</pre>

For netdrms_9.1.1 as we are going to use new remote sums (written in python) 

we only need to do the following

<pre>

make show_info 

make show_series 

</pre>

Checking the modif on Db from NetDrms 2.4 we have figured out that some index where added

The other modif are for sum_rm use ...

Not needed at IAS  

On sdo and sdo3 modification in the DB ( index addition)

<pre>

production@sdo:~$ psql ias_sdo_sums -p5434 

ias_sdo_sums=# CREATE INDEX sum_partn_alloc_wd_idx ON public.sum_partn_alloc(wd) ;

</pre>

Install OK

h2. Starting  Netdrms 

<pre>

Starting it I have got :

production@sdo3:/usr/local/netdrms_current$ sum_start.NetDRMS

Number of SUM process sets to spawn = 1

Max number of SUM process can spawn = 8

SUMSERVER is sdo3

SUMS_MANAGER is production

SUMS db is ias_sdo_sums

SUMS log dir is /usr/local/netdrms_current/logs/SUM

sum_start.NetDRMS at 2017.07.12_19:41:38

** NOTE: SUMS is already running

Do you want me to do a sum_stop.NetDRMS followed by a sum_start.NetDRMS (y or n):

y

SUMS log dir is /usr/local/netdrms_current/logs/SUM

sum_stop.NetDRMS at 2017.07.12_19:41:53

NOTE: This will stop sum_svc and related processes. Make sure you want to do this

Enter reason for stop (terminate with single '.' or ^D):

>> some reason

>> .

Please identify yourself:

>> Pablo

Timed-out waiting for sums_procck.py (pid 12640) to exit.

** Now starting SUMS on ias_sdo_sums

SUM NetDRMS Start Complete

production@sdo3:/usr/local/netdrms_current$ Running on SUMS-server host sdo3.

Connection to SUMS DB ias_sdo_sums.

Using SUMS-log file sum_svc_2017.07.12.192435.logR.

Using SUMS-rm-log file 2017.07.12.192435.log.

Process sum_svc is not running.

Starting process sum_svc.

Please wait for sum_svc and tape inventory (optional) to initialize...

Can't open config file /usr/local/netdrms_current/logs/SUM/sum_rm.cfg. Using defaults...

rm: impossible de supprimer « /SUM12/D355906038/Records.txt »: Système de fichiers accessible en lecture seulement

rm: impossible de supprimer « /SUM12/D355906038/S00000/spikes.fits »: Système de fichiers accessible en lecture seulement

rm: impossible de supprimer « /SUM12/D355906038/S00000/image_lev1.fits »: Système de fichiers accessible en lecture seulement

(...)

</pre>

So I killed netdrms to fix that point 

<pre>

Shutting down SUMS...

Killing [sum_rm](pid 17376).

Killing sum_svc(pid 17371).

Removing pidfile /usr/local/netdrms_current/logs/SUM/sums.pidfile

Exiting process via shutdown() with return value 0.

</pre>

The sum_rm is running although in config.local I have set SUMRM_NOOP to 1

<pre>

# If the value is set to anything other than 0, then sum_rm is rendered inactive. Otherwise, sum_rm is active.

SUMRM_NOOP                             1

</pre>

So I put an old sum_rm.cfg in /home/production/netdrms_current/logs/SUM/ then it was ok

<pre>

production@sdo3:/usr/local/netdrms_current$ sum_start.NetDRMS

Number of SUM process sets to spawn = 1

Max number of SUM process can spawn = 8

SUMSERVER is sdo3

SUMS_MANAGER is production

SUMS db is ias_sdo_sums

SUMS log dir is /usr/local/netdrms_current/logs/SUM

sum_start.NetDRMS at 2017.07.12_19:46:51

** Now starting SUMS on ias_sdo_sums

SUM NetDRMS Start Complete

production@sdo3:/usr/local/netdrms_current$ Running on SUMS-server host sdo3.

Connection to SUMS DB ias_sdo_sums.

Using SUMS-log file sum_svc_2017.07.12.194651.log.

Using SUMS-rm-log file 2017.07.12.194651.log.

Process sum_svc is not running.

Starting process sum_svc.

Please wait for sum_svc and tape inventory (optional) to initialize...

Can't open the log file /home/production/netdrms_current/logs/SUM/sum_rm.log.2017.07.12.194651 for sum_rm on sdo3

## sum_rm for production  pid = 17672 ##

You are configured as the active user

cmd: cat /tmp/cmdgrep.17672.log | wc -l 1> /tmp/sum_rm.17672.log

Connect to Postgres host=sdo3 db=ias_sdo_sums user=production Ok

DS_PavailRequest

Called alrm_sig()

Jul 12 19:46:53

NOOP

production@sdo3:/usr/local/netdrms_current$ sum_svc now available

</pre>

h2. Test Netdrms_9.0

Quick test on the thing that bothered us with netdrms_2.4 and lead us to think about an update :

<pre>

production@sdo3:/usr/local/netdrms_current$ show_info -r -S -P 'hmi.sharp_720s_nrt[5398][! DATE__OBS>$(2016-12-20T23:00:00Z) AND DATE__OBS<$(2016-12-21T01:00:00Z) !]' key='DATE__OBS'

recnum    sunum    DATE__OBS    SUDIR

2190405    884413026    2016-12-20T23:10:30.80Z    /SUM13/D884413026/S00000

2190409    884414163    2016-12-20T23:22:30.80Z    /SUM13/D884414163/S00000

2190413    884416279    2016-12-20T23:34:30.80Z    /SUM13/D884416279/S00000

2190417    884417281    2016-12-20T23:46:30.80Z    /SUM13/D884417281/S00000

2190421    884419003    2016-12-20T23:58:30.80Z    /SUM13/D884419003/S00000

2190425    884421131    2016-12-21T00:10:30.90Z    /SUM13/D884421131/S00000

2190429    884422598    2016-12-21T00:22:30.90Z    /SUM13/D884422598/S00000

2190433    884424336    2016-12-21T00:34:30.90Z    /SUM13/D884424336/S00000

2190437    884425290    2016-12-21T00:46:30.90Z    /SUM13/D884425290/S00000

2190441    884427329    2016-12-21T00:58:30.90Z    /SUM13/D884427329/S00000

</pre>

h1. Use sumsd.py ,  rsumsd.py & sums-clientd.py 

* sumsd.py is the replacement for sum_svc. It runs on the 'sums server'. This is sdo1.

sumsd.py will open socket to communicate from the client to the server 

* rsumsd.py wiil need a couple of table to work  so let's create it 

This is remote sums; it also runs on sdo1. To use this, both the client machine (sol-calcul4) and the server machine (sdo1) will need access to the DB running on sdo1. 

So, for this to be working, you should be able to do a psql -h sdo1 ... from sol-calcul4 and connect.

* sums-clientd.py

 will take sunum in the table drms.ingested_sunums  to insert them into drms.rs_requests 

h2. create missing tables 

To be done too on sdo1 

<pre>

/usr/local/netdrms_current$ base/drms/scripts/rscreatetabs.py op=create  tabs=req,su

</pre>

ok

h2. ssh-agent

ssh-agent was not working so we did regenerate public key 

<pre>

ssh-keygen -t rsa

</pre>

then copy public key id_rsa.pub into ~/.ssh/authorized_keys on the targeted server  jsocport.stanford.edu

jsoc side, on  jsocport.stanford.edu

make sure sdo3 ip is referenced in /etc/hosts.netdrms

h2. Launch sumsd.py 

As 'production'

<pre>

sumsd.py --loglevel=debug &

</pre>

h2. Launch rsumsd.py 

The new remote sums server

As 'production'

<pre>

rsumsd.py --loglevel=debug &

</pre>

h2. Launch sums-clientd.py 

daemon that ingest sunums into drms.rs_requests table from drms.ingested_sunums

As 'production'

<pre>

sums-clientd.py --loglevel=debug & 

</pre>