History

Installation Netdrms_9.0¶

Create dir¶

Create the directory to untar netdrms_9.0 from http://jsoc.stanford.edu/~arta/for-pablo/netdrms-9.X.20170530tar.gz

mkdir /usr/local/netdrms_9.0

Create the directory netdmrs_9.1.1 (location Art's home directory at Stanford ) as root

mkdir /usr/local/netdrms_9.0

Change owner for production:ias

chown -R production:ias netdrms_9.1.1

Configuration file¶

cp config.local.template config.local

Config from netdmrs_8.12
Warning : Config not done for remote sum as jmd is used

File configure modif¶

In file /usr/local/netdrms_current/configure

Replace all " $? !=0 " by " $status !=0 " in file configure then tape

File jsoc_machine.csh modif¶

In file /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.11 )

ligne 28 :

case "x86_64":
      echo linux_x86_64
      breaksw

File make_basic.mk¶

In file make_basic.mk line 223

look for "GCC_LF_ALL ="
add "-lcrypto"

Execution configure¶

./configure

Build¶

Type

make
make sums

For netdrms_9.1.1 as we are going to use new remote sums (written in python)
we only need to do the following

make show_info 
make show_series

Checking the modif on Db from NetDrms 2.4 we have figured out that some index where added
The other modif are for sum_rm use ...
Not needed at IAS

On sdo and sdo3 modification in the DB ( index addition)

production@sdo:~$ psql ias_sdo_sums -p5434 
ias_sdo_sums=# CREATE INDEX sum_partn_alloc_wd_idx ON public.sum_partn_alloc(wd) ;

Install OK

Starting Netdrms¶

Starting it I have got :
production@sdo3:/usr/local/netdrms_current$ sum_start.NetDRMS
Number of SUM process sets to spawn = 1
Max number of SUM process can spawn = 8
SUMSERVER is sdo3
SUMS_MANAGER is production
SUMS db is ias_sdo_sums
SUMS log dir is /usr/local/netdrms_current/logs/SUM
sum_start.NetDRMS at 2017.07.12_19:41:38

** NOTE: SUMS is already running
Do you want me to do a sum_stop.NetDRMS followed by a sum_start.NetDRMS (y or n):
y
SUMS log dir is /usr/local/netdrms_current/logs/SUM

sum_stop.NetDRMS at 2017.07.12_19:41:53

NOTE: This will stop sum_svc and related processes. Make sure you want to do this
Enter reason for stop (terminate with single '.' or ^D):
>> some reason
>> .
Please identify yourself:
>> Pablo
Timed-out waiting for sums_procck.py (pid 12640) to exit.
** Now starting SUMS on ias_sdo_sums
SUM NetDRMS Start Complete
production@sdo3:/usr/local/netdrms_current$ Running on SUMS-server host sdo3.
Connection to SUMS DB ias_sdo_sums.
Using SUMS-log file sum_svc_2017.07.12.192435.logR.
Using SUMS-rm-log file 2017.07.12.192435.log.
Process sum_svc is not running.
Starting process sum_svc.

Please wait for sum_svc and tape inventory (optional) to initialize...
Can't open config file /usr/local/netdrms_current/logs/SUM/sum_rm.cfg. Using defaults...
rm: impossible de supprimer « /SUM12/D355906038/Records.txt »: Système de fichiers accessible en lecture seulement
rm: impossible de supprimer « /SUM12/D355906038/S00000/spikes.fits »: Système de fichiers accessible en lecture seulement
rm: impossible de supprimer « /SUM12/D355906038/S00000/image_lev1.fits »: Système de fichiers accessible en lecture seulement
(...)

So I killed netdrms to fix that point

Shutting down SUMS...
Killing [sum_rm](pid 17376).
Killing sum_svc(pid 17371).
Removing pidfile /usr/local/netdrms_current/logs/SUM/sums.pidfile
Exiting process via shutdown() with return value 0.

The sum_rm is running although in config.local I have set SUMRM_NOOP to 1

# If the value is set to anything other than 0, then sum_rm is rendered inactive. Otherwise, sum_rm is active.
SUMRM_NOOP                             1

So I put an old sum_rm.cfg in /home/production/netdrms_current/logs/SUM/ then it was ok

production@sdo3:/usr/local/netdrms_current$ sum_start.NetDRMS
Number of SUM process sets to spawn = 1
Max number of SUM process can spawn = 8
SUMSERVER is sdo3
SUMS_MANAGER is production
SUMS db is ias_sdo_sums
SUMS log dir is /usr/local/netdrms_current/logs/SUM
sum_start.NetDRMS at 2017.07.12_19:46:51

** Now starting SUMS on ias_sdo_sums
SUM NetDRMS Start Complete
production@sdo3:/usr/local/netdrms_current$ Running on SUMS-server host sdo3.
Connection to SUMS DB ias_sdo_sums.
Using SUMS-log file sum_svc_2017.07.12.194651.log.
Using SUMS-rm-log file 2017.07.12.194651.log.
Process sum_svc is not running.
Starting process sum_svc.

Please wait for sum_svc and tape inventory (optional) to initialize...
Can't open the log file /home/production/netdrms_current/logs/SUM/sum_rm.log.2017.07.12.194651 for sum_rm on sdo3

## sum_rm for production  pid = 17672 ##
You are configured as the active user
cmd: cat /tmp/cmdgrep.17672.log | wc -l 1> /tmp/sum_rm.17672.log
Connect to Postgres host=sdo3 db=ias_sdo_sums user=production Ok
DS_PavailRequest
Called alrm_sig()
Jul 12 19:46:53
NOOP

production@sdo3:/usr/local/netdrms_current$ sum_svc now available

Test Netdrms_9.0¶

Quick test on the thing that bothered us with netdrms_2.4 and lead us to think about an update :

production@sdo3:/usr/local/netdrms_current$ show_info -r -S -P 'hmi.sharp_720s_nrt[5398][! DATE__OBS>$(2016-12-20T23:00:00Z) AND DATE__OBS<$(2016-12-21T01:00:00Z) !]' key='DATE__OBS'
recnum    sunum    DATE__OBS    SUDIR
2190405    884413026    2016-12-20T23:10:30.80Z    /SUM13/D884413026/S00000
2190409    884414163    2016-12-20T23:22:30.80Z    /SUM13/D884414163/S00000
2190413    884416279    2016-12-20T23:34:30.80Z    /SUM13/D884416279/S00000
2190417    884417281    2016-12-20T23:46:30.80Z    /SUM13/D884417281/S00000
2190421    884419003    2016-12-20T23:58:30.80Z    /SUM13/D884419003/S00000
2190425    884421131    2016-12-21T00:10:30.90Z    /SUM13/D884421131/S00000
2190429    884422598    2016-12-21T00:22:30.90Z    /SUM13/D884422598/S00000
2190433    884424336    2016-12-21T00:34:30.90Z    /SUM13/D884424336/S00000
2190437    884425290    2016-12-21T00:46:30.90Z    /SUM13/D884425290/S00000
2190441    884427329    2016-12-21T00:58:30.90Z    /SUM13/D884427329/S00000

Use sumsd.py , rsumsd.py & sums-clientd.py¶

sumsd.py is the replacement for sum_svc. It runs on the 'sums server'. This is sdo1.
sumsd.py will open socket to communicate from the client to the server

rsumsd.py wiil need a couple of table to work so let's create it
This is remote sums; it also runs on sdo1. To use this, both the client machine (sol-calcul4) and the server machine (sdo1) will need access to the DB running on sdo1.
So, for this to be working, you should be able to do a psql -h sdo1 ... from sol-calcul4 and connect.

sums-clientd.py
will take sunum in the table drms.ingested_sunums to insert them into drms.rs_requests

create missing tables¶

To be done too on sdo1

/usr/local/netdrms_current$ base/drms/scripts/rscreatetabs.py op=create  tabs=req,su

ssh-agent¶

ssh-agent was not working so we did regenerate public key

ssh-keygen -t rsa

then copy public key id_rsa.pub into ~/.ssh/authorized_keys on the targeted server jsocport.stanford.edu

jsoc side, on jsocport.stanford.edu
make sure sdo3 ip is referenced in /etc/hosts.netdrms

Launch sumsd.py¶

As 'production'

sumsd.py --loglevel=debug &

Launch rsumsd.py¶

The new remote sums server

As 'production'

rsumsd.py --loglevel=debug &

Launch sums-clientd.py¶

daemon that ingest sunums into drms.rs_requests table from drms.ingested_sunums

As 'production'

sums-clientd.py --loglevel=debug &

Files

Project

General

Profile

MEDOC - Multi Experiment Data & Operation Center » NetDrms_9.0

Wiki

Installation Netdrms_9.0¶

Create dir¶

Configuration file¶

File configure modif¶

File jsoc_machine.csh modif¶

File make_basic.mk¶

Execution configure¶

Build¶

Starting Netdrms¶

Test Netdrms_9.0¶

Use sumsd.py , rsumsd.py & sums-clientd.py¶

create missing tables¶

ssh-agent¶

Launch sumsd.py¶

Launch rsumsd.py¶

Launch sums-clientd.py¶