MEDOC - Multi Experiment Data & Operation Center » Netdrms_9.1.1

h1. NetDrms2.4 => Netdrms 9.1.1  

h2. Posgres 9.4.13 

h3. Install additional modules

We need dblink for triggers between db 

<pre>

root@sdo1:~# cd /usr/share/postgresql/9.4/extension

root@sdo1:/usr/share/postgresql/9.4/extension# ls -al  dblink--1.1.sql

-rw-r--r-- 1 root root 5831 Aug 10 14:53 dblink--1.1.sql

</pre>

h4. ias_sdo

<pre>

root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -p5432 -U postgres < dblink--1.1.sql

Use "CREATE EXTENSION dblink" to load this file.

root@sdo1:/usr/share/postgresql/9.4/extension# 

root@sdo1:/usr/share/postgresql/9.4/extension# 

root@sdo1:/usr/share/postgresql/9.4/extension# 

root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -U postgres 

psql (9.4.13)

Type "help" for help.

ias_sdo=# CREATE EXTENSION dblink ;

ERROR:  extension "dblink" already exists

ias_sdo=# \q

</pre>

Seems ok !

h4. ias_sdo_sums 

<pre>

root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres < dblink--1.1.sql 

Use "CREATE EXTENSION dblink" to load this file.

root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres 

psql (9.4.13)

Type "help" for help.

ias_sdo_sums=#  CREATE EXTENSION dblink ;

ERROR:  extension "dblink" already exists

ias_sdo_sums=# \q

</pre>

Seems ok too !

h4. ias_sdo_monitor 

<pre>

root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres < dblink--1.1.sql 

Use "CREATE EXTENSION dblink" to load this file.

root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres

psql (9.4.13)

Type "help" for help.

ias_sdo_monitor=# CREATE EXTENSION dblink ;

ERROR:  extension "dblink" already exists

</pre>

Ok too!

h3. Ingest last dumps from sdo1

<pre>

postgres@sdo1:~/dumps$ gunzip < ias_sdo_sums_2017-09-28.gz | psql ias_sdo_sums  -p5434 -U postgres

postgres@sdo1:~/dumps$ gunzip < ias_sdo_2017-09-28.gz | psql ias_sdo -p5432 -U postgres

postgres@sdo1:~/dumps$ gunzip < ias_sdo_monitor_2017-09-28.gz | psql ias_sdo_monitor -p5436 -U postgres

</pre>

h3. Creation of missing tables & functions

TABLE drms.ingested_sunums

--When a client ingests a Slony log, each SUNUM of each data series record is copied

-- into this table. The client can use this table of SUNUMs to prefetch SUs from the

-- providing site. This SQL is ingested as the pg_user database user, the same

-- user that will be ingesting Slony logs and prefetching SUs, so permissions on

-- this table will be correct without having to execute a GRANT statement.

-- 

-- The namespace drms is required, and is created by NetDRMS.sql during the

-- NetDRMS installation process.

-- 

<pre>

DROP TABLE IF EXISTS drms.ingested_sunums;

CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL);

CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime);

</pre>

FUNCTION drms.capturesunum

-- 

-- For each table under replication, a trigger is created that calls this function, which

-- then copies SUNUMs into the underlying table, drms.ingested_sunums.

-- 

-- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create

-- this table). If this table does not exist, then this function is a no-op.

-- 

<pre>

CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS

$capturesunumtrig$

BEGIN

    IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN

      IF (TG_OP='INSERT' AND new.sunum > 0) THEN

        IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN

          RETURN NULL;

        END IF;

        INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp());

      END IF;

    END IF;

  RETURN NEW;

END

$capturesunumtrig$ LANGUAGE plpgsql;

</pre>

Add a trigger on every hmi table like this to use the function:

CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum();

> -- TABLE drms.ingested_sunums

> -- 

> -- When a client ingests a Slony log, each SUNUM of each data series record is copied

> -- into this table. The client can use this table of SUNUMs to prefetch SUs from the

> -- providing site. This SQL is ingested as the pg_user database user, the same

> -- user that will be ingesting Slony logs and prefetching SUs, so permissions on

> -- this table will be correct without having to execute a GRANT statement.

> -- 

> -- The namespace drms is required, and is created by NetDRMS.sql during the

> -- NetDRMS installation process.

> -- 

<pre>

> DROP TABLE IF EXISTS drms.ingested_sunums;

> CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL);

> CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime);

</pre>

> -- FUNCTION drms.capturesunum

> -- 

> -- For each table under replication, a trigger is created that calls this function, which

> -- then copies SUNUMs into the underlying table, drms.ingested_sunums.

> -- 

> -- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create

> -- this table). If this table does not exist, then this function is a no-op.

> -- 

<pre>

> CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS

> $capturesunumtrig$

> BEGIN

>     IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN

>       IF (TG_OP='INSERT' AND new.sunum > 0) THEN

>         IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN

>           RETURN NULL;

>         END IF;

>         INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp());

>       END IF;

>     END IF;

>

>   RETURN NEW;

> END

> $capturesunumtrig$ LANGUAGE plpgsql;

</pre>

 Add a trigger on every hmi table like this to use the function:

<pre>

CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum();

</pre>

NOT for aia.lev1 series because we are taking 1/5 data we don't want everything

h3. Create shadow & speed up the usage of aia.lev1 table 

h4. postgresql-plperl

In order to run createshadow we need postgresql-plperl

<pre>

apt-get install postgresql-plperl 

</pre>

h4. Install language plperl

Install plperl within postgres

<pre>

createlang -p5432 -U postgres plperlu ias_sdo

</pre>

h4. Create function updateshadow()

Create function updateshadow()

<pre>

psql ias_sdo -p5432 -f /usr/local/netdrms_current/base/drms/libs/pg/updateshadow.sql

</pre>

So now we are able to launch createshadow()

<pre>

createshadow series=aia.lev1

</pre>

h3. Get data from JSOC 

h4. Create readonly user 

<pre>

production@sdo1:/usr/local/netdrms_current$ psql ias_sdo_sums -p5434 -U postgres 

psql (9.4.13)

Type "help" for help.

ias_sdo_sums=# CREATE USER readonlyuser;

CREATE ROLE

ias_sdo_sums=# GRANT CONNECT ON DATABASE ias_sdo_sums To readonlyuser ; 

GRANT

</pre>

h2. Installation Netdrms_9.1.1

h3. Create netdrms directory 

Create the directory to untar netdrms_9.1.1  from  http://jsoc.stanford.edu/~arta/for-pablo/netdrms-9.1.1.tar.gz

<pre>

mkdir /usr/local/netdrms_9.1.1

</pre>

Change owner for production:ias 

<pre>

chown -R production:ias netdrms_9.1.1 

</pre>

h3. Configuration file 

<pre>

cp config.local.template config.local

</pre>

Warning : Config has done for remote sum, indeed jmd is not used anymore

h3. Modification file 'configure' 

In file /usr/local/netdrms_current/configure

Replace all " $? !=0 " by " $status !=0 " in file configure then tape

see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/0fb3fa60d2b6124500739087c8edeaba91559449

h3. Modiffication file jsoc_machine.csh  

In file /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.11 , NetDrms 9.0)

ligne 28 :

<pre>

case "x86_64":

      echo linux_x86_64

      breaksw

</pre>

see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/e75212b1bc8ab493ae15ef3f8b50ebfbad065bc8

h3. File make_basic.mk 

In file make_basic.mk line 223

look for "GCC_LF_ALL ="

add "-lcrypto"

h3. Execution configure

<pre>

./configure

</pre>

h2. Build 

As we are going to use new remote sums (written in python) 

we only need to do the following

<pre>

make show_info 

make show_series 

</pre>

Checking the modif on Db from NetDrms 2.4 we have figured out that some index where added

The other modif are for sum_rm use ...

Not needed at IAS  

On sdo and sdo3 modification in the DB ( index addition)

<pre>

production@sdo:~$ psql ias_sdo_sums -p5434 

ias_sdo_sums=# CREATE INDEX sum_partn_alloc_wd_idx ON public.sum_partn_alloc(wd) ;

</pre>

h2. Start NetDrms 

h3. Launch sums.py 

<pre>

sums.py --loglevel=debug &

</pre>

h3. Launch rsumsd-.py 

<pre>

rsumsd.py --loglevel=debug &

</pre>

h3. Launch rsums-clientd.py 

<pre>

rsums-clientd.py --loglevel=debug &

</pre>

h2. Stop NetDrms 

h3. list processes and their pid

<pre>

ps aux | grep python

</pre>

h3. Stop rsums-clientd.py 

<pre>

kill -2 #rsums-clientd.py_pid

</pre>

h3. Stop rsumsd-.py 

<pre>

kill -2 #rsumsd.py _pid 

</pre>

h3. Stop sums.py 

<pre>

kill -2 #sums.py_pid 

</pre>

h1. NetDrms 9.1.1 FAQ 

h2. rsumsd.py keep crashing 

Error message :

If you 've got sthg like in rslog_20171204.txt (for exemple) 

<pre>

2017-12-04 14:35:56 - DEBUG - rsumsd.py:423 - Exiting TerminationHandler.

2017-12-04 14:35:56 - ERROR - rsumsd.py:441 - (get) No SU-table record exists for SU 995061214.

2017-12-04 14:35:56 - INFO - rsumsd.py:429 - Exiting with return status 15.

</pre>

Workaround  : 

1) Delete from ingested_sunums table 

ias_sdo=# select * from drms.ingested_sunums where sunum=995061214;                                                                                  

 sunum   |           starttime           

-----------+-------------------------------

 995061214 | 2017-11-30 18:02:18.085994+01

(1 row)

ias_sdo=# delete from drms.ingested_sunums where sunum=995061214;

DELETE 1

2)Delete from rs.requests 

<pre>

ias_sdo=# select * from drms.rs_requests where sunums like '%995061214%' ;

ias_sdo=# delete from drms.rs_requests where sunums like '%995061214%' ;

DELETE 132

</pre>

3) Restart daemons sumsd.py , rsumsd.py, rsums-clientd.py  as production 

<pre>

sumsd.py &

rsums.py &

rsums-clientd.py &

</pre>