Wiki » History » Version 11
Version 10 (Pablo Alingery, 29/12/2017 13:47) → Version 11/14 (Pablo Alingery, 03/04/2018 16:02)
h1. NetDrms2.4 => Netdrms 9.1.1
h2. Posgres 9.4.13
h3. Install additional modules
We need dblink for triggers between db
<pre>
root@sdo1:~# cd /usr/share/postgresql/9.4/extension
root@sdo1:/usr/share/postgresql/9.4/extension# ls -al dblink--1.1.sql
-rw-r--r-- 1 root root 5831 Aug 10 14:53 dblink--1.1.sql
</pre>
h4. ias_sdo
<pre>
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -p5432 -U postgres < dblink--1.1.sql
Use "CREATE EXTENSION dblink" to load this file.
root@sdo1:/usr/share/postgresql/9.4/extension#
root@sdo1:/usr/share/postgresql/9.4/extension#
root@sdo1:/usr/share/postgresql/9.4/extension#
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
ERROR: extension "dblink" already exists
ias_sdo=# \q
</pre>
Seems ok !
h4. ias_sdo_sums
<pre>
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres < dblink--1.1.sql
Use "CREATE EXTENSION dblink" to load this file.
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo_sums=# CREATE EXTENSION dblink ;
ERROR: extension "dblink" already exists
ias_sdo_sums=# \q
</pre>
Seems ok too !
h4. ias_sdo_monitor
<pre>
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres < dblink--1.1.sql
Use "CREATE EXTENSION dblink" to load this file.
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo_monitor=# CREATE EXTENSION dblink ;
ERROR: extension "dblink" already exists
</pre>
Ok too!
h3. Ingest last dumps from sdo1
<pre>
postgres@sdo1:~/dumps$ gunzip < ias_sdo_sums_2017-09-28.gz | psql ias_sdo_sums -p5434 -U postgres
postgres@sdo1:~/dumps$ gunzip < ias_sdo_2017-09-28.gz | psql ias_sdo -p5432 -U postgres
postgres@sdo1:~/dumps$ gunzip < ias_sdo_monitor_2017-09-28.gz | psql ias_sdo_monitor -p5436 -U postgres
</pre>
h3. Creation of missing tables & functions
TABLE drms.ingested_sunums
--When a client ingests a Slony log, each SUNUM of each data series record is copied
-- into this table. The client can use this table of SUNUMs to prefetch SUs from the
-- providing site. This SQL is ingested as the pg_user database user, the same
-- user that will be ingesting Slony logs and prefetching SUs, so permissions on
-- this table will be correct without having to execute a GRANT statement.
--
-- The namespace drms is required, and is created by NetDRMS.sql during the
-- NetDRMS installation process.
--
<pre>
DROP TABLE IF EXISTS drms.ingested_sunums;
CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL);
CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime);
</pre>
FUNCTION drms.capturesunum
--
-- For each table under replication, a trigger is created that calls this function, which
-- then copies SUNUMs into the underlying table, drms.ingested_sunums.
--
-- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create
-- this table). If this table does not exist, then this function is a no-op.
--
<pre>
CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS
$capturesunumtrig$
BEGIN
IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN
IF (TG_OP='INSERT' AND new.sunum > 0) THEN
IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN
RETURN NULL;
END IF;
INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp());
END IF;
END IF;
RETURN NEW;
END
$capturesunumtrig$ LANGUAGE plpgsql;
</pre>
Add a trigger on every hmi table like this to use the function:
<pre>
CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum();
</pre>
NOT for aia.lev1 series because we are taking 1/5 data we don't want everything
h3. Create shadow & speed up the usage of aia.lev1 table
h4. postgresql-plperl
In order to run createshadow we need postgresql-plperl
<pre>
apt-get install postgresql-plperl
</pre>
h4. Install language plperl
Install plperl within postgres
<pre>
createlang -p5432 -U postgres plperlu ias_sdo
</pre>
h4. Create function updateshadow()
Create function updateshadow()
<pre>
psql ias_sdo -p5432 -f /usr/local/netdrms_current/base/drms/libs/pg/updateshadow.sql
</pre>
So now we are able to launch createshadow()
<pre>
createshadow series=aia.lev1
</pre>
h3. Get data from JSOC
h4. Create readonly user
<pre>
production@sdo1:/usr/local/netdrms_current$ psql ias_sdo_sums -p5434 -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo_sums=# CREATE USER readonlyuser;
CREATE ROLE
ias_sdo_sums=# GRANT CONNECT ON DATABASE ias_sdo_sums To readonlyuser ;
GRANT
</pre>
h2. Installation Netdrms_9.1.1
h3. Create netdrms directory
Create the directory to untar netdrms_9.1.1 from http://jsoc.stanford.edu/~arta/for-pablo/netdrms-9.1.1.tar.gz
<pre>
mkdir /usr/local/netdrms_9.1.1
</pre>
Change owner for production:ias
<pre>
chown -R production:ias netdrms_9.1.1
</pre>
h3. Configuration file
<pre>
cp config.local.template config.local
</pre>
Warning : Config has done for remote sum, indeed jmd is not used anymore
h3. Modification file 'configure'
In file /usr/local/netdrms_current/configure
Replace all " $? !=0 " by " $status !=0 " in file configure then tape
see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/0fb3fa60d2b6124500739087c8edeaba91559449
h3. Modiffication file jsoc_machine.csh
In file /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.11 , NetDrms 9.0)
ligne 28 :
<pre>
case "x86_64":
echo linux_x86_64
breaksw
</pre>
see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/e75212b1bc8ab493ae15ef3f8b50ebfbad065bc8
h3. File make_basic.mk
In file make_basic.mk line 223
look for "GCC_LF_ALL ="
add "-lcrypto"
h3. Execution configure
<pre>
./configure
</pre>
h2. Build
As we are going to use new remote sums (written in python)
we only need to do the following
<pre>
make show_info
make show_series
</pre>
Checking the modif on Db from NetDrms 2.4 we have figured out that some index where added
The other modif are for sum_rm use ...
Not needed at IAS
On sdo and sdo3 modification in the DB ( index addition)
<pre>
production@sdo:~$ psql ias_sdo_sums -p5434
ias_sdo_sums=# CREATE INDEX sum_partn_alloc_wd_idx ON public.sum_partn_alloc(wd) ;
</pre>
h2. Start NetDrms
h3. Launch sums.py
<pre>
sumsd.py sums.py --loglevel=debug &
</pre>
h3. Launch rsumsd-.py
<pre>
rsumsd.py --loglevel=debug &
</pre>
h3. Launch rsums-clientd.py
<pre>
rsums-clientd.py --loglevel=debug &
</pre>
h2. Stop NetDrms
h3. list processes and their pid
<pre>
ps aux | grep python
</pre>
h3. Stop rsums-clientd.py
<pre>
kill -2 #rsums-clientd.py_pid
</pre>
h3. Stop rsumsd-.py
<pre>
kill -2 #rsumsd.py _pid
</pre>
h3. Stop sums.py
<pre>
kill -2 #sums.py_pid
</pre>
h1. NetDrms 9.1.1 FAQ
h2. rsumsd.py keep crashing
Error message :
If you 've got sthg like in rslog_20171204.txt (for exemple)
<pre>
2017-12-04 14:35:56 - DEBUG - rsumsd.py:423 - Exiting TerminationHandler.
2017-12-04 14:35:56 - ERROR - rsumsd.py:441 - (get) No SU-table record exists for SU 995061214.
2017-12-04 14:35:56 - INFO - rsumsd.py:429 - Exiting with return status 15.
</pre>
Workaround :
1) Delete from ingested_sunums table
ias_sdo=# select * from drms.ingested_sunums where sunum=995061214;
sunum | starttime
-----------+-------------------------------
995061214 | 2017-11-30 18:02:18.085994+01
(1 row)
ias_sdo=# delete from drms.ingested_sunums where sunum=995061214;
DELETE 1
2)Delete from rs.requests
<pre>
ias_sdo=# select * from drms.rs_requests where sunums like '%995061214%' ;
ias_sdo=# delete from drms.rs_requests where sunums like '%995061214%' ;
DELETE 132
</pre>
3) Restart daemons sumsd.py , rsumsd.py, rsums-clientd.py as production
<pre>
sumsd.py &
rsums.py &
rsums-clientd.py &
</pre>
h2. Posgres 9.4.13
h3. Install additional modules
We need dblink for triggers between db
<pre>
root@sdo1:~# cd /usr/share/postgresql/9.4/extension
root@sdo1:/usr/share/postgresql/9.4/extension# ls -al dblink--1.1.sql
-rw-r--r-- 1 root root 5831 Aug 10 14:53 dblink--1.1.sql
</pre>
h4. ias_sdo
<pre>
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -p5432 -U postgres < dblink--1.1.sql
Use "CREATE EXTENSION dblink" to load this file.
root@sdo1:/usr/share/postgresql/9.4/extension#
root@sdo1:/usr/share/postgresql/9.4/extension#
root@sdo1:/usr/share/postgresql/9.4/extension#
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo=# CREATE EXTENSION dblink ;
ERROR: extension "dblink" already exists
ias_sdo=# \q
</pre>
Seems ok !
h4. ias_sdo_sums
<pre>
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres < dblink--1.1.sql
Use "CREATE EXTENSION dblink" to load this file.
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo_sums=# CREATE EXTENSION dblink ;
ERROR: extension "dblink" already exists
ias_sdo_sums=# \q
</pre>
Seems ok too !
h4. ias_sdo_monitor
<pre>
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres < dblink--1.1.sql
Use "CREATE EXTENSION dblink" to load this file.
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo_monitor=# CREATE EXTENSION dblink ;
ERROR: extension "dblink" already exists
</pre>
Ok too!
h3. Ingest last dumps from sdo1
<pre>
postgres@sdo1:~/dumps$ gunzip < ias_sdo_sums_2017-09-28.gz | psql ias_sdo_sums -p5434 -U postgres
postgres@sdo1:~/dumps$ gunzip < ias_sdo_2017-09-28.gz | psql ias_sdo -p5432 -U postgres
postgres@sdo1:~/dumps$ gunzip < ias_sdo_monitor_2017-09-28.gz | psql ias_sdo_monitor -p5436 -U postgres
</pre>
h3. Creation of missing tables & functions
TABLE drms.ingested_sunums
--When a client ingests a Slony log, each SUNUM of each data series record is copied
-- into this table. The client can use this table of SUNUMs to prefetch SUs from the
-- providing site. This SQL is ingested as the pg_user database user, the same
-- user that will be ingesting Slony logs and prefetching SUs, so permissions on
-- this table will be correct without having to execute a GRANT statement.
--
-- The namespace drms is required, and is created by NetDRMS.sql during the
-- NetDRMS installation process.
--
<pre>
DROP TABLE IF EXISTS drms.ingested_sunums;
CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL);
CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime);
</pre>
FUNCTION drms.capturesunum
--
-- For each table under replication, a trigger is created that calls this function, which
-- then copies SUNUMs into the underlying table, drms.ingested_sunums.
--
-- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create
-- this table). If this table does not exist, then this function is a no-op.
--
<pre>
CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS
$capturesunumtrig$
BEGIN
IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN
IF (TG_OP='INSERT' AND new.sunum > 0) THEN
IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN
RETURN NULL;
END IF;
INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp());
END IF;
END IF;
RETURN NEW;
END
$capturesunumtrig$ LANGUAGE plpgsql;
</pre>
Add a trigger on every hmi table like this to use the function:
<pre>
CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum();
</pre>
NOT for aia.lev1 series because we are taking 1/5 data we don't want everything
h3. Create shadow & speed up the usage of aia.lev1 table
h4. postgresql-plperl
In order to run createshadow we need postgresql-plperl
<pre>
apt-get install postgresql-plperl
</pre>
h4. Install language plperl
Install plperl within postgres
<pre>
createlang -p5432 -U postgres plperlu ias_sdo
</pre>
h4. Create function updateshadow()
Create function updateshadow()
<pre>
psql ias_sdo -p5432 -f /usr/local/netdrms_current/base/drms/libs/pg/updateshadow.sql
</pre>
So now we are able to launch createshadow()
<pre>
createshadow series=aia.lev1
</pre>
h3. Get data from JSOC
h4. Create readonly user
<pre>
production@sdo1:/usr/local/netdrms_current$ psql ias_sdo_sums -p5434 -U postgres
psql (9.4.13)
Type "help" for help.
ias_sdo_sums=# CREATE USER readonlyuser;
CREATE ROLE
ias_sdo_sums=# GRANT CONNECT ON DATABASE ias_sdo_sums To readonlyuser ;
GRANT
</pre>
h2. Installation Netdrms_9.1.1
h3. Create netdrms directory
Create the directory to untar netdrms_9.1.1 from http://jsoc.stanford.edu/~arta/for-pablo/netdrms-9.1.1.tar.gz
<pre>
mkdir /usr/local/netdrms_9.1.1
</pre>
Change owner for production:ias
<pre>
chown -R production:ias netdrms_9.1.1
</pre>
h3. Configuration file
<pre>
cp config.local.template config.local
</pre>
Warning : Config has done for remote sum, indeed jmd is not used anymore
h3. Modification file 'configure'
In file /usr/local/netdrms_current/configure
Replace all " $? !=0 " by " $status !=0 " in file configure then tape
see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/0fb3fa60d2b6124500739087c8edeaba91559449
h3. Modiffication file jsoc_machine.csh
In file /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.11 , NetDrms 9.0)
ligne 28 :
<pre>
case "x86_64":
echo linux_x86_64
breaksw
</pre>
see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/e75212b1bc8ab493ae15ef3f8b50ebfbad065bc8
h3. File make_basic.mk
In file make_basic.mk line 223
look for "GCC_LF_ALL ="
add "-lcrypto"
h3. Execution configure
<pre>
./configure
</pre>
h2. Build
As we are going to use new remote sums (written in python)
we only need to do the following
<pre>
make show_info
make show_series
</pre>
Checking the modif on Db from NetDrms 2.4 we have figured out that some index where added
The other modif are for sum_rm use ...
Not needed at IAS
On sdo and sdo3 modification in the DB ( index addition)
<pre>
production@sdo:~$ psql ias_sdo_sums -p5434
ias_sdo_sums=# CREATE INDEX sum_partn_alloc_wd_idx ON public.sum_partn_alloc(wd) ;
</pre>
h2. Start NetDrms
h3. Launch sums.py
<pre>
sumsd.py sums.py --loglevel=debug &
</pre>
h3. Launch rsumsd-.py
<pre>
rsumsd.py --loglevel=debug &
</pre>
h3. Launch rsums-clientd.py
<pre>
rsums-clientd.py --loglevel=debug &
</pre>
h2. Stop NetDrms
h3. list processes and their pid
<pre>
ps aux | grep python
</pre>
h3. Stop rsums-clientd.py
<pre>
kill -2 #rsums-clientd.py_pid
</pre>
h3. Stop rsumsd-.py
<pre>
kill -2 #rsumsd.py _pid
</pre>
h3. Stop sums.py
<pre>
kill -2 #sums.py_pid
</pre>
h1. NetDrms 9.1.1 FAQ
h2. rsumsd.py keep crashing
Error message :
If you 've got sthg like in rslog_20171204.txt (for exemple)
<pre>
2017-12-04 14:35:56 - DEBUG - rsumsd.py:423 - Exiting TerminationHandler.
2017-12-04 14:35:56 - ERROR - rsumsd.py:441 - (get) No SU-table record exists for SU 995061214.
2017-12-04 14:35:56 - INFO - rsumsd.py:429 - Exiting with return status 15.
</pre>
Workaround :
1) Delete from ingested_sunums table
ias_sdo=# select * from drms.ingested_sunums where sunum=995061214;
sunum | starttime
-----------+-------------------------------
995061214 | 2017-11-30 18:02:18.085994+01
(1 row)
ias_sdo=# delete from drms.ingested_sunums where sunum=995061214;
DELETE 1
2)Delete from rs.requests
<pre>
ias_sdo=# select * from drms.rs_requests where sunums like '%995061214%' ;
ias_sdo=# delete from drms.rs_requests where sunums like '%995061214%' ;
DELETE 132
</pre>
3) Restart daemons sumsd.py , rsumsd.py, rsums-clientd.py as production
<pre>
sumsd.py &
rsums.py &
rsums-clientd.py &
</pre>