Project

General

Profile

Wiki » History » Version 13

Pablo Alingery, 03/04/2018 16:05

1 9 Pablo Alingery
h1. NetDrms2.4 => Netdrms 9.1.1  
2 1 Pablo Alingery
3 1 Pablo Alingery
h2. Posgres 9.4.13 
4 1 Pablo Alingery
5 9 Pablo Alingery
6 1 Pablo Alingery
h3. Install additional modules
7 1 Pablo Alingery
8 1 Pablo Alingery
We need dblink for triggers between db 
9 1 Pablo Alingery
10 1 Pablo Alingery
<pre>
11 1 Pablo Alingery
root@sdo1:~# cd /usr/share/postgresql/9.4/extension
12 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# ls -al  dblink--1.1.sql
13 1 Pablo Alingery
-rw-r--r-- 1 root root 5831 Aug 10 14:53 dblink--1.1.sql
14 1 Pablo Alingery
</pre>
15 1 Pablo Alingery
16 1 Pablo Alingery
h4. ias_sdo
17 1 Pablo Alingery
18 1 Pablo Alingery
<pre>
19 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -p5432 -U postgres < dblink--1.1.sql
20 1 Pablo Alingery
Use "CREATE EXTENSION dblink" to load this file.
21 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# 
22 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# 
23 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# 
24 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -U postgres 
25 1 Pablo Alingery
psql (9.4.13)
26 1 Pablo Alingery
Type "help" for help.
27 4 Pablo Alingery
28 4 Pablo Alingery
29 1 Pablo Alingery
ias_sdo=# CREATE EXTENSION dblink ;
30 1 Pablo Alingery
ERROR:  extension "dblink" already exists
31 1 Pablo Alingery
ias_sdo=# \q
32 1 Pablo Alingery
</pre>
33 1 Pablo Alingery
34 1 Pablo Alingery
Seems ok !
35 1 Pablo Alingery
36 1 Pablo Alingery
h4. ias_sdo_sums 
37 1 Pablo Alingery
38 1 Pablo Alingery
<pre>
39 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres < dblink--1.1.sql 
40 1 Pablo Alingery
Use "CREATE EXTENSION dblink" to load this file.
41 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres 
42 1 Pablo Alingery
psql (9.4.13)
43 1 Pablo Alingery
Type "help" for help.
44 1 Pablo Alingery
45 1 Pablo Alingery
ias_sdo_sums=#  CREATE EXTENSION dblink ;
46 1 Pablo Alingery
ERROR:  extension "dblink" already exists
47 1 Pablo Alingery
ias_sdo_sums=# \q
48 1 Pablo Alingery
</pre>
49 1 Pablo Alingery
50 1 Pablo Alingery
Seems ok too !
51 1 Pablo Alingery
52 1 Pablo Alingery
h4. ias_sdo_monitor 
53 1 Pablo Alingery
54 1 Pablo Alingery
<pre>
55 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres < dblink--1.1.sql 
56 1 Pablo Alingery
Use "CREATE EXTENSION dblink" to load this file.
57 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres
58 1 Pablo Alingery
psql (9.4.13)
59 1 Pablo Alingery
Type "help" for help.
60 1 Pablo Alingery
61 1 Pablo Alingery
ias_sdo_monitor=# CREATE EXTENSION dblink ;
62 1 Pablo Alingery
ERROR:  extension "dblink" already exists
63 1 Pablo Alingery
</pre>
64 1 Pablo Alingery
65 1 Pablo Alingery
Ok too!
66 2 Pablo Alingery
67 2 Pablo Alingery
68 5 Pablo Alingery
h3. Ingest last dumps from sdo1
69 1 Pablo Alingery
70 5 Pablo Alingery
<pre>
71 5 Pablo Alingery
postgres@sdo1:~/dumps$ gunzip < ias_sdo_sums_2017-09-28.gz | psql ias_sdo_sums  -p5434 -U postgres
72 5 Pablo Alingery
73 5 Pablo Alingery
postgres@sdo1:~/dumps$ gunzip < ias_sdo_2017-09-28.gz | psql ias_sdo -p5432 -U postgres
74 5 Pablo Alingery
75 5 Pablo Alingery
postgres@sdo1:~/dumps$ gunzip < ias_sdo_monitor_2017-09-28.gz | psql ias_sdo_monitor -p5436 -U postgres
76 5 Pablo Alingery
</pre>
77 5 Pablo Alingery
78 5 Pablo Alingery
h3. Creation of missing tables & functions
79 5 Pablo Alingery
80 2 Pablo Alingery
TABLE drms.ingested_sunums
81 2 Pablo Alingery
82 2 Pablo Alingery
--When a client ingests a Slony log, each SUNUM of each data series record is copied
83 2 Pablo Alingery
-- into this table. The client can use this table of SUNUMs to prefetch SUs from the
84 2 Pablo Alingery
-- providing site. This SQL is ingested as the pg_user database user, the same
85 2 Pablo Alingery
-- user that will be ingesting Slony logs and prefetching SUs, so permissions on
86 2 Pablo Alingery
-- this table will be correct without having to execute a GRANT statement.
87 2 Pablo Alingery
-- 
88 2 Pablo Alingery
-- The namespace drms is required, and is created by NetDRMS.sql during the
89 2 Pablo Alingery
-- NetDRMS installation process.
90 2 Pablo Alingery
-- 
91 2 Pablo Alingery
92 2 Pablo Alingery
<pre>
93 2 Pablo Alingery
DROP TABLE IF EXISTS drms.ingested_sunums;
94 2 Pablo Alingery
CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL);
95 2 Pablo Alingery
CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime);
96 2 Pablo Alingery
</pre>
97 2 Pablo Alingery
98 2 Pablo Alingery
99 2 Pablo Alingery
100 2 Pablo Alingery
FUNCTION drms.capturesunum
101 2 Pablo Alingery
-- 
102 2 Pablo Alingery
-- For each table under replication, a trigger is created that calls this function, which
103 2 Pablo Alingery
-- then copies SUNUMs into the underlying table, drms.ingested_sunums.
104 2 Pablo Alingery
-- 
105 2 Pablo Alingery
-- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create
106 2 Pablo Alingery
-- this table). If this table does not exist, then this function is a no-op.
107 2 Pablo Alingery
-- 
108 2 Pablo Alingery
109 2 Pablo Alingery
<pre>
110 2 Pablo Alingery
CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS
111 2 Pablo Alingery
$capturesunumtrig$
112 2 Pablo Alingery
BEGIN
113 2 Pablo Alingery
    IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN
114 2 Pablo Alingery
      IF (TG_OP='INSERT' AND new.sunum > 0) THEN
115 2 Pablo Alingery
        IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN
116 2 Pablo Alingery
          RETURN NULL;
117 2 Pablo Alingery
        END IF;
118 2 Pablo Alingery
        INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp());
119 2 Pablo Alingery
      END IF;
120 2 Pablo Alingery
    END IF;
121 2 Pablo Alingery
122 2 Pablo Alingery
  RETURN NEW;
123 2 Pablo Alingery
END
124 2 Pablo Alingery
$capturesunumtrig$ LANGUAGE plpgsql;
125 2 Pablo Alingery
</pre>
126 3 Pablo Alingery
127 3 Pablo Alingery
128 3 Pablo Alingery
Add a trigger on every hmi table like this to use the function:
129 3 Pablo Alingery
130 3 Pablo Alingery
<pre>
131 4 Pablo Alingery
CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum();
132 1 Pablo Alingery
</pre>
133 5 Pablo Alingery
134 6 Pablo Alingery
NOT for aia.lev1 series because we are taking 1/5 data we don't want everything
135 6 Pablo Alingery
136 6 Pablo Alingery
h3. Create shadow & speed up the usage of aia.lev1 table 
137 6 Pablo Alingery
138 6 Pablo Alingery
h4. postgresql-plperl
139 6 Pablo Alingery
140 6 Pablo Alingery
In order to run createshadow we need postgresql-plperl
141 6 Pablo Alingery
142 6 Pablo Alingery
<pre>
143 6 Pablo Alingery
apt-get install postgresql-plperl 
144 6 Pablo Alingery
</pre>
145 6 Pablo Alingery
146 6 Pablo Alingery
h4. Install language plperl
147 6 Pablo Alingery
148 6 Pablo Alingery
Install plperl within postgres
149 6 Pablo Alingery
150 6 Pablo Alingery
<pre>
151 6 Pablo Alingery
createlang -p5432 -U postgres plperlu ias_sdo
152 6 Pablo Alingery
</pre>
153 6 Pablo Alingery
154 6 Pablo Alingery
h4. Create function updateshadow()
155 6 Pablo Alingery
156 6 Pablo Alingery
Create function updateshadow()
157 6 Pablo Alingery
158 6 Pablo Alingery
<pre>
159 6 Pablo Alingery
psql ias_sdo -p5432 -f /usr/local/netdrms_current/base/drms/libs/pg/updateshadow.sql
160 6 Pablo Alingery
</pre>
161 6 Pablo Alingery
162 6 Pablo Alingery
163 6 Pablo Alingery
So now we are able to launch createshadow()
164 6 Pablo Alingery
165 6 Pablo Alingery
<pre>
166 6 Pablo Alingery
createshadow series=aia.lev1
167 6 Pablo Alingery
</pre>
168 7 Pablo Alingery
169 7 Pablo Alingery
170 7 Pablo Alingery
h3. Get data from JSOC 
171 7 Pablo Alingery
172 7 Pablo Alingery
h4. Create readonly user 
173 7 Pablo Alingery
174 7 Pablo Alingery
<pre>
175 7 Pablo Alingery
production@sdo1:/usr/local/netdrms_current$ psql ias_sdo_sums -p5434 -U postgres 
176 7 Pablo Alingery
psql (9.4.13)
177 7 Pablo Alingery
Type "help" for help.
178 7 Pablo Alingery
179 7 Pablo Alingery
ias_sdo_sums=# CREATE USER readonlyuser;
180 7 Pablo Alingery
CREATE ROLE
181 7 Pablo Alingery
ias_sdo_sums=# GRANT CONNECT ON DATABASE ias_sdo_sums To readonlyuser ; 
182 7 Pablo Alingery
GRANT
183 1 Pablo Alingery
184 1 Pablo Alingery
</pre>
185 1 Pablo Alingery
186 1 Pablo Alingery
187 1 Pablo Alingery
188 9 Pablo Alingery
h2. Installation Netdrms_9.1.1
189 9 Pablo Alingery
190 9 Pablo Alingery
h3. Create netdrms directory 
191 9 Pablo Alingery
192 9 Pablo Alingery
Create the directory to untar netdrms_9.1.1  from  http://jsoc.stanford.edu/~arta/for-pablo/netdrms-9.1.1.tar.gz
193 9 Pablo Alingery
194 1 Pablo Alingery
<pre>
195 9 Pablo Alingery
mkdir /usr/local/netdrms_9.1.1
196 1 Pablo Alingery
</pre>
197 9 Pablo Alingery
198 9 Pablo Alingery
Change owner for production:ias 
199 9 Pablo Alingery
200 9 Pablo Alingery
<pre>
201 9 Pablo Alingery
chown -R production:ias netdrms_9.1.1 
202 9 Pablo Alingery
</pre>
203 9 Pablo Alingery
204 9 Pablo Alingery
205 9 Pablo Alingery
h3. Configuration file 
206 9 Pablo Alingery
207 9 Pablo Alingery
<pre>
208 9 Pablo Alingery
cp config.local.template config.local
209 9 Pablo Alingery
</pre>
210 9 Pablo Alingery
211 9 Pablo Alingery
Warning : Config has done for remote sum, indeed jmd is not used anymore
212 9 Pablo Alingery
213 9 Pablo Alingery
h3. Modification file 'configure' 
214 9 Pablo Alingery
215 9 Pablo Alingery
In file /usr/local/netdrms_current/configure
216 9 Pablo Alingery
217 9 Pablo Alingery
Replace all " $? !=0 " by " $status !=0 " in file configure then tape
218 9 Pablo Alingery
219 9 Pablo Alingery
see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/0fb3fa60d2b6124500739087c8edeaba91559449
220 9 Pablo Alingery
221 9 Pablo Alingery
222 9 Pablo Alingery
h3. Modiffication file jsoc_machine.csh  
223 9 Pablo Alingery
224 9 Pablo Alingery
In file /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.11 , NetDrms 9.0)
225 9 Pablo Alingery
226 9 Pablo Alingery
ligne 28 :
227 9 Pablo Alingery
228 9 Pablo Alingery
<pre>
229 9 Pablo Alingery
case "x86_64":
230 9 Pablo Alingery
      echo linux_x86_64
231 9 Pablo Alingery
      breaksw
232 9 Pablo Alingery
</pre>
233 9 Pablo Alingery
234 9 Pablo Alingery
see https://git.ias.u-psud.fr/palinger/netdrms_9.1.1/commit/e75212b1bc8ab493ae15ef3f8b50ebfbad065bc8
235 9 Pablo Alingery
236 9 Pablo Alingery
h3. File make_basic.mk 
237 9 Pablo Alingery
238 9 Pablo Alingery
In file make_basic.mk line 223
239 9 Pablo Alingery
240 9 Pablo Alingery
look for "GCC_LF_ALL ="
241 9 Pablo Alingery
add "-lcrypto"
242 9 Pablo Alingery
243 9 Pablo Alingery
244 9 Pablo Alingery
245 9 Pablo Alingery
h3. Execution configure
246 9 Pablo Alingery
247 9 Pablo Alingery
<pre>
248 9 Pablo Alingery
./configure
249 9 Pablo Alingery
</pre>
250 9 Pablo Alingery
251 9 Pablo Alingery
h2. Build 
252 9 Pablo Alingery
253 9 Pablo Alingery
As we are going to use new remote sums (written in python) 
254 9 Pablo Alingery
we only need to do the following
255 9 Pablo Alingery
256 9 Pablo Alingery
<pre>
257 9 Pablo Alingery
make show_info 
258 9 Pablo Alingery
make show_series 
259 9 Pablo Alingery
</pre>
260 9 Pablo Alingery
261 9 Pablo Alingery
Checking the modif on Db from NetDrms 2.4 we have figured out that some index where added
262 9 Pablo Alingery
The other modif are for sum_rm use ...
263 9 Pablo Alingery
Not needed at IAS  
264 9 Pablo Alingery
265 9 Pablo Alingery
On sdo and sdo3 modification in the DB ( index addition)
266 9 Pablo Alingery
267 9 Pablo Alingery
<pre>
268 9 Pablo Alingery
production@sdo:~$ psql ias_sdo_sums -p5434 
269 9 Pablo Alingery
ias_sdo_sums=# CREATE INDEX sum_partn_alloc_wd_idx ON public.sum_partn_alloc(wd) ;
270 9 Pablo Alingery
</pre>
271 9 Pablo Alingery
272 11 Pablo Alingery
h2. Start NetDrms
273 9 Pablo Alingery
274 9 Pablo Alingery
h3. Launch sums.py 
275 9 Pablo Alingery
276 9 Pablo Alingery
<pre>
277 11 Pablo Alingery
sumsd.py --loglevel=debug &
278 9 Pablo Alingery
</pre>
279 9 Pablo Alingery
280 9 Pablo Alingery
h3. Launch rsumsd-.py 
281 9 Pablo Alingery
282 9 Pablo Alingery
<pre>
283 9 Pablo Alingery
rsumsd.py --loglevel=debug &
284 9 Pablo Alingery
</pre>
285 9 Pablo Alingery
286 9 Pablo Alingery
287 9 Pablo Alingery
h3. Launch rsums-clientd.py 
288 9 Pablo Alingery
289 9 Pablo Alingery
<pre>
290 9 Pablo Alingery
rsums-clientd.py --loglevel=debug &
291 9 Pablo Alingery
</pre>
292 9 Pablo Alingery
293 9 Pablo Alingery
h2. Stop NetDrms 
294 9 Pablo Alingery
295 9 Pablo Alingery
h3. list processes and their pid
296 9 Pablo Alingery
297 9 Pablo Alingery
<pre>
298 9 Pablo Alingery
ps aux | grep python
299 9 Pablo Alingery
</pre>
300 9 Pablo Alingery
301 9 Pablo Alingery
h3. Stop rsums-clientd.py 
302 9 Pablo Alingery
303 9 Pablo Alingery
<pre>
304 9 Pablo Alingery
kill -2 #rsums-clientd.py_pid
305 9 Pablo Alingery
</pre>
306 9 Pablo Alingery
307 13 Pablo Alingery
h3. Stop rsumsd.py 
308 9 Pablo Alingery
309 9 Pablo Alingery
<pre>
310 12 Pablo Alingery
kill -2 #rsumsd.py_pid 
311 9 Pablo Alingery
</pre>
312 9 Pablo Alingery
313 9 Pablo Alingery
h3. Stop sums.py 
314 9 Pablo Alingery
315 9 Pablo Alingery
<pre>
316 9 Pablo Alingery
kill -2 #sums.py_pid 
317 9 Pablo Alingery
</pre>
318 9 Pablo Alingery
319 9 Pablo Alingery
320 9 Pablo Alingery
321 8 Pablo Alingery
322 8 Pablo Alingery
323 8 Pablo Alingery
324 8 Pablo Alingery
h1. NetDrms 9.1.1 FAQ 
325 8 Pablo Alingery
326 8 Pablo Alingery
h2. rsumsd.py keep crashing 
327 8 Pablo Alingery
328 8 Pablo Alingery
Error message :
329 8 Pablo Alingery
If you 've got sthg like in rslog_20171204.txt (for exemple) 
330 8 Pablo Alingery
<pre>
331 8 Pablo Alingery
2017-12-04 14:35:56 - DEBUG - rsumsd.py:423 - Exiting TerminationHandler.
332 8 Pablo Alingery
2017-12-04 14:35:56 - ERROR - rsumsd.py:441 - (get) No SU-table record exists for SU 995061214.
333 8 Pablo Alingery
2017-12-04 14:35:56 - INFO - rsumsd.py:429 - Exiting with return status 15.
334 8 Pablo Alingery
</pre>
335 8 Pablo Alingery
336 8 Pablo Alingery
Workaround  : 
337 8 Pablo Alingery
338 8 Pablo Alingery
1) Delete from ingested_sunums table 
339 8 Pablo Alingery
ias_sdo=# select * from drms.ingested_sunums where sunum=995061214;                                                                                  
340 8 Pablo Alingery
 sunum   |           starttime           
341 8 Pablo Alingery
-----------+-------------------------------
342 8 Pablo Alingery
 995061214 | 2017-11-30 18:02:18.085994+01
343 8 Pablo Alingery
(1 row)
344 8 Pablo Alingery
345 8 Pablo Alingery
ias_sdo=# delete from drms.ingested_sunums where sunum=995061214;
346 8 Pablo Alingery
DELETE 1
347 8 Pablo Alingery
348 8 Pablo Alingery
2)Delete from rs.requests 
349 8 Pablo Alingery
350 8 Pablo Alingery
<pre>
351 8 Pablo Alingery
ias_sdo=# select * from drms.rs_requests where sunums like '%995061214%' ;
352 8 Pablo Alingery
ias_sdo=# delete from drms.rs_requests where sunums like '%995061214%' ;
353 8 Pablo Alingery
DELETE 132
354 8 Pablo Alingery
</pre>
355 8 Pablo Alingery
356 8 Pablo Alingery
3) Restart daemons sumsd.py , rsumsd.py, rsums-clientd.py  as production 
357 8 Pablo Alingery
358 8 Pablo Alingery
<pre>
359 8 Pablo Alingery
sumsd.py &
360 8 Pablo Alingery
rsums.py &
361 8 Pablo Alingery
rsums-clientd.py &
362 8 Pablo Alingery
363 8 Pablo Alingery
</pre>