Project

General

Profile

Wiki » History » Version 7

Pablo Alingery, 29/09/2017 12:22

1 1 Pablo Alingery
h1. Netdrms 9.1.1  
2 1 Pablo Alingery
3 1 Pablo Alingery
h2. Posgres 9.4.13 
4 1 Pablo Alingery
5 1 Pablo Alingery
h3. Install additional modules
6 1 Pablo Alingery
7 1 Pablo Alingery
We need dblink for triggers between db 
8 1 Pablo Alingery
9 1 Pablo Alingery
<pre>
10 1 Pablo Alingery
root@sdo1:~# cd /usr/share/postgresql/9.4/extension
11 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# ls -al  dblink--1.1.sql
12 1 Pablo Alingery
-rw-r--r-- 1 root root 5831 Aug 10 14:53 dblink--1.1.sql
13 1 Pablo Alingery
</pre>
14 1 Pablo Alingery
15 1 Pablo Alingery
h4. ias_sdo
16 1 Pablo Alingery
17 1 Pablo Alingery
<pre>
18 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -p5432 -U postgres < dblink--1.1.sql
19 1 Pablo Alingery
Use "CREATE EXTENSION dblink" to load this file.
20 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# 
21 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# 
22 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# 
23 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -U postgres 
24 1 Pablo Alingery
psql (9.4.13)
25 1 Pablo Alingery
Type "help" for help.
26 1 Pablo Alingery
27 4 Pablo Alingery
28 4 Pablo Alingery
29 4 Pablo Alingery
30 4 Pablo Alingery
31 4 Pablo Alingery
32 1 Pablo Alingery
ias_sdo=# CREATE EXTENSION dblink ;
33 1 Pablo Alingery
ERROR:  extension "dblink" already exists
34 1 Pablo Alingery
ias_sdo=# \q
35 1 Pablo Alingery
</pre>
36 1 Pablo Alingery
37 1 Pablo Alingery
Seems ok !
38 1 Pablo Alingery
39 1 Pablo Alingery
h4. ias_sdo_sums 
40 1 Pablo Alingery
41 1 Pablo Alingery
<pre>
42 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres < dblink--1.1.sql 
43 1 Pablo Alingery
Use "CREATE EXTENSION dblink" to load this file.
44 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres 
45 1 Pablo Alingery
psql (9.4.13)
46 1 Pablo Alingery
Type "help" for help.
47 1 Pablo Alingery
48 1 Pablo Alingery
ias_sdo_sums=#  CREATE EXTENSION dblink ;
49 1 Pablo Alingery
ERROR:  extension "dblink" already exists
50 1 Pablo Alingery
ias_sdo_sums=# \q
51 1 Pablo Alingery
</pre>
52 1 Pablo Alingery
53 1 Pablo Alingery
Seems ok too !
54 1 Pablo Alingery
55 1 Pablo Alingery
h4. ias_sdo_monitor 
56 1 Pablo Alingery
57 1 Pablo Alingery
<pre>
58 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres < dblink--1.1.sql 
59 1 Pablo Alingery
Use "CREATE EXTENSION dblink" to load this file.
60 1 Pablo Alingery
root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres
61 1 Pablo Alingery
psql (9.4.13)
62 1 Pablo Alingery
Type "help" for help.
63 1 Pablo Alingery
64 1 Pablo Alingery
ias_sdo_monitor=# CREATE EXTENSION dblink ;
65 1 Pablo Alingery
ERROR:  extension "dblink" already exists
66 1 Pablo Alingery
</pre>
67 1 Pablo Alingery
68 1 Pablo Alingery
Ok too!
69 2 Pablo Alingery
70 2 Pablo Alingery
71 5 Pablo Alingery
h3. Ingest last dumps from sdo1
72 1 Pablo Alingery
73 5 Pablo Alingery
<pre>
74 5 Pablo Alingery
postgres@sdo1:~/dumps$ gunzip < ias_sdo_sums_2017-09-28.gz | psql ias_sdo_sums  -p5434 -U postgres
75 5 Pablo Alingery
76 5 Pablo Alingery
postgres@sdo1:~/dumps$ gunzip < ias_sdo_2017-09-28.gz | psql ias_sdo -p5432 -U postgres
77 5 Pablo Alingery
78 5 Pablo Alingery
postgres@sdo1:~/dumps$ gunzip < ias_sdo_monitor_2017-09-28.gz | psql ias_sdo_monitor -p5436 -U postgres
79 5 Pablo Alingery
</pre>
80 5 Pablo Alingery
81 5 Pablo Alingery
h3. Creation of missing tables & functions
82 5 Pablo Alingery
83 2 Pablo Alingery
TABLE drms.ingested_sunums
84 2 Pablo Alingery
85 2 Pablo Alingery
--When a client ingests a Slony log, each SUNUM of each data series record is copied
86 2 Pablo Alingery
-- into this table. The client can use this table of SUNUMs to prefetch SUs from the
87 2 Pablo Alingery
-- providing site. This SQL is ingested as the pg_user database user, the same
88 2 Pablo Alingery
-- user that will be ingesting Slony logs and prefetching SUs, so permissions on
89 2 Pablo Alingery
-- this table will be correct without having to execute a GRANT statement.
90 2 Pablo Alingery
-- 
91 2 Pablo Alingery
-- The namespace drms is required, and is created by NetDRMS.sql during the
92 2 Pablo Alingery
-- NetDRMS installation process.
93 2 Pablo Alingery
-- 
94 2 Pablo Alingery
95 2 Pablo Alingery
<pre>
96 2 Pablo Alingery
DROP TABLE IF EXISTS drms.ingested_sunums;
97 2 Pablo Alingery
CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL);
98 2 Pablo Alingery
CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime);
99 2 Pablo Alingery
</pre>
100 2 Pablo Alingery
101 2 Pablo Alingery
102 2 Pablo Alingery
103 2 Pablo Alingery
FUNCTION drms.capturesunum
104 2 Pablo Alingery
-- 
105 2 Pablo Alingery
-- For each table under replication, a trigger is created that calls this function, which
106 2 Pablo Alingery
-- then copies SUNUMs into the underlying table, drms.ingested_sunums.
107 2 Pablo Alingery
-- 
108 2 Pablo Alingery
-- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create
109 2 Pablo Alingery
-- this table). If this table does not exist, then this function is a no-op.
110 2 Pablo Alingery
-- 
111 2 Pablo Alingery
112 2 Pablo Alingery
<pre>
113 2 Pablo Alingery
CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS
114 2 Pablo Alingery
$capturesunumtrig$
115 2 Pablo Alingery
BEGIN
116 2 Pablo Alingery
    IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN
117 2 Pablo Alingery
      IF (TG_OP='INSERT' AND new.sunum > 0) THEN
118 2 Pablo Alingery
        IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN
119 2 Pablo Alingery
          RETURN NULL;
120 2 Pablo Alingery
        END IF;
121 2 Pablo Alingery
        INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp());
122 2 Pablo Alingery
      END IF;
123 2 Pablo Alingery
    END IF;
124 2 Pablo Alingery
125 2 Pablo Alingery
  RETURN NEW;
126 2 Pablo Alingery
END
127 2 Pablo Alingery
$capturesunumtrig$ LANGUAGE plpgsql;
128 2 Pablo Alingery
</pre>
129 3 Pablo Alingery
130 3 Pablo Alingery
131 3 Pablo Alingery
132 3 Pablo Alingery
Add a trigger on every hmi table like this to use the function:
133 3 Pablo Alingery
134 3 Pablo Alingery
CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum();
135 3 Pablo Alingery
136 3 Pablo Alingery
137 3 Pablo Alingery
> -- TABLE drms.ingested_sunums
138 3 Pablo Alingery
> -- 
139 3 Pablo Alingery
> -- When a client ingests a Slony log, each SUNUM of each data series record is copied
140 3 Pablo Alingery
> -- into this table. The client can use this table of SUNUMs to prefetch SUs from the
141 3 Pablo Alingery
> -- providing site. This SQL is ingested as the pg_user database user, the same
142 3 Pablo Alingery
> -- user that will be ingesting Slony logs and prefetching SUs, so permissions on
143 3 Pablo Alingery
> -- this table will be correct without having to execute a GRANT statement.
144 3 Pablo Alingery
> -- 
145 3 Pablo Alingery
> -- The namespace drms is required, and is created by NetDRMS.sql during the
146 3 Pablo Alingery
> -- NetDRMS installation process.
147 3 Pablo Alingery
> -- 
148 3 Pablo Alingery
149 3 Pablo Alingery
<pre>
150 3 Pablo Alingery
> DROP TABLE IF EXISTS drms.ingested_sunums;
151 3 Pablo Alingery
> CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL);
152 3 Pablo Alingery
> CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime);
153 3 Pablo Alingery
</pre>
154 3 Pablo Alingery
155 3 Pablo Alingery
> -- FUNCTION drms.capturesunum
156 3 Pablo Alingery
> -- 
157 3 Pablo Alingery
> -- For each table under replication, a trigger is created that calls this function, which
158 3 Pablo Alingery
> -- then copies SUNUMs into the underlying table, drms.ingested_sunums.
159 3 Pablo Alingery
> -- 
160 3 Pablo Alingery
> -- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create
161 3 Pablo Alingery
> -- this table). If this table does not exist, then this function is a no-op.
162 3 Pablo Alingery
> -- 
163 3 Pablo Alingery
164 3 Pablo Alingery
<pre>
165 3 Pablo Alingery
> CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS
166 3 Pablo Alingery
> $capturesunumtrig$
167 3 Pablo Alingery
> BEGIN
168 3 Pablo Alingery
>     IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN
169 3 Pablo Alingery
>       IF (TG_OP='INSERT' AND new.sunum > 0) THEN
170 3 Pablo Alingery
>         IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN
171 3 Pablo Alingery
>           RETURN NULL;
172 3 Pablo Alingery
>         END IF;
173 3 Pablo Alingery
>         INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp());
174 3 Pablo Alingery
>       END IF;
175 3 Pablo Alingery
>     END IF;
176 3 Pablo Alingery
>
177 3 Pablo Alingery
>   RETURN NEW;
178 3 Pablo Alingery
> END
179 3 Pablo Alingery
> $capturesunumtrig$ LANGUAGE plpgsql;
180 4 Pablo Alingery
</pre>
181 4 Pablo Alingery
182 4 Pablo Alingery
183 4 Pablo Alingery
 Add a trigger on every hmi table like this to use the function:
184 4 Pablo Alingery
185 4 Pablo Alingery
<pre>
186 4 Pablo Alingery
CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum();
187 1 Pablo Alingery
</pre>
188 5 Pablo Alingery
189 6 Pablo Alingery
NOT for aia.lev1 series because we are taking 1/5 data we don't want everything
190 6 Pablo Alingery
191 6 Pablo Alingery
192 6 Pablo Alingery
h3. Create shadow & speed up the usage of aia.lev1 table 
193 6 Pablo Alingery
194 6 Pablo Alingery
h4. postgresql-plperl
195 6 Pablo Alingery
196 6 Pablo Alingery
In order to run createshadow we need postgresql-plperl
197 6 Pablo Alingery
198 6 Pablo Alingery
<pre>
199 6 Pablo Alingery
apt-get install postgresql-plperl 
200 6 Pablo Alingery
</pre>
201 6 Pablo Alingery
202 6 Pablo Alingery
h4. Install language plperl
203 6 Pablo Alingery
204 6 Pablo Alingery
Install plperl within postgres
205 6 Pablo Alingery
206 6 Pablo Alingery
<pre>
207 6 Pablo Alingery
createlang -p5432 -U postgres plperlu ias_sdo
208 6 Pablo Alingery
</pre>
209 6 Pablo Alingery
210 6 Pablo Alingery
h4. Create function updateshadow()
211 6 Pablo Alingery
212 6 Pablo Alingery
Create function updateshadow()
213 6 Pablo Alingery
214 6 Pablo Alingery
<pre>
215 6 Pablo Alingery
psql ias_sdo -p5432 -f /usr/local/netdrms_current/base/drms/libs/pg/updateshadow.sql
216 6 Pablo Alingery
</pre>
217 6 Pablo Alingery
218 6 Pablo Alingery
219 6 Pablo Alingery
So now we are able to launch createshadow()
220 6 Pablo Alingery
221 6 Pablo Alingery
<pre>
222 6 Pablo Alingery
createshadow series=aia.lev1
223 6 Pablo Alingery
</pre>
224 7 Pablo Alingery
225 7 Pablo Alingery
226 7 Pablo Alingery
h3. Get data from JSOC 
227 7 Pablo Alingery
228 7 Pablo Alingery
h4. Create readonly user 
229 7 Pablo Alingery
230 7 Pablo Alingery
<pre>
231 7 Pablo Alingery
production@sdo1:/usr/local/netdrms_current$ psql ias_sdo_sums -p5434 -U postgres 
232 7 Pablo Alingery
psql (9.4.13)
233 7 Pablo Alingery
Type "help" for help.
234 7 Pablo Alingery
235 7 Pablo Alingery
ias_sdo_sums=# CREATE USER readonlyuser;
236 7 Pablo Alingery
CREATE ROLE
237 7 Pablo Alingery
ias_sdo_sums=# GRANT CONNECT ON DATABASE ias_sdo_sums To readonlyuser ; 
238 7 Pablo Alingery
GRANT
239 7 Pablo Alingery
240 7 Pablo Alingery
</pre>
241 7 Pablo Alingery
242 7 Pablo Alingery
243 7 Pablo Alingery
h4. Launch rsums-client.py 
244 7 Pablo Alingery
245 7 Pablo Alingery
<pre>
246 7 Pablo Alingery
python3 base/drms/replication/subscribe_series/rsums-clientd.py --loglevel=debug &
247 7 Pablo Alingery
</pre>