Wiki » History » Version 7
Pablo Alingery, 29/09/2017 12:22
1 | 1 | Pablo Alingery | h1. Netdrms 9.1.1 |
---|---|---|---|
2 | 1 | Pablo Alingery | |
3 | 1 | Pablo Alingery | h2. Posgres 9.4.13 |
4 | 1 | Pablo Alingery | |
5 | 1 | Pablo Alingery | h3. Install additional modules |
6 | 1 | Pablo Alingery | |
7 | 1 | Pablo Alingery | We need dblink for triggers between db |
8 | 1 | Pablo Alingery | |
9 | 1 | Pablo Alingery | <pre> |
10 | 1 | Pablo Alingery | root@sdo1:~# cd /usr/share/postgresql/9.4/extension |
11 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# ls -al dblink--1.1.sql |
12 | 1 | Pablo Alingery | -rw-r--r-- 1 root root 5831 Aug 10 14:53 dblink--1.1.sql |
13 | 1 | Pablo Alingery | </pre> |
14 | 1 | Pablo Alingery | |
15 | 1 | Pablo Alingery | h4. ias_sdo |
16 | 1 | Pablo Alingery | |
17 | 1 | Pablo Alingery | <pre> |
18 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -p5432 -U postgres < dblink--1.1.sql |
19 | 1 | Pablo Alingery | Use "CREATE EXTENSION dblink" to load this file. |
20 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# |
21 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# |
22 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# |
23 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo -U postgres |
24 | 1 | Pablo Alingery | psql (9.4.13) |
25 | 1 | Pablo Alingery | Type "help" for help. |
26 | 1 | Pablo Alingery | |
27 | 4 | Pablo Alingery | |
28 | 4 | Pablo Alingery | |
29 | 4 | Pablo Alingery | |
30 | 4 | Pablo Alingery | |
31 | 4 | Pablo Alingery | |
32 | 1 | Pablo Alingery | ias_sdo=# CREATE EXTENSION dblink ; |
33 | 1 | Pablo Alingery | ERROR: extension "dblink" already exists |
34 | 1 | Pablo Alingery | ias_sdo=# \q |
35 | 1 | Pablo Alingery | </pre> |
36 | 1 | Pablo Alingery | |
37 | 1 | Pablo Alingery | Seems ok ! |
38 | 1 | Pablo Alingery | |
39 | 1 | Pablo Alingery | h4. ias_sdo_sums |
40 | 1 | Pablo Alingery | |
41 | 1 | Pablo Alingery | <pre> |
42 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres < dblink--1.1.sql |
43 | 1 | Pablo Alingery | Use "CREATE EXTENSION dblink" to load this file. |
44 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_sums -p5434 -U postgres |
45 | 1 | Pablo Alingery | psql (9.4.13) |
46 | 1 | Pablo Alingery | Type "help" for help. |
47 | 1 | Pablo Alingery | |
48 | 1 | Pablo Alingery | ias_sdo_sums=# CREATE EXTENSION dblink ; |
49 | 1 | Pablo Alingery | ERROR: extension "dblink" already exists |
50 | 1 | Pablo Alingery | ias_sdo_sums=# \q |
51 | 1 | Pablo Alingery | </pre> |
52 | 1 | Pablo Alingery | |
53 | 1 | Pablo Alingery | Seems ok too ! |
54 | 1 | Pablo Alingery | |
55 | 1 | Pablo Alingery | h4. ias_sdo_monitor |
56 | 1 | Pablo Alingery | |
57 | 1 | Pablo Alingery | <pre> |
58 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres < dblink--1.1.sql |
59 | 1 | Pablo Alingery | Use "CREATE EXTENSION dblink" to load this file. |
60 | 1 | Pablo Alingery | root@sdo1:/usr/share/postgresql/9.4/extension# psql ias_sdo_monitor -p5436 -U postgres |
61 | 1 | Pablo Alingery | psql (9.4.13) |
62 | 1 | Pablo Alingery | Type "help" for help. |
63 | 1 | Pablo Alingery | |
64 | 1 | Pablo Alingery | ias_sdo_monitor=# CREATE EXTENSION dblink ; |
65 | 1 | Pablo Alingery | ERROR: extension "dblink" already exists |
66 | 1 | Pablo Alingery | </pre> |
67 | 1 | Pablo Alingery | |
68 | 1 | Pablo Alingery | Ok too! |
69 | 2 | Pablo Alingery | |
70 | 2 | Pablo Alingery | |
71 | 5 | Pablo Alingery | h3. Ingest last dumps from sdo1 |
72 | 1 | Pablo Alingery | |
73 | 5 | Pablo Alingery | <pre> |
74 | 5 | Pablo Alingery | postgres@sdo1:~/dumps$ gunzip < ias_sdo_sums_2017-09-28.gz | psql ias_sdo_sums -p5434 -U postgres |
75 | 5 | Pablo Alingery | |
76 | 5 | Pablo Alingery | postgres@sdo1:~/dumps$ gunzip < ias_sdo_2017-09-28.gz | psql ias_sdo -p5432 -U postgres |
77 | 5 | Pablo Alingery | |
78 | 5 | Pablo Alingery | postgres@sdo1:~/dumps$ gunzip < ias_sdo_monitor_2017-09-28.gz | psql ias_sdo_monitor -p5436 -U postgres |
79 | 5 | Pablo Alingery | </pre> |
80 | 5 | Pablo Alingery | |
81 | 5 | Pablo Alingery | h3. Creation of missing tables & functions |
82 | 5 | Pablo Alingery | |
83 | 2 | Pablo Alingery | TABLE drms.ingested_sunums |
84 | 2 | Pablo Alingery | |
85 | 2 | Pablo Alingery | --When a client ingests a Slony log, each SUNUM of each data series record is copied |
86 | 2 | Pablo Alingery | -- into this table. The client can use this table of SUNUMs to prefetch SUs from the |
87 | 2 | Pablo Alingery | -- providing site. This SQL is ingested as the pg_user database user, the same |
88 | 2 | Pablo Alingery | -- user that will be ingesting Slony logs and prefetching SUs, so permissions on |
89 | 2 | Pablo Alingery | -- this table will be correct without having to execute a GRANT statement. |
90 | 2 | Pablo Alingery | -- |
91 | 2 | Pablo Alingery | -- The namespace drms is required, and is created by NetDRMS.sql during the |
92 | 2 | Pablo Alingery | -- NetDRMS installation process. |
93 | 2 | Pablo Alingery | -- |
94 | 2 | Pablo Alingery | |
95 | 2 | Pablo Alingery | <pre> |
96 | 2 | Pablo Alingery | DROP TABLE IF EXISTS drms.ingested_sunums; |
97 | 2 | Pablo Alingery | CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL); |
98 | 2 | Pablo Alingery | CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime); |
99 | 2 | Pablo Alingery | </pre> |
100 | 2 | Pablo Alingery | |
101 | 2 | Pablo Alingery | |
102 | 2 | Pablo Alingery | |
103 | 2 | Pablo Alingery | FUNCTION drms.capturesunum |
104 | 2 | Pablo Alingery | -- |
105 | 2 | Pablo Alingery | -- For each table under replication, a trigger is created that calls this function, which |
106 | 2 | Pablo Alingery | -- then copies SUNUMs into the underlying table, drms.ingested_sunums. |
107 | 2 | Pablo Alingery | -- |
108 | 2 | Pablo Alingery | -- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create |
109 | 2 | Pablo Alingery | -- this table). If this table does not exist, then this function is a no-op. |
110 | 2 | Pablo Alingery | -- |
111 | 2 | Pablo Alingery | |
112 | 2 | Pablo Alingery | <pre> |
113 | 2 | Pablo Alingery | CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS |
114 | 2 | Pablo Alingery | $capturesunumtrig$ |
115 | 2 | Pablo Alingery | BEGIN |
116 | 2 | Pablo Alingery | IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN |
117 | 2 | Pablo Alingery | IF (TG_OP='INSERT' AND new.sunum > 0) THEN |
118 | 2 | Pablo Alingery | IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN |
119 | 2 | Pablo Alingery | RETURN NULL; |
120 | 2 | Pablo Alingery | END IF; |
121 | 2 | Pablo Alingery | INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp()); |
122 | 2 | Pablo Alingery | END IF; |
123 | 2 | Pablo Alingery | END IF; |
124 | 2 | Pablo Alingery | |
125 | 2 | Pablo Alingery | RETURN NEW; |
126 | 2 | Pablo Alingery | END |
127 | 2 | Pablo Alingery | $capturesunumtrig$ LANGUAGE plpgsql; |
128 | 2 | Pablo Alingery | </pre> |
129 | 3 | Pablo Alingery | |
130 | 3 | Pablo Alingery | |
131 | 3 | Pablo Alingery | |
132 | 3 | Pablo Alingery | Add a trigger on every hmi table like this to use the function: |
133 | 3 | Pablo Alingery | |
134 | 3 | Pablo Alingery | CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum(); |
135 | 3 | Pablo Alingery | |
136 | 3 | Pablo Alingery | |
137 | 3 | Pablo Alingery | > -- TABLE drms.ingested_sunums |
138 | 3 | Pablo Alingery | > -- |
139 | 3 | Pablo Alingery | > -- When a client ingests a Slony log, each SUNUM of each data series record is copied |
140 | 3 | Pablo Alingery | > -- into this table. The client can use this table of SUNUMs to prefetch SUs from the |
141 | 3 | Pablo Alingery | > -- providing site. This SQL is ingested as the pg_user database user, the same |
142 | 3 | Pablo Alingery | > -- user that will be ingesting Slony logs and prefetching SUs, so permissions on |
143 | 3 | Pablo Alingery | > -- this table will be correct without having to execute a GRANT statement. |
144 | 3 | Pablo Alingery | > -- |
145 | 3 | Pablo Alingery | > -- The namespace drms is required, and is created by NetDRMS.sql during the |
146 | 3 | Pablo Alingery | > -- NetDRMS installation process. |
147 | 3 | Pablo Alingery | > -- |
148 | 3 | Pablo Alingery | |
149 | 3 | Pablo Alingery | <pre> |
150 | 3 | Pablo Alingery | > DROP TABLE IF EXISTS drms.ingested_sunums; |
151 | 3 | Pablo Alingery | > CREATE TABLE drms.ingested_sunums (sunum bigint PRIMARY KEY, starttime timestamp with time zone NOT NULL); |
152 | 3 | Pablo Alingery | > CREATE INDEX ingested_sunums_starttime ON drms.ingested_sunums(starttime); |
153 | 3 | Pablo Alingery | </pre> |
154 | 3 | Pablo Alingery | |
155 | 3 | Pablo Alingery | > -- FUNCTION drms.capturesunum |
156 | 3 | Pablo Alingery | > -- |
157 | 3 | Pablo Alingery | > -- For each table under replication, a trigger is created that calls this function, which |
158 | 3 | Pablo Alingery | > -- then copies SUNUMs into the underlying table, drms.ingested_sunums. |
159 | 3 | Pablo Alingery | > -- |
160 | 3 | Pablo Alingery | > -- drms.ingested_sunums may not exist (older NetDRMSs did not receive the SQL to create |
161 | 3 | Pablo Alingery | > -- this table). If this table does not exist, then this function is a no-op. |
162 | 3 | Pablo Alingery | > -- |
163 | 3 | Pablo Alingery | |
164 | 3 | Pablo Alingery | <pre> |
165 | 3 | Pablo Alingery | > CREATE OR REPLACE FUNCTION drms.capturesunum() RETURNS TRIGGER AS |
166 | 3 | Pablo Alingery | > $capturesunumtrig$ |
167 | 3 | Pablo Alingery | > BEGIN |
168 | 3 | Pablo Alingery | > IF EXISTS (SELECT n.nspname, c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'drms' AND c.relname = 'ingested_sunums') THEN |
169 | 3 | Pablo Alingery | > IF (TG_OP='INSERT' AND new.sunum > 0) THEN |
170 | 3 | Pablo Alingery | > IF EXISTS (SELECT 1 FROM drms.ingested_sunums WHERE sunum = new.sunum) THEN |
171 | 3 | Pablo Alingery | > RETURN NULL; |
172 | 3 | Pablo Alingery | > END IF; |
173 | 3 | Pablo Alingery | > INSERT INTO drms.ingested_sunums (sunum, starttime) VALUES (new.sunum, clock_timestamp()); |
174 | 3 | Pablo Alingery | > END IF; |
175 | 3 | Pablo Alingery | > END IF; |
176 | 3 | Pablo Alingery | > |
177 | 3 | Pablo Alingery | > RETURN NEW; |
178 | 3 | Pablo Alingery | > END |
179 | 3 | Pablo Alingery | > $capturesunumtrig$ LANGUAGE plpgsql; |
180 | 4 | Pablo Alingery | </pre> |
181 | 4 | Pablo Alingery | |
182 | 4 | Pablo Alingery | |
183 | 4 | Pablo Alingery | Add a trigger on every hmi table like this to use the function: |
184 | 4 | Pablo Alingery | |
185 | 4 | Pablo Alingery | <pre> |
186 | 4 | Pablo Alingery | CREATE TRIGGER capturesunumtrig AFTER INSERT ON hmi.bharp_720s FOR EACH ROW EXECUTE PROCEDURE drms.capturesunum(); |
187 | 1 | Pablo Alingery | </pre> |
188 | 5 | Pablo Alingery | |
189 | 6 | Pablo Alingery | NOT for aia.lev1 series because we are taking 1/5 data we don't want everything |
190 | 6 | Pablo Alingery | |
191 | 6 | Pablo Alingery | |
192 | 6 | Pablo Alingery | h3. Create shadow & speed up the usage of aia.lev1 table |
193 | 6 | Pablo Alingery | |
194 | 6 | Pablo Alingery | h4. postgresql-plperl |
195 | 6 | Pablo Alingery | |
196 | 6 | Pablo Alingery | In order to run createshadow we need postgresql-plperl |
197 | 6 | Pablo Alingery | |
198 | 6 | Pablo Alingery | <pre> |
199 | 6 | Pablo Alingery | apt-get install postgresql-plperl |
200 | 6 | Pablo Alingery | </pre> |
201 | 6 | Pablo Alingery | |
202 | 6 | Pablo Alingery | h4. Install language plperl |
203 | 6 | Pablo Alingery | |
204 | 6 | Pablo Alingery | Install plperl within postgres |
205 | 6 | Pablo Alingery | |
206 | 6 | Pablo Alingery | <pre> |
207 | 6 | Pablo Alingery | createlang -p5432 -U postgres plperlu ias_sdo |
208 | 6 | Pablo Alingery | </pre> |
209 | 6 | Pablo Alingery | |
210 | 6 | Pablo Alingery | h4. Create function updateshadow() |
211 | 6 | Pablo Alingery | |
212 | 6 | Pablo Alingery | Create function updateshadow() |
213 | 6 | Pablo Alingery | |
214 | 6 | Pablo Alingery | <pre> |
215 | 6 | Pablo Alingery | psql ias_sdo -p5432 -f /usr/local/netdrms_current/base/drms/libs/pg/updateshadow.sql |
216 | 6 | Pablo Alingery | </pre> |
217 | 6 | Pablo Alingery | |
218 | 6 | Pablo Alingery | |
219 | 6 | Pablo Alingery | So now we are able to launch createshadow() |
220 | 6 | Pablo Alingery | |
221 | 6 | Pablo Alingery | <pre> |
222 | 6 | Pablo Alingery | createshadow series=aia.lev1 |
223 | 6 | Pablo Alingery | </pre> |
224 | 7 | Pablo Alingery | |
225 | 7 | Pablo Alingery | |
226 | 7 | Pablo Alingery | h3. Get data from JSOC |
227 | 7 | Pablo Alingery | |
228 | 7 | Pablo Alingery | h4. Create readonly user |
229 | 7 | Pablo Alingery | |
230 | 7 | Pablo Alingery | <pre> |
231 | 7 | Pablo Alingery | production@sdo1:/usr/local/netdrms_current$ psql ias_sdo_sums -p5434 -U postgres |
232 | 7 | Pablo Alingery | psql (9.4.13) |
233 | 7 | Pablo Alingery | Type "help" for help. |
234 | 7 | Pablo Alingery | |
235 | 7 | Pablo Alingery | ias_sdo_sums=# CREATE USER readonlyuser; |
236 | 7 | Pablo Alingery | CREATE ROLE |
237 | 7 | Pablo Alingery | ias_sdo_sums=# GRANT CONNECT ON DATABASE ias_sdo_sums To readonlyuser ; |
238 | 7 | Pablo Alingery | GRANT |
239 | 7 | Pablo Alingery | |
240 | 7 | Pablo Alingery | </pre> |
241 | 7 | Pablo Alingery | |
242 | 7 | Pablo Alingery | |
243 | 7 | Pablo Alingery | h4. Launch rsums-client.py |
244 | 7 | Pablo Alingery | |
245 | 7 | Pablo Alingery | <pre> |
246 | 7 | Pablo Alingery | python3 base/drms/replication/subscribe_series/rsums-clientd.py --loglevel=debug & |
247 | 7 | Pablo Alingery | </pre> |