Project

General

Profile

Wiki » History » Version 18

Pablo Alingery, 16/10/2017 16:24

1 1 Pablo Alingery
h1. Installation Netdrms_9.0
2 1 Pablo Alingery
3 1 Pablo Alingery
4 5 Pablo Alingery
5 1 Pablo Alingery
h2. Create dir 
6 1 Pablo Alingery
7 2 Pablo Alingery
Create the directory to untar netdrms_9.0  from  http://jsoc.stanford.edu/~arta/for-pablo/netdrms-9.X.20170530tar.gz
8 5 Pablo Alingery
9 1 Pablo Alingery
<pre>
10 1 Pablo Alingery
mkdir /usr/local/netdrms_9.0
11 5 Pablo Alingery
</pre>
12 5 Pablo Alingery
13 5 Pablo Alingery
Create the directory netdmrs_9.1.1 (location Art's home directory at Stanford )  as root 
14 5 Pablo Alingery
15 5 Pablo Alingery
<pre>
16 5 Pablo Alingery
mkdir /usr/local/netdrms_9.0
17 5 Pablo Alingery
</pre>
18 5 Pablo Alingery
19 5 Pablo Alingery
Change owner for production:ias 
20 5 Pablo Alingery
21 5 Pablo Alingery
<pre>
22 5 Pablo Alingery
chown -R production:ias netdrms_9.1.1 
23 1 Pablo Alingery
</pre>
24 1 Pablo Alingery
25 1 Pablo Alingery
26 1 Pablo Alingery
h2. Configuration file 
27 1 Pablo Alingery
28 1 Pablo Alingery
<pre>
29 1 Pablo Alingery
cp config.local.template config.local
30 1 Pablo Alingery
</pre>
31 1 Pablo Alingery
32 1 Pablo Alingery
Config from netdmrs_8.12 
33 1 Pablo Alingery
Warning : Config not done for remote sum as jmd is used 
34 1 Pablo Alingery
35 1 Pablo Alingery
36 1 Pablo Alingery
h2. File configure modif
37 1 Pablo Alingery
38 1 Pablo Alingery
In file /usr/local/netdrms_current/configure
39 1 Pablo Alingery
40 1 Pablo Alingery
Replace all " $? !=0 " by " $status !=0 " in file configure then tape
41 1 Pablo Alingery
42 1 Pablo Alingery
43 1 Pablo Alingery
h2. File jsoc_machine.csh modif 
44 1 Pablo Alingery
45 1 Pablo Alingery
In file /usr/local/netdrms_current/build/jsoc_machine.csh (Same issue in NetDrms 8.11 )
46 1 Pablo Alingery
47 1 Pablo Alingery
ligne 28 :
48 1 Pablo Alingery
49 1 Pablo Alingery
<pre>
50 1 Pablo Alingery
case "x86_64":
51 1 Pablo Alingery
      echo linux_x86_64
52 1 Pablo Alingery
      breaksw
53 1 Pablo Alingery
</pre>
54 1 Pablo Alingery
55 1 Pablo Alingery
h2. File make_basic.mk 
56 1 Pablo Alingery
57 1 Pablo Alingery
In file make_basic.mk line 223
58 1 Pablo Alingery
59 1 Pablo Alingery
look for "GCC_LF_ALL ="
60 1 Pablo Alingery
add "-lcrypto"
61 1 Pablo Alingery
62 1 Pablo Alingery
h2. Execution configure
63 4 Pablo Alingery
64 3 Pablo Alingery
<pre>
65 3 Pablo Alingery
./configure
66 3 Pablo Alingery
</pre>
67 1 Pablo Alingery
68 4 Pablo Alingery
h2. Build 
69 4 Pablo Alingery
70 3 Pablo Alingery
Type 
71 1 Pablo Alingery
72 3 Pablo Alingery
<pre>
73 3 Pablo Alingery
make
74 3 Pablo Alingery
make sums
75 3 Pablo Alingery
</pre>
76 1 Pablo Alingery
77 6 Pablo Alingery
78 6 Pablo Alingery
79 12 Pablo Alingery
For netdrms_9.1.1 as we are going to use new remote sums (written in python) 
80 11 Pablo Alingery
we only need to do the following
81 6 Pablo Alingery
82 6 Pablo Alingery
<pre>
83 6 Pablo Alingery
make show_info 
84 6 Pablo Alingery
make show_series 
85 6 Pablo Alingery
</pre>
86 6 Pablo Alingery
87 6 Pablo Alingery
Checking the modif on Db from NetDrms 2.4 we have figured out that some index where added
88 6 Pablo Alingery
The other modif are for sum_rm use ...
89 6 Pablo Alingery
Not needed at IAS  
90 6 Pablo Alingery
91 13 Pablo Alingery
On sdo and sdo3 modification in the DB ( index addition)
92 6 Pablo Alingery
93 6 Pablo Alingery
<pre>
94 6 Pablo Alingery
production@sdo:~$ psql ias_sdo_sums -p5434 
95 6 Pablo Alingery
ias_sdo_sums=# CREATE INDEX sum_partn_alloc_wd_idx ON public.sum_partn_alloc(wd) ;
96 6 Pablo Alingery
</pre>
97 6 Pablo Alingery
98 1 Pablo Alingery
99 6 Pablo Alingery
100 11 Pablo Alingery
Install OK
101 1 Pablo Alingery
102 1 Pablo Alingery
h2. Starting  Netdrms 
103 1 Pablo Alingery
104 1 Pablo Alingery
<pre>
105 1 Pablo Alingery
Starting it I have got :
106 1 Pablo Alingery
production@sdo3:/usr/local/netdrms_current$ sum_start.NetDRMS
107 1 Pablo Alingery
Number of SUM process sets to spawn = 1
108 1 Pablo Alingery
Max number of SUM process can spawn = 8
109 1 Pablo Alingery
SUMSERVER is sdo3
110 1 Pablo Alingery
SUMS_MANAGER is production
111 1 Pablo Alingery
SUMS db is ias_sdo_sums
112 1 Pablo Alingery
SUMS log dir is /usr/local/netdrms_current/logs/SUM
113 1 Pablo Alingery
sum_start.NetDRMS at 2017.07.12_19:41:38
114 1 Pablo Alingery
115 1 Pablo Alingery
** NOTE: SUMS is already running
116 1 Pablo Alingery
Do you want me to do a sum_stop.NetDRMS followed by a sum_start.NetDRMS (y or n):
117 1 Pablo Alingery
y
118 1 Pablo Alingery
SUMS log dir is /usr/local/netdrms_current/logs/SUM
119 1 Pablo Alingery
120 1 Pablo Alingery
sum_stop.NetDRMS at 2017.07.12_19:41:53
121 1 Pablo Alingery
122 1 Pablo Alingery
NOTE: This will stop sum_svc and related processes. Make sure you want to do this
123 1 Pablo Alingery
Enter reason for stop (terminate with single '.' or ^D):
124 1 Pablo Alingery
>> some reason
125 1 Pablo Alingery
>> .
126 1 Pablo Alingery
Please identify yourself:
127 1 Pablo Alingery
>> Pablo
128 1 Pablo Alingery
Timed-out waiting for sums_procck.py (pid 12640) to exit.
129 1 Pablo Alingery
** Now starting SUMS on ias_sdo_sums
130 1 Pablo Alingery
SUM NetDRMS Start Complete
131 1 Pablo Alingery
production@sdo3:/usr/local/netdrms_current$ Running on SUMS-server host sdo3.
132 1 Pablo Alingery
Connection to SUMS DB ias_sdo_sums.
133 1 Pablo Alingery
Using SUMS-log file sum_svc_2017.07.12.192435.logR.
134 1 Pablo Alingery
Using SUMS-rm-log file 2017.07.12.192435.log.
135 1 Pablo Alingery
Process sum_svc is not running.
136 1 Pablo Alingery
Starting process sum_svc.
137 1 Pablo Alingery
138 1 Pablo Alingery
Please wait for sum_svc and tape inventory (optional) to initialize...
139 1 Pablo Alingery
Can't open config file /usr/local/netdrms_current/logs/SUM/sum_rm.cfg. Using defaults...
140 1 Pablo Alingery
rm: impossible de supprimer « /SUM12/D355906038/Records.txt »: Système de fichiers accessible en lecture seulement
141 1 Pablo Alingery
rm: impossible de supprimer « /SUM12/D355906038/S00000/spikes.fits »: Système de fichiers accessible en lecture seulement
142 1 Pablo Alingery
rm: impossible de supprimer « /SUM12/D355906038/S00000/image_lev1.fits »: Système de fichiers accessible en lecture seulement
143 1 Pablo Alingery
(...)
144 1 Pablo Alingery
</pre>
145 1 Pablo Alingery
146 1 Pablo Alingery
So I killed netdrms to fix that point 
147 1 Pablo Alingery
148 1 Pablo Alingery
<pre>
149 1 Pablo Alingery
Shutting down SUMS...
150 1 Pablo Alingery
Killing [sum_rm](pid 17376).
151 1 Pablo Alingery
Killing sum_svc(pid 17371).
152 1 Pablo Alingery
Removing pidfile /usr/local/netdrms_current/logs/SUM/sums.pidfile
153 1 Pablo Alingery
Exiting process via shutdown() with return value 0.
154 1 Pablo Alingery
</pre>
155 1 Pablo Alingery
156 1 Pablo Alingery
157 1 Pablo Alingery
The sum_rm is running although in config.local I have set SUMRM_NOOP to 1
158 1 Pablo Alingery
159 1 Pablo Alingery
<pre>
160 1 Pablo Alingery
# If the value is set to anything other than 0, then sum_rm is rendered inactive. Otherwise, sum_rm is active.
161 1 Pablo Alingery
SUMRM_NOOP                             1
162 1 Pablo Alingery
</pre>
163 1 Pablo Alingery
164 1 Pablo Alingery
So I put an old sum_rm.cfg in /home/production/netdrms_current/logs/SUM/ then it was ok
165 1 Pablo Alingery
166 1 Pablo Alingery
<pre>
167 1 Pablo Alingery
production@sdo3:/usr/local/netdrms_current$ sum_start.NetDRMS
168 1 Pablo Alingery
Number of SUM process sets to spawn = 1
169 1 Pablo Alingery
Max number of SUM process can spawn = 8
170 1 Pablo Alingery
SUMSERVER is sdo3
171 1 Pablo Alingery
SUMS_MANAGER is production
172 1 Pablo Alingery
SUMS db is ias_sdo_sums
173 1 Pablo Alingery
SUMS log dir is /usr/local/netdrms_current/logs/SUM
174 1 Pablo Alingery
sum_start.NetDRMS at 2017.07.12_19:46:51
175 1 Pablo Alingery
176 1 Pablo Alingery
** Now starting SUMS on ias_sdo_sums
177 1 Pablo Alingery
SUM NetDRMS Start Complete
178 1 Pablo Alingery
production@sdo3:/usr/local/netdrms_current$ Running on SUMS-server host sdo3.
179 1 Pablo Alingery
Connection to SUMS DB ias_sdo_sums.
180 1 Pablo Alingery
Using SUMS-log file sum_svc_2017.07.12.194651.log.
181 1 Pablo Alingery
Using SUMS-rm-log file 2017.07.12.194651.log.
182 1 Pablo Alingery
Process sum_svc is not running.
183 1 Pablo Alingery
Starting process sum_svc.
184 1 Pablo Alingery
185 1 Pablo Alingery
Please wait for sum_svc and tape inventory (optional) to initialize...
186 1 Pablo Alingery
Can't open the log file /home/production/netdrms_current/logs/SUM/sum_rm.log.2017.07.12.194651 for sum_rm on sdo3
187 1 Pablo Alingery
188 1 Pablo Alingery
189 1 Pablo Alingery
## sum_rm for production  pid = 17672 ##
190 1 Pablo Alingery
You are configured as the active user
191 1 Pablo Alingery
cmd: cat /tmp/cmdgrep.17672.log | wc -l 1> /tmp/sum_rm.17672.log
192 1 Pablo Alingery
Connect to Postgres host=sdo3 db=ias_sdo_sums user=production Ok
193 1 Pablo Alingery
DS_PavailRequest
194 1 Pablo Alingery
Called alrm_sig()
195 1 Pablo Alingery
Jul 12 19:46:53
196 1 Pablo Alingery
NOOP
197 1 Pablo Alingery
198 1 Pablo Alingery
production@sdo3:/usr/local/netdrms_current$ sum_svc now available
199 1 Pablo Alingery
</pre>
200 1 Pablo Alingery
201 1 Pablo Alingery
h2. Test Netdrms_9.0
202 1 Pablo Alingery
203 1 Pablo Alingery
Quick test on the thing that bothered us with netdrms_2.4 and lead us to think about an update :
204 1 Pablo Alingery
205 1 Pablo Alingery
<pre>
206 1 Pablo Alingery
production@sdo3:/usr/local/netdrms_current$ show_info -r -S -P 'hmi.sharp_720s_nrt[5398][! DATE__OBS>$(2016-12-20T23:00:00Z) AND DATE__OBS<$(2016-12-21T01:00:00Z) !]' key='DATE__OBS'
207 1 Pablo Alingery
recnum    sunum    DATE__OBS    SUDIR
208 1 Pablo Alingery
2190405    884413026    2016-12-20T23:10:30.80Z    /SUM13/D884413026/S00000
209 1 Pablo Alingery
2190409    884414163    2016-12-20T23:22:30.80Z    /SUM13/D884414163/S00000
210 1 Pablo Alingery
2190413    884416279    2016-12-20T23:34:30.80Z    /SUM13/D884416279/S00000
211 1 Pablo Alingery
2190417    884417281    2016-12-20T23:46:30.80Z    /SUM13/D884417281/S00000
212 1 Pablo Alingery
2190421    884419003    2016-12-20T23:58:30.80Z    /SUM13/D884419003/S00000
213 1 Pablo Alingery
2190425    884421131    2016-12-21T00:10:30.90Z    /SUM13/D884421131/S00000
214 1 Pablo Alingery
2190429    884422598    2016-12-21T00:22:30.90Z    /SUM13/D884422598/S00000
215 1 Pablo Alingery
2190433    884424336    2016-12-21T00:34:30.90Z    /SUM13/D884424336/S00000
216 1 Pablo Alingery
2190437    884425290    2016-12-21T00:46:30.90Z    /SUM13/D884425290/S00000
217 1 Pablo Alingery
2190441    884427329    2016-12-21T00:58:30.90Z    /SUM13/D884427329/S00000
218 1 Pablo Alingery
</pre>
219 7 Pablo Alingery
220 17 Pablo Alingery
h1. Use sumsd.py ,  rsumsd.py & sums-clientd.py 
221 1 Pablo Alingery
222 17 Pablo Alingery
* sumsd.py is the replacement for sum_svc. It runs on the 'sums server'. This is sdo1.
223 1 Pablo Alingery
sumsd.py will open socket to communicate from the client to the server 
224 17 Pablo Alingery
225 17 Pablo Alingery
* rsumsd.py wiil need a couple of table to work  so let's create it 
226 17 Pablo Alingery
This is remote sums; it also runs on sdo1. To use this, both the client machine (sol-calcul4) and the server machine (sdo1) will need access to the DB running on sdo1. 
227 17 Pablo Alingery
So, for this to be working, you should be able to do a psql -h sdo1 ... from sol-calcul4 and connect.
228 17 Pablo Alingery
229 17 Pablo Alingery
* sums-clientd.py
230 17 Pablo Alingery
 will take sunum in the table drms.ingested_sunums  to insert them into drms.rs_requests 
231 1 Pablo Alingery
232 1 Pablo Alingery
h2. create missing tables 
233 9 Pablo Alingery
234 9 Pablo Alingery
To be done too on sdo1 
235 9 Pablo Alingery
236 1 Pablo Alingery
<pre>
237 7 Pablo Alingery
/usr/local/netdrms_current$ base/drms/scripts/rscreatetabs.py op=create  tabs=req,su
238 7 Pablo Alingery
</pre>
239 9 Pablo Alingery
240 10 Pablo Alingery
ok
241 10 Pablo Alingery
242 8 Pablo Alingery
h2. ssh-agent
243 9 Pablo Alingery
244 9 Pablo Alingery
ssh-agent was not working so we did regenerate public key 
245 9 Pablo Alingery
246 1 Pablo Alingery
<pre>
247 8 Pablo Alingery
ssh-keygen -t rsa
248 8 Pablo Alingery
</pre>
249 8 Pablo Alingery
250 1 Pablo Alingery
251 8 Pablo Alingery
then copy public key id_rsa.pub into ~/.ssh/authorized_keys on the targeted server  jsocport.stanford.edu
252 8 Pablo Alingery
253 10 Pablo Alingery
jsoc side, on  jsocport.stanford.edu
254 10 Pablo Alingery
make sure sdo3 ip is referenced in /etc/hosts.netdrms
255 14 Pablo Alingery
256 14 Pablo Alingery
257 14 Pablo Alingery
h2. Launch sumsd.py 
258 1 Pablo Alingery
259 15 Pablo Alingery
As 'production'
260 15 Pablo Alingery
261 1 Pablo Alingery
<pre>
262 18 Pablo Alingery
sumsd.py --loglevel=debug &
263 15 Pablo Alingery
</pre>
264 15 Pablo Alingery
265 15 Pablo Alingery
266 15 Pablo Alingery
267 15 Pablo Alingery
h2. Launch rsumsd.py 
268 15 Pablo Alingery
269 15 Pablo Alingery
The new remote sums server
270 15 Pablo Alingery
271 15 Pablo Alingery
As 'production'
272 15 Pablo Alingery
 
273 15 Pablo Alingery
<pre>
274 18 Pablo Alingery
rsumsd.py --loglevel=debug &
275 14 Pablo Alingery
</pre>
276 1 Pablo Alingery
277 1 Pablo Alingery
278 18 Pablo Alingery
h2. Launch sums-clientd.py 
279 1 Pablo Alingery
280 18 Pablo Alingery
deamon that ingest sunums into drms.rs_requests table from drms.ingested_sunums
281 18 Pablo Alingery
282 18 Pablo Alingery
As 'production'
283 18 Pablo Alingery
 
284 18 Pablo Alingery
<pre>
285 18 Pablo Alingery
sums-clientd.py --loglevel=debug & 
286 18 Pablo Alingery
</pre>