Project

General

Profile

NetDRMS Installation » History » Version 207

Pablo Alingery, 07/06/2016 11:34

1 50 Herve Ballans
{{toc}}
2 1 Herve Ballans
3 1 Herve Ballans
h1. NetDRMS Installation
4 1 Herve Ballans
5 1 Herve Ballans
h2. Requirements
6 1 Herve Ballans
7 47 Pablo Alingery
h3. Configuring production environment
8 1 Herve Ballans
9 121 Pablo Alingery
Adding user production (en root)
10 47 Pablo Alingery
11 1 Herve Ballans
<pre>
12 1 Herve Ballans
# adduser production
13 1 Herve Ballans
# addgroup ias
14 1 Herve Ballans
# adduser production ias
15 1 Herve Ballans
</pre>
16 1 Herve Ballans
17 49 Herve Ballans
h3. Configure /home/production/.profile
18 1 Herve Ballans
19 49 Herve Ballans
Adding environment variables
20 1 Herve Ballans
21 1 Herve Ballans
<pre>
22 122 Herve Ballans
#Modif Herve.B Pablo.A 12/05/2016
23 122 Herve Ballans
24 1 Herve Ballans
PATH="/usr/local/netdrms_current/bin/linux_x86_64:$PATH"
25 49 Herve Ballans
PATH="/usr/local/netdrms_current/scripts:$PATH"
26 122 Herve Ballans
PATH="/usr/local/jmd/bin:$PATH"
27 122 Herve Ballans
PATH="/usr/local/jmd/scripts:$PATH"
28 141 Pablo Alingery
#The following is not needed as installed with 'apt-get install' 
29 141 Pablo Alingery
#PATH="/usr/local/pgsql/bin:$PATH"
30 122 Herve Ballans
PATH="/usr/local/netdrms-tools/scripts:$PATH"
31 49 Herve Ballans
#2014-12-17 Avec Herve et Pablo on pense que la ligne suivante est intutile as it is f.. install in /usr/local/lib
32 140 Pablo Alingery
PATH="/usr/local/cfitsio:$PATH"
33 1 Herve Ballans
#Done
34 49 Herve Ballans
35 122 Herve Ballans
DERBY_HOME=/usr/local/jmd/databases/derby/derbyBD
36 49 Herve Ballans
export DERBY_HOME
37 49 Herve Ballans
38 49 Herve Ballans
export CVSROOT=:pserver:anonymous@solarch.tuc.noao.edu:2401/vtarc1/vso/cvsroot
39 49 Herve Ballans
</pre>
40 49 Herve Ballans
41 162 Pablo Alingery
h3. Configure /varl/lib/postgres/.profile
42 161 Pablo Alingery
43 163 Pablo Alingery
Add to $PATH so postgres can exeute command like initdb
44 161 Pablo Alingery
45 161 Pablo Alingery
<pre>
46 161 Pablo Alingery
#Modif Herve.B Pablo.A 19/05/2016
47 161 Pablo Alingery
PATH="/usr/lib/postgresql/9.4/bin:$PATH"
48 161 Pablo Alingery
</pre>
49 161 Pablo Alingery
50 125 Pablo Alingery
h3. Setting privileges for SUMS_MANAGER
51 87 Herve Ballans
52 126 Pablo Alingery
Please modify file /etc/sudoers so user 'production' will be able to execute  'sum_chown' that we will locate in the dir : /usr/local/bin . See below
53 125 Pablo Alingery
Add to /etc/sudoers the following line :
54 1 Herve Ballans
	production host=NOPASSWD:/usr/local/bin/sum_chmown
55 1 Herve Ballans
56 127 Pablo Alingery
h3. Set passwords file for db postgres 
57 1 Herve Ballans
58 127 Pablo Alingery
Please create file .pgpass with the following information for user 'production" so he will be able to connect without password as 'production' or 'postgres' to 2 db that we will create later ias_sdo and ias_sdo_sums.
59 1 Herve Ballans
60 1 Herve Ballans
61 1 Herve Ballans
.pgpass content :
62 1 Herve Ballans
<pre>
63 128 Pablo Alingery
	#hostname:5432:ias_sdo:production: 
64 128 Pablo Alingery
	#hostname:5434:ias_sdo_sums:production: 
65 128 Pablo Alingery
	#hostname:5432:ias_sdo:postgres: 
66 128 Pablo Alingery
	#hostname:5434:ias_sdo_sums:postgres
67 1 Herve Ballans
</pre>
68 128 Pablo Alingery
69 129 Pablo Alingery
where #hostname is the name of your server
70 129 Pablo Alingery
71 129 Pablo Alingery
72 131 Pablo Alingery
%{color:red}WARNING : change hostname when migrate sdo-new to sdo%
73 1 Herve Ballans
74 143 Pablo Alingery
75 132 Pablo Alingery
76 133 Pablo Alingery
h3. Install missing libraries 
77 1 Herve Ballans
78 133 Pablo Alingery
• libreadline-dev (but already installed with postgresql)
79 133 Pablo Alingery
• libssl-dev 
80 133 Pablo Alingery
• libpam0g-dev 
81 133 Pablo Alingery
• libperl-dev 
82 1 Herve Ballans
83 133 Pablo Alingery
NB : Already done in Debian 8.4 Jessy
84 1 Herve Ballans
<pre>
85 1 Herve Ballans
# apt-get install $library_name
86 1 Herve Ballans
</pre> 
87 1 Herve Ballans
where
88 1 Herve Ballans
	_$library_name_ is the name of library
89 19 Pablo Alingery
90 1 Herve Ballans
Then to update path on libraries, please type :
91 1 Herve Ballans
<pre>
92 19 Pablo Alingery
# ldconfig
93 1 Herve Ballans
</pre>
94 1 Herve Ballans
95 1 Herve Ballans
h3. Reconfigure locale
96 1 Herve Ballans
97 1 Herve Ballans
<pre>
98 1 Herve Ballans
# dpkg-reconfigure locales
99 1 Herve Ballans
</pre>
100 1 Herve Ballans
101 1 Herve Ballans
select fr_FR.UTF8 UTF8 and unselect en_US.UTF-8
102 1 Herve Ballans
103 1 Herve Ballans
localedef -i en_US -f UTF-8 en_US.UTF-8
104 1 Herve Ballans
105 1 Herve Ballans
(see here : [[http://www.thomas-krenn.com/en/wiki/Perl_warning_Setting_locale_failed_in_Debian]])
106 1 Herve Ballans
107 134 Pablo Alingery
h3. Install gfortran packages
108 1 Herve Ballans
109 1 Herve Ballans
<pre>
110 1 Herve Ballans
# apt-get install gfortran swig
111 1 Herve Ballans
</pre>
112 1 Herve Ballans
113 1 Herve Ballans
h3. Installation of cfitsio library
114 1 Herve Ballans
115 135 Pablo Alingery
In /usr/local dir
116 135 Pablo Alingery
117 135 Pablo Alingery
Version 3.39 (currently on sdo : 3.24)
118 1 Herve Ballans
<pre>
119 135 Pablo Alingery
# wget ftp://heasarc.gsfc.nasa.gov/software/fitsio/c/cfitsio3390.tar.gz
120 1 Herve Ballans
</pre>
121 137 Pablo Alingery
Untar and  decompression
122 1 Herve Ballans
<pre>
123 136 Pablo Alingery
# tar -xzvf cfitsio3390.tar.gz
124 1 Herve Ballans
</pre> 
125 1 Herve Ballans
126 139 Pablo Alingery
Go into /usr/local and type
127 1 Herve Ballans
128 1 Herve Ballans
su - root and :
129 1 Herve Ballans
<pre>
130 1 Herve Ballans
# ./configure --prefix=/usr/local
131 1 Herve Ballans
# make 
132 1 Herve Ballans
# make install
133 135 Pablo Alingery
</pre>
134 118 Pablo Alingery
135 144 Pablo Alingery
h3. Installation of apache2 and lib perl
136 118 Pablo Alingery
137 118 Pablo Alingery
<pre>
138 118 Pablo Alingery
# apt-get install apache2
139 118 Pablo Alingery
140 118 Pablo Alingery
# apt-get install libjson-perl
141 118 Pablo Alingery
142 118 Pablo Alingery
# apt-get install libapache2-mod-perl2
143 118 Pablo Alingery
</pre>
144 1 Herve Ballans
145 183 Herve Ballans
h3. Installation of postgresql
146 1 Herve Ballans
147 145 Pablo Alingery
Into 'root'
148 145 Pablo Alingery
 
149 145 Pablo Alingery
<pre>
150 147 Pablo Alingery
apt-get install postgresql-9.4 postgresql-client-9.4 
151 145 Pablo Alingery
</pre>
152 145 Pablo Alingery
153 1 Herve Ballans
h2. NetDRMS databases
154 1 Herve Ballans
155 156 Pablo Alingery
into /var/lib/postgresql  (hard mount point separated,  7 disks 15000 tr/min in raid5 in order to distribute read and write) 
156 1 Herve Ballans
157 1 Herve Ballans
<pre>
158 158 Pablo Alingery
# cd /var/lib/postgresql
159 157 Pablo Alingery
# mkdir data  data_sums data_monitor
160 157 Pablo Alingery
# chown -R postgres:postgres * 
161 1 Herve Ballans
</pre>
162 1 Herve Ballans
163 160 Pablo Alingery
h3. Initialize 3 servers data,  data_sums, data_monitor
164 1 Herve Ballans
165 159 Pablo Alingery
Into postgres
166 1 Herve Ballans
<pre>
167 1 Herve Ballans
# su - postgres
168 1 Herve Ballans
169 1 Herve Ballans
$ initdb --locale=C -D data -A md5 -W
170 1 Herve Ballans
(no password)
171 2 Herve Ballans
172 2 Herve Ballans
Success. You can now start the database server using:
173 2 Herve Ballans
174 2 Herve Ballans
    postgres -D data
175 2 Herve Ballans
or
176 2 Herve Ballans
    pg_ctl -D data -l logfile start
177 2 Herve Ballans
178 1 Herve Ballans
179 1 Herve Ballans
$ initdb --locale=C -D data_sums -A md5 -W
180 2 Herve Ballans
(no password)
181 2 Herve Ballans
182 2 Herve Ballans
Success. You can now start the database server using:
183 2 Herve Ballans
184 2 Herve Ballans
    postgres -D data_sums
185 2 Herve Ballans
or
186 2 Herve Ballans
    pg_ctl -D data_sums -l logfile start
187 8 Pablo Alingery
188 8 Pablo Alingery
$ initdb --locale=C -D data_monitor -A md5 -W
189 8 Pablo Alingery
(no password)
190 8 Pablo Alingery
191 8 Pablo Alingery
Success. You can now start the database server using:
192 8 Pablo Alingery
193 8 Pablo Alingery
    postgres -D data_monitor
194 8 Pablo Alingery
or
195 8 Pablo Alingery
    pg_ctl -D data_monitor -l logfile start
196 2 Herve Ballans
</pre>
197 2 Herve Ballans
198 1 Herve Ballans
199 1 Herve Ballans
200 1 Herve Ballans
Edition des fichiers postgresql.conf pour configurer les bons ports d'écoute :
201 1 Herve Ballans
listen_addresses = '*'
202 1 Herve Ballans
data -> port 5432
203 10 Pablo Alingery
data_sums -> port 5434
204 1 Herve Ballans
data_monitor -> port 5436
205 10 Pablo Alingery
206 165 Pablo Alingery
Edit files data*/pg_hba.conf : replace 'md5' by 'trust'
207 1 Herve Ballans
<pre>
208 1 Herve Ballans
#"local"
209 1 Herve Ballans
local   all             all                                     trust
210 1 Herve Ballans
#"IPv4"
211 1 Herve Ballans
host   all             all                                     trust
212 1 Herve Ballans
</pre>
213 166 Pablo Alingery
214 166 Pablo Alingery
h3. Tunning postgres 
215 166 Pablo Alingery
216 166 Pablo Alingery
In file postgres.conf ( according to https://www.qwant.com/?q=Tunning+postgres+server+9.4+&client=firefox  )
217 166 Pablo Alingery
218 168 Pablo Alingery
shared_buffer=4096MB %{color:red}So 1/12 of the total RAM memory available%
219 1 Herve Ballans
220 169 Pablo Alingery
work_mem = 24MB
221 168 Pablo Alingery
222 168 Pablo Alingery
maintenance_work_mem = 4096MB
223 1 Herve Ballans
224 169 Pablo Alingery
wal_buffers = 16MB
225 169 Pablo Alingery
226 169 Pablo Alingery
checkpoint_segments = 32
227 169 Pablo Alingery
228 169 Pablo Alingery
checkpoint_completion_target = 0.9
229 169 Pablo Alingery
230 169 Pablo Alingery
random_page_cost = 1.0
231 169 Pablo Alingery
232 169 Pablo Alingery
effective_cache_size = 16384MB so 1/4 of the max memory 
233 169 Pablo Alingery
234 170 Pablo Alingery
235 170 Pablo Alingery
236 170 Pablo Alingery
#log
237 169 Pablo Alingery
logging_collector = on 
238 169 Pablo Alingery
log_directory = 'pg_log' 
239 169 Pablo Alingery
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,
240 169 Pablo Alingery
log_truncate_on_rotation = on           # If on, an existing log file with the
241 169 Pablo Alingery
log_rotation_age = 1d                   # Automatic rotation of logfiles will
242 170 Pablo Alingery
client_min_messages = error
243 170 Pablo Alingery
log_min_messages = error 
244 170 Pablo Alingery
log_min_duration_statement = 0
245 170 Pablo Alingery
246 170 Pablo Alingery
log_connections = on
247 170 Pablo Alingery
log_disconnections = on
248 170 Pablo Alingery
log_duration = off
249 170 Pablo Alingery
log_hostname = on
250 170 Pablo Alingery
log_line_prefix = '%t [%p]: %u@%h - %d :'  
251 170 Pablo Alingery
252 170 Pablo Alingery
track_counts = on
253 170 Pablo Alingery
autovacuum = on 
254 170 Pablo Alingery
255 170 Pablo Alingery
extra_float_digits = 3
256 170 Pablo Alingery
257 177 Pablo Alingery
h3. Start the 3 databases :
258 1 Herve Ballans
259 1 Herve Ballans
<pre>
260 1 Herve Ballans
# su - postgres
261 196 Herve Ballans
$ cd /var/lib/postgresql
262 172 Pablo Alingery
$ pg_ctl -D data  start
263 172 Pablo Alingery
$ pg_ctl -D data_sums start
264 172 Pablo Alingery
$ pg_ctl -D data_monitor start
265 1 Herve Ballans
</pre>
266 174 Pablo Alingery
267 1 Herve Ballans
h3. Create the 3 databases :
268 181 Pablo Alingery
269 178 Pablo Alingery
Into user 'postgres'
270 1 Herve Ballans
271 179 Pablo Alingery
* createdb --locale C -E LATIN1 -T template0 ias_sdo
272 179 Pablo Alingery
* createdb --locale C -E LATIN1 -T template0 -p 5434 ias_sdo_sums
273 179 Pablo Alingery
* createdb --locale C -E LATIN1 -T template0 -p 5436 ias_sdo_monitor
274 54 Herve Ballans
275 182 Herve Ballans
h3. Automatic startup
276 182 Herve Ballans
277 182 Herve Ballans
Add following lines in the /etc/rc.local file :
278 182 Herve Ballans
279 182 Herve Ballans
<pre>
280 197 Herve Ballans
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data start" postgres
281 197 Herve Ballans
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_sums start" postgres
282 197 Herve Ballans
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_monitor start" postgres
283 182 Herve Ballans
</pre>
284 182 Herve Ballans
285 184 Herve Ballans
h2. System tuning
286 1 Herve Ballans
287 187 Herve Ballans
Current configuration :
288 185 Herve Ballans
<pre>
289 185 Herve Ballans
# sysctl -a
290 185 Herve Ballans
</pre>
291 1 Herve Ballans
292 187 Herve Ballans
In the following file :
293 185 Herve Ballans
<pre>
294 185 Herve Ballans
# vi /etc/sysctl.conf
295 185 Herve Ballans
</pre>
296 1 Herve Ballans
297 188 Herve Ballans
we change some values (default values are commented with '#')
298 194 Herve Ballans
299 195 Herve Ballans
(resources links [[Links|here]])
300 194 Herve Ballans
301 185 Herve Ballans
<pre>
302 191 Herve Ballans
# Semaphore is a object that is used to control utilization of a particular process.
303 191 Herve Ballans
# kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI
304 191 Herve Ballans
# SEMMSL        maximum number of semaphores per array  (min 128)
305 191 Herve Ballans
# SEMMNS        maximum semaphores system-wide  
306 191 Herve Ballans
# SEMOPM        maximum operations per semop call       
307 191 Herve Ballans
# SEMMNI        maximum arrays 
308 191 Herve Ballans
#semop incresed from 32 (default value)operation to 100 (3rd value )
309 191 Herve Ballans
# kernel.sem = 250      32000   32      128
310 191 Herve Ballans
kernel.sem = 250 32000 100 128
311 191 Herve Ballans
312 191 Herve Ballans
In order to  apply changes :#This sets the  OS receive buffer size for all types of connections (default value 212992)
313 191 Herve Ballans
#net.core.rmem_default = 212992
314 191 Herve Ballans
#net.core.rmem_max = 212992
315 191 Herve Ballans
net.core.rmem_default = 33554432
316 191 Herve Ballans
net.core.rmem_max = 33554432
317 191 Herve Ballans
318 191 Herve Ballans
#This is the OS send buffer size for all types of connections (212992 default value) 
319 191 Herve Ballans
#net.core.wmem_default = 212992
320 191 Herve Ballans
#net.core.wmem_max = 212992
321 191 Herve Ballans
net.core.wmem_default = 33554432
322 191 Herve Ballans
net.core.wmem_max = 33554432
323 191 Herve Ballans
324 193 Pablo Alingery
#TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always
325 193 Pablo Alingery
# allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive
326 193 Pablo Alingery
# buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ...
327 192 Pablo Alingery
# The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."
328 1 Herve Ballans
#net.ipv4.tcp_rmem = 4096       87380   6291456
329 1 Herve Ballans
net.ipv4.tcp_rmem = 10240 87380 33554432
330 1 Herve Ballans
331 193 Pablo Alingery
#TCP Autotuning setting. "This variable takes 3 different values which holds information on how much TCP sendbuffer memory space
332 193 Pablo Alingery
# each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three 
333 193 Pablo Alingery
#values are used under different conditions. ... The first value in this variable tells the minimum TCP send buffer space available
334 193 Pablo Alingery
# for a single TCP socket. ... The second value in the variable tells us the default buffer space allowed for a single TCP socket to use. ... 
335 193 Pablo Alingery
#The third value tells the kernel the maximum TCP send buffer space." 
336 191 Herve Ballans
#net.ipv4.tcp_wmem = 4096       16384   4194304
337 191 Herve Ballans
net.ipv4.tcp_wmem = 10240 87380 33554432
338 191 Herve Ballans
339 191 Herve Ballans
#Disable cache metrics so the initial conditions of the closed connections will not be saved to be used in near future connections
340 191 Herve Ballans
#net.ipv4.tcp_no_metrics_save = 0
341 191 Herve Ballans
net.ipv4.tcp_no_metrics_save = 1
342 191 Herve Ballans
343 1 Herve Ballans
# Increase number of incoming connections backlog
344 1 Herve Ballans
# net.core.netdev_max_backlog = 1000
345 1 Herve Ballans
net.core.netdev_max_backlog = 5000
346 1 Herve Ballans
347 1 Herve Ballans
# The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage. ... 
348 192 Pablo Alingery
#The first value specified in the tcp_mem variable tells the kernel the low threshold. 
349 192 Pablo Alingery
#Below this point, the TCP stack do not bother at all about putting any pressure on the memory usage by different TCP sockets. ... 
350 193 Pablo Alingery
#The second value tells the kernel at which point to start pressuring memory usage down. ... The final value tells the kernel how many
351 193 Pablo Alingery
# memory pages it may use maximally
352 1 Herve Ballans
# net.ipv4.tcp_mem = 1543347    2057796 3086694
353 191 Herve Ballans
net.ipv4.tcp_mem = 786432 1048576 26777216
354 191 Herve Ballans
355 192 Pablo Alingery
# local port range that is used by TCP and UDP traffic to choose the local port. You will see in the parameters of this file two numbers:
356 192 Pablo Alingery
# The first number is the first local port allowed for TCP and UDP traffic on the server, the second is the last local port number.
357 191 Herve Ballans
# net.ipv4.ip_local_port_range = 32768  61000
358 191 Herve Ballans
net.ipv4.ip_local_port_range = 1024 65535
359 191 Herve Ballans
360 191 Herve Ballans
# maximum number of sockets in TIME-WAIT to be held simultaneously.
361 191 Herve Ballans
# net.ipv4.tcp_max_tw_buckets = 262144
362 191 Herve Ballans
net.ipv4.tcp_max_tw_buckets = 360000
363 1 Herve Ballans
</pre>
364 185 Herve Ballans
365 191 Herve Ballans
Then we run the following command to make our change take effect:
366 191 Herve Ballans
367 185 Herve Ballans
<pre>
368 185 Herve Ballans
# sysctl -p
369 185 Herve Ballans
</pre>
370 176 Pablo Alingery
371 1 Herve Ballans
h2. NetDRMS tool Install
372 23 Pablo Alingery
373 199 Pablo Alingery
The following installed is based on : [[http://inf-redmine.ias.u-psud.fr/redmine/attachments/download/114/Netdrms_First_Install.odt]]
374 1 Herve Ballans
375 207 Pablo Alingery
The sources are in : http://jsoc.stanford.edu/netdrms/dist/ 
376 207 Pablo Alingery
Locally on sdo-new in  : /home/production/sources
377 1 Herve Ballans
378 199 Pablo Alingery
Symbolic link created 'netdrms_current' that corresponds to the last release
379 1 Herve Ballans
ln -s netdrms_8.10/ netdrms_current 
380 1 Herve Ballans
381 199 Pablo Alingery
Edit file config.local
382 27 Pablo Alingery
<pre>
383 27 Pablo Alingery
cp config.local.template config.local
384 1 Herve Ballans
</pre>
385 29 Pablo Alingery
386 203 Pablo Alingery
%{color:red}Warning WEB_DBUSER      apache  We think that it should value www maybe , to be checked testing the cgi%
387 204 Pablo Alingery
%{color:red}So check the cgito figure out which user is used to connect to bd%
388 32 Herve Ballans
389 199 Pablo Alingery
Create 2 followings dirs : 
390 1 Herve Ballans
<pre>
391 32 Herve Ballans
mkdir /usr/local/netdrms_8.10/logs/SUM
392 28 Pablo Alingery
mkdir /usr/local/netdrms_8.10/logs/slony
393 27 Pablo Alingery
</pre>
394 1 Herve Ballans
395 205 Pablo Alingery
%{color:red}Warning config.local from line 150 =>222 Remote config to provide data, series to externals%
396 205 Pablo Alingery
%{color:red}To be checked for PLATO%
397 1 Herve Ballans
398 1 Herve Ballans
Création d'un alias netdrms dans /root/.bashrc
399 27 Pablo Alingery
<pre>
400 1 Herve Ballans
alias netdrms='cd /usr/local/netdrms_current'
401 1 Herve Ballans
</pre>
402 31 Pablo Alingery
403 206 Pablo Alingery
%{color:red}In config.local we are at line 223%
404 206 Pablo Alingery
%{color:red}------------------------------------------------------------------------------------------------------------------%
405 206 Pablo Alingery
406 1 Herve Ballans
h3. Modification du fichier /usr/local/netdrms_current/configure (Not Applied in NetDrms_8.10)
407 1 Herve Ballans
408 1 Herve Ballans
<pre>
409 1 Herve Ballans
# The system configuration for configure script
410 1 Herve Ballans
set PERLBIN = "/usr/bin/perl"
411 1 Herve Ballans
set PYBIN = "/usr/bin/python"
412 200 Pablo Alingery
</pre>
413 200 Pablo Alingery
414 34 Herve Ballans
415 34 Herve Ballans
h3. Compilation problem due to uncorrect variable types
416 35 Herve Ballans
417 34 Herve Ballans
Art (art.amezcua@stanford.edu) purpose us a workaround and send us 2 files with correct declaration :
418 42 Herve Ballans
419 42 Herve Ballans
SUMLIB_RmDo.pgc (diff with original file : [[diff_SUMLIB_RmDo.pgc]])
420 34 Herve Ballans
SUMLIB_RmDoX.pgc (diff with original file : [[diff_SUMLIB_RmDoX.pgc]])
421 109 Pablo Alingery
422 109 Pablo Alingery
Dans le répertoire base/sums/libs/pg renommer les anciens en *.orig et placer les nouveau fichiers SUMLIB_RmDo.pgc et SUMLIB_RmDoX.pgc
423 36 Herve Ballans
424 36 Herve Ballans
<pre>
425 36 Herve Ballans
# wget http://jsoc.stanford.edu/~arta/SUMLIB_RmDo.pgc
426 38 Herve Ballans
# wget http://jsoc.stanford.edu/~arta/SUMLIB_RmDoX.pgc
427 34 Herve Ballans
</pre>
428 1 Herve Ballans
429 1 Herve Ballans
h3. Modification du fichier /usr/local/netdrms_current/build/jsoc_machine.csh
430 1 Herve Ballans
431 1 Herve Ballans
ligne 28 :
432 1 Herve Ballans
433 1 Herve Ballans
<pre>
434 1 Herve Ballans
  case "x86_64":
435 1 Herve Ballans
      echo linux_x86_64
436 1 Herve Ballans
      breaksw
437 109 Pablo Alingery
</pre>
438 109 Pablo Alingery
439 109 Pablo Alingery
Overwriting the existing files in the right place (base/sums/libs/pg) & modif in jsoc_machine.csh 
440 109 Pablo Alingery
=> the command 'make' works well.
441 109 Pablo Alingery
442 109 Pablo Alingery
But there is still problem with 'make sums'
443 109 Pablo Alingery
444 109 Pablo Alingery
h3. Workaround to solve make sums problem
445 109 Pablo Alingery
446 109 Pablo Alingery
Modify [[sum_rpc.h]] in order to match the type of some variables
447 109 Pablo Alingery
448 1 Herve Ballans
449 43 Pablo Alingery
450 1 Herve Ballans
h3. Copier /usr/local/netdrms_2.4_release/base/drms/apps/serverdefs.h dans /usr/local/netdrms_current/base/drms/apps (not applied in NetDrms_8.10)
451 44 Pablo Alingery
452 1 Herve Ballans
h3. Modification du fichier /usr/local/netdrms_current/configure (not applied in NetDrms_8.10)
453 1 Herve Ballans
454 1 Herve Ballans
au lieu de $? en csh on va mettre plutot $status
455 1 Herve Ballans
456 1 Herve Ballans
ligne 110 :
457 1 Herve Ballans
458 1 Herve Ballans
<pre>
459 1 Herve Ballans
  if ($status==0) 
460 1 Herve Ballans
</pre>
461 4 Herve Ballans
462 1 Herve Ballans
et ligne 123 :
463 1 Herve Ballans
464 1 Herve Ballans
<pre>
465 1 Herve Ballans
  if ($status==0) 
466 1 Herve Ballans
</pre>
467 4 Herve Ballans
468 4 Herve Ballans
Lancement :
469 4 Herve Ballans
<pre>
470 1 Herve Ballans
./configure
471 1 Herve Ballans
</pre>
472 1 Herve Ballans
473 1 Herve Ballans
h3. Installation des libraries libtar-dev, libcurl3-dev et libecpg-dev
474 1 Herve Ballans
475 4 Herve Ballans
<pre>
476 1 Herve Ballans
apt-get install libtar libtar-dev
477 1 Herve Ballans
</pre>
478 1 Herve Ballans
479 1 Herve Ballans
<pre>
480 1 Herve Ballans
apt-get install libcurl3-dev
481 1 Herve Ballans
</pre>
482 1 Herve Ballans
483 1 Herve Ballans
<pre>
484 1 Herve Ballans
apt-get install libecpg-dev
485 1 Herve Ballans
</pre>
486 45 Pablo Alingery
487 1 Herve Ballans
h3. #H4ck 2 Malade #PabloR0cks# (Not Applied in NetDrm_8.10)
488 1 Herve Ballans
489 1 Herve Ballans
Modification du fichier /usr/include/stdint.h
490 17 Herve Ballans
491 17 Herve Ballans
ligne 55 : long int remplacé par long long int
492 17 Herve Ballans
493 17 Herve Ballans
<pre>
494 17 Herve Ballans
#if __WORDSIZE == 64
495 17 Herve Ballans
typedef unsigned long int       uint64_t; --> ici long long int
496 17 Herve Ballans
#else
497 17 Herve Ballans
__extension__
498 17 Herve Ballans
typedef unsigned long long int  uint64_t;
499 17 Herve Ballans
#endif
500 1 Herve Ballans
</pre>
501 1 Herve Ballans
502 1 Herve Ballans
"Allo Linus ?!?.."
503 1 Herve Ballans
504 1 Herve Ballans
h3. Compilation
505 1 Herve Ballans
506 1 Herve Ballans
<pre>
507 1 Herve Ballans
# make
508 53 Herve Ballans
# make sums
509 1 Herve Ballans
# make jsoc_fetch
510 1 Herve Ballans
</pre>
511 1 Herve Ballans
512 6 Pablo Alingery
NB : lib64 does not exist on debian 8. so ln -s /usr/lib lib64
513 46 Pablo Alingery
514 1 Herve Ballans
h2. make sums KO  (Not applied in NetDrms_8.10)
515 6 Pablo Alingery
516 6 Pablo Alingery
Solution found : Take old lib libcrypto instead libssl
517 16 Herve Ballans
518 6 Pablo Alingery
dans le fichier base/sums/apps/Rules.mk ligne 51 
519 55 Herve Ballans
replace -lssl by -lcrypto LD flag
520 57 Herve Ballans
521 55 Herve Ballans
h2. NetDRMS replication
522 56 Herve Ballans
523 56 Herve Ballans
<pre>
524 60 Herve Ballans
# cd base/drms/replication/etc
525 60 Herve Ballans
# scp production@sdo:/usr/local/netdrms/install-config-files/ias.subscribe_list.cfg  .
526 104 Pablo Alingery
# scp production@sdo:/usr/local/netdrms/install-config-files/ias.repclient.cfg  .
527 104 Pablo Alingery
WRONG !!!!!
528 104 Pablo Alingery
To be done cp repclient.template.cfg into ias.subscribe_list.cfg because new field in config file can't appear 
529 104 Pablo Alingery
Correct that step in NetDrms doc 
530 104 Pablo Alingery
The existing ias.repclient.cfg can be used tough to fill params 
531 104 Pablo Alingery
532 56 Herve Ballans
533 104 Pablo Alingery
</pre>
534 6 Pablo Alingery
535 59 Herve Ballans
536 59 Herve Ballans
Create tmp dir in netdrms working directory (/usr/local/netdrms_current)
537 59 Herve Ballans
<pre>
538 59 Herve Ballans
# mkdir tmp
539 59 Herve Ballans
</pre>
540 61 Herve Ballans
541 61 Herve Ballans
Prevent from deleting files
542 61 Herve Ballans
<pre>
543 61 Herve Ballans
cd logs/SUM ; scp production@sdo:/usr/local/netdrms/install-config-files/sum_rm.cfg .
544 61 Herve Ballans
</pre>
545 117 Pablo Alingery
546 112 Herve Ballans
h2. JMD installation 
547 115 Pablo Alingery
548 115 Pablo Alingery
Donwnload new JMD package build by Niles Oien 2016-04-07 following thoses actions :
549 115 Pablo Alingery
550 116 Pablo Alingery
<pre>
551 116 Pablo Alingery
cd /urs/local
552 115 Pablo Alingery
553 115 Pablo Alingery
[oien@spsc-nso19-12 ~]$ ftp gong2.nso.edu
554 115 Pablo Alingery
Name (gong2.nso.edu:oien): anonymous
555 115 Pablo Alingery
Password: <---------------- use your email as the password
556 115 Pablo Alingery
ftp> cd outgoing/oien
557 115 Pablo Alingery
ftp> get pablo_jmd.tar.gz
558 115 Pablo Alingery
ftp> quit
559 115 Pablo Alingery
</pre>
560 115 Pablo Alingery
561 115 Pablo Alingery
Install new JMD package following thoses actions 
562 115 Pablo Alingery
563 115 Pablo Alingery
<pre>
564 1 Herve Ballans
[oien@spsc-nso19-12 ~]$ gunzip -vf pablo_jmd.tar.gz
565 1 Herve Ballans
pablo_jmd.tar.gz:      8.3% -- replaced with pablo_jmd.tar
566 116 Pablo Alingery
[oien@spsc-nso19-12 ~]$ tar xf pablo_jmd.tar
567 116 Pablo Alingery
[oien@spsc-nso19-12 ~]$ cd pablo_jmd
568 116 Pablo Alingery
[oien@spsc-nso19-12 pablo_jmd]$ less PABLO_README.txt
569 115 Pablo Alingery
cd ..
570 116 Pablo Alingery
mv jmd jmd_old
571 115 Pablo Alingery
mv pablo_jmd jmd 
572 112 Herve Ballans
</pre>
573 119 Pablo Alingery
574 112 Herve Ballans
h2. JMD Configuration
575 112 Herve Ballans
576 112 Herve Ballans
Configure webserver  to request jsoc_fetch
577 112 Herve Ballans
(following the mail from nilesoien@gmail.com)
578 113 Pablo Alingery
579 113 Pablo Alingery
config d 'apache2 fichier sdo3.ias.u-psud.fr 
580 113 Pablo Alingery
<pre>
581 113 Pablo Alingery
582 113 Pablo Alingery
<VirtualHost *:80>
583 113 Pablo Alingery
        ServerName sdo3.ias.u-psud.fr
584 113 Pablo Alingery
        ServerAdmin pablo.alingery@ias.u-psud.fr
585 113 Pablo Alingery
        #ErrorLog /var/log/apache2/error.log
586 113 Pablo Alingery
        DocumentRoot /var/www
587 113 Pablo Alingery
588 113 Pablo Alingery
        # Possible values include: debug, info, notice, warn, error, crit,
589 113 Pablo Alingery
        # alert, emerg.
590 113 Pablo Alingery
        LogLevel debug
591 113 Pablo Alingery
592 113 Pablo Alingery
        #CustomLog /var/log/apache2/VSO/access.log combined
593 113 Pablo Alingery
594 113 Pablo Alingery
        Alias /VSO/DRMS/cgi-bin/ "/home/production/netdrms-tools/scripts/cgi-bin/"
595 113 Pablo Alingery
596 113 Pablo Alingery
        <Location /VSO/DRMS/cgi-bin>
597 113 Pablo Alingery
                SetHandler perl-script
598 113 Pablo Alingery
                PerlResponseHandler ModPerl::Registry
599 113 Pablo Alingery
                PerlOptions +ParseHeaders
600 113 Pablo Alingery
                Options +ExecCGI
601 113 Pablo Alingery
                Order allow,deny
602 113 Pablo Alingery
                Allow from all
603 112 Herve Ballans
        </Location>
604 112 Herve Ballans
</VirtualHost>
605 112 Herve Ballans
606 112 Herve Ballans
</pre>
607 1 Herve Ballans
608 120 Pablo Alingery
609 1 Herve Ballans
Execution test cgi with URL :
610 112 Herve Ballans
http://sdo3.ias.u-psud.fr/ VSO/DRMS/cgi-bin/vso_jsoc_fetch.cgi
611 120 Pablo Alingery
612 112 Herve Ballans
expected result  :
613 112 Herve Ballans
<pre>
614 112 Herve Ballans
{
615 112 Herve Ballans
   "wait" : 0,
616 112 Herve Ballans
   "requestid" : "",
617 112 Herve Ballans
   "data" : {},
618 112 Herve Ballans
   "method" : "url_quick",
619 112 Herve Ballans
   "size" : 0,
620 112 Herve Ballans
   "errormsg" : "Empty query",
621 112 Herve Ballans
   "protocol" : "as-is",
622 112 Herve Ballans
   "status" : 1,
623 112 Herve Ballans
   "space_ratio" : 0.1,
624 112 Herve Ballans
   "load_ratio" : 0.066875,
625 112 Herve Ballans
   "dir" : "",
626 112 Herve Ballans
   "count" : 0
627 112 Herve Ballans
}
628 91 Herve Ballans
</pre>
629 91 Herve Ballans
630 91 Herve Ballans
h2. NetDRMS Start script
631 91 Herve Ballans
632 91 Herve Ballans
<pre>
633 91 Herve Ballans
sum_start.NetDRMS
634 91 Herve Ballans
</pre>
635 91 Herve Ballans
636 91 Herve Ballans
h2. Database repair (from Art)
637 91 Herve Ballans
638 91 Herve Ballans
On ias_sdo
639 91 Herve Ballans
640 91 Herve Ballans
<pre>
641 91 Herve Ballans
delete from admin.ns where name = 'lm_jps';
642 91 Herve Ballans
delete from admin.ns where name = 'aia_test';
643 91 Herve Ballans
</pre>
644 93 Pablo Alingery
645 93 Pablo Alingery
h2. Private key Public key generation 
646 93 Pablo Alingery
647 93 Pablo Alingery
<pre>
648 93 Pablo Alingery
ssh-keygen -t rsa
649 93 Pablo Alingery
</pre>
650 103 Herve Ballans
651 103 Herve Ballans
Not necessary cause we recover the ssh keys of the current server.
652 103 Herve Ballans
653 103 Herve Ballans
However, we have to retrieve the private key of production account on sdo3
654 103 Herve Ballans
655 103 Herve Ballans
From sdo :
656 103 Herve Ballans
<pre>
657 103 Herve Ballans
rsync -av id_rsa production@sdo3:/home/production/.ssh/
658 103 Herve Ballans
</pre>
659 94 Herve Ballans
660 94 Herve Ballans
h2. Node definition
661 94 Herve Ballans
662 94 Herve Ballans
file : /usr/local/netdrms_current/base/drms/replication/etc/ias.repclient.cfg
663 94 Herve Ballans
664 94 Herve Ballans
<pre>
665 1 Herve Ballans
node=IAStest
666 96 Herve Ballans
</pre>
667 96 Herve Ballans
668 96 Herve Ballans
h2. show_series and delete_series binaries
669 96 Herve Ballans
670 97 Pablo Alingery
Remark : these 2 sources are not compiled during installation
671 96 Herve Ballans
Issue : in netdrms_current 8.10 , do :
672 96 Herve Ballans
$ make delete_series
673 94 Herve Ballans
$ make show_series
674 98 Pablo Alingery
675 1 Herve Ballans
h2. SSH-HPN install 
676 98 Pablo Alingery
677 1 Herve Ballans
Intallation de openssh 6.3p1 car le dernier patch hpn dispo est 6.3hpn13.V6 
678 99 Herve Ballans
679 99 Herve Ballans
web site for open ssh : http://ftp2.fr.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-6.3p1.tar.gz
680 1 Herve Ballans
patch hpn : http://www.psc.edu/index.php/hpn-ssh-patches/hpn-14-kitchen-sink-patches/viewdownload/24-hpn-14-kitchen-sink-patches/102-openssh-6-3p1-hpnssh14v2-kitchen-sink-patch
681 99 Herve Ballans
682 99 Herve Ballans
<pre>
683 99 Herve Ballans
# tar -xzvf openssh-6.3p1.tar.gz
684 99 Herve Ballans
# gunzip openssh-6.3p1-hpnssh14v2.diff.gz
685 99 Herve Ballans
# cd openssh-6.3p1
686 99 Herve Ballans
# zcat ../openssh-6.3p1-hpnssh14v2.diff.gz | patch -p1
687 99 Herve Ballans
# ./configure --prefix=/usr/local/hpn-ssh --with-pam --with-md5-passwords --without-zlib-version-check --with-tcp-wrappers
688 99 Herve Ballans
# make
689 99 Herve Ballans
# make install
690 1 Herve Ballans
</pre>
691 99 Herve Ballans
692 99 Herve Ballans
Depuis sdo :
693 99 Herve Ballans
<pre>
694 99 Herve Ballans
# cd /usr/local/hpn-ssh/etc/
695 99 Herve Ballans
# rsync -av *key* netadm@sdo3:/home/netadm/
696 99 Herve Ballans
</pre>
697 99 Herve Ballans
698 99 Herve Ballans
Sur sdo3 :
699 1 Herve Ballans
<pre>
700 100 Herve Ballans
rsync -av /home/netadm/*key* /usr/local/hpn-ssh/etc/
701 100 Herve Ballans
</pre>
702 100 Herve Ballans
703 100 Herve Ballans
We have to configure the port number to 55000
704 100 Herve Ballans
<pre>
705 100 Herve Ballans
# vi ssh_config
706 100 Herve Ballans
</pre>
707 101 Herve Ballans
708 100 Herve Ballans
and add :
709 100 Herve Ballans
<pre>
710 100 Herve Ballans
#Port 22
711 100 Herve Ballans
Port 55000
712 1 Herve Ballans
</pre>
713 101 Herve Ballans
714 100 Herve Ballans
We configure also the server hpn (even we don't use it yet)
715 100 Herve Ballans
<pre>
716 100 Herve Ballans
# vi sshd_config
717 100 Herve Ballans
</pre>
718 100 Herve Ballans
719 100 Herve Ballans
and add :
720 100 Herve Ballans
<pre>
721 100 Herve Ballans
#Port 22
722 100 Herve Ballans
Port 55000
723 100 Herve Ballans
724 100 Herve Ballans
#PidFile /var/run/sshd.pid
725 100 Herve Ballans
PidFile /var/run/sshd.55000.pid
726 100 Herve Ballans
727 100 Herve Ballans
# allow the use of the none cipher
728 100 Herve Ballans
#NoneEnabled no
729 99 Herve Ballans
NoneEnabled yes
730 98 Pablo Alingery
</pre>
731 106 Pablo Alingery
732 105 Pablo Alingery
h2. Correction
733 105 Pablo Alingery
734 105 Pablo Alingery
h3. Modification du fichier ias.repclient.cfg
735 105 Pablo Alingery
736 105 Pablo Alingery
Copier le template existant repclient.template.cfg dans le répertoire [netdrms_current]/base/drms/replication/etc
737 105 Pablo Alingery
 
738 105 Pablo Alingery
<pre>
739 105 Pablo Alingery
production@sdo3:/usr/local/netdrms_current/base/drms/replication/etc$ cp repclient.template.cfg ias.repclient.cfg
740 105 Pablo Alingery
</pre>
741 1 Herve Ballans
742 106 Pablo Alingery
Nouveau / netdrms2.4 
743 105 Pablo Alingery
<pre>
744 105 Pablo Alingery
# Apps
745 106 Pablo Alingery
kRSPerl=<path to Perl binary to use when Perl scripts are executed>
746 105 Pablo Alingery
</pre>
747 107 Pablo Alingery
748 107 Pablo Alingery
h2. Subscription test 
749 107 Pablo Alingery
750 107 Pablo Alingery
h3. Librairies manquantes
751 107 Pablo Alingery
752 107 Pablo Alingery
Errors 
753 107 Pablo Alingery
<pre>
754 107 Pablo Alingery
Failure to apply SQL file 'IAStest.subscribe_series.sql' from server: ABORTING!
755 107 Pablo Alingery
Can't locate DBI.pm 
756 107 Pablo Alingery
.
757 107 Pablo Alingery
.
758 107 Pablo Alingery
.
759 107 Pablo Alingery
Failure to apply SQL file 'IAStest.subscribe_series.sql' from server: ABORTING!
760 107 Pablo Alingery
Can't locate DBD/Pg.pm
761 107 Pablo Alingery
</pre>
762 1 Herve Ballans
763 1 Herve Ballans
<pre>
764 1 Herve Ballans
root@sdo3:/usr/local/netdrms_8.10/base/drms/replication/subscribe_series# apt-get install libdbi-perl 
765 1 Herve Ballans
root@sdo3:/usr/local/netdrms_8.10/base/drms/replication/subscribe_series# apt-get install libdbd-pg-perl
766 1 Herve Ballans
</pre>
767 110 Herve Ballans
768 110 Herve Ballans
h3. Series souscrites
769 110 Herve Ballans
770 110 Herve Ballans
hmi.sharp_720s_nrt
771 110 Herve Ballans
hmi.mharp_720s_nrt
772 110 Herve Ballans
hmi.bharp_720s_nrt
773 110 Herve Ballans
774 110 Herve Ballans
<pre>
775 110 Herve Ballans
production@sdo3:~$ show_info -s hmi.sharp_720s_nrt
776 110 Herve Ballans
First Record: hmi.sharp_720s_nrt[476][2012.09.14_02:12:00_TAI], Recnum = 829707
777 110 Herve Ballans
Last Record:  hmi.sharp_720s_nrt[4943][2016.03.22_18:24:00_TAI], Recnum = 2015202
778 110 Herve Ballans
Last Recnum:  2015202
779 110 Herve Ballans
Has shadow table: no
780 107 Pablo Alingery
</pre>
781 111 Herve Ballans
782 111 Herve Ballans
h3. Lancement de jetty
783 111 Herve Ballans
784 111 Herve Ballans
<pre>
785 111 Herve Ballans
$ jetty.sh start
786 111 Herve Ballans
</pre>
787 111 Herve Ballans
788 111 Herve Ballans
789 111 Herve Ballans
790 108 Pablo Alingery
791 108 Pablo Alingery
h2. Test recuperation de meta data 
792 108 Pablo Alingery
793 108 Pablo Alingery
librairies manquantes 
794 108 Pablo Alingery
apt-get install libnet-ssh-perl
795 108 Pablo Alingery
apt-get install libstring-shellquote-perl
796 108 Pablo Alingery
797 152 Pablo Alingery
h2. TO BE DONE FURTHER
798 108 Pablo Alingery
799 107 Pablo Alingery
800 152 Pablo Alingery
%{color:red} configure sytem nbr de fichier ouvert en meme temps%
801 1 Herve Ballans
802 152 Pablo Alingery
%{color:red} configure sytem taille  buffer%
803 152 Pablo Alingery
804 155 Pablo Alingery
%{color:red} swap désactivé ? discussion Stephane et Gilles%
805 152 Pablo Alingery
806 80 Pablo Alingery
807 1 Herve Ballans
h3. Modification du fichir config.local pour la souscription au jsoc
808 1 Herve Ballans
809 1 Herve Ballans
810 1 Herve Ballans
<pre>
811 1 Herve Ballans
# NetDRMS users can request subdirectories from the Stanford proj directories 
812 1 Herve Ballans
# (e.g., JSOC/proj/util). These users will need to contact Stanford and obtain
813 1 Herve Ballans
# the source subdirectories desired. They will then place these subdirectories
814 1 Herve Ballans
# in the JSOC/proj directory of their NetDRMS release. To properly build 
815 1 Herve Ballans
# targets in these subdirectories, using the JSOC make system, the user needs
816 1 Herve Ballans
# to add entries to this configuration file - one entry for each subdirectory
817 1 Herve Ballans
# that contains source code to be compiled. Each entry is a space-separated pair
818 1 Herve Ballans
# of strings: the string "PROJDIR" followed by a subdirectory (of the proj 
819 1 Herve Ballans
# directory).  For example:
820 1 Herve Ballans
#
821 1 Herve Ballans
# PROJDIR     util/apps
822 1 Herve Ballans
# PROJDIR     util/libs
823 1 Herve Ballans
# 
824 1 Herve Ballans
# NOTE: It is not sufficient to list only a parent directory, like util. Source code
825 18 Herve Ballans
# in child directories will not automatically be compiled.
826 18 Herve Ballans
</pre>
827 18 Herve Ballans
828 18 Herve Ballans
h3. Python library for multi-threaded SUMS
829 1 Herve Ballans
830 1 Herve Ballans
Installation de la librairie python3.4m