Project

General

Profile

NetDRMS Installation » History » Version 298

Herve Ballans, 10/06/2016 14:56

1 50 Herve Ballans
{{toc}}
2 1 Herve Ballans
3 1 Herve Ballans
h1. NetDRMS Installation
4 1 Herve Ballans
5 276 Herve Ballans
h2. Preliminary note
6 275 Herve Ballans
7 275 Herve Ballans
In the following sections, when there is a command line :
8 289 Herve Ballans
_#\_ means that command is executed into _root_ user
9 289 Herve Ballans
_$\_ means that command is executed into _production_ user
10 275 Herve Ballans
11 1 Herve Ballans
h2. Requirements
12 1 Herve Ballans
13 47 Pablo Alingery
h3. Configuring production environment
14 1 Herve Ballans
15 284 Herve Ballans
Adding user production
16 47 Pablo Alingery
17 1 Herve Ballans
<pre>
18 1 Herve Ballans
# adduser production
19 1 Herve Ballans
# addgroup ias
20 1 Herve Ballans
# adduser production ias
21 1 Herve Ballans
</pre>
22 1 Herve Ballans
23 49 Herve Ballans
h3. Configure /home/production/.profile
24 1 Herve Ballans
25 49 Herve Ballans
Adding environment variables
26 1 Herve Ballans
27 1 Herve Ballans
<pre>
28 122 Herve Ballans
#Modif Herve.B Pablo.A 12/05/2016
29 122 Herve Ballans
30 1 Herve Ballans
PATH="/usr/local/netdrms_current/bin/linux_x86_64:$PATH"
31 49 Herve Ballans
PATH="/usr/local/netdrms_current/scripts:$PATH"
32 122 Herve Ballans
PATH="/usr/local/jmd/bin:$PATH"
33 122 Herve Ballans
PATH="/usr/local/jmd/scripts:$PATH"
34 141 Pablo Alingery
#The following is not needed as installed with 'apt-get install' 
35 141 Pablo Alingery
#PATH="/usr/local/pgsql/bin:$PATH"
36 122 Herve Ballans
PATH="/usr/local/netdrms-tools/scripts:$PATH"
37 49 Herve Ballans
#2014-12-17 Avec Herve et Pablo on pense que la ligne suivante est intutile as it is f.. install in /usr/local/lib
38 140 Pablo Alingery
PATH="/usr/local/cfitsio:$PATH"
39 1 Herve Ballans
#Done
40 49 Herve Ballans
41 122 Herve Ballans
DERBY_HOME=/usr/local/jmd/databases/derby/derbyBD
42 49 Herve Ballans
export DERBY_HOME
43 49 Herve Ballans
44 49 Herve Ballans
export CVSROOT=:pserver:anonymous@solarch.tuc.noao.edu:2401/vtarc1/vso/cvsroot
45 49 Herve Ballans
</pre>
46 49 Herve Ballans
47 162 Pablo Alingery
h3. Configure /varl/lib/postgres/.profile
48 161 Pablo Alingery
49 163 Pablo Alingery
Add to $PATH so postgres can exeute command like initdb
50 161 Pablo Alingery
51 161 Pablo Alingery
<pre>
52 161 Pablo Alingery
#Modif Herve.B Pablo.A 19/05/2016
53 161 Pablo Alingery
PATH="/usr/lib/postgresql/9.4/bin:$PATH"
54 161 Pablo Alingery
</pre>
55 161 Pablo Alingery
56 125 Pablo Alingery
h3. Setting privileges for SUMS_MANAGER
57 87 Herve Ballans
58 126 Pablo Alingery
Please modify file /etc/sudoers so user 'production' will be able to execute  'sum_chown' that we will locate in the dir : /usr/local/bin . See below
59 125 Pablo Alingery
Add to /etc/sudoers the following line :
60 1 Herve Ballans
	production host=NOPASSWD:/usr/local/bin/sum_chmown
61 1 Herve Ballans
62 127 Pablo Alingery
h3. Set passwords file for db postgres 
63 1 Herve Ballans
64 127 Pablo Alingery
Please create file .pgpass with the following information for user 'production" so he will be able to connect without password as 'production' or 'postgres' to 2 db that we will create later ias_sdo and ias_sdo_sums.
65 1 Herve Ballans
66 1 Herve Ballans
67 1 Herve Ballans
.pgpass content :
68 1 Herve Ballans
<pre>
69 128 Pablo Alingery
	#hostname:5432:ias_sdo:production: 
70 128 Pablo Alingery
	#hostname:5434:ias_sdo_sums:production: 
71 128 Pablo Alingery
	#hostname:5432:ias_sdo:postgres: 
72 128 Pablo Alingery
	#hostname:5434:ias_sdo_sums:postgres
73 1 Herve Ballans
</pre>
74 128 Pablo Alingery
75 129 Pablo Alingery
where #hostname is the name of your server
76 129 Pablo Alingery
77 129 Pablo Alingery
78 131 Pablo Alingery
%{color:red}WARNING : change hostname when migrate sdo-new to sdo%
79 1 Herve Ballans
80 133 Pablo Alingery
h3. Install missing libraries 
81 1 Herve Ballans
82 293 Herve Ballans
* libreadline-dev (but already installed with postgresql)
83 293 Herve Ballans
* libssl-dev 
84 293 Herve Ballans
* libpam0g-dev 
85 293 Herve Ballans
* libperl-dev 
86 293 Herve Ballans
* libnet-ssh-perl
87 293 Herve Ballans
* libstring-shellquote-perl
88 296 Herve Ballans
* libdbi-perl 
89 296 Herve Ballans
* libdbd-pg-perl
90 1 Herve Ballans
91 133 Pablo Alingery
NB : Already done in Debian 8.4 Jessy
92 1 Herve Ballans
<pre>
93 1 Herve Ballans
# apt-get install $library_name
94 1 Herve Ballans
</pre> 
95 1 Herve Ballans
where
96 1 Herve Ballans
	_$library_name_ is the name of library
97 19 Pablo Alingery
98 1 Herve Ballans
Then to update path on libraries, please type :
99 1 Herve Ballans
<pre>
100 19 Pablo Alingery
# ldconfig
101 1 Herve Ballans
</pre>
102 1 Herve Ballans
103 1 Herve Ballans
h3. Reconfigure locale
104 1 Herve Ballans
105 1 Herve Ballans
<pre>
106 1 Herve Ballans
# dpkg-reconfigure locales
107 1 Herve Ballans
</pre>
108 1 Herve Ballans
109 1 Herve Ballans
select fr_FR.UTF8 UTF8 and unselect en_US.UTF-8
110 1 Herve Ballans
111 1 Herve Ballans
localedef -i en_US -f UTF-8 en_US.UTF-8
112 1 Herve Ballans
113 1 Herve Ballans
(see here : [[http://www.thomas-krenn.com/en/wiki/Perl_warning_Setting_locale_failed_in_Debian]])
114 1 Herve Ballans
115 134 Pablo Alingery
h3. Install gfortran packages
116 1 Herve Ballans
117 1 Herve Ballans
<pre>
118 1 Herve Ballans
# apt-get install gfortran swig
119 1 Herve Ballans
</pre>
120 1 Herve Ballans
121 1 Herve Ballans
h3. Installation of cfitsio library
122 1 Herve Ballans
123 135 Pablo Alingery
In /usr/local dir
124 135 Pablo Alingery
125 135 Pablo Alingery
Version 3.39 (currently on sdo : 3.24)
126 1 Herve Ballans
<pre>
127 135 Pablo Alingery
# wget ftp://heasarc.gsfc.nasa.gov/software/fitsio/c/cfitsio3390.tar.gz
128 1 Herve Ballans
</pre>
129 137 Pablo Alingery
Untar and  decompression
130 1 Herve Ballans
<pre>
131 136 Pablo Alingery
# tar -xzvf cfitsio3390.tar.gz
132 1 Herve Ballans
</pre> 
133 1 Herve Ballans
134 139 Pablo Alingery
Go into /usr/local and type
135 1 Herve Ballans
136 1 Herve Ballans
su - root and :
137 1 Herve Ballans
<pre>
138 1 Herve Ballans
# ./configure --prefix=/usr/local
139 1 Herve Ballans
# make 
140 1 Herve Ballans
# make install
141 135 Pablo Alingery
</pre>
142 118 Pablo Alingery
143 237 Herve Ballans
h3. Installation des libraries libtar-dev, libcurl3-dev et libecpg-dev
144 237 Herve Ballans
145 237 Herve Ballans
<pre>
146 237 Herve Ballans
apt-get install libtar libtar-dev
147 237 Herve Ballans
</pre>
148 237 Herve Ballans
149 238 Pablo Alingery
Note, selecting 'libcurl4-openssl-dev' instead of 'libcurl3-dev'
150 238 Pablo Alingery
libcurl4-openssl-dev is already the newest version
151 237 Herve Ballans
<pre>
152 237 Herve Ballans
apt-get install libcurl3-dev
153 237 Herve Ballans
</pre>
154 237 Herve Ballans
155 237 Herve Ballans
<pre>
156 237 Herve Ballans
apt-get install libecpg-dev
157 237 Herve Ballans
</pre>
158 237 Herve Ballans
159 144 Pablo Alingery
h3. Installation of apache2 and lib perl
160 118 Pablo Alingery
161 118 Pablo Alingery
<pre>
162 118 Pablo Alingery
# apt-get install apache2
163 118 Pablo Alingery
164 118 Pablo Alingery
# apt-get install libjson-perl
165 118 Pablo Alingery
166 118 Pablo Alingery
# apt-get install libapache2-mod-perl2
167 118 Pablo Alingery
</pre>
168 1 Herve Ballans
169 183 Herve Ballans
h3. Installation of postgresql
170 1 Herve Ballans
171 145 Pablo Alingery
Into 'root'
172 145 Pablo Alingery
 
173 145 Pablo Alingery
<pre>
174 147 Pablo Alingery
apt-get install postgresql-9.4 postgresql-client-9.4 
175 145 Pablo Alingery
</pre>
176 145 Pablo Alingery
177 213 Herve Ballans
h3. Installation of python3
178 209 Pablo Alingery
179 209 Pablo Alingery
Into 'root'
180 209 Pablo Alingery
 
181 209 Pablo Alingery
<pre>
182 209 Pablo Alingery
apt-get install python3 
183 1 Herve Ballans
</pre>
184 210 Pablo Alingery
185 211 Pablo Alingery
Add some modules psycopg2 (postgres connector) and pySmartDL (Download manager) 
186 210 Pablo Alingery
187 210 Pablo Alingery
<pre>
188 210 Pablo Alingery
apt-get install python3-psycopg2
189 210 Pablo Alingery
</pre>
190 210 Pablo Alingery
191 210 Pablo Alingery
<pre>
192 210 Pablo Alingery
apt-get install python3-psycopg2
193 210 Pablo Alingery
</pre>
194 210 Pablo Alingery
195 210 Pablo Alingery
Add python3-pip for modules within python3
196 210 Pablo Alingery
197 210 Pablo Alingery
<pre>
198 210 Pablo Alingery
apt-get install python3-pip
199 1 Herve Ballans
</pre>
200 212 Pablo Alingery
201 212 Pablo Alingery
<pre>
202 212 Pablo Alingery
pip3 install pySmartDL
203 212 Pablo Alingery
</pre>
204 1 Herve Ballans
205 213 Herve Ballans
Both python2.7 and 3.4 versions are installed on the system.
206 213 Herve Ballans
By default, the python2.7 is used.
207 213 Herve Ballans
In order to change in 3.4 by default, type :
208 213 Herve Ballans
<pre>
209 213 Herve Ballans
update-alternatives --config python
210 213 Herve Ballans
</pre>
211 209 Pablo Alingery
212 1 Herve Ballans
h3. Installation of sdo_scripts
213 218 Pablo Alingery
214 219 Pablo Alingery
Install monitoring scripts for sdo
215 218 Pablo Alingery
216 218 Pablo Alingery
<pre>
217 268 Pablo Alingery
rsync -av scripts production@sdo-new:/home/production/netdrms-tools/
218 218 Pablo Alingery
</pre>
219 218 Pablo Alingery
220 1 Herve Ballans
h2. NetDRMS databases
221 1 Herve Ballans
222 156 Pablo Alingery
into /var/lib/postgresql  (hard mount point separated,  7 disks 15000 tr/min in raid5 in order to distribute read and write) 
223 1 Herve Ballans
224 1 Herve Ballans
<pre>
225 158 Pablo Alingery
# cd /var/lib/postgresql
226 157 Pablo Alingery
# mkdir data  data_sums data_monitor
227 157 Pablo Alingery
# chown -R postgres:postgres * 
228 1 Herve Ballans
</pre>
229 1 Herve Ballans
230 160 Pablo Alingery
h3. Initialize 3 servers data,  data_sums, data_monitor
231 1 Herve Ballans
232 159 Pablo Alingery
Into postgres
233 1 Herve Ballans
<pre>
234 1 Herve Ballans
# su - postgres
235 1 Herve Ballans
236 1 Herve Ballans
$ initdb --locale=C -D data -A md5 -W
237 1 Herve Ballans
(no password)
238 2 Herve Ballans
239 2 Herve Ballans
Success. You can now start the database server using:
240 2 Herve Ballans
241 2 Herve Ballans
    postgres -D data
242 2 Herve Ballans
or
243 2 Herve Ballans
    pg_ctl -D data -l logfile start
244 2 Herve Ballans
245 1 Herve Ballans
246 1 Herve Ballans
$ initdb --locale=C -D data_sums -A md5 -W
247 2 Herve Ballans
(no password)
248 2 Herve Ballans
249 2 Herve Ballans
Success. You can now start the database server using:
250 2 Herve Ballans
251 2 Herve Ballans
    postgres -D data_sums
252 2 Herve Ballans
or
253 2 Herve Ballans
    pg_ctl -D data_sums -l logfile start
254 8 Pablo Alingery
255 8 Pablo Alingery
$ initdb --locale=C -D data_monitor -A md5 -W
256 8 Pablo Alingery
(no password)
257 8 Pablo Alingery
258 8 Pablo Alingery
Success. You can now start the database server using:
259 8 Pablo Alingery
260 8 Pablo Alingery
    postgres -D data_monitor
261 8 Pablo Alingery
or
262 8 Pablo Alingery
    pg_ctl -D data_monitor -l logfile start
263 2 Herve Ballans
</pre>
264 2 Herve Ballans
265 1 Herve Ballans
266 1 Herve Ballans
267 1 Herve Ballans
Edition des fichiers postgresql.conf pour configurer les bons ports d'écoute :
268 1 Herve Ballans
listen_addresses = '*'
269 1 Herve Ballans
data -> port 5432
270 10 Pablo Alingery
data_sums -> port 5434
271 1 Herve Ballans
data_monitor -> port 5436
272 10 Pablo Alingery
273 165 Pablo Alingery
Edit files data*/pg_hba.conf : replace 'md5' by 'trust'
274 1 Herve Ballans
<pre>
275 1 Herve Ballans
#"local"
276 1 Herve Ballans
local   all             all                                     trust
277 1 Herve Ballans
#"IPv4"
278 1 Herve Ballans
host   all             all                                     trust
279 1 Herve Ballans
</pre>
280 166 Pablo Alingery
281 166 Pablo Alingery
h3. Tunning postgres 
282 166 Pablo Alingery
283 166 Pablo Alingery
In file postgres.conf ( according to https://www.qwant.com/?q=Tunning+postgres+server+9.4+&client=firefox  )
284 166 Pablo Alingery
285 168 Pablo Alingery
shared_buffer=4096MB %{color:red}So 1/12 of the total RAM memory available%
286 1 Herve Ballans
287 169 Pablo Alingery
work_mem = 24MB
288 168 Pablo Alingery
289 168 Pablo Alingery
maintenance_work_mem = 4096MB
290 1 Herve Ballans
291 169 Pablo Alingery
wal_buffers = 16MB
292 169 Pablo Alingery
293 169 Pablo Alingery
checkpoint_segments = 32
294 169 Pablo Alingery
295 169 Pablo Alingery
checkpoint_completion_target = 0.9
296 169 Pablo Alingery
297 169 Pablo Alingery
random_page_cost = 1.0
298 169 Pablo Alingery
299 169 Pablo Alingery
effective_cache_size = 16384MB so 1/4 of the max memory 
300 169 Pablo Alingery
301 170 Pablo Alingery
302 170 Pablo Alingery
303 170 Pablo Alingery
#log
304 169 Pablo Alingery
logging_collector = on 
305 169 Pablo Alingery
log_directory = 'pg_log' 
306 169 Pablo Alingery
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,
307 169 Pablo Alingery
log_truncate_on_rotation = on           # If on, an existing log file with the
308 169 Pablo Alingery
log_rotation_age = 1d                   # Automatic rotation of logfiles will
309 170 Pablo Alingery
client_min_messages = error
310 170 Pablo Alingery
log_min_messages = error 
311 170 Pablo Alingery
log_min_duration_statement = 0
312 170 Pablo Alingery
313 170 Pablo Alingery
log_connections = on
314 170 Pablo Alingery
log_disconnections = on
315 170 Pablo Alingery
log_duration = off
316 170 Pablo Alingery
log_hostname = on
317 170 Pablo Alingery
log_line_prefix = '%t [%p]: %u@%h - %d :'  
318 170 Pablo Alingery
319 170 Pablo Alingery
track_counts = on
320 170 Pablo Alingery
autovacuum = on 
321 170 Pablo Alingery
322 170 Pablo Alingery
extra_float_digits = 3
323 170 Pablo Alingery
324 177 Pablo Alingery
h3. Start the 3 databases :
325 1 Herve Ballans
326 1 Herve Ballans
<pre>
327 1 Herve Ballans
# su - postgres
328 196 Herve Ballans
$ cd /var/lib/postgresql
329 172 Pablo Alingery
$ pg_ctl -D data  start
330 172 Pablo Alingery
$ pg_ctl -D data_sums start
331 172 Pablo Alingery
$ pg_ctl -D data_monitor start
332 1 Herve Ballans
</pre>
333 174 Pablo Alingery
334 1 Herve Ballans
h3. Create the 3 databases :
335 181 Pablo Alingery
336 178 Pablo Alingery
Into user 'postgres'
337 1 Herve Ballans
338 179 Pablo Alingery
* createdb --locale C -E LATIN1 -T template0 ias_sdo
339 179 Pablo Alingery
* createdb --locale C -E LATIN1 -T template0 -p 5434 ias_sdo_sums
340 179 Pablo Alingery
* createdb --locale C -E LATIN1 -T template0 -p 5436 ias_sdo_monitor
341 54 Herve Ballans
342 182 Herve Ballans
h3. Automatic startup
343 182 Herve Ballans
344 182 Herve Ballans
Add following lines in the /etc/rc.local file :
345 182 Herve Ballans
346 182 Herve Ballans
<pre>
347 197 Herve Ballans
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data start" postgres
348 197 Herve Ballans
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_sums start" postgres
349 197 Herve Ballans
su -l -c "/usr/lib/postgresql/9.4/bin/pg_ctl -D /var/lib/postgresql/data_monitor start" postgres
350 182 Herve Ballans
</pre>
351 182 Herve Ballans
352 184 Herve Ballans
h2. System tuning
353 1 Herve Ballans
354 187 Herve Ballans
Current configuration :
355 185 Herve Ballans
<pre>
356 185 Herve Ballans
# sysctl -a
357 185 Herve Ballans
</pre>
358 1 Herve Ballans
359 187 Herve Ballans
In the following file :
360 185 Herve Ballans
<pre>
361 185 Herve Ballans
# vi /etc/sysctl.conf
362 185 Herve Ballans
</pre>
363 1 Herve Ballans
364 188 Herve Ballans
we change some values (default values are commented with '#')
365 194 Herve Ballans
366 195 Herve Ballans
(resources links [[Links|here]])
367 194 Herve Ballans
368 185 Herve Ballans
<pre>
369 191 Herve Ballans
# Semaphore is a object that is used to control utilization of a particular process.
370 191 Herve Ballans
# kernel.sem = SEMMSL SEMMNS SEMOPM SEMMNI
371 191 Herve Ballans
# SEMMSL        maximum number of semaphores per array  (min 128)
372 191 Herve Ballans
# SEMMNS        maximum semaphores system-wide  
373 191 Herve Ballans
# SEMOPM        maximum operations per semop call       
374 191 Herve Ballans
# SEMMNI        maximum arrays 
375 191 Herve Ballans
#semop incresed from 32 (default value)operation to 100 (3rd value )
376 191 Herve Ballans
# kernel.sem = 250      32000   32      128
377 191 Herve Ballans
kernel.sem = 250 32000 100 128
378 191 Herve Ballans
379 191 Herve Ballans
In order to  apply changes :#This sets the  OS receive buffer size for all types of connections (default value 212992)
380 191 Herve Ballans
#net.core.rmem_default = 212992
381 191 Herve Ballans
#net.core.rmem_max = 212992
382 191 Herve Ballans
net.core.rmem_default = 33554432
383 191 Herve Ballans
net.core.rmem_max = 33554432
384 191 Herve Ballans
385 191 Herve Ballans
#This is the OS send buffer size for all types of connections (212992 default value) 
386 191 Herve Ballans
#net.core.wmem_default = 212992
387 191 Herve Ballans
#net.core.wmem_max = 212992
388 191 Herve Ballans
net.core.wmem_default = 33554432
389 191 Herve Ballans
net.core.wmem_max = 33554432
390 191 Herve Ballans
391 193 Pablo Alingery
#TCP Autotuning setting. "The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always
392 193 Pablo Alingery
# allocated to a TCP socket, even under high pressure on the system. ... The second value specified tells the kernel the default receive
393 193 Pablo Alingery
# buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols. ...
394 192 Pablo Alingery
# The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket."
395 1 Herve Ballans
#net.ipv4.tcp_rmem = 4096       87380   6291456
396 1 Herve Ballans
net.ipv4.tcp_rmem = 10240 87380 33554432
397 1 Herve Ballans
398 193 Pablo Alingery
#TCP Autotuning setting. "This variable takes 3 different values which holds information on how much TCP sendbuffer memory space
399 193 Pablo Alingery
# each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three 
400 193 Pablo Alingery
#values are used under different conditions. ... The first value in this variable tells the minimum TCP send buffer space available
401 193 Pablo Alingery
# for a single TCP socket. ... The second value in the variable tells us the default buffer space allowed for a single TCP socket to use. ... 
402 193 Pablo Alingery
#The third value tells the kernel the maximum TCP send buffer space." 
403 191 Herve Ballans
#net.ipv4.tcp_wmem = 4096       16384   4194304
404 191 Herve Ballans
net.ipv4.tcp_wmem = 10240 87380 33554432
405 191 Herve Ballans
406 191 Herve Ballans
#Disable cache metrics so the initial conditions of the closed connections will not be saved to be used in near future connections
407 191 Herve Ballans
#net.ipv4.tcp_no_metrics_save = 0
408 191 Herve Ballans
net.ipv4.tcp_no_metrics_save = 1
409 191 Herve Ballans
410 1 Herve Ballans
# Increase number of incoming connections backlog
411 1 Herve Ballans
# net.core.netdev_max_backlog = 1000
412 1 Herve Ballans
net.core.netdev_max_backlog = 5000
413 1 Herve Ballans
414 1 Herve Ballans
# The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage. ... 
415 192 Pablo Alingery
#The first value specified in the tcp_mem variable tells the kernel the low threshold. 
416 192 Pablo Alingery
#Below this point, the TCP stack do not bother at all about putting any pressure on the memory usage by different TCP sockets. ... 
417 193 Pablo Alingery
#The second value tells the kernel at which point to start pressuring memory usage down. ... The final value tells the kernel how many
418 193 Pablo Alingery
# memory pages it may use maximally
419 1 Herve Ballans
# net.ipv4.tcp_mem = 1543347    2057796 3086694
420 191 Herve Ballans
net.ipv4.tcp_mem = 786432 1048576 26777216
421 191 Herve Ballans
422 192 Pablo Alingery
# local port range that is used by TCP and UDP traffic to choose the local port. You will see in the parameters of this file two numbers:
423 192 Pablo Alingery
# The first number is the first local port allowed for TCP and UDP traffic on the server, the second is the last local port number.
424 191 Herve Ballans
# net.ipv4.ip_local_port_range = 32768  61000
425 191 Herve Ballans
net.ipv4.ip_local_port_range = 1024 65535
426 191 Herve Ballans
427 191 Herve Ballans
# maximum number of sockets in TIME-WAIT to be held simultaneously.
428 191 Herve Ballans
# net.ipv4.tcp_max_tw_buckets = 262144
429 191 Herve Ballans
net.ipv4.tcp_max_tw_buckets = 360000
430 1 Herve Ballans
</pre>
431 185 Herve Ballans
432 191 Herve Ballans
Then we run the following command to make our change take effect:
433 191 Herve Ballans
434 185 Herve Ballans
<pre>
435 185 Herve Ballans
# sysctl -p
436 185 Herve Ballans
</pre>
437 176 Pablo Alingery
438 216 Pablo Alingery
h2. JMD Installation 
439 215 Herve Ballans
440 217 Pablo Alingery
Download new JMD package build by Niles Oien 2016-04-07 following thoses actions :
441 215 Herve Ballans
442 215 Herve Ballans
<pre>
443 215 Herve Ballans
cd /urs/local
444 215 Herve Ballans
445 215 Herve Ballans
[oien@spsc-nso19-12 ~]$ ftp gong2.nso.edu
446 215 Herve Ballans
Name (gong2.nso.edu:oien): anonymous
447 215 Herve Ballans
Password: <---------------- use your email as the password
448 215 Herve Ballans
ftp> cd outgoing/oien
449 215 Herve Ballans
ftp> get pablo_jmd.tar.gz
450 215 Herve Ballans
ftp> quit
451 215 Herve Ballans
</pre>
452 215 Herve Ballans
453 215 Herve Ballans
Install new JMD package following thoses actions 
454 215 Herve Ballans
455 215 Herve Ballans
<pre>
456 215 Herve Ballans
[oien@spsc-nso19-12 ~]$ gunzip -vf pablo_jmd.tar.gz
457 215 Herve Ballans
pablo_jmd.tar.gz:      8.3% -- replaced with pablo_jmd.tar
458 215 Herve Ballans
[oien@spsc-nso19-12 ~]$ tar xf pablo_jmd.tar
459 215 Herve Ballans
[oien@spsc-nso19-12 ~]$ cd pablo_jmd
460 215 Herve Ballans
[oien@spsc-nso19-12 pablo_jmd]$ less PABLO_README.txt
461 215 Herve Ballans
cd ..
462 215 Herve Ballans
mv jmd jmd_old
463 215 Herve Ballans
mv pablo_jmd jmd 
464 215 Herve Ballans
</pre>
465 215 Herve Ballans
466 266 Herve Ballans
h2. Apache Configuration for JMD
467 215 Herve Ballans
468 215 Herve Ballans
Configure webserver  to request jsoc_fetch
469 215 Herve Ballans
(following the mail from nilesoien@gmail.com)
470 215 Herve Ballans
471 222 Pablo Alingery
%{color:red}WARNING : change hostname when migrate sdo-new to sdo%
472 222 Pablo Alingery
473 220 Pablo Alingery
config d 'apache2 fichier sdo-new.ias.u-psud.fr 
474 215 Herve Ballans
<pre>
475 215 Herve Ballans
476 215 Herve Ballans
<VirtualHost *:80>
477 220 Pablo Alingery
        ServerName sdo-new.ias.u-psud.fr
478 215 Herve Ballans
        ServerAdmin pablo.alingery@ias.u-psud.fr
479 215 Herve Ballans
        #ErrorLog /var/log/apache2/error.log
480 215 Herve Ballans
        DocumentRoot /var/www
481 215 Herve Ballans
482 215 Herve Ballans
        # Possible values include: debug, info, notice, warn, error, crit,
483 215 Herve Ballans
        # alert, emerg.
484 215 Herve Ballans
        LogLevel debug
485 215 Herve Ballans
486 215 Herve Ballans
        #CustomLog /var/log/apache2/VSO/access.log combined
487 215 Herve Ballans
488 215 Herve Ballans
        Alias /VSO/DRMS/cgi-bin/ "/home/production/netdrms-tools/scripts/cgi-bin/"
489 215 Herve Ballans
490 215 Herve Ballans
        <Location /VSO/DRMS/cgi-bin>
491 215 Herve Ballans
                SetHandler perl-script
492 215 Herve Ballans
                PerlResponseHandler ModPerl::Registry
493 215 Herve Ballans
                PerlOptions +ParseHeaders
494 215 Herve Ballans
                Options +ExecCGI
495 215 Herve Ballans
                Order allow,deny
496 215 Herve Ballans
                Allow from all
497 215 Herve Ballans
        </Location>
498 215 Herve Ballans
</VirtualHost>
499 215 Herve Ballans
500 215 Herve Ballans
</pre>
501 215 Herve Ballans
502 223 Herve Ballans
Activate the new virtualhost and disable the default one :
503 223 Herve Ballans
<pre>
504 225 Herve Ballans
# a2ensite sdo-new.ias.u-psud.fr.conf
505 225 Herve Ballans
# a2dissite 000-default.conf
506 225 Herve Ballans
# service apache2 restart
507 225 Herve Ballans
</pre>
508 225 Herve Ballans
509 225 Herve Ballans
Allow symlink for the cgi scripts dierctory in apache2.conf :
510 225 Herve Ballans
<pre>
511 225 Herve Ballans
<Directory /home/production/netdrms-tools/scripts/>
512 225 Herve Ballans
        Options FollowSymLinks
513 225 Herve Ballans
        AllowOverride None
514 225 Herve Ballans
        Require all granted
515 225 Herve Ballans
</Directory>
516 225 Herve Ballans
</pre>
517 225 Herve Ballans
518 225 Herve Ballans
<pre>
519 225 Herve Ballans
# service apache2 restart
520 223 Herve Ballans
</pre>
521 223 Herve Ballans
 
522 226 Pablo Alingery
%{color:red}To be  added futher to test cgi%
523 1 Herve Ballans
Execution test cgi with URL :
524 226 Pablo Alingery
525 226 Pablo Alingery
make sure my $hostname=in the cgi file vso_jsoc_fetch.cgi is correct and test
526 224 Pablo Alingery
http://sdo-new.ias.u-psud.fr/VSO/DRMS/cgi-bin/vso_jsoc_fetch.cgi
527 215 Herve Ballans
528 215 Herve Ballans
expected result  :
529 215 Herve Ballans
<pre>
530 215 Herve Ballans
{
531 215 Herve Ballans
   "wait" : 0,
532 215 Herve Ballans
   "requestid" : "",
533 215 Herve Ballans
   "data" : {},
534 215 Herve Ballans
   "method" : "url_quick",
535 215 Herve Ballans
   "size" : 0,
536 215 Herve Ballans
   "errormsg" : "Empty query",
537 215 Herve Ballans
   "protocol" : "as-is",
538 215 Herve Ballans
   "status" : 1,
539 215 Herve Ballans
   "space_ratio" : 0.1,
540 215 Herve Ballans
   "load_ratio" : 0.066875,
541 215 Herve Ballans
   "dir" : "",
542 215 Herve Ballans
   "count" : 0
543 215 Herve Ballans
}
544 215 Herve Ballans
</pre>
545 215 Herve Ballans
546 267 Herve Ballans
h2. JMD configuration
547 267 Herve Ballans
548 291 Herve Ballans
Modify the file [[/usr/local/jmd/cfg/JMD.cfg]]
549 267 Herve Ballans
550 267 Herve Ballans
551 271 Herve Ballans
%{color:red}Warning : when migrate production server, change _sdo-new_ by _sdo_%
552 271 Herve Ballans
553 1 Herve Ballans
h2. NetDRMS tool Install
554 23 Pablo Alingery
555 199 Pablo Alingery
The following installed is based on : [[http://inf-redmine.ias.u-psud.fr/redmine/attachments/download/114/Netdrms_First_Install.odt]]
556 1 Herve Ballans
557 207 Pablo Alingery
The sources are in : http://jsoc.stanford.edu/netdrms/dist/ 
558 207 Pablo Alingery
Locally on sdo-new in  : /home/production/sources
559 1 Herve Ballans
560 199 Pablo Alingery
Symbolic link created 'netdrms_current' that corresponds to the last release
561 227 Pablo Alingery
ln -s netdrms_8.11/ netdrms_current 
562 1 Herve Ballans
563 199 Pablo Alingery
Edit file config.local
564 27 Pablo Alingery
<pre>
565 27 Pablo Alingery
cp config.local.template config.local
566 1 Herve Ballans
</pre>
567 29 Pablo Alingery
568 203 Pablo Alingery
%{color:red}Warning WEB_DBUSER      apache  We think that it should value www maybe , to be checked testing the cgi%
569 208 Pablo Alingery
%{color:red}So check the cgi to figure out which user is used to connect to bd%
570 32 Herve Ballans
571 199 Pablo Alingery
Create 2 followings dirs : 
572 1 Herve Ballans
<pre>
573 227 Pablo Alingery
mkdir /usr/local/netdrms_8.11/logs/SUM
574 227 Pablo Alingery
mkdir /usr/local/netdrms_8.11/logs/slony
575 27 Pablo Alingery
</pre>
576 1 Herve Ballans
577 205 Pablo Alingery
%{color:red}Warning config.local from line 150 =>222 Remote config to provide data, series to externals%
578 205 Pablo Alingery
%{color:red}To be checked for PLATO%
579 1 Herve Ballans
580 1 Herve Ballans
Création d'un alias netdrms dans /root/.bashrc
581 27 Pablo Alingery
<pre>
582 1 Herve Ballans
alias netdrms='cd /usr/local/netdrms_current'
583 1 Herve Ballans
</pre>
584 31 Pablo Alingery
585 1 Herve Ballans
h3. Prepare compiling
586 1 Herve Ballans
587 252 Herve Ballans
Into _production_ profile
588 252 Herve Ballans
589 246 Herve Ballans
<pre>
590 252 Herve Ballans
$ ./configure
591 246 Herve Ballans
</pre>
592 246 Herve Ballans
593 113 Pablo Alingery
h3. Compilation
594 113 Pablo Alingery
595 249 Herve Ballans
Creating lib64 link (lib64 does not exist on debian 8)
596 113 Pablo Alingery
<pre>
597 249 Herve Ballans
# cd /usr
598 249 Herve Ballans
# ln -s lib/ lib64
599 249 Herve Ballans
</pre>
600 249 Herve Ballans
601 249 Herve Ballans
<pre>
602 253 Herve Ballans
$ make
603 253 Herve Ballans
$ make sums
604 113 Pablo Alingery
</pre>
605 113 Pablo Alingery
606 255 Herve Ballans
h2. SSH-HPN install 
607 255 Herve Ballans
608 258 Herve Ballans
Intallation de openssh 7.2p2 car le dernier patch hpn dispo est 7.2hpn14.V10 
609 255 Herve Ballans
610 258 Herve Ballans
web site for open ssh : http://ftp2.fr.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.2p2.tar.gz
611 258 Herve Ballans
patch hpn : https://sourceforge.net/projects/hpnssh/files/HPN-SSH%2014v10%207.2p2/openssh-7_2_P2-hpn-14.10.diff
612 255 Herve Ballans
613 256 Pablo Alingery
In the dir /home/production/sources into  'root'
614 255 Herve Ballans
615 255 Herve Ballans
<pre>
616 258 Herve Ballans
# tar -xzvf openssh-7.2p2.tar.gz
617 258 Herve Ballans
# cd openssh-7.2p2/
618 258 Herve Ballans
# cat ../openssh-7_2_P2-hpn-14.10.diff |patch -p1
619 255 Herve Ballans
# ./configure --prefix=/usr/local/hpn-ssh --with-pam --with-md5-passwords --without-zlib-version-check --with-tcp-wrappers
620 255 Herve Ballans
# make
621 255 Herve Ballans
# make install
622 255 Herve Ballans
</pre>
623 255 Herve Ballans
624 255 Herve Ballans
Depuis sdo :
625 255 Herve Ballans
<pre>
626 255 Herve Ballans
# cd /usr/local/hpn-ssh/etc/
627 259 Herve Ballans
# rsync -av *key* production@sdo-new:/home/production/
628 255 Herve Ballans
</pre>
629 255 Herve Ballans
630 273 Herve Ballans
Following the link : http://vso.stanford.edu/netdrms/rmtsums.html
631 1 Herve Ballans
<pre>
632 274 Herve Ballans
$ cd /home/production/
633 273 Herve Ballans
$ ssh-agent -c > ~/.ssh-agent_rs
634 285 Pablo Alingery
</pre>
635 286 Pablo Alingery
NB : correct file   ~/.ssh-agent_rs it should look like :
636 285 Pablo Alingery
<pre>
637 285 Pablo Alingery
#!/bin/csh
638 285 Pablo Alingery
export SSH_AUTH_SOCK=/tmp/ssh-9POrTXobhLR4/agent.74272;
639 285 Pablo Alingery
export SSH_AGENT_PID=74273;
640 285 Pablo Alingery
echo Agent pid 74273;
641 285 Pablo Alingery
</pre>
642 285 Pablo Alingery
643 285 Pablo Alingery
<pre>
644 273 Herve Ballans
$ source ~/.ssh-agent_rs
645 273 Herve Ballans
$ ssh-add ~/.ssh/id_rsa
646 269 Herve Ballans
</pre>
647 269 Herve Ballans
648 259 Herve Ballans
Sur sdo-new :
649 255 Herve Ballans
<pre>
650 260 Herve Ballans
# mv /home/production/*key* /usr/local/hpn-ssh/etc/
651 255 Herve Ballans
</pre>
652 255 Herve Ballans
653 255 Herve Ballans
We have to configure the port number to 55000
654 255 Herve Ballans
<pre>
655 255 Herve Ballans
# vi ssh_config
656 255 Herve Ballans
</pre>
657 255 Herve Ballans
658 255 Herve Ballans
and add :
659 255 Herve Ballans
<pre>
660 255 Herve Ballans
#Port 22
661 255 Herve Ballans
Port 55000
662 255 Herve Ballans
</pre>
663 255 Herve Ballans
664 255 Herve Ballans
We configure also the server hpn (even we don't use it yet)
665 255 Herve Ballans
<pre>
666 255 Herve Ballans
# vi sshd_config
667 255 Herve Ballans
</pre>
668 255 Herve Ballans
669 255 Herve Ballans
and add :
670 255 Herve Ballans
<pre>
671 255 Herve Ballans
#Port 22
672 255 Herve Ballans
Port 55000
673 255 Herve Ballans
674 255 Herve Ballans
#PidFile /var/run/sshd.pid
675 255 Herve Ballans
PidFile /var/run/sshd.55000.pid
676 255 Herve Ballans
677 255 Herve Ballans
# allow the use of the none cipher
678 255 Herve Ballans
#NoneEnabled no
679 255 Herve Ballans
NoneEnabled yes
680 255 Herve Ballans
</pre>
681 255 Herve Ballans
682 265 Herve Ballans
In order to force the use of SSH-hpn by JMD :
683 265 Herve Ballans
684 265 Herve Ballans
<pre>
685 265 Herve Ballans
$ vi /usr/local/jmd/cfg/JMD.cfg
686 265 Herve Ballans
</pre>
687 265 Herve Ballans
688 265 Herve Ballans
and replace the path of scp binary :
689 265 Herve Ballans
690 265 Herve Ballans
<pre>
691 265 Herve Ballans
SCPCommand=/usr/local/hpn-ssh/bin/scp -o NoneSwitch=yes -o NoneEnabled=yes
692 265 Herve Ballans
</pre>
693 265 Herve Ballans
694 262 Herve Ballans
h2. Private key Public key generation 
695 262 Herve Ballans
696 262 Herve Ballans
JSOC need your public key to directly connect to your netdrms server. In order to generate your RSA public/private key pair :
697 262 Herve Ballans
698 262 Herve Ballans
<pre>
699 262 Herve Ballans
$ ssh-keygen -t rsa
700 262 Herve Ballans
</pre>
701 262 Herve Ballans
702 263 Pablo Alingery
Not necessary in our case cause we recover the ssh keys of the current server 'sdo'.
703 262 Herve Ballans
704 262 Herve Ballans
However, we have to retrieve the private key of production account on sdo-new
705 262 Herve Ballans
706 262 Herve Ballans
From sdo :
707 262 Herve Ballans
<pre>
708 262 Herve Ballans
rsync -av id_rsa production@sdo-new:/home/production/.ssh/
709 262 Herve Ballans
</pre>
710 262 Herve Ballans
711 112 Herve Ballans
h2. NetDRMS replication
712 112 Herve Ballans
713 1 Herve Ballans
<pre>
714 120 Pablo Alingery
# cd base/drms/replication/etc
715 1 Herve Ballans
# scp production@sdo:/usr/local/netdrms/install-config-files/ias.subscribe_list.cfg  .
716 112 Herve Ballans
# scp production@sdo:/usr/local/netdrms/install-config-files/ias.repclient.cfg  .
717 120 Pablo Alingery
WRONG !!!!!
718 112 Herve Ballans
To be done cp repclient.template.cfg into ias.subscribe_list.cfg because new field in config file can't appear 
719 112 Herve Ballans
Correct that step in NetDrms doc 
720 112 Herve Ballans
The existing ias.repclient.cfg can be used tough to fill params 
721 112 Herve Ballans
722 112 Herve Ballans
723 112 Herve Ballans
</pre>
724 112 Herve Ballans
725 112 Herve Ballans
726 112 Herve Ballans
Create tmp dir in netdrms working directory (/usr/local/netdrms_current)
727 112 Herve Ballans
<pre>
728 112 Herve Ballans
# mkdir tmp
729 112 Herve Ballans
</pre>
730 112 Herve Ballans
731 112 Herve Ballans
Prevent from deleting files
732 112 Herve Ballans
<pre>
733 112 Herve Ballans
cd logs/SUM ; scp production@sdo:/usr/local/netdrms/install-config-files/sum_rm.cfg .
734 91 Herve Ballans
</pre>
735 91 Herve Ballans
736 91 Herve Ballans
h2. NetDRMS Start script
737 91 Herve Ballans
738 91 Herve Ballans
<pre>
739 91 Herve Ballans
sum_start.NetDRMS
740 91 Herve Ballans
</pre>
741 91 Herve Ballans
742 91 Herve Ballans
h2. Database repair (from Art)
743 91 Herve Ballans
744 91 Herve Ballans
On ias_sdo
745 91 Herve Ballans
746 91 Herve Ballans
<pre>
747 91 Herve Ballans
delete from admin.ns where name = 'lm_jps';
748 91 Herve Ballans
delete from admin.ns where name = 'aia_test';
749 91 Herve Ballans
</pre>
750 93 Pablo Alingery
751 94 Herve Ballans
h2. Node definition
752 94 Herve Ballans
753 94 Herve Ballans
file : /usr/local/netdrms_current/base/drms/replication/etc/ias.repclient.cfg
754 94 Herve Ballans
755 94 Herve Ballans
<pre>
756 1 Herve Ballans
node=IAStest
757 96 Herve Ballans
</pre>
758 96 Herve Ballans
759 105 Pablo Alingery
h2. Correction
760 105 Pablo Alingery
761 105 Pablo Alingery
h3. Modification du fichier ias.repclient.cfg
762 105 Pablo Alingery
763 105 Pablo Alingery
Copier le template existant repclient.template.cfg dans le répertoire [netdrms_current]/base/drms/replication/etc
764 105 Pablo Alingery
 
765 105 Pablo Alingery
<pre>
766 105 Pablo Alingery
production@sdo3:/usr/local/netdrms_current/base/drms/replication/etc$ cp repclient.template.cfg ias.repclient.cfg
767 105 Pablo Alingery
</pre>
768 1 Herve Ballans
769 106 Pablo Alingery
Nouveau / netdrms2.4 
770 105 Pablo Alingery
<pre>
771 105 Pablo Alingery
# Apps
772 106 Pablo Alingery
kRSPerl=<path to Perl binary to use when Perl scripts are executed>
773 105 Pablo Alingery
</pre>
774 107 Pablo Alingery
775 107 Pablo Alingery
h2. Subscription test 
776 107 Pablo Alingery
777 297 Pablo Alingery
h3. Test show_info
778 110 Herve Ballans
779 110 Herve Ballans
<pre>
780 110 Herve Ballans
production@sdo3:~$ show_info -s hmi.sharp_720s_nrt
781 110 Herve Ballans
First Record: hmi.sharp_720s_nrt[476][2012.09.14_02:12:00_TAI], Recnum = 829707
782 110 Herve Ballans
Last Record:  hmi.sharp_720s_nrt[4943][2016.03.22_18:24:00_TAI], Recnum = 2015202
783 110 Herve Ballans
Last Recnum:  2015202
784 110 Herve Ballans
Has shadow table: no
785 107 Pablo Alingery
</pre>
786 111 Herve Ballans
787 111 Herve Ballans
h3. Lancement de jetty
788 111 Herve Ballans
789 111 Herve Ballans
<pre>
790 111 Herve Ballans
$ jetty.sh start
791 111 Herve Ballans
</pre>
792 111 Herve Ballans
793 292 Pablo Alingery
h2. TO BE DONE BEFORE MIGRATION
794 108 Pablo Alingery
795 107 Pablo Alingery
796 152 Pablo Alingery
%{color:red} configure sytem nbr de fichier ouvert en meme temps%
797 1 Herve Ballans
798 152 Pablo Alingery
%{color:red} configure sytem taille  buffer%
799 152 Pablo Alingery
800 155 Pablo Alingery
%{color:red} swap désactivé ? discussion Stephane et Gilles%
801 1 Herve Ballans
802 295 Herve Ballans
%{color:red}Test ssh connection to JSOC , NSO CFA before subcription and just after name sdo-new into sdo%
803 80 Pablo Alingery
804 1 Herve Ballans
h3. Modification du fichir config.local pour la souscription au jsoc
805 1 Herve Ballans
806 1 Herve Ballans
807 1 Herve Ballans
<pre>
808 1 Herve Ballans
# NetDRMS users can request subdirectories from the Stanford proj directories 
809 1 Herve Ballans
# (e.g., JSOC/proj/util). These users will need to contact Stanford and obtain
810 1 Herve Ballans
# the source subdirectories desired. They will then place these subdirectories
811 1 Herve Ballans
# in the JSOC/proj directory of their NetDRMS release. To properly build 
812 1 Herve Ballans
# targets in these subdirectories, using the JSOC make system, the user needs
813 1 Herve Ballans
# to add entries to this configuration file - one entry for each subdirectory
814 1 Herve Ballans
# that contains source code to be compiled. Each entry is a space-separated pair
815 1 Herve Ballans
# of strings: the string "PROJDIR" followed by a subdirectory (of the proj 
816 1 Herve Ballans
# directory).  For example:
817 1 Herve Ballans
#
818 1 Herve Ballans
# PROJDIR     util/apps
819 1 Herve Ballans
# PROJDIR     util/libs
820 1 Herve Ballans
# 
821 1 Herve Ballans
# NOTE: It is not sufficient to list only a parent directory, like util. Source code
822 18 Herve Ballans
# in child directories will not automatically be compiled.
823 18 Herve Ballans
</pre>