Pyros applications

Version 41 (Paul Carensac, 07/08/2016 11:30 am)

1 1 Paul Carensac
h1. Pyros applications
2 1 Paul Carensac
3 2 Paul Carensac
List and details of all the pyros applications.
4 2 Paul Carensac
5 17 Paul Carensac
{{>toc}}
6 17 Paul Carensac
7 2 Paul Carensac
---
8 2 Paul Carensac
9 2 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}pyrosapp%
10 2 Paul Carensac
11 3 Paul Carensac
h3. Purpose
12 2 Paul Carensac
13 2 Paul Carensac
 * Contains all the database Models
14 2 Paul Carensac
 * Basic tests in tests.py
15 2 Paul Carensac
 * Backoffice configuration in admin.py
16 2 Paul Carensac
17 3 Paul Carensac
h3. Notes
18 2 Paul Carensac
19 2 Paul Carensac
20 2 Paul Carensac
---
21 2 Paul Carensac
22 2 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}dashboard%
23 2 Paul Carensac
24 3 Paul Carensac
h3. Purpose
25 2 Paul Carensac
26 2 Paul Carensac
 * Interface for all external users
27 2 Paul Carensac
 * Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)
28 2 Paul Carensac
29 4 Paul Carensac
h3. Evolution
30 4 Paul Carensac
31 4 Paul Carensac
 * Creating application
32 4 Paul Carensac
33 4 Paul Carensac
    * manage.py startapp dashboard
34 5 Paul Carensac
    * added 'dashboard' in settings.py -> INSTALLED_APPS
35 5 Paul Carensac
    * created a urls.py file in dashboard module
36 5 Paul Carensac
    * added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
37 5 Paul Carensac
    * created templates/ and templates/dashboard/ folders in dashboard module
38 5 Paul Carensac
39 5 Paul Carensac
 * Main page
40 5 Paul Carensac
41 5 Paul Carensac
    * added 'home' view in views.py
42 5 Paul Carensac
    * linked 'home' view to 8000/dashboard URL
43 5 Paul Carensac
    * created a template for the homepage in templates/dashboard/ (with bootstrap)
44 5 Paul Carensac
    * created views, views linking (urls.py) and templates for the dashboard modules
45 5 Paul Carensac
    * redirected mainpage buttons to Admin interface (temporary)
46 4 Paul Carensac
47 22 Paul Carensac
 * System page (logs)
48 22 Paul Carensac
49 22 Paul Carensac
    * Retrieve logs from the db
50 26 Paul Carensac
    * Print logs and automatically update every second via an ajax request
51 22 Paul Carensac
52 3 Paul Carensac
h3. Notes
53 2 Paul Carensac
54 2 Paul Carensac
 * The buttons lead to the Admin interface for the moment
55 7 Paul Carensac
 * Added bootstrap3 module (see Installation)
56 7 Paul Carensac
 * Added Django Template Editor (see Eclipse configuration)
57 2 Paul Carensac
58 3 Paul Carensac
h3. TODO
59 2 Paul Carensac
60 2 Paul Carensac
 * Create the backoffice views as the modules are integrated in pyros
61 2 Paul Carensac
 * Think about a system of permissions
62 2 Paul Carensac
63 2 Paul Carensac
---
64 8 Paul Carensac
65 8 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}scheduler%
66 8 Paul Carensac
67 8 Paul Carensac
h3. Purpose
68 8 Paul Carensac
69 8 Paul Carensac
 * Creates the planning with the OBSERVABLE sequences
70 40 Paul Carensac
 * Give access to a web page to see the current & old plannings
71 8 Paul Carensac
72 8 Paul Carensac
h3. Evolution
73 8 Paul Carensac
74 8 Paul Carensac
 * Creating application
75 8 Paul Carensac
76 8 Paul Carensac
    * manage.py startapp scheduler
77 8 Paul Carensac
    * added 'scheduler' in settings.py -> INSTALLED_APPS
78 8 Paul Carensac
    * created a urls.py file in scheduler module
79 8 Paul Carensac
    * added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
80 8 Paul Carensac
    * created templates/ and templates/scheduler/ folders in scheduler module
81 8 Paul Carensac
82 9 Paul Carensac
 * Model modifications
83 1 Paul Carensac
84 9 Paul Carensac
    * Schedule
85 9 Paul Carensac
86 9 Paul Carensac
        * Remove day_start
87 9 Paul Carensac
        * Remove day_stop
88 9 Paul Carensac
        * Add plan_start
89 13 Paul Carensac
        * Add plan_end
90 9 Paul Carensac
        * Enum system for the status 
91 9 Paul Carensac
    * ScheduleHistory
92 9 Paul Carensac
93 9 Paul Carensac
        * Remove day_start
94 9 Paul Carensac
        * Remove day_stop
95 9 Paul Carensac
        * Add plan_start
96 13 Paul Carensac
        * Add plan_end
97 9 Paul Carensac
    * Sequence
98 9 Paul Carensac
99 9 Paul Carensac
        * Remove exec_start
100 9 Paul Carensac
        * Remove exec_stop
101 9 Paul Carensac
        * Add tsp
102 9 Paul Carensac
        * Add tep
103 9 Paul Carensac
        * Add jd1
104 9 Paul Carensac
        * Add jd2
105 9 Paul Carensac
        * Add deltaTL
106 9 Paul Carensac
        * Add deltaTR
107 9 Paul Carensac
        * Add t_prefered
108 9 Paul Carensac
        * Changed duration from Float to DecimalField (more precise)
109 15 Paul Carensac
        * Add overhead
110 15 Paul Carensac
111 10 Paul Carensac
    * manage.py makemigrations sheduler ; manage.py migrate
112 10 Paul Carensac
113 10 Paul Carensac
 * Creation of Scheduler and Interval classes in models.py
114 10 Paul Carensac
115 10 Paul Carensac
 * Implementation of the Interval class
116 10 Paul Carensac
117 12 Paul Carensac
 * Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
118 9 Paul Carensac
119 14 Paul Carensac
 * Creation of the web interface
120 14 Paul Carensac
121 14 Paul Carensac
    * Added current_schedule.html in template/scheduler folder
122 14 Paul Carensac
    * Created view and url linking to this template (with current planning retrieving)
123 14 Paul Carensac
124 14 Paul Carensac
 * Creation of the simulator
125 14 Paul Carensac
126 14 Paul Carensac
    * Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
127 14 Paul Carensac
    * Created a simulator module in the scheduler
128 14 Paul Carensac
        
129 14 Paul Carensac
        * Added the MyHTMLParser class (easy implementation of HTMLParser)
130 23 Paul Carensac
        * Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
131 14 Paul Carensac
    * Created a second view linked to schedule/simulation to show simulation results
132 1 Paul Carensac
 
133 29 Paul Carensac
 * Supression du système d'agent et ajout de la tâche (celery) de scheduling
134 29 Paul Carensac
135 23 Paul Carensac
 * Main update 19/05/2016 : overheads & historic system
136 23 Paul Carensac
137 23 Paul Carensac
    * Deleted ScheduleHistory
138 23 Paul Carensac
    * Transformed the Sequence - Schedule relation to ManyToMany
139 23 Paul Carensac
    * Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
140 23 Paul Carensac
    * Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))
141 23 Paul Carensac
142 23 Paul Carensac
h3. Tests
143 23 Paul Carensac
144 30 Paul Carensac
 * Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
145 24 Paul Carensac
 * A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html
146 24 Paul Carensac
147 8 Paul Carensac
h3. Notes
148 1 Paul Carensac
149 1 Paul Carensac
 * Priorities and quotas are default-calculated (for the moment)
150 9 Paul Carensac
 * What is the 'flag' attribute in the Schedule model ?
151 1 Paul Carensac
152 9 Paul Carensac
h3. TODO
153 13 Paul Carensac
154 38 Paul Carensac
 * views
155 39 Paul Carensac
156 38 Paul Carensac
  * Link the main page to the current schedule instead of the simulation page
157 38 Paul Carensac
  * Show user sequences in the schedules (with links)
158 38 Paul Carensac
  * Give accces to old schedules (day / days before / all end-night plannings / all plannings)
159 38 Paul Carensac
  * Give access to the refused sequences of a schedule, and the reasons of rejects
160 39 Paul Carensac
161 38 Paul Carensac
 * scheduler
162 39 Paul Carensac
163 38 Paul Carensac
  * Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)
164 38 Paul Carensac
  * Store the reasons of rejects (create a new attribute)
165 38 Paul Carensac
  * What is the 'flag' attribute ? (@AK)
166 38 Paul Carensac
  * Do not create the execute_sequence tasks if it's not the night (- 120 seconds)
167 38 Paul Carensac
  * Priority and quota computing
168 38 Paul Carensac
  * Quotas evolution
169 38 Paul Carensac
  * Blank space filling
170 38 Paul Carensac
  * At the end of a scheduling send, it to the IC ?
171 18 Paul Carensac
172 18 Paul Carensac
---
173 18 Paul Carensac
174 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}alert_manager%
175 18 Paul Carensac
176 18 Paul Carensac
h3. Purpose
177 18 Paul Carensac
178 18 Paul Carensac
 * Listen to VOEvent network
179 18 Paul Carensac
 * Sort interesting events
180 18 Paul Carensac
 * Create requests according to these events
181 18 Paul Carensac
 * Manage the requests created via the alerts
182 35 Paul Carensac
 * Change the observation strategies for alerts
183 18 Paul Carensac
184 18 Paul Carensac
h3. Evolution
185 18 Paul Carensac
186 18 Paul Carensac
 * Creating application
187 18 Paul Carensac
188 18 Paul Carensac
    * See scheduler documentation above
189 18 Paul Carensac
190 18 Paul Carensac
 * Implementation of the agent system
191 18 Paul Carensac
192 18 Paul Carensac
    * Created agent.py whith a "class AlertManagerAgent(Agent):" inside
193 18 Paul Carensac
    * Overriding agent's 'work' and 'shutdown' method
194 18 Paul Carensac
    * Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
195 18 Paul Carensac
    * Added the Agent's launch system in apps.py
196 18 Paul Carensac
 
197 18 Paul Carensac
 * Implementation of the VOEventListener Thread
198 18 Paul Carensac
199 18 Paul Carensac
    * Created a "class VOEventListener(Thread):" in agent.py
200 18 Paul Carensac
    * Overriding the run method
201 18 Paul Carensac
202 28 Paul Carensac
 * 19/05/2016 : deleted the agent system (deleted from the applications below too)
203 1 Paul Carensac
204 27 Paul Carensac
 * Added some VOEvents in events_to_send
205 27 Paul Carensac
206 27 Paul Carensac
 * Installed Comet (see in [[Technical components]] section)
207 1 Paul Carensac
208 28 Paul Carensac
 * Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
209 28 Paul Carensac
 
210 28 Paul Carensac
 * Added alert_listener launch at server start
211 28 Paul Carensac
212 28 Paul Carensac
 * Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
213 28 Paul Carensac
214 28 Paul Carensac
 * Added a simulator system ( see section Tests below )
215 28 Paul Carensac
216 35 Paul Carensac
 * Added the strategy_change page
217 35 Paul Carensac
218 28 Paul Carensac
h3. Tests
219 28 Paul Carensac
220 28 Paul Carensac
 * Simulator : 
221 28 Paul Carensac
222 28 Paul Carensac
  * You first need to set settings.SIMULATION to True (in pyros/settings.py)
223 28 Paul Carensac
  * Start celery workers (scripts/start_celery_workers.sh)
224 28 Paul Carensac
  * Start server (manage.py runserver)
225 28 Paul Carensac
  * Go to localhost:8000/alert_manager/simulation, and press the button !
226 28 Paul Carensac
  * This will read into the simulation_sequences file.
227 28 Paul Carensac
228 41 Paul Carensac
 * Test AlertListenerTestsCelery.test_alert_reception :
229 28 Paul Carensac
230 28 Paul Carensac
  * Tests if the alert_listener is working well.
231 28 Paul Carensac
  * +Be careful :+ this will act on the production database, and will start the entire alert workflow
232 1 Paul Carensac
  * To start it : script/celery_test.sh alert_manager
233 41 Paul Carensac
234 41 Paul Carensac
 * Tests TestStrategyChange.* :
235 41 Paul Carensac
  * Test the changing of the observation strategy with errors and different parameters
236 28 Paul Carensac
237 27 Paul Carensac
h3. Notes
238 1 Paul Carensac
239 28 Paul Carensac
 * The request created by alert_listener is hard-coded
240 28 Paul Carensac
241 1 Paul Carensac
h3. TODO
242 28 Paul Carensac
243 28 Paul Carensac
 * Analysis of VOEvents to create real requests
244 18 Paul Carensac
245 18 Paul Carensac
---
246 18 Paul Carensac
247 20 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}analyzer%
248 18 Paul Carensac
249 18 Paul Carensac
h3. Purpose
250 18 Paul Carensac
251 36 Paul Carensac
 * Apply calibration files to the images taken by the observation manager
252 36 Paul Carensac
 * Analyze the images produced by the calibrations
253 18 Paul Carensac
254 18 Paul Carensac
h3. Evolution
255 18 Paul Carensac
256 18 Paul Carensac
 * Creating application
257 18 Paul Carensac
258 18 Paul Carensac
    * See scheduler documentation above
259 18 Paul Carensac
260 18 Paul Carensac
 * Implementation of the agent system
261 18 Paul Carensac
262 18 Paul Carensac
    * See alert_manager above
263 1 Paul Carensac
    * 'shutdown' method is not implemented here because the Agent's default one is enough 
264 36 Paul Carensac
265 36 Paul Carensac
 * 
266 18 Paul Carensac
267 18 Paul Carensac
h3. Notes
268 18 Paul Carensac
269 18 Paul Carensac
h3. TODO
270 18 Paul Carensac
271 18 Paul Carensac
---
272 18 Paul Carensac
273 19 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}majordome%
274 18 Paul Carensac
275 18 Paul Carensac
h3. Purpose
276 18 Paul Carensac
277 18 Paul Carensac
h3. Evolution
278 18 Paul Carensac
279 18 Paul Carensac
 * Creating application
280 18 Paul Carensac
281 18 Paul Carensac
    * See scheduler documentation above
282 18 Paul Carensac
283 21 Paul Carensac
 * Implementation of the agent system
284 1 Paul Carensac
285 21 Paul Carensac
    * See alert_manager above
286 21 Paul Carensac
287 21 Paul Carensac
 * Implementation of the MajordomeExecutor Thread
288 21 Paul Carensac
289 21 Paul Carensac
    * Created a "class MajordomeExecutor(Thread):" in agent.py
290 21 Paul Carensac
    * Overriding the run method
291 21 Paul Carensac
292 21 Paul Carensac
293 18 Paul Carensac
h3. Notes
294 18 Paul Carensac
295 18 Paul Carensac
h3. TODO
296 18 Paul Carensac
297 18 Paul Carensac
---
298 18 Paul Carensac
299 19 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}monitoring%
300 18 Paul Carensac
301 18 Paul Carensac
h3. Purpose
302 18 Paul Carensac
303 18 Paul Carensac
h3. Evolution
304 18 Paul Carensac
305 18 Paul Carensac
 * Creating application
306 1 Paul Carensac
307 18 Paul Carensac
    * See scheduler documentation above
308 1 Paul Carensac
309 21 Paul Carensac
 * Implementation of the agent system
310 1 Paul Carensac
311 21 Paul Carensac
    * See alert_manager above
312 21 Paul Carensac
    * 'shutdown' method is not implemented here because the Agent's default one is enough 
313 21 Paul Carensac
314 18 Paul Carensac
 
315 18 Paul Carensac
h3. Notes
316 18 Paul Carensac
317 18 Paul Carensac
h3. TODO
318 18 Paul Carensac
319 18 Paul Carensac
---
320 19 Paul Carensac
321 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}observation_manager%
322 18 Paul Carensac
323 18 Paul Carensac
h3. Purpose
324 18 Paul Carensac
325 18 Paul Carensac
h3. Evolution
326 18 Paul Carensac
327 1 Paul Carensac
 * Creating application
328 18 Paul Carensac
329 1 Paul Carensac
    * See scheduler documentation above
330 1 Paul Carensac
331 21 Paul Carensac
 * Implementation of the agent system
332 1 Paul Carensac
333 21 Paul Carensac
    * See alert_manager above
334 21 Paul Carensac
335 21 Paul Carensac
 * Implementation of the ObservationExecutor Thread
336 21 Paul Carensac
337 21 Paul Carensac
    * Created a "class ObservationExecutor(Thread):" in agent.py
338 21 Paul Carensac
    * Overriding the run method
339 21 Paul Carensac
340 18 Paul Carensac
 
341 18 Paul Carensac
h3. Notes
342 18 Paul Carensac
343 18 Paul Carensac
h3. TODO
344 18 Paul Carensac
345 19 Paul Carensac
---
346 18 Paul Carensac
347 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}routine_manager%
348 18 Paul Carensac
349 18 Paul Carensac
h3. Purpose
350 18 Paul Carensac
351 18 Paul Carensac
h3. Evolution
352 18 Paul Carensac
353 18 Paul Carensac
 * Creating application
354 18 Paul Carensac
355 18 Paul Carensac
    * See scheduler documentation above
356 18 Paul Carensac
357 18 Paul Carensac
 
358 18 Paul Carensac
h3. Notes
359 18 Paul Carensac
360 18 Paul Carensac
h3. TODO
361 18 Paul Carensac
362 19 Paul Carensac
---
363 18 Paul Carensac
364 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}user_manager%
365 18 Paul Carensac
366 18 Paul Carensac
h3. Purpose
367 18 Paul Carensac
368 31 Paul Carensac
 * Handle user create / delete / login / logout
369 31 Paul Carensac
 * Allow to see user profile and make password recovery
370 31 Paul Carensac
371 18 Paul Carensac
h3. Evolution
372 18 Paul Carensac
373 18 Paul Carensac
 * Creating application
374 18 Paul Carensac
375 18 Paul Carensac
    * See scheduler documentation above
376 18 Paul Carensac
377 32 Paul Carensac
 * Taking and adapting templates from Alexandru project
378 32 Paul Carensac
379 32 Paul Carensac
 * User creation
380 32 Paul Carensac
381 32 Paul Carensac
  * Form in admin.py
382 33 Paul Carensac
  * Functions to verify if email doesn't exist and if passwords match
383 33 Paul Carensac
  * Added check to pass login page if user is already autheticated
384 33 Paul Carensac
  * Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required
385 18 Paul Carensac
 
386 18 Paul Carensac
h3. Notes
387 18 Paul Carensac
388 18 Paul Carensac
h3. TODO
389 31 Paul Carensac
390 31 Paul Carensac
 * Password recovery
391 31 Paul Carensac
 * Profile page
392 31 Paul Carensac
 * User validation by administrator / commission
393 18 Paul Carensac
394 18 Paul Carensac
---
395 18 Paul Carensac
396 20 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}common%
397 18 Paul Carensac
398 18 Paul Carensac
h3. Purpose
399 18 Paul Carensac
400 18 Paul Carensac
 * Regroups common data and functions shared by applications
401 18 Paul Carensac
402 18 Paul Carensac
h3. Content
403 18 Paul Carensac
404 25 Paul Carensac
 * RequestBuilder - class to build and save in DB the requests
405 18 Paul Carensac
406 25 Paul Carensac
  * You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
407 25 Paul Carensac
  * Each function returns the ID of the element it created (to give it to the children classes)
408 25 Paul Carensac
  * To save a request, you need to use validate_request
409 18 Paul Carensac
 
410 18 Paul Carensac
h3. Notes
411 25 Paul Carensac
412 25 Paul Carensac
 * Will probably contain the DB models at last ...
413 18 Paul Carensac
414 18 Paul Carensac
h3. TODO