Pyros applications

Version 43 (Paul Carensac, 07/08/2016 11:39 am)

1 1 Paul Carensac
h1. Pyros applications
2 1 Paul Carensac
3 2 Paul Carensac
List and details of all the pyros applications.
4 2 Paul Carensac
5 17 Paul Carensac
{{>toc}}
6 17 Paul Carensac
7 2 Paul Carensac
---
8 2 Paul Carensac
9 2 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}pyrosapp%
10 2 Paul Carensac
11 3 Paul Carensac
h3. Purpose
12 2 Paul Carensac
13 2 Paul Carensac
 * Contains all the database Models
14 2 Paul Carensac
 * Basic tests in tests.py
15 2 Paul Carensac
 * Backoffice configuration in admin.py
16 2 Paul Carensac
17 3 Paul Carensac
h3. Notes
18 2 Paul Carensac
19 2 Paul Carensac
20 2 Paul Carensac
---
21 2 Paul Carensac
22 2 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}dashboard%
23 2 Paul Carensac
24 3 Paul Carensac
h3. Purpose
25 2 Paul Carensac
26 2 Paul Carensac
 * Interface for all external users
27 2 Paul Carensac
 * Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)
28 2 Paul Carensac
29 4 Paul Carensac
h3. Evolution
30 4 Paul Carensac
31 4 Paul Carensac
 * Creating application
32 4 Paul Carensac
33 4 Paul Carensac
    * manage.py startapp dashboard
34 5 Paul Carensac
    * added 'dashboard' in settings.py -> INSTALLED_APPS
35 5 Paul Carensac
    * created a urls.py file in dashboard module
36 5 Paul Carensac
    * added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
37 5 Paul Carensac
    * created templates/ and templates/dashboard/ folders in dashboard module
38 5 Paul Carensac
39 5 Paul Carensac
 * Main page
40 5 Paul Carensac
41 5 Paul Carensac
    * added 'home' view in views.py
42 5 Paul Carensac
    * linked 'home' view to 8000/dashboard URL
43 5 Paul Carensac
    * created a template for the homepage in templates/dashboard/ (with bootstrap)
44 5 Paul Carensac
    * created views, views linking (urls.py) and templates for the dashboard modules
45 5 Paul Carensac
    * redirected mainpage buttons to Admin interface (temporary)
46 4 Paul Carensac
47 22 Paul Carensac
 * System page (logs)
48 22 Paul Carensac
49 22 Paul Carensac
    * Retrieve logs from the db
50 26 Paul Carensac
    * Print logs and automatically update every second via an ajax request
51 22 Paul Carensac
52 3 Paul Carensac
h3. Notes
53 2 Paul Carensac
54 2 Paul Carensac
 * The buttons lead to the Admin interface for the moment
55 7 Paul Carensac
 * Added bootstrap3 module (see Installation)
56 7 Paul Carensac
 * Added Django Template Editor (see Eclipse configuration)
57 2 Paul Carensac
58 3 Paul Carensac
h3. TODO
59 2 Paul Carensac
60 2 Paul Carensac
 * Create the backoffice views as the modules are integrated in pyros
61 2 Paul Carensac
 * Think about a system of permissions
62 2 Paul Carensac
63 2 Paul Carensac
---
64 8 Paul Carensac
65 8 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}scheduler%
66 8 Paul Carensac
67 8 Paul Carensac
h3. Purpose
68 8 Paul Carensac
69 8 Paul Carensac
 * Creates the planning with the OBSERVABLE sequences
70 40 Paul Carensac
 * Give access to a web page to see the current & old plannings
71 8 Paul Carensac
72 8 Paul Carensac
h3. Evolution
73 8 Paul Carensac
74 8 Paul Carensac
 * Creating application
75 8 Paul Carensac
76 8 Paul Carensac
    * manage.py startapp scheduler
77 8 Paul Carensac
    * added 'scheduler' in settings.py -> INSTALLED_APPS
78 8 Paul Carensac
    * created a urls.py file in scheduler module
79 8 Paul Carensac
    * added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
80 8 Paul Carensac
    * created templates/ and templates/scheduler/ folders in scheduler module
81 8 Paul Carensac
82 9 Paul Carensac
 * Model modifications
83 1 Paul Carensac
84 9 Paul Carensac
    * Schedule
85 9 Paul Carensac
86 9 Paul Carensac
        * Remove day_start
87 9 Paul Carensac
        * Remove day_stop
88 9 Paul Carensac
        * Add plan_start
89 13 Paul Carensac
        * Add plan_end
90 9 Paul Carensac
        * Enum system for the status 
91 9 Paul Carensac
    * ScheduleHistory
92 9 Paul Carensac
93 9 Paul Carensac
        * Remove day_start
94 9 Paul Carensac
        * Remove day_stop
95 9 Paul Carensac
        * Add plan_start
96 13 Paul Carensac
        * Add plan_end
97 9 Paul Carensac
    * Sequence
98 9 Paul Carensac
99 9 Paul Carensac
        * Remove exec_start
100 9 Paul Carensac
        * Remove exec_stop
101 9 Paul Carensac
        * Add tsp
102 9 Paul Carensac
        * Add tep
103 9 Paul Carensac
        * Add jd1
104 9 Paul Carensac
        * Add jd2
105 9 Paul Carensac
        * Add deltaTL
106 9 Paul Carensac
        * Add deltaTR
107 9 Paul Carensac
        * Add t_prefered
108 9 Paul Carensac
        * Changed duration from Float to DecimalField (more precise)
109 15 Paul Carensac
        * Add overhead
110 15 Paul Carensac
111 10 Paul Carensac
    * manage.py makemigrations sheduler ; manage.py migrate
112 10 Paul Carensac
113 10 Paul Carensac
 * Creation of Scheduler and Interval classes in models.py
114 10 Paul Carensac
115 10 Paul Carensac
 * Implementation of the Interval class
116 10 Paul Carensac
117 12 Paul Carensac
 * Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
118 9 Paul Carensac
119 14 Paul Carensac
 * Creation of the web interface
120 14 Paul Carensac
121 14 Paul Carensac
    * Added current_schedule.html in template/scheduler folder
122 14 Paul Carensac
    * Created view and url linking to this template (with current planning retrieving)
123 14 Paul Carensac
124 14 Paul Carensac
 * Creation of the simulator
125 14 Paul Carensac
126 14 Paul Carensac
    * Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
127 14 Paul Carensac
    * Created a simulator module in the scheduler
128 14 Paul Carensac
        
129 14 Paul Carensac
        * Added the MyHTMLParser class (easy implementation of HTMLParser)
130 23 Paul Carensac
        * Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
131 14 Paul Carensac
    * Created a second view linked to schedule/simulation to show simulation results
132 1 Paul Carensac
 
133 29 Paul Carensac
 * Supression du système d'agent et ajout de la tâche (celery) de scheduling
134 29 Paul Carensac
135 23 Paul Carensac
 * Main update 19/05/2016 : overheads & historic system
136 23 Paul Carensac
137 23 Paul Carensac
    * Deleted ScheduleHistory
138 23 Paul Carensac
    * Transformed the Sequence - Schedule relation to ManyToMany
139 23 Paul Carensac
    * Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
140 23 Paul Carensac
    * Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))
141 23 Paul Carensac
142 23 Paul Carensac
h3. Tests
143 23 Paul Carensac
144 30 Paul Carensac
 * Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
145 24 Paul Carensac
 * A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html
146 24 Paul Carensac
147 8 Paul Carensac
h3. Notes
148 1 Paul Carensac
149 1 Paul Carensac
 * Priorities and quotas are default-calculated (for the moment)
150 9 Paul Carensac
 * What is the 'flag' attribute in the Schedule model ?
151 1 Paul Carensac
152 9 Paul Carensac
h3. TODO
153 13 Paul Carensac
154 38 Paul Carensac
 * views
155 39 Paul Carensac
156 38 Paul Carensac
  * Link the main page to the current schedule instead of the simulation page
157 38 Paul Carensac
  * Show user sequences in the schedules (with links)
158 38 Paul Carensac
  * Give accces to old schedules (day / days before / all end-night plannings / all plannings)
159 38 Paul Carensac
  * Give access to the refused sequences of a schedule, and the reasons of rejects
160 39 Paul Carensac
161 38 Paul Carensac
 * scheduler
162 39 Paul Carensac
163 38 Paul Carensac
  * Change the system to determine night start/end (they must be given in parameter, only if first_schedule is True)
164 38 Paul Carensac
  * Store the reasons of rejects (create a new attribute)
165 38 Paul Carensac
  * What is the 'flag' attribute ? (@AK)
166 38 Paul Carensac
  * Do not create the execute_sequence tasks if it's not the night (- 120 seconds)
167 38 Paul Carensac
  * Priority and quota computing
168 38 Paul Carensac
  * Quotas evolution
169 38 Paul Carensac
  * Blank space filling
170 38 Paul Carensac
  * At the end of a scheduling send, it to the IC ?
171 18 Paul Carensac
172 18 Paul Carensac
---
173 18 Paul Carensac
174 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}alert_manager%
175 18 Paul Carensac
176 18 Paul Carensac
h3. Purpose
177 18 Paul Carensac
178 18 Paul Carensac
 * Listen to VOEvent network
179 18 Paul Carensac
 * Sort interesting events
180 18 Paul Carensac
 * Create requests according to these events
181 18 Paul Carensac
 * Manage the requests created via the alerts
182 35 Paul Carensac
 * Change the observation strategies for alerts
183 18 Paul Carensac
184 18 Paul Carensac
h3. Evolution
185 18 Paul Carensac
186 18 Paul Carensac
 * Creating application
187 18 Paul Carensac
188 18 Paul Carensac
    * See scheduler documentation above
189 18 Paul Carensac
190 18 Paul Carensac
 * Implementation of the agent system
191 18 Paul Carensac
192 18 Paul Carensac
    * Created agent.py whith a "class AlertManagerAgent(Agent):" inside
193 18 Paul Carensac
    * Overriding agent's 'work' and 'shutdown' method
194 18 Paul Carensac
    * Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
195 18 Paul Carensac
    * Added the Agent's launch system in apps.py
196 18 Paul Carensac
 
197 18 Paul Carensac
 * Implementation of the VOEventListener Thread
198 18 Paul Carensac
199 18 Paul Carensac
    * Created a "class VOEventListener(Thread):" in agent.py
200 18 Paul Carensac
    * Overriding the run method
201 18 Paul Carensac
202 28 Paul Carensac
 * 19/05/2016 : deleted the agent system (deleted from the applications below too)
203 1 Paul Carensac
204 27 Paul Carensac
 * Added some VOEvents in events_to_send
205 27 Paul Carensac
206 27 Paul Carensac
 * Installed Comet (see in [[Technical components]] section)
207 1 Paul Carensac
208 28 Paul Carensac
 * Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
209 28 Paul Carensac
 
210 28 Paul Carensac
 * Added alert_listener launch at server start
211 28 Paul Carensac
212 28 Paul Carensac
 * Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
213 28 Paul Carensac
214 28 Paul Carensac
 * Added a simulator system ( see section Tests below )
215 28 Paul Carensac
216 35 Paul Carensac
 * Added the strategy_change page
217 35 Paul Carensac
218 28 Paul Carensac
h3. Tests
219 28 Paul Carensac
220 28 Paul Carensac
 * Simulator : 
221 28 Paul Carensac
222 28 Paul Carensac
  * You first need to set settings.SIMULATION to True (in pyros/settings.py)
223 28 Paul Carensac
  * Start celery workers (scripts/start_celery_workers.sh)
224 28 Paul Carensac
  * Start server (manage.py runserver)
225 28 Paul Carensac
  * Go to localhost:8000/alert_manager/simulation, and press the button !
226 28 Paul Carensac
  * This will read into the simulation_sequences file.
227 28 Paul Carensac
228 41 Paul Carensac
 * Test AlertListenerTestsCelery.test_alert_reception :
229 28 Paul Carensac
230 28 Paul Carensac
  * Tests if the alert_listener is working well.
231 28 Paul Carensac
  * +Be careful :+ this will act on the production database, and will start the entire alert workflow
232 1 Paul Carensac
  * To start it : script/celery_test.sh alert_manager
233 41 Paul Carensac
234 41 Paul Carensac
 * Tests TestStrategyChange.* :
235 41 Paul Carensac
  * Test the changing of the observation strategy with errors and different parameters
236 28 Paul Carensac
237 27 Paul Carensac
h3. Notes
238 1 Paul Carensac
239 28 Paul Carensac
 * The request created by alert_listener is hard-coded
240 28 Paul Carensac
241 1 Paul Carensac
h3. TODO
242 28 Paul Carensac
243 42 Paul Carensac
 * Web :
244 42 Paul Carensac
245 42 Paul Carensac
  * Print if there is an alert in progress in the main page
246 42 Paul Carensac
  * Link the alerts to their status and results
247 42 Paul Carensac
248 42 Paul Carensac
 * Connect to a real VOEvent broker
249 42 Paul Carensac
250 42 Paul Carensac
 * Determine the communication with FSC for strategy change
251 42 Paul Carensac
252 42 Paul Carensac
 * VOEvents :
253 42 Paul Carensac
254 42 Paul Carensac
  * Extract the good fields (see AK Q&A 07/01/2016)
255 42 Paul Carensac
  * Fill the request & alerts objects
256 42 Paul Carensac
  * Use strategies to build a request
257 42 Paul Carensac
  * Handle VOEvent updates
258 42 Paul Carensac
  * Be careful to not create 2 alerts for a same GRB, seen by 2 different satellites 
259 18 Paul Carensac
260 18 Paul Carensac
---
261 18 Paul Carensac
262 20 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}analyzer%
263 18 Paul Carensac
264 18 Paul Carensac
h3. Purpose
265 18 Paul Carensac
266 36 Paul Carensac
 * Apply calibration files to the images taken by the observation manager
267 36 Paul Carensac
 * Analyze the images produced by the calibrations
268 18 Paul Carensac
269 18 Paul Carensac
h3. Evolution
270 18 Paul Carensac
271 18 Paul Carensac
 * Creating application
272 18 Paul Carensac
273 18 Paul Carensac
    * See scheduler documentation above
274 18 Paul Carensac
275 43 Paul Carensac
 * Implemented the skeleton of the module in the task Analysis (call  a function for calibrations then a function for analyses)
276 1 Paul Carensac
277 1 Paul Carensac
h3. Notes
278 1 Paul Carensac
279 36 Paul Carensac
h3. TODO
280 43 Paul Carensac
281 43 Paul Carensac
 * Apply the calibrations in the right function
282 43 Paul Carensac
 * Apply the analyses only if it's a GRB
283 43 Paul Carensac
 * Implement the analyses
284 43 Paul Carensac
 * Send analyses to FSC
285 18 Paul Carensac
286 18 Paul Carensac
---
287 18 Paul Carensac
288 19 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}majordome%
289 18 Paul Carensac
290 18 Paul Carensac
h3. Purpose
291 18 Paul Carensac
292 18 Paul Carensac
h3. Evolution
293 18 Paul Carensac
294 18 Paul Carensac
 * Creating application
295 18 Paul Carensac
296 18 Paul Carensac
    * See scheduler documentation above
297 18 Paul Carensac
298 21 Paul Carensac
 * Implementation of the agent system
299 1 Paul Carensac
300 21 Paul Carensac
    * See alert_manager above
301 21 Paul Carensac
302 21 Paul Carensac
 * Implementation of the MajordomeExecutor Thread
303 21 Paul Carensac
304 21 Paul Carensac
    * Created a "class MajordomeExecutor(Thread):" in agent.py
305 21 Paul Carensac
    * Overriding the run method
306 21 Paul Carensac
307 21 Paul Carensac
308 18 Paul Carensac
h3. Notes
309 18 Paul Carensac
310 18 Paul Carensac
h3. TODO
311 18 Paul Carensac
312 18 Paul Carensac
---
313 18 Paul Carensac
314 19 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}monitoring%
315 18 Paul Carensac
316 18 Paul Carensac
h3. Purpose
317 18 Paul Carensac
318 18 Paul Carensac
h3. Evolution
319 18 Paul Carensac
320 18 Paul Carensac
 * Creating application
321 1 Paul Carensac
322 18 Paul Carensac
    * See scheduler documentation above
323 1 Paul Carensac
324 21 Paul Carensac
 * Implementation of the agent system
325 1 Paul Carensac
326 21 Paul Carensac
    * See alert_manager above
327 21 Paul Carensac
    * 'shutdown' method is not implemented here because the Agent's default one is enough 
328 21 Paul Carensac
329 18 Paul Carensac
 
330 18 Paul Carensac
h3. Notes
331 18 Paul Carensac
332 18 Paul Carensac
h3. TODO
333 18 Paul Carensac
334 18 Paul Carensac
---
335 19 Paul Carensac
336 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}observation_manager%
337 18 Paul Carensac
338 18 Paul Carensac
h3. Purpose
339 18 Paul Carensac
340 18 Paul Carensac
h3. Evolution
341 18 Paul Carensac
342 1 Paul Carensac
 * Creating application
343 18 Paul Carensac
344 1 Paul Carensac
    * See scheduler documentation above
345 1 Paul Carensac
346 21 Paul Carensac
 * Implementation of the agent system
347 1 Paul Carensac
348 21 Paul Carensac
    * See alert_manager above
349 21 Paul Carensac
350 21 Paul Carensac
 * Implementation of the ObservationExecutor Thread
351 21 Paul Carensac
352 21 Paul Carensac
    * Created a "class ObservationExecutor(Thread):" in agent.py
353 21 Paul Carensac
    * Overriding the run method
354 21 Paul Carensac
355 18 Paul Carensac
 
356 18 Paul Carensac
h3. Notes
357 18 Paul Carensac
358 18 Paul Carensac
h3. TODO
359 18 Paul Carensac
360 19 Paul Carensac
---
361 18 Paul Carensac
362 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}routine_manager%
363 18 Paul Carensac
364 18 Paul Carensac
h3. Purpose
365 18 Paul Carensac
366 18 Paul Carensac
h3. Evolution
367 18 Paul Carensac
368 18 Paul Carensac
 * Creating application
369 18 Paul Carensac
370 18 Paul Carensac
    * See scheduler documentation above
371 18 Paul Carensac
372 18 Paul Carensac
 
373 18 Paul Carensac
h3. Notes
374 18 Paul Carensac
375 18 Paul Carensac
h3. TODO
376 18 Paul Carensac
377 19 Paul Carensac
---
378 18 Paul Carensac
379 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}user_manager%
380 18 Paul Carensac
381 18 Paul Carensac
h3. Purpose
382 18 Paul Carensac
383 31 Paul Carensac
 * Handle user create / delete / login / logout
384 31 Paul Carensac
 * Allow to see user profile and make password recovery
385 31 Paul Carensac
386 18 Paul Carensac
h3. Evolution
387 18 Paul Carensac
388 18 Paul Carensac
 * Creating application
389 18 Paul Carensac
390 18 Paul Carensac
    * See scheduler documentation above
391 18 Paul Carensac
392 32 Paul Carensac
 * Taking and adapting templates from Alexandru project
393 32 Paul Carensac
394 32 Paul Carensac
 * User creation
395 32 Paul Carensac
396 32 Paul Carensac
  * Form in admin.py
397 33 Paul Carensac
  * Functions to verify if email doesn't exist and if passwords match
398 33 Paul Carensac
  * Added check to pass login page if user is already autheticated
399 33 Paul Carensac
  * Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required
400 18 Paul Carensac
 
401 18 Paul Carensac
h3. Notes
402 18 Paul Carensac
403 18 Paul Carensac
h3. TODO
404 31 Paul Carensac
405 31 Paul Carensac
 * Password recovery
406 31 Paul Carensac
 * Profile page
407 31 Paul Carensac
 * User validation by administrator / commission
408 18 Paul Carensac
409 18 Paul Carensac
---
410 18 Paul Carensac
411 20 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}common%
412 18 Paul Carensac
413 18 Paul Carensac
h3. Purpose
414 18 Paul Carensac
415 18 Paul Carensac
 * Regroups common data and functions shared by applications
416 18 Paul Carensac
417 18 Paul Carensac
h3. Content
418 18 Paul Carensac
419 25 Paul Carensac
 * RequestBuilder - class to build and save in DB the requests
420 18 Paul Carensac
421 25 Paul Carensac
  * You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
422 25 Paul Carensac
  * Each function returns the ID of the element it created (to give it to the children classes)
423 25 Paul Carensac
  * To save a request, you need to use validate_request
424 18 Paul Carensac
 
425 18 Paul Carensac
h3. Notes
426 25 Paul Carensac
427 25 Paul Carensac
 * Will probably contain the DB models at last ...
428 18 Paul Carensac
429 18 Paul Carensac
h3. TODO