Pyros applications

Version 59 (Paul Carensac, 08/09/2016 04:44 pm)

1 1 Paul Carensac
h1. Pyros applications
2 1 Paul Carensac
3 2 Paul Carensac
List and details of all the pyros applications.
4 2 Paul Carensac
5 17 Paul Carensac
{{>toc}}
6 17 Paul Carensac
7 2 Paul Carensac
---
8 2 Paul Carensac
9 2 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}dashboard%
10 2 Paul Carensac
11 3 Paul Carensac
h3. Purpose
12 2 Paul Carensac
13 2 Paul Carensac
 * Interface for all external users
14 2 Paul Carensac
 * Leads to displays and actions for all the pyros modules (users, requests, system execution, ...)
15 2 Paul Carensac
16 4 Paul Carensac
h3. Evolution
17 4 Paul Carensac
18 4 Paul Carensac
 * Creating application
19 4 Paul Carensac
20 4 Paul Carensac
    * manage.py startapp dashboard
21 5 Paul Carensac
    * added 'dashboard' in settings.py -> INSTALLED_APPS
22 5 Paul Carensac
    * created a urls.py file in dashboard module
23 5 Paul Carensac
    * added _url(r'^dashboard/', include('dashboard.urls'))_ in pyros/urls.py -> urlpatterns
24 5 Paul Carensac
    * created templates/ and templates/dashboard/ folders in dashboard module
25 5 Paul Carensac
26 5 Paul Carensac
 * Main page
27 5 Paul Carensac
28 5 Paul Carensac
    * added 'home' view in views.py
29 5 Paul Carensac
    * linked 'home' view to 8000/dashboard URL
30 5 Paul Carensac
    * created a template for the homepage in templates/dashboard/ (with bootstrap)
31 5 Paul Carensac
    * created views, views linking (urls.py) and templates for the dashboard modules
32 5 Paul Carensac
    * redirected mainpage buttons to Admin interface (temporary)
33 4 Paul Carensac
34 22 Paul Carensac
 * System page (logs)
35 22 Paul Carensac
36 22 Paul Carensac
    * Retrieve logs from the db
37 26 Paul Carensac
    * Print logs and automatically update every second via an ajax request
38 22 Paul Carensac
39 3 Paul Carensac
h3. Notes
40 2 Paul Carensac
41 2 Paul Carensac
 * The buttons lead to the Admin interface for the moment
42 7 Paul Carensac
 * Added bootstrap3 module (see Installation)
43 7 Paul Carensac
 * Added Django Template Editor (see Eclipse configuration)
44 2 Paul Carensac
45 2 Paul Carensac
---
46 8 Paul Carensac
47 8 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}scheduler%
48 8 Paul Carensac
49 8 Paul Carensac
h3. Purpose
50 8 Paul Carensac
51 8 Paul Carensac
 * Creates the planning with the OBSERVABLE sequences
52 40 Paul Carensac
 * Give access to a web page to see the current & old plannings
53 8 Paul Carensac
54 8 Paul Carensac
h3. Evolution
55 8 Paul Carensac
56 8 Paul Carensac
 * Creating application
57 8 Paul Carensac
58 8 Paul Carensac
    * manage.py startapp scheduler
59 8 Paul Carensac
    * added 'scheduler' in settings.py -> INSTALLED_APPS
60 8 Paul Carensac
    * created a urls.py file in scheduler module
61 8 Paul Carensac
    * added _url(r'^scheduler/', include('scheduler.urls'))_ in pyros/urls.py -> urlpatterns
62 8 Paul Carensac
    * created templates/ and templates/scheduler/ folders in scheduler module
63 8 Paul Carensac
64 9 Paul Carensac
 * Model modifications
65 1 Paul Carensac
66 9 Paul Carensac
    * Schedule
67 9 Paul Carensac
68 9 Paul Carensac
        * Remove day_start
69 9 Paul Carensac
        * Remove day_stop
70 9 Paul Carensac
        * Add plan_start
71 13 Paul Carensac
        * Add plan_end
72 9 Paul Carensac
        * Enum system for the status 
73 9 Paul Carensac
    * ScheduleHistory
74 9 Paul Carensac
75 9 Paul Carensac
        * Remove day_start
76 9 Paul Carensac
        * Remove day_stop
77 9 Paul Carensac
        * Add plan_start
78 13 Paul Carensac
        * Add plan_end
79 9 Paul Carensac
    * Sequence
80 9 Paul Carensac
81 9 Paul Carensac
        * Remove exec_start
82 9 Paul Carensac
        * Remove exec_stop
83 9 Paul Carensac
        * Add tsp
84 9 Paul Carensac
        * Add tep
85 9 Paul Carensac
        * Add jd1
86 9 Paul Carensac
        * Add jd2
87 9 Paul Carensac
        * Add deltaTL
88 9 Paul Carensac
        * Add deltaTR
89 9 Paul Carensac
        * Add t_prefered
90 9 Paul Carensac
        * Changed duration from Float to DecimalField (more precise)
91 15 Paul Carensac
        * Add overhead
92 15 Paul Carensac
93 10 Paul Carensac
    * manage.py makemigrations sheduler ; manage.py migrate
94 10 Paul Carensac
95 10 Paul Carensac
 * Creation of Scheduler and Interval classes in models.py
96 10 Paul Carensac
97 10 Paul Carensac
 * Implementation of the Interval class
98 10 Paul Carensac
99 12 Paul Carensac
 * Implementation of the Scheduler's 'make_schedule' function (and children). This is the only entry point for now. This function creates the planning (organizes the observable sequences).
100 9 Paul Carensac
101 14 Paul Carensac
 * Creation of the web interface
102 14 Paul Carensac
103 14 Paul Carensac
    * Added current_schedule.html in template/scheduler folder
104 14 Paul Carensac
    * Created view and url linking to this template (with current planning retrieving)
105 14 Paul Carensac
106 14 Paul Carensac
 * Creation of the simulator
107 14 Paul Carensac
108 14 Paul Carensac
    * Created a second entry point in the Scheduler class (with a few minor adaptations to handle SIMULATION mode)
109 14 Paul Carensac
    * Created a simulator module in the scheduler
110 14 Paul Carensac
        
111 14 Paul Carensac
        * Added the MyHTMLParser class (easy implementation of HTMLParser)
112 23 Paul Carensac
        * Added Simulator class. It parses a file given in parametr to retrieve sequences and create a schedule
113 14 Paul Carensac
    * Created a second view linked to schedule/simulation to show simulation results
114 1 Paul Carensac
 
115 29 Paul Carensac
 * Supression du système d'agent et ajout de la tâche (celery) de scheduling
116 29 Paul Carensac
117 23 Paul Carensac
 * Main update 19/05/2016 : overheads & historic system
118 23 Paul Carensac
119 23 Paul Carensac
    * Deleted ScheduleHistory
120 23 Paul Carensac
    * Transformed the Sequence - Schedule relation to ManyToMany
121 23 Paul Carensac
    * Moved the scheduling information into the M2M relationship class (shs = ScheduleHasSequences)
122 23 Paul Carensac
    * Added a static overhead into scheduling (can be set from the calling code, through scheduler.max_overhead = blabla(float))
123 23 Paul Carensac
124 23 Paul Carensac
h3. Tests
125 23 Paul Carensac
126 30 Paul Carensac
 * Some tests in test.py to test the different functionalities of the scheduler. These tests include the simulation.
127 24 Paul Carensac
 * A simulator at localhost:8000/scheduler/simulation, taking sequences from scheduler/sequences_cador.html
128 24 Paul Carensac
129 8 Paul Carensac
h3. Notes
130 1 Paul Carensac
131 1 Paul Carensac
 * Priorities and quotas are default-calculated (for the moment)
132 9 Paul Carensac
 * What is the 'flag' attribute in the Schedule model ?
133 1 Paul Carensac
134 13 Paul Carensac
135 18 Paul Carensac
---
136 18 Paul Carensac
137 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}alert_manager%
138 18 Paul Carensac
139 18 Paul Carensac
h3. Purpose
140 18 Paul Carensac
141 18 Paul Carensac
 * Listen to VOEvent network
142 18 Paul Carensac
 * Sort interesting events
143 18 Paul Carensac
 * Create requests according to these events
144 18 Paul Carensac
 * Manage the requests created via the alerts
145 35 Paul Carensac
 * Change the observation strategies for alerts
146 18 Paul Carensac
147 18 Paul Carensac
h3. Evolution
148 18 Paul Carensac
149 18 Paul Carensac
 * Creating application
150 18 Paul Carensac
151 18 Paul Carensac
    * See scheduler documentation above
152 18 Paul Carensac
153 18 Paul Carensac
 * Implementation of the agent system
154 18 Paul Carensac
155 18 Paul Carensac
    * Created agent.py whith a "class AlertManagerAgent(Agent):" inside
156 18 Paul Carensac
    * Overriding agent's 'work' and 'shutdown' method
157 18 Paul Carensac
    * Added message(s) to agent.actions_by_message dictionnary, with the functions associated to these messages
158 18 Paul Carensac
    * Added the Agent's launch system in apps.py
159 18 Paul Carensac
 
160 18 Paul Carensac
 * Implementation of the VOEventListener Thread
161 18 Paul Carensac
162 18 Paul Carensac
    * Created a "class VOEventListener(Thread):" in agent.py
163 18 Paul Carensac
    * Overriding the run method
164 18 Paul Carensac
165 28 Paul Carensac
 * 19/05/2016 : deleted the agent system (deleted from the applications below too)
166 1 Paul Carensac
167 27 Paul Carensac
 * Added some VOEvents in events_to_send
168 27 Paul Carensac
169 27 Paul Carensac
 * Installed Comet (see in [[Technical components]] section)
170 1 Paul Carensac
171 28 Paul Carensac
 * Created 2 celery tasks in tesks.py : alert_listener and change_obs_strategy
172 28 Paul Carensac
 
173 28 Paul Carensac
 * Added alert_listener launch at server start
174 28 Paul Carensac
175 28 Paul Carensac
 * Implemented alert_listener : it waits for an event file to be created in events_received, then read it and creates a request
176 28 Paul Carensac
177 28 Paul Carensac
 * Added a simulator system ( see section Tests below )
178 28 Paul Carensac
179 35 Paul Carensac
 * Added the strategy_change page
180 35 Paul Carensac
181 45 Paul Carensac
 * Removed launching at start : it is now started by the monitoring task
182 45 Paul Carensac
183 28 Paul Carensac
h3. Tests
184 28 Paul Carensac
185 28 Paul Carensac
 * Simulator : 
186 28 Paul Carensac
187 28 Paul Carensac
  * You first need to set settings.SIMULATION to True (in pyros/settings.py)
188 28 Paul Carensac
  * Start celery workers (scripts/start_celery_workers.sh)
189 28 Paul Carensac
  * Start server (manage.py runserver)
190 28 Paul Carensac
  * Go to localhost:8000/alert_manager/simulation, and press the button !
191 28 Paul Carensac
  * This will read into the simulation_sequences file.
192 28 Paul Carensac
193 41 Paul Carensac
 * Test AlertListenerTestsCelery.test_alert_reception :
194 28 Paul Carensac
195 28 Paul Carensac
  * Tests if the alert_listener is working well.
196 28 Paul Carensac
  * +Be careful :+ this will act on the production database, and will start the entire alert workflow
197 1 Paul Carensac
  * To start it : script/celery_test.sh alert_manager
198 41 Paul Carensac
199 41 Paul Carensac
 * Tests TestStrategyChange.* :
200 41 Paul Carensac
  * Test the changing of the observation strategy with errors and different parameters
201 28 Paul Carensac
202 27 Paul Carensac
h3. Notes
203 1 Paul Carensac
204 28 Paul Carensac
 * The request created by alert_listener is hard-coded
205 28 Paul Carensac
206 18 Paul Carensac
---
207 18 Paul Carensac
208 20 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}analyzer%
209 18 Paul Carensac
210 18 Paul Carensac
h3. Purpose
211 18 Paul Carensac
212 36 Paul Carensac
 * Apply calibration files to the images taken by the observation manager
213 36 Paul Carensac
 * Analyze the images produced by the calibrations
214 18 Paul Carensac
215 18 Paul Carensac
h3. Evolution
216 18 Paul Carensac
217 18 Paul Carensac
 * Creating application
218 18 Paul Carensac
219 18 Paul Carensac
    * See scheduler documentation above
220 18 Paul Carensac
221 43 Paul Carensac
 * Implemented the skeleton of the module in the task Analysis (call  a function for calibrations then a function for analyses)
222 1 Paul Carensac
223 1 Paul Carensac
h3. Notes
224 1 Paul Carensac
225 18 Paul Carensac
---
226 18 Paul Carensac
227 19 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}majordome%
228 18 Paul Carensac
229 18 Paul Carensac
h3. Purpose
230 18 Paul Carensac
231 44 Paul Carensac
 * Handles the execution of the sequences in the right order during the night
232 44 Paul Carensac
 * Handles the system start & stop
233 44 Paul Carensac
 * Handles the changes of observation conditions
234 46 Paul Carensac
 * Stops the celery tasks when needed
235 44 Paul Carensac
236 18 Paul Carensac
h3. Evolution
237 18 Paul Carensac
238 21 Paul Carensac
 * Creating application
239 1 Paul Carensac
240 46 Paul Carensac
  * See scheduler documentation above
241 21 Paul Carensac
242 46 Paul Carensac
  * Created all the tasks in tasks.py
243 21 Paul Carensac
244 46 Paul Carensac
  * Created the TaskManager to stop the sequence & plan executions when an alert / routine is received
245 21 Paul Carensac
246 46 Paul Carensac
  * Implemented execute_sequence (not finished)
247 1 Paul Carensac
248 1 Paul Carensac
h3. Notes
249 1 Paul Carensac
250 18 Paul Carensac
---
251 18 Paul Carensac
252 19 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}monitoring%
253 18 Paul Carensac
254 18 Paul Carensac
h3. Purpose
255 18 Paul Carensac
256 47 Paul Carensac
 * It is the only tasks launched at the system's start
257 47 Paul Carensac
 * Make all the starting actions : instruments configuration, connection to the PLC, updating software versions, determine night start/end & start a scheduling
258 47 Paul Carensac
 * Starts the alert_listener
259 47 Paul Carensac
 * Infinite loop to regularly check the instruments and PLC status (and send them to the IC)
260 47 Paul Carensac
 * Handles night start & end
261 47 Paul Carensac
262 18 Paul Carensac
h3. Evolution
263 18 Paul Carensac
264 1 Paul Carensac
 * Creating application
265 21 Paul Carensac
266 1 Paul Carensac
    * See scheduler documentation above
267 1 Paul Carensac
268 47 Paul Carensac
 * System to launch the task at the system's start
269 1 Paul Carensac
270 47 Paul Carensac
 * Implement skeleton of task
271 1 Paul Carensac
272 47 Paul Carensac
 * Implement starting actions (with some empty functions)
273 47 Paul Carensac
274 47 Paul Carensac
 * Main loop to check PLC, instruments and night start / end (not finished)
275 18 Paul Carensac
 
276 18 Paul Carensac
h3. Notes
277 18 Paul Carensac
278 18 Paul Carensac
279 19 Paul Carensac
---
280 18 Paul Carensac
281 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}observation_manager%
282 18 Paul Carensac
283 18 Paul Carensac
h3. Purpose
284 18 Paul Carensac
285 49 Paul Carensac
 * Give orders the instruments to execute the plans
286 49 Paul Carensac
 * Take the calibration images and computes the 'super' calibration images
287 49 Paul Carensac
288 18 Paul Carensac
h3. Evolution
289 1 Paul Carensac
290 18 Paul Carensac
 * Creating application
291 1 Paul Carensac
292 21 Paul Carensac
    * See scheduler documentation above
293 1 Paul Carensac
294 49 Paul Carensac
 * Created the tasks : execute_plan_(nir/vis) & create_calibrations
295 1 Paul Carensac
296 49 Paul Carensac
 * Implemented the execute_plan tasks with some incomplete functions
297 1 Paul Carensac
 
298 1 Paul Carensac
h3. Notes
299 1 Paul Carensac
300 49 Paul Carensac
301 49 Paul Carensac
302 18 Paul Carensac
---
303 18 Paul Carensac
304 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}routine_manager%
305 18 Paul Carensac
306 18 Paul Carensac
h3. Purpose
307 18 Paul Carensac
308 50 Paul Carensac
 * Enables astronomers to create routines via the website
309 50 Paul Carensac
 * Import / export of routines (XML)
310 50 Paul Carensac
 * Display observation results for routines
311 50 Paul Carensac
312 18 Paul Carensac
h3. Evolution
313 18 Paul Carensac
314 18 Paul Carensac
 * Creating application
315 18 Paul Carensac
316 18 Paul Carensac
    * See scheduler documentation above
317 18 Paul Carensac
318 50 Paul Carensac
 * Created forms online for routine creation
319 50 Paul Carensac
320 50 Paul Carensac
 * Added import / export
321 50 Paul Carensac
322 50 Paul Carensac
 * Added routine submit / unsubmit
323 18 Paul Carensac
 
324 52 Paul Carensac
h3. Tests
325 52 Paul Carensac
326 52 Paul Carensac
 * Import fake file
327 52 Paul Carensac
 * Import invalid file
328 52 Paul Carensac
 * Import good file
329 52 Paul Carensac
 * Export incomplete
330 52 Paul Carensac
 * Export valid
331 52 Paul Carensac
332 18 Paul Carensac
h3. Notes
333 18 Paul Carensac
334 50 Paul Carensac
335 50 Paul Carensac
336 18 Paul Carensac
---
337 18 Paul Carensac
338 18 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}user_manager%
339 18 Paul Carensac
340 31 Paul Carensac
h3. Purpose
341 31 Paul Carensac
342 31 Paul Carensac
 * Handle user create / delete / login / logout
343 18 Paul Carensac
 * Allow to see user profile and make password recovery
344 18 Paul Carensac
345 18 Paul Carensac
h3. Evolution
346 18 Paul Carensac
347 18 Paul Carensac
 * Creating application
348 18 Paul Carensac
349 32 Paul Carensac
    * See scheduler documentation above
350 32 Paul Carensac
351 32 Paul Carensac
 * Taking and adapting templates from Alexandru project
352 32 Paul Carensac
353 32 Paul Carensac
 * User creation
354 33 Paul Carensac
355 33 Paul Carensac
  * Form in admin.py
356 33 Paul Carensac
  * Functions to verify if email doesn't exist and if passwords match
357 18 Paul Carensac
  * Added check to pass login page if user is already autheticated
358 18 Paul Carensac
  * Restricted access to all pyros views to authenticated users (except from user creation and login) with @login_required
359 18 Paul Carensac
 
360 53 Paul Carensac
h3. Tests
361 53 Paul Carensac
362 53 Paul Carensac
 * Creation
363 53 Paul Carensac
 * Login
364 1 Paul Carensac
 * wrong email
365 1 Paul Carensac
 * wrong password
366 1 Paul Carensac
 * logout
367 53 Paul Carensac
368 53 Paul Carensac
h3. Notes
369 53 Paul Carensac
370 53 Paul Carensac
371 18 Paul Carensac
---
372 31 Paul Carensac
373 31 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}common%
374 20 Paul Carensac
375 18 Paul Carensac
h3. Purpose
376 18 Paul Carensac
377 18 Paul Carensac
 * Regroups common data and functions shared by applications
378 59 Paul Carensac
 * Contains all the database Models
379 59 Paul Carensac
 * Basic tests in tests.py
380 59 Paul Carensac
 * Backoffice configuration in admin.py
381 18 Paul Carensac
382 25 Paul Carensac
h3. Content
383 18 Paul Carensac
384 25 Paul Carensac
 * RequestBuilder - class to build and save in DB the requests
385 1 Paul Carensac
386 25 Paul Carensac
  * You can start a request, then add sequences, albums and plans, given their parent class (ex: you need to give a sequence id to create an album)
387 25 Paul Carensac
  * Each function returns the ID of the element it created (to give it to the children classes)
388 18 Paul Carensac
  * To save a request, you need to use validate_request
389 18 Paul Carensac
390 54 Paul Carensac
 * Device and children
391 54 Paul Carensac
392 54 Paul Carensac
  * Implementation of devices communication, messages and initialization
393 54 Paul Carensac
  * All classes inherit from the Device class
394 54 Paul Carensac
  * All classes re-implement the list of messages
395 54 Paul Carensac
 
396 25 Paul Carensac
h3. Tests
397 54 Paul Carensac
398 54 Paul Carensac
 * Test the creation of a full request by the RequestBuilder
399 59 Paul Carensac
 * Basic tests for models
400 54 Paul Carensac
401 25 Paul Carensac
h3. Notes
402 18 Paul Carensac
403 18 Paul Carensac
 * Will probably contain the DB models at last ...