Technical components

Version 10 (Paul Carensac, 05/04/2016 04:38 pm)

1 1 Paul Carensac
h1. Technical components
2 2 Paul Carensac
3 8 Paul Carensac
Explanations about the technical components of the project : the ones we have created (internal), and the imported ones (external).
4 2 Paul Carensac
5 2 Paul Carensac
---
6 2 Paul Carensac
7 2 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}Internal components%
8 2 Paul Carensac
9 2 Paul Carensac
h3. Agent
10 1 Paul Carensac
11 4 Paul Carensac
The Agent class is in the common.agent.py file.
12 4 Paul Carensac
13 4 Paul Carensac
 * *I - Purpose*
14 4 Paul Carensac
15 4 Paul Carensac
    * Generically handles and creates the asynchronous modules
16 4 Paul Carensac
    * Uses the threading library (see below in External components) to make all modules independent
17 4 Paul Carensac
    * Provides an abstract class to be inherited
18 4 Paul Carensac
19 4 Paul Carensac
 * *II - Features*
20 4 Paul Carensac
21 4 Paul Carensac
    * Uses a config file (pyros_agent_config.ini) to set the network communication interface of all agents
22 4 Paul Carensac
    * Provides a 'work' method to override : this is the entry method of the newly created thread (see 'How to use it' section below)
23 4 Paul Carensac
    * Provides the 'receive' and 'analyze_message' methods to generically receive messages from network and analyze them 
24 4 Paul Carensac
25 4 Paul Carensac
 * *III - How to use it ?*
26 4 Paul Carensac
27 4 Paul Carensac
    Each of these points are +NECESSARY+
28 4 Paul Carensac
29 4 Paul Carensac
    * Create a new class that inherits from Agent
30 4 Paul Carensac
    * In the __init__ method, first call the __init__ method of Agent, passing the name of the agent as second parameter (they are defined in the Agent class, eg: Agent.SCHEDULER)
31 4 Paul Carensac
    * Inside the class, define the messages your agent can receive (eg: MSG_OBS_FINISHED = "Observation finished")
32 4 Paul Carensac
    * Create a method to be called for every message you created
33 4 Paul Carensac
    * In the __init__, after calling the Agent's __init__, associate each message to its associated function in the 'self.actions_by_message' dictionary (eg: self.actions_by_message[self.MSG_OBS_FINISHED] = self.observation_finished)
34 4 Paul Carensac
    * Override the method work : this will be the entry function of the new thread, so do whatever you need. This MUST NOT be an infinite loop, because Agent's receive method will be called after this one
35 4 Paul Carensac
    * If ever needed, override the 'shutdown' method, it will be called when your agent receive the Agent.SHUTDOWN message (eg: if you created another thread in the 'work' method, you need to close it)
36 5 Paul Carensac
    * To start the agent, just instantiate your class and do MyClass.start() (the 'work' method will be called)
37 1 Paul Carensac
38 5 Paul Carensac
    The main points to understand are that you can do whatever you want (but non-blocking) in work method (like creating new threads or variables' initialization), then the only entry points are the message-associated methods
39 4 Paul Carensac
40 7 Paul Carensac
 * *IV - Important : pyros agents launching*
41 7 Paul Carensac
42 7 Paul Carensac
    * In pyros, there is maximum 1 agent per application
43 7 Paul Carensac
    * The agent must be started at application start :
44 7 Paul Carensac
        
45 7 Paul Carensac
        * In the MyApp.apps.py file, create a class inheriting from django.apps.AppConfig
46 7 Paul Carensac
        * Define the 'name' attribute in it, giving it the name of the agent
47 7 Paul Carensac
        * Create a 'ready(self)' method
48 7 Paul Carensac
        * in the ready method, import your agent implementation, instantiate it and start it
49 7 Paul Carensac
<pre>
50 7 Paul Carensac
from django.apps import AppConfig
51 7 Paul Carensac
52 7 Paul Carensac
53 7 Paul Carensac
class AlertManagerConfig(AppConfig):
54 7 Paul Carensac
    name = 'alert_manager'
55 7 Paul Carensac
    
56 7 Paul Carensac
    def ready(self):
57 7 Paul Carensac
        from alert_manager.agent import AlertManagerAgent
58 7 Paul Carensac
        self.agent = AlertManagerAgent()
59 7 Paul Carensac
        self.agent.start()
60 7 Paul Carensac
</pre>
61 7 Paul Carensac
62 2 Paul Carensac
h3. Sender
63 1 Paul Carensac
64 4 Paul Carensac
The Sender class is in the common.sender.py file
65 4 Paul Carensac
66 4 Paul Carensac
 * *I - Purpose*
67 4 Paul Carensac
68 4 Paul Carensac
    * Send a given message to an agent
69 4 Paul Carensac
70 4 Paul Carensac
 * *II - Features*
71 4 Paul Carensac
72 4 Paul Carensac
    * Uses the 'pyros_agent_config.ini' file to get the agents' network interface configuration (ip and port)
73 4 Paul Carensac
    * Provide a 'send_to' static method to send the messages
74 4 Paul Carensac
75 4 Paul Carensac
 * *III - How to use it ?*
76 4 Paul Carensac
77 4 Paul Carensac
    * The targeted agent must be described in 'pyros_agent_config.ini'
78 4 Paul Carensac
    * Use Sender.send_to method, giving as first parameter the name of the targeted agent (eg: Agent.SCHEDULER), and as second parameter the message (eg: Agent.SHUTDOWN)
79 4 Paul Carensac
    * /!\ send_to is a static method, you don't need to instantiate a Sender (just do Sender.send_to(...))
80 9 Paul Carensac
81 2 Paul Carensac
---
82 2 Paul Carensac
83 2 Paul Carensac
h2. %{margin-left:0px; font-weight:bold; font-size:25px;  display:block; color:red;}External components%
84 2 Paul Carensac
85 10 Paul Carensac
h3. Celery
86 10 Paul Carensac
87 10 Paul Carensac
 * *I - Purpose*
88 10 Paul Carensac
    Celery is used to create and handle tasks in different processes/threads.
89 10 Paul Carensac
    Its use is very easy.
90 10 Paul Carensac
91 10 Paul Carensac
 * *II - Features*
92 10 Paul Carensac
93 10 Paul Carensac
    * Create personalized tasks asynchronously
94 10 Paul Carensac
    * Has ETA and countdowns
95 10 Paul Carensac
    * Lots of configurations are possible
96 10 Paul Carensac
97 10 Paul Carensac
 * *III - How to use it ?*
98 10 Paul Carensac
99 10 Paul Carensac
100 10 Paul Carensac
---
101 10 Paul Carensac
102 1 Paul Carensac
h3. Threading library
103 5 Paul Carensac
104 5 Paul Carensac
 * *I - Purpose*
105 5 Paul Carensac
106 5 Paul Carensac
    * Simply create threads with basic communication
107 5 Paul Carensac
    * Allows to handle concurrent access
108 5 Paul Carensac
109 5 Paul Carensac
 * *II - Features*
110 5 Paul Carensac
111 5 Paul Carensac
    Provides :
112 5 Paul Carensac
113 5 Paul Carensac
    * A Thread class to inherit from, with a run() method that will be called when the thread starts
114 5 Paul Carensac
    * An Event class to set/unset a boolean in order to transmit message to the thread
115 5 Paul Carensac
    * Lock and RLock object to handle concurrent access
116 5 Paul Carensac
117 5 Paul Carensac
 * *III - How to use it ?*
118 5 Paul Carensac
119 5 Paul Carensac
    <pre>from threading import Thread, Event</pre>
120 5 Paul Carensac
121 5 Paul Carensac
    * Thread
122 5 Paul Carensac
123 5 Paul Carensac
        * Create a class inheriting from Thread
124 5 Paul Carensac
        * Override 'run' method, that will be called at thread start
125 5 Paul Carensac
        * Instantiate your class, and do MyClass.start() to create the thread
126 5 Paul Carensac
127 5 Paul Carensac
    * Event
128 5 Paul Carensac
129 5 Paul Carensac
        * Create an Event variable in your Thread-inheriting class (eg: 'self.stop_event = Event()')
130 5 Paul Carensac
        * After thread starts, you can set/unset the event by doing MyClass.stop_event.set() / .clear()
131 5 Paul Carensac
        * There are a few useful methods, see this link for further information : https://docs.python.org/3/library/threading.html#threading.Event
132 5 Paul Carensac
    * Lock / RLock
133 5 Paul Carensac
134 5 Paul Carensac
        * Still not used, see online documentation : https://docs.python.org/3/library/threading.html#lock-objects
135 10 Paul Carensac
136 10 Paul Carensac
---
137 2 Paul Carensac
138 2 Paul Carensac
h3. Socket library
139 2 Paul Carensac
140 6 Paul Carensac
 * *I - Purpose*
141 6 Paul Carensac
142 6 Paul Carensac
    * Handle network communication, just giving IP and Port of the interlocutors
143 6 Paul Carensac
144 6 Paul Carensac
 * *II - Features*
145 6 Paul Carensac
146 6 Paul Carensac
    * 'server' system to create an interface, waiting for client connections and sending / receiving data from them
147 6 Paul Carensac
    * 'client' system to connect to a server, and send/receive data from it
148 6 Paul Carensac
149 6 Paul Carensac
 * *III - How to use it ?*
150 6 Paul Carensac
151 6 Paul Carensac
    * Server
152 6 Paul Carensac
153 6 Paul Carensac
        * Instantiate socket and wait for connections
154 6 Paul Carensac
<pre>
155 6 Paul Carensac
self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)   # create the socket
156 6 Paul Carensac
self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # for the port to be immediately re-usable after closing the socket
157 6 Paul Carensac
self.server_socket.bind((self.ip, self.receive_port))                    # associate the socket to an ip and a port
158 6 Paul Carensac
self.server_socket.listen(12)                                            # wait for connections (here, 12 connections can be simultaneously waiting for acceptance)
159 6 Paul Carensac
</pre>
160 6 Paul Carensac
        * Accept connections
161 6 Paul Carensac
<pre>
162 6 Paul Carensac
conn, addr = self.server_socket.accept() # conn is a new socket created at the connection
163 6 Paul Carensac
</pre>
164 6 Paul Carensac
        * Exchanging messages
165 6 Paul Carensac
<pre>
166 6 Paul Carensac
conn.send(bytes(message, 'UTF-8'))          # sending
167 6 Paul Carensac
data = conn.recv(self.buffer_size).decode() # receiving
168 6 Paul Carensac
</pre>
169 6 Paul Carensac
        * Closing sockets when you're done with them
170 6 Paul Carensac
<pre>
171 6 Paul Carensac
conn.close()
172 6 Paul Carensac
...
173 6 Paul Carensac
server_socket.close()
174 6 Paul Carensac
</pre>
175 6 Paul Carensac
176 6 Paul Carensac
    * Client
177 6 Paul Carensac
178 6 Paul Carensac
        * Instantiate the socket and connect to a server
179 6 Paul Carensac
<pre>
180 6 Paul Carensac
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
181 6 Paul Carensac
client_socket.connect((dest_ip, dest_receive_port))
182 6 Paul Carensac
</pre>
183 6 Paul Carensac
        * Exchanging messages
184 6 Paul Carensac
<pre>
185 6 Paul Carensac
client_socket.send(bytes(message, 'UTF-8'))          # sending
186 6 Paul Carensac
data = client_socket.recv(self.buffer_size).decode() # receiving
187 6 Paul Carensac
</pre>
188 6 Paul Carensac
        * Closing sockets when you're done with them
189 6 Paul Carensac
<pre>
190 6 Paul Carensac
client_socket.close()
191 6 Paul Carensac
</pre>
192 6 Paul Carensac
193 2 Paul Carensac
---