videoflow.engines package

Submodules

videoflow.engines.batch module

class videoflow.engines.batch.BatchExecutionEngine

Bases: videoflow.core.engine.ExecutionEngine

join_task_processes()

Blocking method. It is supposed to make the calling process sleep until all task processes have finished processing.

signal_flow_termination()

Signals the execution environment that the flow needs to stop. When this signal is received, all consumer tasks will pick it on and pass it together with the flow until they reach every task in the graph and everyone stops working.

class videoflow.engines.batch.BatchprocessingQueueMessenger(computation_node: videoflow.core.node.Node, task_queue: multiprocessing.context.BaseContext.Queue, parent_task_queue: multiprocessing.context.BaseContext.Queue, termination_event: multiprocessing.context.BaseContext.Event)

Bases: videoflow.core.engine.Messenger

BatchprocessingQueueMessenger is a messenger that communicates through queues of type multiprocessing.Queue. It is not real time, which means that if a queue is full when publishing a message to it, it will block until the queue can process it.

check_for_termination() → bool

Checks if someone has set a termination event.

passthrough_message()

Used when the task has received a message that might be needed by another task below it, but when the task itself does not produces output needed by tasks below it. (i.e.: videoflow.core.task.ConsumerTask)

Depending on the kind of environment, this method might drop the message if the receiving container (usually a queue) is full.

passthrough_termination_message()

Same as passthrough_message, but this method will never drop the message regardless of environment, which means that sometimes this method might block until it can deliver the message.

publish_message(message, metadata=None)

Publishes output message to a place where the child task will receive it. Will drop the message is the receiving queue is full.

publish_termination_message(message, metadata=None)

This method is identical to publish message

receive_message()

This method blocks. It waits until a message has been received.

• Returns:

  • message: the message received from parent task in topological sort.

receive_raw_message()

videoflow.engines.realtime module

class videoflow.engines.realtime.RealtimeExecutionEngine

Bases: videoflow.core.engine.ExecutionEngine

join_task_processes()

Blocking method. It is supposed to make the calling process sleep until all task processes have finished processing.

signal_flow_termination()

Signals the execution environment that the flow needs to stop. When this signal is received, all consumer tasks will pick it on and pass it together with the flow until they reach every task in the graph and everyone stops working.

class videoflow.engines.realtime.RealtimeQueueMessenger(computation_node: videoflow.core.node.Node, task_queue: multiprocessing.context.BaseContext.Queue, parent_task_queue: multiprocessing.context.BaseContext.Queue, termination_event: multiprocessing.context.BaseContext.Event)

Bases: videoflow.core.engine.Messenger

RealtimeQueueMessenger is a messenquer that communicates through queues of type multiprocessing.Queue. It is a real time, which means that if a queue is full when publishing a message to it, it will drop the message and not block. The methods that publish and passthrough termination messages will block and not drop.

check_for_termination() → bool

Checks if someone has set a termination event.

passthrough_message()

Used when the task has received a message that might be needed by another task below it, but when the task itself does not produces output needed by tasks below it. (i.e.: videoflow.core.task.ConsumerTask)

Depending on the kind of environment, this method might drop the message if the receiving container (usually a queue) is full.

passthrough_termination_message()

Same as passthrough_message, but this method will never drop the message regardless of environment, which means that sometimes this method might block until it can deliver the message.

publish_message(message, metadata=None)

Publishes output message to a place where the child task will receive it. Will drop the message is the receiving queue is full.

publish_termination_message(message, metadata=None)

This method is identical to publish message, but is blocking Because, the termination message cannot be dropped.

receive_message()

This method blocks. It waits until a message has been received.

• Returns:

  • message: the message received from parent task in topological sort.

videoflow.engines.task_functions module

videoflow.engines.task_functions.create_process_task(task)

videoflow.engines.task_functions.create_process_task_gpu(task, gpu_id)

videoflow.engines.task_functions.task_executor_fn(task: videoflow.core.task.Task)

videoflow.engines.task_functions.task_executor_gpu_fn(task: videoflow.core.task.Task, gpu_id: int)

videoflow.engines.zeromq module

videoflow.engines.zeromq.recv_array(socket: zmq.sugar.socket.Socket, flags: int = 0, copy: int = True, track: bool = False)

Receive a numpy array

videoflow.engines.zeromq.recv_zipped_pickle(socket: zmq.sugar.socket.Socket, flags: int = 0)

Receive a sent zipped pickle.

videoflow.engines.zeromq.send_array(socket: zmq.sugar.socket.Socket, array: numpy.core.multiarray.array, flags: int = 0, copy: bool = True, track: bool = False)

Send a numpy array with metadata, type and shape

videoflow.engines.zeromq.send_zipped_pickle(socket: zmq.sugar.socket.Socket, obj: any, flags: int = 0, protocol: int = -1)

Pickle an object, and zip the pickle before sending it

videoflow.engines.zeromq.socket_from_task_id(task_id: int)

videoflow.engines.zeromq.task_id_from_socket(socket_number: int)

videoflow.engines.zeromq.task_id_from_termination_socket(socket_number: int)

videoflow.engines.zeromq.termination_socket_from_task_id(task_id: int)