ae.transfer_service

transfer client and server services

this ae portion is providing client and server services to transfer files and text messages between two devices in the same local network.

the number of parallel running file and message transfers is only limited by the available resources of the involved devices.

if a file transfers gets interrupted it can be recovered later and without the need to resend the already transferred file content.

standard file paths - like e.g. the documents or downloads folders - are getting automatically adopted to the specific path structures of each involved device and operating system.

transfer service life cycle

the transfer service can be invoked in different ways: standalone as a separate process or attached and embedded into a controlling application.

run transfer service in standalone mode

execute this module to run the transfer services as a separate standalone process via:

python transfer_service.py [--bind=...] [--port=...] [--buf_len=...]

the following command line options are overwriting the default server address and socket buffer length (see also service_factory()):

  • ‘bind’ to restrict the incoming connections to an ip address/range (overwriting the default SERVER_BIND).

  • ‘port’ to specify the socket port (overwriting the default port SERVER_PORT).

  • ‘buf_len’ to specify the socket buffer length (overwriting the default buffer length SOCKET_BUF_LEN).

after that the transfer service will be able to receive files send from another process or device.

Note

on Android a standalone transfer service has to be started as android service.

run transfer service attached to any app

alternatively you can run the transfer service server in a separate thread within respectively attached to your application:

from ae.transfer_service import service_factory

transfer_service_app = service_factory()
transfer_service_app.set_option('port', 12345, save_to_config=False)
transfer_service_app.set_option('buf_len', 34567, save_to_config=False)
transfer_service_app.start_server(threaded=True)

pause or stop transfer service

to manually pause the transfer service, store the app instance of the transfer service app (transfer_service_app in the example above) and call its stop_server() method:

transfer_service_app.stop_server()

to fully stop the transfer service and terminate the transfer service app call instead its shutdown() method:

transfer_service_app.shutdown()

Hint

the shutdown() method of the base app instance (AppBase) automatically ensures a clean shutdown of the transfer service server on app quit/exit.

send file to another transfer service server

to send files from one transfer server to another running transfer server, a separate client process has to be started on the device storing the file to be send.

to initiate the file transfer the client process has to make a tcp connection to the transfer server running on the same device, specifying the path of the file to send and the remote ip of the receiving transfer server and finally call the remote procedure send_file like shown in the following example:

import socket
from ae.transfer_service import connect_and_request

request_kwargs = dict(method_name='send_file', file_path='path_to_file/file_name.ext', remote_ip='192.168.3.123')
with socket.socket() as sock:
    response_kwargs = connect_and_request(sock, request_kwargs)

if 'error' in response_kwargs:
    # handle error (e.g. display to user or add to a log file)

if the file transfer failed then the transfer kwargs dict returned by connect_and_request() will contain an error key containing the error message text.

implemented remote procedures

the following remote procedures are provided by the transfer service server:

  • cancel_request: cancel running file transfer.

  • pending_requests: get log info the progress/status of all currently running file transfers.

  • recv_file: receive file from other transfer service server.

  • recv_message: receive text message from other transfer service server.

  • send_file: send file to other transfer service server.

  • send_message: send text message to other transfer service server.

Hint

the demo app ComPartY is using all provided remote procedures.

Module Attributes

CONNECTION_TIMEOUT

default timeout (in seconds) to connect and request a server process

CONNECT_ERR_PREFIX

error message string prefix if error happened directly in connect_and_request() helper (no protocol error).

ENCODING_KWARGS

default encoding and encoding error handling strategy

SERVER_BIND

setting BIND to '' or None to allow connections for all available interfaces

SERVER_PORT

server listening port

SHUTDOWN_TIMEOUT

timeout (in seconds) to shutdown/stop the console app

SOCKET_BUF_LEN

buf length for socket receives and sends

TRANSFER_KWARGS_DATE_TIME_NAME_PARTS

kwarg name suffix for values automatically converted

TRANSFER_KWARGS_LINE_END_CHAR

end of line/command character as string

TRANSFER_KWARGS_LINE_END_BYTE

TransferKwargs

command/action format of requests and responses

requests_lock

locking requests TransferKwargs in TransferServiceApp.reqs_and_logs

server_app

transfer service server app

Functions

clean_log_str(log_str)

remove high-commas and backslashes from the passed string to add it to the logs (preventing duplicates).

connect_and_request(sock, request_kwargs[, ...])

connect to remote, send first command/action and return response as transfer kwargs dict.

recv_bytes(sock[, buf_len])

receive all bytes from the passed client socket instance until connection lost or line end reached.

service_factory([task_id_func])

create server app instance including the command line options bind and port.

transfer_kwargs_error(transfer_kwargs, err_msg)

add/append error to transfer kwargs dict without overwriting any previous error.

transfer_kwargs_from_literal(transfer_kwargs_lit)

convert dict literal (created with transfer_kwargs_literal()) to transfer kwargs dict.

transfer_kwargs_literal(transfer_kwargs)

convert dict to str literal to be sent via sockets (re-instantiable via transfer_kwargs_from_literal()).

transfer_kwargs_update(*variables, **kwargs)

update multiple transfer dicts with the same keys/values (locking with requests_lock).

Classes

ThreadedTCPRequestHandler(request, ...)

server request handler.

TransferServiceApp([app_title, app_name, ...])

server service app class

CONNECTION_TIMEOUT = 2.7

default timeout (in seconds) to connect and request a server process

CONNECT_ERR_PREFIX = 'transfer_service.connect_and_request() exception '

error message string prefix if error happened directly in connect_and_request() helper (no protocol error).

ENCODING_KWARGS = {'encoding': 'UTF-8', 'errors': 'ignore'}

default encoding and encoding error handling strategy

SERVER_BIND = ''

setting BIND to ‘’ or None to allow connections for all available interfaces

SERVER_PORT = 36969

server listening port

SHUTDOWN_TIMEOUT = 3.9

timeout (in seconds) to shutdown/stop the console app

SOCKET_BUF_LEN = 16384

buf length for socket receives and sends

TRANSFER_KWARGS_DATE_TIME_NAME_PARTS = ('_date', '_time')

kwarg name suffix for values automatically converted

TRANSFER_KWARGS_LINE_END_CHAR = '\n'

end of line/command character as string

TRANSFER_KWARGS_LINE_END_BYTE = b'\n'
TransferKwargs

command/action format of requests and responses

alias of Dict[str, Any]

requests_lock = <unlocked _thread.lock object>

locking requests TransferKwargs in TransferServiceApp.reqs_and_logs

clean_log_str(log_str)[source]

remove high-commas and backslashes from the passed string to add it to the logs (preventing duplicates).

Parameters:

log_str (Union[str, bytes]) – log string or bytes array to clean up.

Return type:

str

Returns:

cleaned log string.

connect_and_request(sock, request_kwargs, buf_len=16384, timeout=2.7)[source]

connect to remote, send first command/action and return response as transfer kwargs dict.

Parameters:
  • sock (socket) – new socket instance.

  • request_kwargs (Dict[str, Any]) – first/initial request kwargs dict. if the key ‘server_address’ is not provided (with the server address as (host, ip) tuple), then (‘localhost’, SERVER_PORT) is used. if the key ‘local_ip’ is not specified then the local ip address will be used.

  • buf_len (int) – socket buffer length. if not passed then SOCKET_BUF_LEN will be used. pass zero/0 to use the buf length defined via the ‘buf_len’ command line option.

  • timeout (Optional[float]) – timeout in seconds or None to use socket/system default timeout. if not passed then the default timeout specified by CONNECTION_TIMEOUT will be used.

Return type:

Dict[str, Any]

Returns:

response transfer kwargs dict.

recv_bytes(sock, buf_len=16384)[source]

receive all bytes from the passed client socket instance until connection lost or line end reached.

Parameters:
  • sock (socket) – socket instance.

  • buf_len (int) – socket buffer length. if not passed then SOCKET_BUF_LEN will be used. pass zero/0 to use the buf length defined via the ‘buf_len’ command line option.

Return type:

bytes

Returns:

received bytes.

service_factory(task_id_func=None)[source]

create server app instance including the command line options bind and port.

Parameters:

task_id_func (Optional[Callable[[str, str, str], str]]) – callable to return an unique id for a transfer request task.

Return type:

TransferServiceApp

Returns:

transfer service app instance.

transfer_kwargs_error(transfer_kwargs, err_msg)[source]

add/append error to transfer kwargs dict without overwriting any previous error. :type _sphinx_paramlinks_ae.transfer_service.transfer_kwargs_error.transfer_kwargs: Dict[str, Any] :param _sphinx_paramlinks_ae.transfer_service.transfer_kwargs_error.transfer_kwargs: request/response transfer kwargs dict. :type _sphinx_paramlinks_ae.transfer_service.transfer_kwargs_error.err_msg: str :param _sphinx_paramlinks_ae.transfer_service.transfer_kwargs_error.err_msg: error message to add.

transfer_kwargs_from_literal(transfer_kwargs_lit)[source]

convert dict literal (created with transfer_kwargs_literal()) to transfer kwargs dict.

Parameters:

transfer_kwargs_lit (str) – request/response dict literal string.

Return type:

Dict[str, Any]

Returns:

transfer kwargs dict.

transfer_kwargs_literal(transfer_kwargs)[source]

convert dict to str literal to be sent via sockets (re-instantiable via transfer_kwargs_from_literal()).

Note

to ensure security (and prevent injections) only the following basic types can be used: int, float, boolean, str, bytes, list, tuple and dict. date/time values are only allowed as dict value and of the type datetime.datetime; additionally the key of this dict value has to contain one of the fragments defined in TRANSFER_KWARGS_DATE_TIME_NAME_PARTS.

Parameters:

transfer_kwargs (Dict[str, Any]) – request/response transfer kwargs dict.

Return type:

str

Returns:

literal string of transfer kwargs dict terminated with TRANSFER_KWARGS_LINE_END_CHAR.

transfer_kwargs_update(*variables, **kwargs)[source]

update multiple transfer dicts with the same keys/values (locking with requests_lock).

Parameters:
  • variables – transfer kwargs variables/dicts.

  • kwargs – kwargs to update.

class ThreadedTCPRequestHandler(request, client_address, server)[source]

Bases: StreamRequestHandler

server request handler.

self.rfile is a file-like object created by the handler; to use e.g. readline() instead of raw recv() likewise, self.wfile is a file-like object used to write back to the client.

handle()[source]

handle a single request

class TransferServiceApp(app_title='', app_name='', app_version='', sys_env_id='', debug_level=0, multi_threading=False, suppress_stdout=False, cfg_opt_eval_vars=None, additional_cfg_files=(), cfg_opt_val_stripper=None, formatter_class=None, epilog='', **logging_params)[source]

Bases: ConsoleApp

server service app class

reqs_and_logs: List[Dict[str, Any]]

list of transfer_kwargs of currently processed requests

server_instance: Optional[ThreadingTCPServer]

server class instance

server_thread: Optional[Thread]

server thread (main or separate thread)

cancel_request(request_kwargs, handler)[source]

cancel running request.

Parameters:
  • request_kwargs (Dict[str, Any]) – request transfer kwargs dict, specifying via rt_id_to_cancel the request to cancel.

  • handler (StreamRequestHandler) – request handler instance.

Return type:

Dict[str, Any]

Returns:

response kwargs, having error key if request could not be found/cancelled.

static id_of_task(action, object_type, object_key)[source]

compile the id of a transfer request task.

Parameters:
  • action (str) – action or log level string.

  • object_type (str) – task object type (‘log’ for log entries, ‘file’ for file transfers, else ‘message’).

  • object_key (str) – task key (file path for file transfers, message string for messages and timestamp for log entries).

Return type:

str

Returns:

unique key identifying the task/log-entry.

log(log_level, message)[source]

print log message and add it to reqs_and_logs (to be read by controller app).

Note

please note that you have to use print() or one of the console print methods of the AppBase (like e.g. verbose_out(), respective self.vpo) instead of this method for the logging of low level transport methods/functions (like e.g. pending_requests(), response_to_request(), handle() or recv_bytes()). this will prevent the duplication of a log message, because each call of this method creates a new entry in reqs_and_logs which will be sent to the controlling app via the low level transport methods (which would recursively grow the sent messages until the system freezes), especially if the transfer kwargs are included into the log message.

Parameters:
  • log_level (str) – ‘print’ always prints, ‘debug’ prints if self.debug, ‘verbose’ prints if self.verbose.

  • message (str) – message to print.

pending_requests(request_kwargs, handler)[source]

determine currently running/pending server requests and debug log messages (in debug mode only).

Parameters:
Return type:

Dict[str, Any]

Returns:

response transfer kwargs dict keys.

recv_file(request_kwargs, handler)[source]

receive binary file content from client.

Parameters:
  • request_kwargs (Dict[str, Any]) –

    request transfer kwargs dict with the following keys:

    • ’file_path’: file path (can contain path placeholders).

    • ’total_bytes’: total file length in bytes.

    • ’series_file’: optionally, pass any value to ensures new file name.

  • handler (StreamRequestHandler) – request handler class instance.

Return type:

Dict[str, Any]

Returns:

response transfer kwargs dict keys (request kwargs extended with additional keys):

  • ’error’: error message string if an error occurred.

  • ’transferred_bytes’: start offset on recovered transfer (previously received bytes).

recv_message(request_kwargs, handler)[source]

receive message from client/peer.

Return type:

Dict[str, Any]

response_to_request(request_lit, handler)[source]

process request to this server and return response string.

Note

this method is running in a separate thread (created by the server to process this request).

Parameters:
  • request_lit (str) – request string, which is a dict literal with ‘method_name’ key.

  • handler (StreamRequestHandler) – stream request handler instance.

Return type:

str

Returns:

response string with transfer kwargs as literal. if an error occurred then the returned response kwargs contains an ‘error’ key which stores the related error message.

send_file(request_kwargs, handler)[source]

send binary file content to remote server.

Return type:

Dict[str, Any]

send_message(request_kwargs, handler)[source]

send message to remote server.

Return type:

Dict[str, Any]

shutdown(exit_code=0, timeout=None)[source]

overwritten to stop a running transfer service server/threads on shutdown of this app instance.

Parameters:
start_server(threaded=False)[source]

start server and run until main app stop_server().

Parameters:

threaded (bool) – optionally pass True to use separate thread for the server process.

Return type:

str

Returns:

empty string/”” if server instance/thread got started else error message string.

stop_server()[source]

stop/pause transfer service server - callable also if not running to reset/prepare this app instance.

server_app: TransferServiceApp | None = None

transfer service server app