xia_engine_mysql.engine.MysqlEngine

class xia_engine_mysql.engine.MysqlEngine

Bases: SqlEngine

SQLite Engine

Connection Parameters:
  • Should be hold in the class attribute class._sqlite = {“db”: “”, “path”: “”}
    • db: Database name, default database name is “”

    • kwarg: Where the database should be stored, default is in memory

__init__()

Methods

__init__()

analyze(document_class, analytic_model)

Run the analytic model

backup(document_class[, location, ...])

Backup data of a model.

batch(operations, originals)

Data Batch Modification

compile(document_class, analytic_request[, ...])

Compile the analysis request

connect([document_class])

Connect to the engine

create(document_class, db_content[, doc_id])

Create a document

create_collection(document_class)

Create Collection if needed

create_table(table_name, field_dict, ...)

Create table

db_to_display(document_class, db_content[, ...])

Convert data from database form to display form

delete(document_class, doc_id)

Delete a document by using id

drop(document_class)

Drop the given collection

fetch(document_class, *args)

Get document one by one from a list of document id

get(document_class, doc_id)

Get Document

get_connection([document_class])

Get engine connection。 Always using existed one when it is possible

get_decoder(field[, inner_field])

Get Decoder for a field

get_encoder(field[, inner_field])

Get Encoder for a field

get_model_sql(document_class, model_condition)

Get SQL to be executed by using provided data model

get_search_sql(document_class, _acl_queries, ...)

Get SQL to be executed by using provided data model

lock(document_class, doc_id[, timeout])

Lock entries for write

merge(document_class[, start, end, purge, ...])

Merge data from log section into main table

parse_content(data_content, data_model, ...)

Parse database result to db content (json format)

parse_data(db_content, data_model, ...)

Split database content to multiple table update instruction

parse_search_option(key)

Reference to search method for the specifications

parse_update_option(key)

Reference to update method for the specifications

replicate(document_class, task_list)

Data replication

restore(document_class[, location, ...])

Restore data of a model

scan(_document_class[, _acl_queries, _limit])

Scan the document class and get the document id list

search(_document_class, *args[, ...])

Searching and yield document by document

set(document_class, doc_id, db_content)

Overwrite whole document

truncate(document_class)

Remove all data from the given collection

unlock(document_class, doc_id)

Release the for write

update(_document_class, _doc_id, **kwargs)

Update a document

update_doc_id(document_class, db_content, ...)

Update document id to new value

Attributes

FIELD_SEPARATOR

KEY_SEPARATOR

MATCH_OPERATIONS

OPERATORS

ORDER_TYPES

TABLE_NAME_SEPARATOR

UPDATE_TYPES

add_column_sql_template

analytic_sql_template

analyzer

backup_coder

backup_storer

create_sql_template

decoders

delete_sql_template

drop_sql_template

encoders

engine_db_shared

engine_default_connector_param

engine_foreign_key_check

engine_param

engine_scope_check

engine_unique_check

insert_sql_template

key_required

lq

paramstyle

rq

scan_and_fetch

select_sql_template

store_embedded_as_table

support_unknown

truncate_sql_template

type_dict

classmethod analyze(document_class: Type[BaseDocument], analytic_model: dict) list

Run the analytic model

Parameters
  • analytic_model – Analyze model

  • document_class – (subclass of BaseDocument): Document definition

Returns

Result as list of dictionary

classmethod backup(document_class: Type[BaseDocument], location: Optional[str] = None, data_encode: Optional[str] = None, data_format: Optional[str] = None, data_store: Optional[str] = None, **kwargs)

Backup data of a model. The real implementation must use kwargs to distribute loads

Parameters
  • document_class (subclass of BaseDocument) – Document definition4

  • data_encode (str) – Backup Data Code

  • data_format (str) – Backup Data Format

  • data_store (str) – Backup Data Store location

  • location (str) – Data location to e used by data store

  • **kwargs – parameter to be passed at engine level

classmethod batch(operations: list, originals: dict)

Data Batch Modification

The data will be updated at once or rolled back

Parameters
  • operations – List of operations to be done * op: Operation type. “S” = set, “I” = create, “D” = delete, “U” = update * cls: Document Class * doc_id: Document ID * content: Document Content in Database form

  • originals – Dictionary (Help to roll back) * class: document class name * id: document id * content: document db form

Returns

return True amd empty message if batch is successful, else False with error message

classmethod compile(document_class: Type[BaseDocument], analytic_request: dict, acl_condition=None)

Compile the analysis request

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • analytic_request – analytic request

  • acl_condition – Extra where condition given by user acl objects

Returns

A analytic model which could be executed by the engine

classmethod connect(document_class: Optional[Type[BaseDocument]] = None)

Connect to the engine

Parameters

document_class – (subclass of BaseDocument): Document definition

Returns

Connection

classmethod create(document_class: Type[BaseDocument], db_content: dict, doc_id: Optional[str] = None)

Create a document

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • db_content (dict) – content to be put to engine

  • doc_id (str) – Having predefined doc id, None means could be generated by engine

Returns

Document ID

classmethod create_collection(document_class: Type[BaseDocument])

Create Collection if needed

Parameters

document_class – document_class

classmethod create_table(table_name: str, field_dict: dict, document_class: Type[BaseDocument])

Create table

Parameters
  • table_name – table name to be created

  • field_dict – dictionary of field information

  • document_class – document_class

classmethod db_to_display(document_class: Type[BaseDocument], db_content: dict, lazy: bool = True, catalog: Optional[dict] = None, show_hidden: bool = False)

Convert data from database form to display form

Parameters
  • document_class – Document class

  • db_content – Database Content

  • lazy – Lazy Mode

  • catalog – Data Catalog

  • show_hidden – Show hidden member or not

Returns

document in display form

classmethod delete(document_class: Type[BaseDocument], doc_id: str)

Delete a document by using id

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • doc_id – Document ID

classmethod drop(document_class: Type[BaseDocument])

Drop the given collection

Parameters

document_class (subclass of BaseDocument) – Document definition

engine_connector

alias of Connection

engine_connector_class

alias of MysqlConnectParam

classmethod fetch(document_class: Type[BaseDocument], *args)

Get document one by one from a list of document id

Returns

An iterator for id, document dictionary pair

Comments:

when doc id is empty, it is probably because that the user only has partial read authorizations

classmethod get(document_class: Type[BaseDocument], doc_id: str)

Get Document

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • doc_id – Document ID

Returns

Document content on python dict

classmethod get_connection(document_class: Optional[Type[BaseDocument]] = None)

Get engine connection。 Always using existed one when it is possible

Parameters

document_class – (subclass of BaseDocument): Document definition

Returns

Connection

classmethod get_decoder(field: type, inner_field: Optional[type] = None) callable

Get Decoder for a field

Parameters
  • field (type) – class type of field class

  • inner_field (type) – class type of inner field (Such qs ListField)

Returns

Decoder function

classmethod get_encoder(field: type, inner_field: Optional[type] = None) callable

Get Encoder for a field

Parameters
  • field (type) – class type of field class

  • inner_field (type) – class type of inner field (Such qs ListField)

Returns

Encoder function

classmethod get_model_sql(document_class: Type[BaseDocument], model_condition: dict, acl_condition: Optional[dict] = None)

Get SQL to be executed by using provided data model

Parameters
  • document_class – document_class

  • model_condition – Search criteria defined by model

  • acl_condition – Search condition given by acl

Returns

SQL statement and the values to be put in the place holder

classmethod get_search_sql(document_class: Type[BaseDocument], _acl_queries: list, _limit: int, **kwargs)

Get SQL to be executed by using provided data model

Parameters
  • document_class – document_class

  • _acl_queries – User Access List Queries

  • _limit – Data limit

  • **kwargs – Search criteria

Returns

SQL statement and the values to be put in the place holder

classmethod lock(document_class: Type[BaseDocument], doc_id: str, timeout: Optional[int] = None)

Lock entries for write

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • doc_id (str) – Having predefined doc id, None means could be generated by engine

  • timeout – Timeout for lock

Returns

return True amd empty message if lock is successful, else false with error message

Comments:

Lock need based engine implementation

classmethod merge(document_class: Type[BaseDocument], start: Optional[float] = None, end: Optional[float] = None, purge: bool = False, criteria: Optional[dict] = None)

Merge data from log section into main table

Parameters
  • document_class – (subclass of BaseDocument): Document definition

  • start (timestamp) – Starting time point

  • end (timestamp) – Ending time point

  • purge – will remove the entries from log table after execution

  • criteria – only merge the given criteria

Comments:

This method is designed to keep a high consistency data. All replicated data is kept on the log table. Only merge the data into main table when passed the consistency check

classmethod parse_content(data_content: dict, data_model: dict, key_values: dict)

Parse database result to db content (json format)

Parameters
  • data_content – dictionary of entries for each table

  • data_model – Parsed data model

  • key_values – Key value to be filtered

Result:

db form of data

classmethod parse_data(db_content: dict, data_model: dict, key_value: dict, data_content: dict)

Split database content to multiple table update instruction

Parameters
  • db_content – Data base content of document

  • data_model – Parsed data model

  • key_value – The key value should be added

  • data_content – Data content to be added

Returns

Parsed data content

classmethod parse_search_option(key: str)

Reference to search method for the specifications

Parameters

key (str) –

Returns

key, operator, order

classmethod parse_update_option(key: str)

Reference to update method for the specifications

Parameters

key (str) –

Returns

key, update

classmethod replicate(document_class: Type[BaseDocument], task_list: list)

Data replication

Parameters
  • document_class – Python class of document

  • task_list – List of dictionary with the following keys: * id: document id * content: document db form * op: operation type: “I” for insert, “D” for delete, “U” for update, “L” for load

Returns

List of dictionary with the following keys:
  • id: document id

  • op: operation type: “I” for insert, “D” for delete, “U” for update, “L” for load

  • time: time when data is replicated

  • status: status code of HTTP protocol

Return type

task_results

classmethod restore(document_class: Type[BaseDocument], location: Optional[str] = None, data_encode: Optional[str] = None, data_format: Optional[str] = None, data_store: Optional[str] = None, **kwargs)

Restore data of a model

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • data_encode (str) – Backup Data Code

  • data_format (str) – Backup Data Format

  • data_store (str) – Backup Data Store location

  • location (str) – Data location to e used by data store

  • **kwargs – parameter to be passed at engine level

classmethod scan(_document_class: Type[BaseDocument], _acl_queries: Optional[list] = None, _limit: int = 1000, **kwargs)

Scan the document class and get the document id list

Parameters
  • _document_class (subclass of BaseDocument) – Document definition

  • _acl_queries (list) – Extra queries calculated from user’s access control list

  • _limit (int) – Limited the scan results

  • **kwargs – Named arguments are search string

Notes for search string:
  • key, str pair: single value search

  • key, list pair: array_contains_any search

  • embedded search: a__b means b component of a. a.b means the key’s name is a.b

  • operators: key is end with __op__. The following op are supported:
    • __eq__: Could ignore because it is a by default behavior

    • __lt__, __le__, __gt__, __ge__, __ne__: as is supposed by the name

    • __asc__, __desc__: the result will be ordered by the fields

Attentions:
  • The complex query might raise compatible issues

classmethod search(_document_class: Type[BaseDocument], *args, _acl_queries: Optional[list] = None, _limit: int = 50, **kwargs)

Searching and yield document by document

Parameters
  • _document_class (subclass of BaseDocument) – Document definition

  • *args – Unnamed arguments are document id

  • _acl_queries – Extra queries calculated from user’s Access Control List

  • _limit – Search result is limited

  • **kwargs – Named arguments are search string

Notes for search string:
  • key, str pair: single value search

  • key, list pair: array_contains_any search

  • embedded search: a__b means b component of a. a.b means the key’s name is a.b

  • operators: key is end with __op__. The following op are supported:
    • __eq__: Could ignore because it is a by default behavior

    • __lt__, __le__, __gt__, __ge__, __ne__: as is supposed by the name

    • __asc__, __desc__: the result will be ordered by the fields

Attentions:
  • The complex query might raise compatible issues

classmethod set(document_class: Type[BaseDocument], doc_id: str, db_content: dict) str

Overwrite whole document

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • doc_id – Document ID

  • db_content – content to be put to engine

Returns

Document ID

classmethod truncate(document_class: Type[BaseDocument])

Remove all data from the given collection

Parameters

document_class (subclass of BaseDocument) – Document definition

classmethod unlock(document_class: Type[BaseDocument], doc_id: str)

Release the for write

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • doc_id (str) – Having predefined doc id, None means could be generated by engine

Returns

return True amd empty message if lock is successful, else false with error message

Comments:

Unlock need based engine implementation

classmethod update(_document_class: Type[BaseDocument], _doc_id: str, **kwargs) dict

Update a document

Parameters
  • _document_class (subclass of BaseDocument) – Document definition

  • _doc_id (str) – Document ID

  • **kwargs – Named keyword for update

Returns

Updated data

Notes for delete string:
  • embedded update: a__b means b component of a. a.b means the key’s name is a.b

  • operators: key is end with __op__. The following op are supported:
    • __append__: Append an item to array

    • __remove__: Remove an item

    • __delete__: Delete the field

classmethod update_doc_id(document_class: Type[BaseDocument], db_content: dict, old_id: str, new_id: str)

Update document id to new value

Parameters
  • document_class (subclass of BaseDocument) – Document definition

  • db_content – content to be put to new engine

  • old_id – old document id

  • new_id – new document id

Returns

new_document_id if the process is successful

Comments:

By default, we return old id(not implemented). When it is implemented in the Engine, will return new document id