![]() |
Qore DataProvider Module Reference 3.1
|
Defines a class for passing data through record processors. More...
#include <DataProviderPipeline.qc.dox.h>
Public Member Functions | |
| abort (*bool ignore_exceptions) | |
| Aborts execution of a pipeline in progress. | |
| bool | aborting () |
| Returns True if the object is aborting. | |
| append (AbstractDataProcessor processor) | |
| Appends a data processor to the default queue. | |
| append (int id, AbstractDataProcessor processor) | |
| Appends a data processor to a queue. | |
| int | appendQueue (int id) |
| Appends a new queue to an existing pipeline and returns the new queue ID. | |
| constructor (*hash< PipelineOptionInfo > opts) | |
| Creates the object with the given options. | |
| copy () | |
| Copy constructor; creates an empty pipeline with the same configuration as the original. | |
| destructor () | |
| Destroys the object. | |
| hash< PipelineInfo > | getInfo () |
| Returns pipeline info. | |
| string | getName () |
| Returns the pipeline name. | |
| bool | hasQueue (int id) |
| Returns True if the given queue exists, False if not. | |
| bool | isProcessing () |
| Returns True if the pipeline is processing data. | |
| logDebug (string fmt,...) | |
| Logs to the debug log, if set. | |
| logError (string fmt,...) | |
| Logs to the error log, if set. | |
| logInfo (string fmt,...) | |
| Logs to the info log, if set. | |
| registerThread (PipelineQueue queue) | |
| Registers a new thread. | |
| reportError (PipelineQueue queue, hash< ExceptionInfo > ex) | |
| Called from a pipeline queue object to report a fatal error durring processing. | |
| reset () | |
| Resets the pipeline. | |
| bool | stopping () |
| Returns True if the object is stopping. | |
| submit (AbstractDataProviderBulkRecordInterface i) | |
| Submits data for processing. | |
| submit (AbstractDataProviderRecordIterator i) | |
| Submits data for processing. | |
| submit (auto _data) | |
| Submits data for processing. | |
| submitData (AbstractIterator i) | |
| Submits data for processing. | |
| waitDone () | |
| Waits for all queues to have processed remaining data. | |
Protected Member Functions | |
| Counter | cnt () |
| Thread counter. | |
| Mutex | lck () |
| Atomic lock. | |
| Sequence | seq (1) |
| Pipeline ID sequence generator. | |
Protected Attributes | |
| bool | abort_flag |
| Abort flag. | |
| *code | debug_log |
| Debug log closure; takes a single format string and then arguments for format placeholders. | |
| bool | do_bulk = True |
| Bulk flag. | |
| list< hash< ExceptionInfo > > | error_list |
| list of exceptions in pipelines | |
| *code | error_log |
| Error log closure; takes a single format string and then arguments for format placeholders. | |
| *code | info_log |
| Info log closure; takes a single format string and then arguments for format placeholders. | |
| bool | locked = False |
| Locked flag. | |
| string | name |
| A descriptive name for logging purposes. | |
| hash< string, PipelineQueue > | pmap |
| Hash of queues keyed by queue ID. | |
| int | record_count = 0 |
| Record count. | |
| date | start_time |
| run start time | |
| bool | stop_flag |
| Stop flag. | |
| date | stop_time |
| run stop time (set in waitDone()) | |
| *code | thread_callback |
| a closure or call reference for setting thread-local data in new pipeline queue threads | |
Private Member Functions | |
| checkLockedIntern () | |
| Throws an exception if the pipeline is locked. | |
| checkSubmitIntern () | |
| Throws an exception if the pipeline cannot be used; locks the pipeline for changes otherwise. | |
| checkUpdatePipelineIntern (int id) | |
| Check if the given queue exists. | |
| PipelineQueue | copyPipeline (PipelineQueue old_queue) |
| Called by the copy constructor to copy the queues. | |
| resetIntern () | |
| Resets the pipeline. | |
| stopIntern () | |
| Stops all background pipeline queues. | |
| stopInternUnlocked () | |
| Stops all background pipeline queues; lock must be held. | |
| submitBulkIntern (AbstractDataProviderBulkRecordInterface i) | |
| Submits bulk data for processing. | |
| submitDataIntern (auto _data) | |
| Submits a single record for processing. | |
| submitIntern (auto _data) | |
| Submits data for processing. | |
| throwPipelineException () | |
| Throws an exception if errors occured in background pipeline processing. | |
Defines a class for passing data through record processors.
Record processing pipelines run in background threads. Each queue has an integer queue ID; a queue with ID 0 is created by default as the initial queue.
| DataProvider::DataProviderPipeline::abort | ( | *bool | ignore_exceptions | ) |
Aborts execution of a pipeline in progress.
| ignore_exceptions | if True then any processing exceptions are ignored |
| PIPELINE-FAILED | thrown if ignore_exceptions is not True and there are any errors in pipeline processing, in which case the exception argument will have a list of exception arguments thrown by pipeline threads |
| DataProvider::DataProviderPipeline::append | ( | AbstractDataProcessor | processor | ) |
Appends a data processor to the default queue.
| DataProvider::DataProviderPipeline::append | ( | int | id, |
| AbstractDataProcessor | processor ) |
Appends a data processor to a queue.
| id | the queue ID as returned from appendQueue() |
| processor | the data processor to append to the pipeline queue |
| PIPELINE-ERROR | invalid queue ID or the queue already terminates in additional queues; the pipeline is locked |
| int DataProvider::DataProviderPipeline::appendQueue | ( | int | id | ) |
Appends a new queue to an existing pipeline and returns the new queue ID.
| id | the queue to which the new queue will be appended |
| PIPELINE-ERROR | the pipeline is locked, or the given queue does not exist |
|
private |
Throws an exception if the pipeline is locked.
Must be called with the lock held
|
private |
Throws an exception if the pipeline cannot be used; locks the pipeline for changes otherwise.
Must be called with the lock held
| DataProvider::DataProviderPipeline::constructor | ( | *hash< PipelineOptionInfo > | opts | ) |
Creates the object with the given options.
| opts | any options for the pipeline; see PipelineOptionInfo for more information |
| DataProvider::DataProviderPipeline::destructor | ( | ) |
Destroys the object.
To ensure that the destructor does not throw a PIPELINE-FAILED exception, call run() or runAsync() and waitDone()
| PIPELINE-FAILED | thrown if there are any errors in pipeline processing, in which case the exception argument will have a list of exception arguments thrown by pipeline queue threads |
| hash< PipelineInfo > DataProvider::DataProviderPipeline::getInfo | ( | ) |
Returns pipeline info.
| DataProvider::DataProviderPipeline::reset | ( | ) |
Resets the pipeline.
| PIPELINE-ERROR | this method cannot be called while the pipeline is processing data; call abort() or waitDone() before resetting if the pipeline is processing data |
|
private |
Resets the pipeline.
Must be called with the lock held
|
private |
Submits bulk data for processing.
Must be called with the lock held
|
private |
Submits a single record for processing.
Must be called with the lock held
|
private |
Submits data for processing.
Must be called with the lock held
|
private |
Throws an exception if errors occured in background pipeline processing.
Must be called with the lock held
| DataProvider::DataProviderPipeline::waitDone | ( | ) |
Waits for all queues to have processed remaining data.
| PIPELINE-FAILED | thrown if there are any errors in pipeline processing, in which case the exception argument will have a list of exception arguments thrown by pipeline queue threads |
|
protected |
Bulk flag.