| Current File : //usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyo |
�
k�6Sc @ s@ d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z
d d l m Z d d l
m
Z
d d l
m Z d d l m Z d d l m Z d d l m Z d d l m Z d d
l m Z y d d l m Z Wn! e k
r)d d l m Z n Xd d l m Z d d
l m Z y d d l m! Z! Wn e k
r}d d l! Z! n Xe" Z# e j$ e% � Z& e" a' a a( a) a* a+ d e, f d � � YZ- d � Z. d f d � � YZ/ d e f d � � YZ0 d � Z1 d e f d � � YZ2 d � Z3 d � Z4 d e f d � � YZ5 d S( s�
Overview
========
The multiprocess plugin enables you to distribute your test run among a set of
worker processes that run tests in parallel. This can speed up CPU-bound test
runs (as long as the number of work processeses is around the number of
processors or cores available), but is mainly useful for IO-bound tests that
spend most of their time waiting for data to arrive from someplace else.
.. note ::
See :doc:`../doc_tests/test_multiprocess/multiprocess` for
additional documentation and examples. Use of this plugin on python
2.5 or earlier requires the multiprocessing_ module, also available
from PyPI.
.. _multiprocessing : http://code.google.com/p/python-multiprocessing/
How tests are distributed
=========================
The ideal case would be to dispatch each test to a worker process
separately. This ideal is not attainable in all cases, however, because many
test suites depend on context (class, module or package) fixtures.
The plugin can't know (unless you tell it -- see below!) if a context fixture
can be called many times concurrently (is re-entrant), or if it can be shared
among tests running in different processes. Therefore, if a context has
fixtures, the default behavior is to dispatch the entire suite to a worker as
a unit.
Controlling distribution
^^^^^^^^^^^^^^^^^^^^^^^^
There are two context-level variables that you can use to control this default
behavior.
If a context's fixtures are re-entrant, set ``_multiprocess_can_split_ = True``
in the context, and the plugin will dispatch tests in suites bound to that
context as if the context had no fixtures. This means that the fixtures will
execute concurrently and multiple times, typically once per test.
If a context's fixtures can be shared by tests running in different processes
-- such as a package-level fixture that starts an external http server or
initializes a shared database -- then set ``_multiprocess_shared_ = True`` in
the context. These fixtures will then execute in the primary nose process, and
tests in those contexts will be individually dispatched to run in parallel.
How results are collected and reported
======================================
As each test or suite executes in a worker process, results (failures, errors,
and specially handled exceptions like SkipTest) are collected in that
process. When the worker process finishes, it returns results to the main
nose process. There, any progress output is printed (dots!), and the
results from the test run are combined into a consolidated result
set. When results have been received for all dispatched tests, or all
workers have died, the result summary is output as normal.
Beware!
=======
Not all test suites will benefit from, or even operate correctly using, this
plugin. For example, CPU-bound tests will run more slowly if you don't have
multiple processors. There are also some differences in plugin
interactions and behaviors due to the way in which tests are dispatched and
loaded. In general, test loading under this plugin operates as if it were
always in directed mode instead of discovered mode. For instance, doctests
in test modules will always be found when using this plugin with the doctest
plugin.
But the biggest issue you will face is probably concurrency. Unless you
have kept your tests as religiously pure unit tests, with no side-effects, no
ordering issues, and no external dependencies, chances are you will experience
odd, intermittent and unexplainable failures and errors when using this
plugin. This doesn't necessarily mean the plugin is broken; it may mean that
your test suite is not safe for concurrency.
New Features in 1.1.0
=====================
* functions generated by test generators are now added to the worker queue
making them multi-threaded.
* fixed timeout functionality, now functions will be terminated with a
TimedOutException exception when they exceed their execution time. The
worker processes are not terminated.
* added ``--process-restartworker`` option to restart workers once they are
done, this helps control memory usage. Sometimes memory leaks can accumulate
making long runs very difficult.
* added global _instantiate_plugins to configure which plugins are started
on the worker processes.
i����N( t TextTestRunner( t failure( t loader( t Plugin( t bytes_( t TextTestResult( t ContextSuite( t test_address( t _WritelnDecorator( t Empty( t warn( t StringIOt TimedOutExceptionc B s e Z d d � Z d � Z RS( s Timed Outc C s
| | _ d S( N( t value( t selfR
( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt __init__� s c C s
t | j � S( N( t reprR
( R ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt __str__� s ( t __name__t
__module__R R ( ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR � s c C s� y� d d l m } m a t j t j t j � } | � } t j t j | � | j | j | j | j | j
f \ a a a a a
Wn t k
r� t d t
� n Xd S( Ni����( t Managert ProcesssK multiprocessing module is not available, multiprocess plugin cannot be used( t multiprocessingR R t signalt SIGINTt SIG_IGNt Queuet Poolt Eventt Valuet Arrayt ImportErrorR
t RuntimeWarning( R t oldt m( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt
_import_mp� s 7
t TestLetc B s, e Z d � Z d � Z d � Z d � Z RS( c C sI y | j � | _ Wn t k
r& n X| j � | _ t | � | _ d S( N( t idt _idt AttributeErrort shortDescriptiont _short_descriptiont strt _str( R t case( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR � s
c C s | j S( N( R& ( R ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR% � s c C s | j S( N( R) ( R ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR( � s c C s | j S( N( R+ ( R ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR � s ( R R R R% R( R ( ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR$ � s t MultiProcessc B s> e Z d Z d Z i Z d � Z d � Z d � Z d � Z RS( sF
Run tests in multiple processes. Requires processing module.
i� c C s� | j d d d d | j d d � d d d d
d d �| j d
d d d | j d d � d d d d d d �| j d d d d | j d t � d d d d �d S( s0
Register command-line options.
s --processest actiont storet defaultt NOSE_PROCESSESi t destt multiprocess_workerst metavart NUMt helpsN Spread test run among this many processes. Set a number equal to the number of processors or cores in your machine for best results. Pass a negative number to have the number of processes automatically set to the number of cores. Passing 0 means to disable parallel testing. Default is 0 unless NOSE_PROCESSES is set. [NOSE_PROCESSES]s --process-timeoutt NOSE_PROCESS_TIMEOUTi
t multiprocess_timeoutt SECONDSsf Set timeout for return of results from each test runner process. Default is 10. [NOSE_PROCESS_TIMEOUT]s --process-restartworkert
store_truet NOSE_PROCESS_RESTARTWORKERt multiprocess_restartworkers� If set, will restart each worker process once their tests are done, this helps control memory leaks from killing the system. [NOSE_PROCESS_RESTARTWORKER]N( t
add_optiont gett False( R t parsert env( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt options� s c C s[ y | j j d � Wn t k
r' n Xt | d � sD t | _ d S| j rQ d S| | _ y t | j � } Wn t
t f k
r� d } n X| rWt � t
d k r� t | _ d S| d k r� y d d l } | j � } Wq� t k
r� t | _ d SXn t | _ | | j _ t | j � } | | j _ t | j � } | | j _ t | j d <n d S( s#
Configure plugin.
t activeR3 Ni i����( t statust popt KeyErrort hasattrR? t enabledt workert configt intR3 t TypeErrort
ValueErrorR# R t NoneR t cpu_countt NotImplementedErrort Truet floatR8 R< ( R RB RJ t workersR t tt r( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt configure� sB
c C s | j | _ d S( sb Remember loader class so MultiProcessTestRunner can instantiate
the right loader.
N( t __class__t loaderClass( R R ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt prepareTestLoader s c C s. t d | j d | j j d | j d | j � S( s9 Replace test runner with MultiProcessTestRunner.
t streamt verbosityRJ RX ( t MultiProcessTestRunnerRZ RJ R[ RX ( R t runner( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt prepareTestRunner
s ( R R t __doc__t scoreRD RB RV RY R^ ( ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR- � s ) c C s
t � � d S( N( R ( t sigt frame( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt
signalhandler s R\ c B s� e Z d Z d � Z d � Z d � Z d � Z d � Z e e � Z d � Z e e � Z d � Z
d � Z e e � Z d � Z d
� Z
RS( g @c K s2 | j d t j � | _ t t | � j | � d S( NRX ( RE R t defaultTestLoaderRX t superR\ R ( R t kw( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR s c
C s� x�| j | � D]�} t j d | t | � � t | t j j � rv t | j t j
� rv t j d � | | � q n t | t � r� | j t j
k r� t j d � | | � q q t | t � r�| j
| � r�t j d | � y | j � WnL t t f k
r � q�t j d t j � � | j | t j � � q�X| j | � | j r�| j j j | g � } x3 | d D]$ } t | d t � rxt | _ qxqxWn | j | | | | | � q | j | | | � } t j d t | � | | � q Wd S( Ns Next batch %s (%s)s Case is a Failures %s has shared fixturess %s setup failedi t _multiprocess_shared_s Queued test %s (%s) to %s( t nextBatcht logt debugt typet
isinstancet noseR, t Testt testR t FailureR t contextt sharedFixturest setUpt KeyboardInterruptt
SystemExitt syst exc_infot addErrort appendt factoryR> t getattrR? RQ t _multiprocess_can_split_t collectt addtaskt len(
R Ro t testQueuet taskst to_teardownt resultR, t ancestorst ant test_addr( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR} s<
!
c C s� t d t d � � } t d t j � � } t � } t d t d | | | | | | | | j | j t j | j
� f
� } | | _ | | _ | | _
t j t j t � }
| j � t j t j |
� | S( Nt ct t dt targett args( R R t timeR R R] RX RW t picklet dumpsRJ t currentaddrt currentstartt keyboardCaughtR t SIGILLRc t start( R t iworkerR� t resultQueuet
shouldStopR� R� R� R� t pR! ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt startProcessE s(
c
C s� t j d | | t j � � | j j j | � } | d k rF | } n | j j j | j � } | d k rv | | _ n t
� } t
� } g } g } g } g } t � }
| j � } t
j
� } | j | | | | | � t j d | j j � xV t | j j � D]B }
| j |
| | |
| � } | j | � t j d |
d � qWt | � } | j j } d } yxY| r�t j d t | � | | � y�| j d | � \ } } } } t j d | | t | � � yY y | j | � Wn$ t k
rt j d | | � n X| t | � 7} | j | � WnA t k
rht j d | � t j d
t t | � d � � n X| j | | g � | j | | � | j j r�| j � r�|
j � Pn | j j! r2t j d | � | | j" d d � |
j# � r2| j$ � r2t j d
| � | j | | | |
| � | | <q2n Wnt% k
r9t j d t | � | j$ � t | � � t& } x�t' | � D]�\ } } | j( � r~t) | j* j+ d � } t
j
� | j, j+ } t | � d k r�| | j j d k r�t j d | � n t- } t | � d k r| | j j d k rt j d | | � t) d � | j* _+ | j. j/ � t
j
� } x� | j. j# � r| j( � rt
j
� | | j0 k r�t j1 d | � | j2 � | j | | | |
| � | | <} Pn t j3 | j4 t5 j6 � t
j7 d � qkWqq~q~W| r:| j$ � r:t j d � Pq:n X| j j } xy | D]q } | j( � rMt | j* j+ � d k rMt
j
� | j, j+ } | | j j k r�t8 | | j j | � } q�qMqMWqmWt j d t | � t | � � WnB t9 t: f k
r*} t j; d � | } | j< | t= j> � � n XyYxe | D]] } t j d | � y | j? � Wq5t9 t: f k
ru� q5| j< | t= j> � � q5Xq5Wt
j
� } | j@ � | jA | | � | j j jB | � | d k r t j d � x3 | D]( } | j( � r�| jC d d t& �q�q�Wn xb t' | � D]T \ } } | j( � r+t j d | � | j" � | j( � rt j d | � qq+q+WWn` t9 t: f k
r�t j; d � x' | D] } | j( � r�| j2 � q�q�W| r�| � q�� n X| S( s
Execute the test (which may be a test suite). If the test is a suite,
distribute it out among as many processes as have been configured, at
as fine a level as is possible given the context fixtures defined in
the suite or any sub-suites.
s %s.run(%s) (%s)s Starting %s workerss Started worker process %si s5 Waiting for results (%s/%s tasks), next timeout=%.3fst timeouts1 Results received for worker %d, %s, new tasks: %ds) worker %s failed to remove from tasks: %ss Got result for unknown task? %ss current: %si s joining worker %ss! starting new process on worker %ss8 Timed out with %s tasks pending (empty testQueue=%r): %st asciig�������?sL worker %d has finished its work item, but is not exiting? do we wait for it?s timed out worker %s: %sR� s terminating worker %ss All workers deads Completed %s tasks (%s remain)s4 parent received ctrl-c when waiting for test resultss# Tearing down shared fixtures for %ss Tell all workers to stopt STOPt blocks failed to join worker %ss= parent received ctrl-c when shutting down: stop all processesN(D Ri Rj t ost getpidRJ t pluginst prepareTestRN t setOutputStreamRZ R R t _makeResultR� R} R3 t rangeR� Ry R R8 R> t removeRM R
t extendRF R* t listt consolidatet stopOnErrort
wasSuccessfult setR< t joint is_sett emptyR R? t enumeratet is_aliveR R� R
R� RQ R� t cleart waitkilltimet errort terminatet killt pidR R� t sleept minRt Ru t infoRx Rv Rw t tearDownt printErrorst printSummaryt finalizet put( R Ro t wrappert wrappedR� R� R� t completedRS R� R� R� R� t iR� t total_taskst nexttimeoutt thrownErrorR� t addrt
newtask_addrst batch_resultt any_alivet wt worker_addrt timeprocessingt
startkilltimet eR, t stopRI ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt run[ s
$
)
#
$!#
c C s� d } t | t j j � rH t | j d � rH d | j _ | j j } n t j
| � } | j | | f d t �| d k r� | t
| � 7} n | d k r� | j | � n | S( Nt argR� ( RN Rl Rm R, Rn RG Ro t
descriptorR� R\ t addressR� R? R* Ry ( R� R� R, R� R� ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR~ s ' c C s t | d � r' | j � \ } } } n: t | d � rQ t | j � \ } } } n t d | � � g } | d k r� | d k r� t d | � � q� | j | � n= t j j | � \ } } | j
d � r� | } n | j | � | d k r� | j | � n d j t t
| � � S( NR� Rq s Unable to convert %s to addresss Unaddressable case %sR t :( RG R� R Rq t ExceptionRN Ry R� t patht splitt
startswithR� t mapR* ( R, t filet modt callt partst dirnamet basename( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR� ! s"
c c s
t | d � r+ t | j d t � s+ d Sn t | t � rL | j | j � sr t | d t � sr t | t j � r� t | t � r� t
| � } t | � d k r� t | d d d � | j k r� | d } q� n | Vn0 x- | D]% } x | j
| � D] } | Vq� Wq� Wd S( NRq t _multiprocess_t can_spliti i ( RG R{ Rq RQ Rl R t hasFixturest
checkCanSplitt unittestt TestSuiteR� R RN Rh ( R Ro t containedR, t batch( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyRh : s"
c C s$ | s
t St | d t � r t St S( s8
Callback that we use to check whether the fixtures found in a
context or ancestor are ones we care about.
Contexts can tell us that their fixtures are reentrant by setting
_multiprocess_can_split_. So if we see that, we return False to
disregard those fixtures.
R| ( R? R{ RQ ( Rq t fixt( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR� Z s
c C s, t | d d � } | s t St | d t � S( NRq Rg ( R{ RN R? ( R R, Rq ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyRr j s c C s1 t j d | � y | \ } } } } } Wn; t k
rf t j d | � t j t j � � | � d SX| j j | � | j | 7_ | j
j | � | j j | � xn | j
� D]` \ } \ } }
} | | j k r� g |
| f | j | <n | j | \ } }
}
| j | � q� Wt j d | | j � d S( Ns batch result is %ss result in unexpected format %ss Ran %s tests (total: %s)( Ri Rj RM R Rp Rv Rw RZ t writet testsRunt failuresR� t errorst itemst errorClasses( R R� R� t outputR� R� R� R� t keyt storaget labelt isfailt mystoraget _junk( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR� p s"
"( R R R� R R} R� R� R~ t staticmethodR� Rh R� Rr R� ( ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR\ s ( � c
C sy yQ y) t | | | | | | | | | | �
SWn! t k
rO t j d | � n XWn! t k
rt t j d | � n Xd S( Ns& Worker %s keyboard interrupt, stoppings% Worker %s timed out waiting for tasks( t __runnerRt Ri Rj R (
t ixR� R� R� R� R� R� RX t resultClassRJ ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR] � s
c
s� t j � � � � j � }
t d k rd x: t D]/ } | � } | j |
i � � j j | � q. Wn � j j � j � � � j j
� t j d | t
j � � | d � � }
t |
j _ � � f d � } � � f d � } d � } x�t | d � D]�\ } } | j � r$t j d | � Pn | � } |
j | g � } � | _ g | _ | | _ t j d | | | � y| | d k r�| t | � } n t | � | _ t j � | _ | | � t d � | _ | j | | | j | | � f � Wn�t k
r�} t | t � } | r | j � n t! | j � r�| r>d
} n d } t j | | | � t d � | _ t" j# t$ j% � � | � | j | | | j | | � f � nJ | r�d } n d
} t j | | | � | j | | | j | | � f � | s�� q�n� t& k
r/t d � | _ t j d | � � na t d � | _ t j d | � t" j# t$ j% � � | � | j | | | j | | � f � n X� j' r� Pq� q� Wt j d | � d S( Ns Worker %s executing, pid=%dRJ c s � j d � j � S( NR� ( R> R8 ( ( RJ R� ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR> � s c sP t t � � } � | d d d � j d � �} � j j | � } | rL | S| S( Nt descriptionsi R[ RJ ( R R R[ R� t prepareTestResult( RZ R� t plug_result( RJ R� ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt
makeResult� s c
S s� g | j D] \ } } t | � | f ^ q
} g | j D] \ } } t | � | f ^ q8 } i } x^ | j j � D]M \ } \ } } } g | D] \ } } t | � | f ^ q� | | f | | <qr W| j j � | j | | | f S( N( R� R$ R� R� R� RZ t getvalueR� (
R� R� t errR� R� R� R� R� R� R� ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR� � s ..%( R� s Worker %d STOPPEDs Worker %s Test is %s (%s)R� s, Worker %s timed out, failing current test %ss5 Worker %s keyboard interrupt, failing current test %ss Worker %s test %s timed outs$ Worker %s test %s keyboard interrupts Worker %s system exits1 Worker %s error running test or returning resultss Worker %s ending(( R� t loadst parserClasst _instantiate_pluginsRN t
addOptionsR� t addPluginRV RB t beginRi Rj R� R� t NoSharedFixtureContextSuitet
suiteClasst iterR� t exceptiont loadTestsFromNamesR� R� R� R* R R
R� R� Rt Rl R R� R R Rp Rv Rw Ru R< ( R� R� R� R� R� R� R� RX R� RJ t dummy_parsert pluginclasst pluginR R> R� R� R� R� R� Ro R� R� t msg( ( RJ R� R� s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR� � s�
&
% "
& R c B s; e Z d Z d Z d Z d Z d � Z d � Z d � Z RS( s
Context suite that never fires shared fixtures.
When a context sets _multiprocess_shared_, fixtures in that context
are executed by the main process. Using this suite class prevents them
from executing in the runner process as well.
c C s0 t | d t � r d St t | � j | � d S( NRg ( R{ R? Re R t setupContext( R Rq ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR � s c C s0 t | d t � r d St t | � j | � d S( NRg ( R{ R? Re R t teardownContext( R Rq ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR s c C s5 t j d t | � | | j � | j rD | j | | � | } } n
| | } } y | j � Wn: t k
ru � n' d | _ | j | | j � � d SXz>x7| j D],} t
| t j j
� r� | j d k r� | j | j _ n | j | _ | j | _ | j | _ | j r#t j d � Pn y | | � Wq� t k
r�} t
| t � } | rad } n d } t j | | | � t t t | � � t j � d f } | j j j | | � | j | | � | s�� q�q� Xq� WWd t | _ y | j � Wn9 t k
r
� n& d | _ | j | | j � � n XXd S( s5 Run tests in suite inside of suite fixtures.
s# suite %s (%s) run called, tests: %st setupNt stoppings( Timeout when running test %s in suite %ss2 KeyboardInterrupt when running test %s in suite %si t teardown( Ri Rj R% t _testst resultProxyRs Rt t
error_contextRx t _exc_infoRl Rm R, Rn R� RN Ro R� R� R� R R* Rv Rw RJ R� RQ t has_runR� ( R R� t origRo R� R� R
R� ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR� s\
N(
R R R_ RN R� R� R� R R R� ( ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyR � s (6 R_ t loggingR� Rv R� t tracebackR� R� R t nose.caseRm t nose.coreR R R t nose.plugins.baseR t nose.pyversionR t nose.resultR t
nose.suiteR t nose.utilR t unittest.runnerR R R R t warningsR
t cStringIOR RN R t getLoggerR Ri R R R R R Rt R R# R$ R- Rc R\ R] R� R ( ( ( s= /usr/lib/python2.7/site-packages/nose/plugins/multiprocess.pyt <module>^ sP
_ � r a