123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515 |
- '''
- Performance Data Helper (PDH) Query Classes
- Wrapper classes for end-users and high-level access to the PDH query
- mechanisms. PDH is a win32-specific mechanism for accessing the
- performance data made available by the system. The Python for Windows
- PDH module does not implement the "Registry" interface, implementing
- the more straightforward Query-based mechanism.
- The basic idea of a PDH Query is an object which can query the system
- about the status of any number of "counters." The counters are paths
- to a particular piece of performance data. For instance, the path
- '\\Memory\\Available Bytes' describes just about exactly what it says
- it does, the amount of free memory on the default computer expressed
- in Bytes. These paths can be considerably more complex than this,
- but part of the point of this wrapper module is to hide that
- complexity from the end-user/programmer.
- EXAMPLE: A more complex Path
- '\\\\RAISTLIN\\PhysicalDisk(_Total)\\Avg. Disk Bytes/Read'
- Raistlin --> Computer Name
- PhysicalDisk --> Object Name
- _Total --> The particular Instance (in this case, all instances, i.e. all drives)
- Avg. Disk Bytes/Read --> The piece of data being monitored.
- EXAMPLE: Collecting Data with a Query
- As an example, the following code implements a logger which allows the
- user to choose what counters they would like to log, and logs those
- counters for 30 seconds, at two-second intervals.
-
- query = Query()
- query.addcounterbybrowsing()
- query.collectdatafor(30,2)
-
- The data is now stored in a list of lists as:
- query.curresults
-
- The counters(paths) which were used to collect the data are:
- query.curpaths
-
- You can use the win32pdh.ParseCounterPath(path) utility function
- to turn the paths into more easily read values for your task, or
- write the data to a file, or do whatever you want with it.
- OTHER NOTABLE METHODS:
- query.collectdatawhile(period) # start a logging thread for collecting data
- query.collectdatawhile_stop() # signal the logging thread to stop logging
- query.collectdata() # run the query only once
- query.addperfcounter(object, counter, machine=None) # add a standard performance counter
- query.addinstcounter(object, counter,machine=None,objtype = 'Process',volatile=1,format = win32pdh.PDH_FMT_LONG) # add a possibly volatile counter
- ### Known bugs and limitations ###
- Due to a problem with threading under the PythonWin interpreter, there
- will be no data logged if the PythonWin window is not the foreground
- application. Workaround: scripts using threading should be run in the
- python.exe interpreter.
- The volatile-counter handlers are possibly buggy, they haven't been
- tested to any extent. The wrapper Query makes it safe to pass invalid
- paths (a -1 will be returned, or the Query will be totally ignored,
- depending on the missing element), so you should be able to work around
- the error by including all possible paths and filtering out the -1's.
- There is no way I know of to stop a thread which is currently sleeping,
- so you have to wait until the thread in collectdatawhile is activated
- again. This might become a problem in situations where the collection
- period is multiple minutes (or hours, or whatever).
- Should make the win32pdh.ParseCounter function available to the Query
- classes as a method or something similar, so that it can be accessed
- by programmes that have just picked up an instance from somewhere.
- Should explicitly mention where QueryErrors can be raised, and create a
- full test set to see if there are any uncaught win32api.error's still
- hanging around.
- When using the python.exe interpreter, the addcounterbybrowsing-
- generated browser window is often hidden behind other windows. No known
- workaround other than Alt-tabing to reach the browser window.
- ### Other References ###
- The win32pdhutil module (which should be in the %pythonroot%/win32/lib
- directory) provides quick-and-dirty utilities for one-off access to
- variables from the PDH. Almost everything in that module can be done
- with a Query object, but it provides task-oriented functions for a
- number of common one-off tasks.
- If you can access the MS Developers Network Library, you can find
- information about the PDH API as MS describes it. For a background article,
- try:
- http://msdn.microsoft.com/library/en-us/dnperfmo/html/msdn_pdhlib.asp
- The reference guide for the PDH API was last spotted at:
- http://msdn.microsoft.com/library/en-us/perfmon/base/using_the_pdh_interface.asp
- In general the Python version of the API is just a wrapper around the
- Query-based version of this API (as far as I can see), so you can learn what
- you need to from there. From what I understand, the MSDN Online
- resources are available for the price of signing up for them. I can't
- guarantee how long that's supposed to last. (Or anything for that
- matter).
- http://premium.microsoft.com/isapi/devonly/prodinfo/msdnprod/msdnlib.idc?theURL=/msdn/library/sdkdoc/perfdata_4982.htm
- The eventual plan is for my (Mike Fletcher's) Starship account to include
- a section on NT Administration, and the Query is the first project
- in this plan. There should be an article describing the creation of
- a simple logger there, but the example above is 90% of the work of
- that project, so don't sweat it if you don't find anything there.
- (currently the account hasn't been set up).
- http://starship.skyport.net/crew/mcfletch/
- If you need to contact me immediately, (why I can't imagine), you can
- email me at mcfletch@golden.net, or just post your question to the
- Python newsgroup with a catchy subject line.
- news:comp.lang.python
- ### Other Stuff ###
- The Query classes are by Mike Fletcher, with the working code
- being corruptions of Mark Hammonds win32pdhutil module.
- Use at your own risk, no warranties, no guarantees, no assurances,
- if you use it, you accept the risk of using it, etceteras.
- '''
- # Feb 12, 98 - MH added "rawaddcounter" so caller can get exception details.
- import win32pdh, win32api,time, _thread,copy
- class BaseQuery:
- '''
- Provides wrapped access to the Performance Data Helper query
- objects, generally you should use the child class Query
- unless you have need of doing weird things :)
- This class supports two major working paradigms. In the first,
- you open the query, and run it as many times as you need, closing
- the query when you're done with it. This is suitable for static
- queries (ones where processes being monitored don't disappear).
- In the second, you allow the query to be opened each time and
- closed afterward. This causes the base query object to be
- destroyed after each call. Suitable for dynamic queries (ones
- which watch processes which might be closed while watching.)
- '''
- def __init__(self,paths=None):
- '''
- The PDH Query object is initialised with a single, optional
- list argument, that must be properly formatted PDH Counter
- paths. Generally this list will only be provided by the class
- when it is being unpickled (removed from storage). Normal
- use is to call the class with no arguments and use the various
- addcounter functions (particularly, for end user's, the use of
- addcounterbybrowsing is the most common approach) You might
- want to provide the list directly if you want to hard-code the
- elements with which your query deals (and thereby avoid the
- overhead of unpickling the class).
- '''
- self.counters = []
- if paths:
- self.paths = paths
- else:
- self.paths = []
- self._base = None
- self.active = 0
- self.curpaths = []
- def addcounterbybrowsing(self, flags = win32pdh.PERF_DETAIL_WIZARD, windowtitle="Python Browser"):
- '''
- Adds possibly multiple paths to the paths attribute of the query,
- does this by calling the standard counter browsing dialogue. Within
- this dialogue, find the counter you want to log, and click: Add,
- repeat for every path you want to log, then click on close. The
- paths are appended to the non-volatile paths list for this class,
- subclasses may create a function which parses the paths and decides
- (via heuristics) whether to add the path to the volatile or non-volatile
- path list.
- e.g.:
- query.addcounter()
- '''
- win32pdh.BrowseCounters(None,0, self.paths.append, flags, windowtitle)
- def rawaddcounter(self,object, counter, instance = None, inum=-1, machine=None):
- '''
- Adds a single counter path, without catching any exceptions.
-
- See addcounter for details.
- '''
- path = win32pdh.MakeCounterPath( (machine,object,instance, None, inum,counter) )
- self.paths.append(path)
-
- def addcounter(self,object, counter, instance = None, inum=-1, machine=None):
- '''
- Adds a single counter path to the paths attribute. Normally
- this will be called by a child class' speciality functions,
- rather than being called directly by the user. (Though it isn't
- hard to call manually, since almost everything is given a default)
- This method is only functional when the query is closed (or hasn't
- yet been opened). This is to prevent conflict in multi-threaded
- query applications).
- e.g.:
- query.addcounter('Memory','Available Bytes')
- '''
- if not self.active:
- try:
- self.rawaddcounter(object, counter, instance, inum, machine)
- return 0
- except win32api.error:
- return -1
- else:
- return -1
-
- def open(self):
- '''
- Build the base query object for this wrapper,
- then add all of the counters required for the query.
- Raise a QueryError if we can't complete the functions.
- If we are already open, then do nothing.
- '''
- if not self.active: # to prevent having multiple open queries
- # curpaths are made accessible here because of the possibility of volatile paths
- # which may be dynamically altered by subclasses.
- self.curpaths = copy.copy(self.paths)
- try:
- base = win32pdh.OpenQuery()
- for path in self.paths:
- try:
- self.counters.append(win32pdh.AddCounter(base, path))
- except win32api.error: # we passed a bad path
- self.counters.append(0)
- pass
- self._base = base
- self.active = 1
- return 0 # open succeeded
- except: # if we encounter any errors, kill the Query
- try:
- self.killbase(base)
- except NameError: # failed in creating query
- pass
- self.active = 0
- self.curpaths = []
- raise QueryError(self)
- return 1 # already open
-
- def killbase(self,base=None):
- '''
- ### This is not a public method
- Mission critical function to kill the win32pdh objects held
- by this object. User's should generally use the close method
- instead of this method, in case a sub-class has overridden
- close to provide some special functionality.
- '''
- # Kill Pythonic references to the objects in this object's namespace
- self._base = None
- counters = self.counters
- self.counters = []
- # we don't kill the curpaths for convenience, this allows the
- # user to close a query and still access the last paths
- self.active = 0
- # Now call the delete functions on all of the objects
- try:
- map(win32pdh.RemoveCounter,counters)
- except:
- pass
- try:
- win32pdh.CloseQuery(base)
- except:
- pass
- del(counters)
- del(base)
- def close(self):
- '''
- Makes certain that the underlying query object has been closed,
- and that all counters have been removed from it. This is
- important for reference counting.
- You should only need to call close if you have previously called
- open. The collectdata methods all can handle opening and
- closing the query. Calling close multiple times is acceptable.
- '''
- try:
- self.killbase(self._base)
- except AttributeError:
- self.killbase()
- __del__ = close
- def collectdata(self,format = win32pdh.PDH_FMT_LONG):
- '''
- Returns the formatted current values for the Query
- '''
- if self._base: # we are currently open, don't change this
- return self.collectdataslave(format)
- else: # need to open and then close the _base, should be used by one-offs and elements tracking application instances
- self.open() # will raise QueryError if couldn't open the query
- temp = self.collectdataslave(format)
- self.close() # will always close
- return temp
- def collectdataslave(self,format = win32pdh.PDH_FMT_LONG):
- '''
- ### Not a public method
- Called only when the Query is known to be open, runs over
- the whole set of counters, appending results to the temp,
- returns the values as a list.
- '''
- try:
- win32pdh.CollectQueryData(self._base)
- temp = []
- for counter in self.counters:
- ok = 0
- try:
- if counter:
- temp.append(win32pdh.GetFormattedCounterValue(counter, format)[1])
- ok = 1
- except win32api.error:
- pass
- if not ok:
- temp.append(-1) # a better way to signal failure???
- return temp
- except win32api.error: # will happen if, for instance, no counters are part of the query and we attempt to collect data for it.
- return [-1] * len(self.counters)
- # pickle functions
- def __getinitargs__(self):
- '''
- ### Not a public method
- '''
- return (self.paths,)
-
- class Query(BaseQuery):
- '''
- Performance Data Helper(PDH) Query object:
-
- Provides a wrapper around the native PDH query object which
- allows for query reuse, query storage, and general maintenance
- functions (adding counter paths in various ways being the most
- obvious ones).
- '''
- def __init__(self,*args,**namedargs):
- '''
- The PDH Query object is initialised with a single, optional
- list argument, that must be properly formatted PDH Counter
- paths. Generally this list will only be provided by the class
- when it is being unpickled (removed from storage). Normal
- use is to call the class with no arguments and use the various
- addcounter functions (particularly, for end user's, the use of
- addcounterbybrowsing is the most common approach) You might
- want to provide the list directly if you want to hard-code the
- elements with which your query deals (and thereby avoid the
- overhead of unpickling the class).
- '''
- self.volatilecounters = []
- BaseQuery.__init__(*(self,)+args, **namedargs)
- def addperfcounter(self, object, counter, machine=None):
- '''
- A "Performance Counter" is a stable, known, common counter,
- such as Memory, or Processor. The use of addperfcounter by
- end-users is deprecated, since the use of
- addcounterbybrowsing is considerably more flexible and general.
- It is provided here to allow the easy development of scripts
- which need to access variables so common we know them by name
- (such as Memory|Available Bytes), and to provide symmetry with
- the add inst counter method.
- usage:
- query.addperfcounter('Memory', 'Available Bytes')
- It is just as easy to access addcounter directly, the following
- has an identicle effect.
- query.addcounter('Memory', 'Available Bytes')
- '''
- BaseQuery.addcounter(self, object=object, counter=counter, machine=machine)
- def addinstcounter(self, object, counter,machine=None,objtype = 'Process',volatile=1,format = win32pdh.PDH_FMT_LONG):
- '''
- The purpose of using an instcounter is to track particular
- instances of a counter object (e.g. a single processor, a single
- running copy of a process). For instance, to track all python.exe
- instances, you would need merely to ask:
- query.addinstcounter('python','Virtual Bytes')
- You can find the names of the objects and their available counters
- by doing an addcounterbybrowsing() call on a query object (or by
- looking in performance monitor's add dialog.)
-
- Beyond merely rearranging the call arguments to make more sense,
- if the volatile flag is true, the instcounters also recalculate
- the paths of the available instances on every call to open the
- query.
- '''
- if volatile:
- self.volatilecounters.append((object,counter,machine,objtype,format))
- else:
- self.paths[len(self.paths):] = self.getinstpaths(object,counter,machine,objtype,format)
-
- def getinstpaths(self,object,counter,machine=None,objtype='Process',format = win32pdh.PDH_FMT_LONG):
- '''
- ### Not an end-user function
- Calculate the paths for an instance object. Should alter
- to allow processing for lists of object-counter pairs.
- '''
- items, instances = win32pdh.EnumObjectItems(None,None,objtype, -1)
- # find out how many instances of this element we have...
- instances.sort()
- try:
- cur = instances.index(object)
- except ValueError:
- return [] # no instances of this object
- temp = [object]
- try:
- while instances[cur+1] == object:
- temp.append(object)
- cur = cur+1
- except IndexError: # if we went over the end
- pass
- paths = []
- for ind in range(len(temp)):
- # can this raise an error?
- paths.append(win32pdh.MakeCounterPath( (machine,'Process',object,None,ind,counter) ) )
- return paths # should also return the number of elements for naming purposes
- def open(self,*args,**namedargs):
- '''
- Explicitly open a query:
- When you are needing to make multiple calls to the same query,
- it is most efficient to open the query, run all of the calls,
- then close the query, instead of having the collectdata method
- automatically open and close the query each time it runs.
- There are currently no arguments to open.
- '''
- # do all the normal opening stuff, self._base is now the query object
- BaseQuery.open(*(self,)+args, **namedargs)
- # should rewrite getinstpaths to take a single tuple
- paths = []
- for tup in self.volatilecounters:
- paths[len(paths):] = self.getinstpaths(*tup)
- for path in paths:
- try:
- self.counters.append(win32pdh.AddCounter(self._base, path))
- self.curpaths.append(path) # if we fail on the line above, this path won't be in the table or the counters
- except win32api.error:
- pass # again, what to do with a malformed path???
- def collectdatafor(self, totalperiod, period=1):
- '''
- Non-threaded collection of performance data:
- This method allows you to specify the total period for which you would
- like to run the Query, and the time interval between individual
- runs. The collected data is stored in query.curresults at the
- _end_ of the run. The pathnames for the query are stored in
- query.curpaths.
- e.g.:
- query.collectdatafor(30,2)
- Will collect data for 30seconds at 2 second intervals
- '''
- tempresults = []
- try:
- self.open()
- for ind in range(totalperiod/period):
- tempresults.append(self.collectdata())
- time.sleep(period)
- self.curresults = tempresults
- finally:
- self.close()
- def collectdatawhile(self, period=1):
- '''
- Threaded collection of performance data:
- This method sets up a simple semaphor system for signalling
- when you would like to start and stop a threaded data collection
- method. The collection runs every period seconds until the
- semaphor attribute is set to a non-true value (which normally
- should be done by calling query.collectdatawhile_stop() .)
- e.g.:
- query.collectdatawhile(2)
- # starts the query running, returns control to the caller immediately
- # is collecting data every two seconds.
- # do whatever you want to do while the thread runs, then call:
- query.collectdatawhile_stop()
- # when you want to deal with the data. It is generally a good idea
- # to sleep for period seconds yourself, since the query will not copy
- # the required data until the next iteration:
- time.sleep(2)
- # now you can access the data from the attributes of the query
- query.curresults
- query.curpaths
- '''
- self.collectdatawhile_active = 1
- _thread.start_new_thread(self.collectdatawhile_slave,(period,))
- def collectdatawhile_stop(self):
- '''
- Signals the collectdatawhile slave thread to stop collecting data
- on the next logging iteration.
- '''
- self.collectdatawhile_active = 0
- def collectdatawhile_slave(self, period):
- '''
- ### Not a public function
- Does the threaded work of collecting the data and storing it
- in an attribute of the class.
- '''
- tempresults = []
- try:
- self.open() # also sets active, so can't be changed.
- while self.collectdatawhile_active:
- tempresults.append(self.collectdata())
- time.sleep(period)
- self.curresults = tempresults
- finally:
- self.close()
-
- # pickle functions
- def __getinitargs__(self):
- return (self.paths,)
- def __getstate__(self):
- return self.volatilecounters
- def __setstate__(self, volatilecounters):
- self.volatilecounters = volatilecounters
- class QueryError:
- def __init__(self, query):
- self.query = query
- def __repr__(self):
- return '<Query Error in %s>'%repr(self.query)
- __str__ = __repr__
-
|