Skip to content

Commit d84537f

Browse files
committed
rework documentation
1 parent 747bca6 commit d84537f

10 files changed

Lines changed: 312 additions & 198 deletions

File tree

‎README.md‎

Lines changed: 79 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,97 @@
11
# VMProf Python package
22

3-
[![Build Status on TravisCI](https://travis-ci.org/vmprof/vmprof-python.svg?branch=master)](https://travis-ci.org/vmprof/vmprof-python)
4-
[![Build Status on TeamCity](https://teamcity.jetbrains.com/app/rest/builds/buildType:(id:VMprofPython_TestsPy27Win)/statusIcon.svg)](https://teamcity.jetbrains.com/project.html?projectId=VMprofPython)
3+
[![Tests](https://github.com/vmprof/vmprof-python/actions/workflows/tests.yml/badge.svg)](https://github.com/vmprof/vmprof-python/actions/workflows/tests.yml)
4+
[![Wheels](https://github.com/vmprof/vmprof-python/actions/workflows/cibuildwheel.yml/badge.svg)](https://github.com/vmprof/vmprof-python/actions/workflows/cibuildwheel.yml)
55
[![Read The Docs](https://readthedocs.org/projects/vmprof/badge/?version=latest)](https://vmprof.readthedocs.org/en/latest/)
6-
[![Build Status on AppVeyor](https://ci.appveyor.com/api/projects/status/github/vmprof/vmprof-python?branch=master&svg=true)](https://ci.appveyor.com/project/planrich/vmprof-python)
76

7+
**VMProf** is a lightweight statistical profiler for CPython and PyPy. It samples
8+
the call stack of a running program and writes a profile file you can open in
9+
several viewers.
810

911
Head over to https://vmprof.readthedocs.org for more info!
1012

1113
## Installation
1214

1315
```console
1416
pip install vmprof
15-
python -m vmprof <your program> <your program args>
1617
```
1718

18-
Our build system ships wheels to PyPI (Linux, Mac OS X). If you build from source you need
19-
to install CPython development headers and libunwind headers (on Linux only).
20-
On Windows this means you need Microsoft Visual C++ Compiler for your Python version.
19+
VMProf 0.6 supports CPython 3.10 through 3.14 and PyPy, on Linux, Mac OS X and
20+
Windows. Native profiling is available on Linux and Mac OS X.
21+
22+
Wheels are published to PyPI for all three platforms with libunwind bundled in.
23+
If you build from source you need the CPython development headers, and on Linux
24+
the libunwind headers as well — on Debian or Ubuntu, `python3-dev` and
25+
`libunwind-dev`. On Windows you need the Microsoft Visual C++ Compiler for your
26+
Python version.
27+
28+
## Quick start
29+
30+
Record a profile:
31+
32+
```console
33+
$ python -m vmprof -o profile.prof <your program> <your program args>
34+
```
35+
36+
Then open `profile.prof` in whichever viewer fits the question you're asking:
37+
38+
| Viewer | Good for | How |
39+
| --- | --- | --- |
40+
| `vmprofshow` | a quick look, no extra installs | `vmprofshow profile.prof tree` |
41+
| [Firefox Profiler](https://profiler.firefox.com) | flame graph, timeline | `python -m vmprofconvert -convert profile.prof` |
42+
| [kcachegrind](https://kcachegrind.github.io/) | callers/callees, call graph | `vmprofshow profile.prof callgrind -o profile.callgrind` |
43+
44+
Running `python -m vmprof` without `-o` prints basic statistics and keeps no
45+
file.
46+
47+
### Firefox Profiler
48+
49+
The [vmprof-firefox-converter](https://github.com/Cskorpion/vmprof-firefox-converter)
50+
converts a profile into a format the Firefox Profiler UI reads, giving you a
51+
flame graph, a stack chart over time and an inverted call tree in the browser.
52+
It understands PyPy's JIT frames too — see
53+
[the announcement post](https://pypy.org/posts/2024/05/vmprof-firefox-converter.html)
54+
for a tour.
55+
56+
```console
57+
$ python -m pip install vmprof-firefox-converter
58+
$ python -m vmprofconvert -convert profile.prof
59+
```
60+
61+
### kcachegrind
62+
63+
`vmprofshow` can write the profile in callgrind format, which kcachegrind (or
64+
`qcachegrind` on Mac OS X and Windows) reads:
65+
66+
```console
67+
$ vmprofshow profile.prof callgrind -o profile.callgrind
68+
$ kcachegrind profile.callgrind
69+
```
70+
71+
The exported event is `Periods`: each sample is weighted by the time since the
72+
previous one, in units of the sampling period, so costs are proportional to time
73+
spent. At the default ~1kHz one unit is about 0.99ms.
74+
75+
Since vmprof samples the stack rather than instrumenting calls, it has no call
76+
counts — every call edge is written as `calls=1`, so ignore kcachegrind's call
77+
count column. Self cost is attributed to the line a function is defined on; use
78+
`vmprofshow profile.prof lines` when you need line-level numbers.
2179

2280
## Development
2381

2482
Setting up development can be done using the following commands:
2583

26-
$ virtualenv -p /usr/bin/python3 vmprof3
84+
$ python3 -m venv vmprof3
2785
$ source vmprof3/bin/activate
2886
$ pip install meson-python meson ninja
2987
$ pip install --no-build-isolation --editable .
3088

3189
You need to install python development packages. In case of e.g. Debian or Ubuntu the package you need is `python3-dev` and `libunwind-dev`.
32-
Now it is time to write a test and implement your feature. If you want
33-
your changes to affect vmprof.com, head over to
34-
https://github.com/vmprof/vmprof-server and follow the setup instructions.
90+
91+
Run the tests with:
92+
93+
$ pip install pytest cffi setuptools
94+
$ python -m pytest vmprof/
3595

3696
Consult our section for development at https://vmprof.readthedocs.org for more
3797
information.
@@ -128,7 +188,7 @@ helpful when functions exist that get called from multiple places, where each
128188
invocation does not consume much time, but all invocations taken together do
129189
amount to a substantial cost.
130190
```console
131-
$ vmprofshow vmprof_cpuburn.dat flat andreask_work@dunkel 15:24
191+
$ vmprofshow vmprof_cpuburn.dat flat
132192
28.895% - _PyFunction_Vectorcall:/home/conda/feedstock_root/build_artifacts/python-split_1608956461873/work/Objects/call.c:389
133193
18.076% - _iterate:cpuburn.py:20
134194
17.298% - _next_rand:cpuburn.py:15
@@ -148,7 +208,7 @@ $ vmprofshow vmprof_cpuburn.dat flat
148208
```
149209
Sometimes it may be desirable to exclude "native" functions:
150210
```console
151-
$ vmprofshow vmprof_cpuburn.dat flat --no-native andreask_work@dunkel 15:27
211+
$ vmprofshow vmprof_cpuburn.dat flat --no-native
152212
53.191% - _next_rand:cpuburn.py:15
153213
46.809% - _iterate:cpuburn.py:20
154214
0.000% - test:cpuburn.py:36
@@ -159,8 +219,8 @@ functions called. (In `--no-native` mode, native-code callees remain included
159219
in the total.)
160220

161221
Sometimes it may also be desirable to get timings *inclusive* of called functions:
162-
```
163-
$ vmprofshow vmprof_cpuburn.dat flat --include-callees andreask_work@dunkel 15:31
222+
```console
223+
$ vmprofshow vmprof_cpuburn.dat flat --include-callees
164224
100.000% - <native symbol 0x7f0dce8cca80>:-:0
165225
100.000% - test:cpuburn.py:36
166226
100.000% - burn:cpuburn.py:27
@@ -179,3 +239,7 @@ $ vmprofshow vmprof_cpuburn.dat flat --include-callees
179239
0.356% - <native symbol 0x563a5f4ed8f1>:/home/conda/feedstock_root/build_artifacts/python-split_1608956461873/work/Objects/longobject.c:3432
180240
```
181241
This view is quite similar to the "tree" view, minus the nesting.
242+
243+
### Callgrind output
244+
245+
See [kcachegrind](#kcachegrind) above.

‎docs/data.rst‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎docs/development.rst‎

Lines changed: 40 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -1,94 +1,65 @@
11
Develop VMProf
22
==============
33

4-
VMProf consists of several projects working together:
5-
6-
* `vmprof-python`_: The PyPI package providing the command line interface to enable vmprof.
7-
* `vmprof-server`_: Webservice hosted at `vmprof.com`_. Hosts and visualizes data uploaded by `vmprof-python`_ package.
8-
* `vmprof-integration`_: Test suite for pulling together all different projects and ensuring that all play together nicely.
9-
* `PyPy`_: A virtual machine for the Python programming language. Most notably it contains an implementation for the logging facility `vmprof-server`_ can display.
10-
11-
The following description helps you to set up a development environment on Linux. For Windows
12-
and MacOSX the instructions might be similar.
4+
vmprof is made up of a Python package and a C extension, built with
5+
`meson-python`_. The `PyPy`_ side of the JIT log support lives in PyPy itself.
136

7+
.. _`meson-python`: https://mesonbuild.com/meson-python/
148
.. _`PyPy`: http://pypy.org
15-
.. _`vmprof.com`: http://vmprof.com
169
.. _`vmprof-python`: https://github.com/vmprof/vmprof-python
17-
.. _`vmprof-server`: https://github.com/vmprof/vmprof-server
18-
.. _`vmprof-integration`: https://github.com/vmprof/vmprof-integration
19-
20-
Develop VMProf on Linux
21-
-----------------------
22-
23-
It is recommended to use Python 3.x for development. Here is a list of requirements
24-
on your system:
25-
26-
* python
27-
* sqlite3
28-
* virtualenv
29-
30-
Please move you shell to the location you store your source code in and setup
31-
a virtual environment::
3210

33-
$ virtualenv -p /usr/bin/python3 vmprof3
34-
$ source vmprof3/bin/activate
35-
36-
All commands from now on assume you have the vmprof3 virutal environment enabled.
37-
38-
Clone the repositories
39-
----------------------
11+
Setting up
12+
----------
4013

41-
::
14+
Create a virtual environment and install vmprof in editable mode::
4215

43-
$ git clone git@github.com:vmprof/vmprof-integration.git
44-
$ git clone git@github.com:vmprof/vmprof-server.git
4516
$ git clone git@github.com:vmprof/vmprof-python.git
46-
# on old mercurial version the following command takes ages. please use a recent version
47-
$ hg clone ssh://hg@bitbucket.org/pypy/pypy # optional, only if you want to hack on pypy as well
48-
49-
VMProf Server
50-
-------------
51-
52-
::
53-
54-
# setup django service
55-
$ cd vmprof-server
56-
$ pip install -r requirements/development.txt
57-
$ python manage.py migrate
58-
# to run the service
59-
$ python manage.py runserver -v 3
60-
61-
VMProf Python
62-
-------------
63-
64-
An optional stage. It is only necessary if you want to co develop `vmprof-python`_ with `vmprof-server`_::
65-
66-
# install vmprof for development (only needed if you want to co develop vmprof-python)
6717
$ cd vmprof-python
18+
$ python3 -m venv vmprof3
19+
$ source vmprof3/bin/activate
6820
$ pip install meson-python meson ninja
6921
$ pip install --no-build-isolation --editable .
7022

23+
Because the build is ``--no-build-isolation``, the C extension is rebuilt on
24+
import when you change anything under ``src/``, so there is no separate build
25+
step while developing.
7126

72-
Now you are able to change both the python package and the server and see the results.
73-
Here are some more hints on how to develop this platform
27+
You need your distribution's Python development headers, and on Linux the
28+
libunwind headers as well. On Debian or Ubuntu those are ``python3-dev`` and
29+
``libunwind-dev``.
7430

75-
Smaller Profiles
76-
----------------
31+
Running the tests
32+
-----------------
33+
34+
::
7735

78-
Some times it is tedious to generate a big log file and develop a new feature with it.
79-
Both for VMProf and JitLog you can generate small log files that ease development.
36+
$ pip install pytest cffi setuptools
37+
$ python -m pytest vmprof/
8038

81-
There are small logs generated by a python script in `vmprof-server/vmlog/test/data/loggen.py`. Use the following command to load those::
39+
Some tests build small C extensions to exercise native profiling, which is why
40+
``cffi`` and a compiler are needed.
8241

83-
$ ./manage.py loaddata vmlog/test/fixtures.yaml
42+
Smaller profiles
43+
----------------
8444

85-
Now open your browser and redirect them to the jitlog. E.g. http://localhost:8000/#/1v1/traces
45+
Reading a profile of a long run is tedious while working on a feature. The
46+
``vmprof/test/`` directory holds small recorded profiles, and
47+
``vmprof/test/cpuburn.py`` generates fresh ones::
8648

87-
Integration Tests
88-
-----------------
49+
$ python -m vmprof -o profile.prof vmprof/test/cpuburn.py
8950

90-
This is a very important test suite to ensure that all packages work together. It is automatically run every day by travis. You can run them locally. If you happen not to run a Debian base distribution, you can provide the following shell variable to prevent the tests from downloading a Debian PyPy::
51+
Working on the output modes
52+
---------------------------
9153

92-
$ TEST_PYPY_EXEC=/path/to/pypy py.test testvmprof/
54+
The viewers in :doc:`viewers` all read the same profile file, so a profile
55+
recorded once can be replayed through every mode while you iterate::
9356

57+
$ vmprofshow profile.prof tree
58+
$ vmprofshow profile.prof flat
59+
$ vmprofshow profile.prof callgrind -o profile.callgrind
9460

61+
The printers live in ``vmprof/show.py``. Each one subclasses
62+
``AbstractPrinter`` and implements ``_show(tree)``, where ``tree`` is the
63+
``Node`` tree built by ``Stats.get_tree()``; see ``vmprof/stats.py`` for what a
64+
node carries. ``vmprof/test/test_show.py`` builds ``Node`` trees by hand, which
65+
is the quickest way to test a new output mode without recording anything.

‎docs/faq.rst‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,22 +9,24 @@ Frequently Asked Questions
99

1010
* **Is it possible to just profile a part of my program?**: Yes here an example how you could do just that::
1111

12-
with open('test.prof', 'w+b') as fd:
12+
with open('profile.prof', 'w+b') as fd:
1313
vmprof.enable(fd.fileno())
1414
my_function_or_program()
1515
vmprof.disable()
1616

17-
Upload it later to vmprof.com if you choose to inspect it further::
17+
Then open ``profile.prof`` in any of the viewers described in
18+
:doc:`viewers`.
1819

19-
$ python -m vmprof.upload test.prof
20-
21-
22-
23-
* **What do the colors on vmprof.com mean?**: For plain CPython there is no particular meaning, we might change
24-
that in the future. For PyPy we have a color coding to show at which state the VM sampled (e.g. JIT, Warmup, ...).
20+
* **Which viewer should I use?**: ``vmprofshow`` is bundled and needs nothing
21+
installed, the Firefox Profiler gives you a flame graph and a timeline, and
22+
kcachegrind gives you caller and callee lists and a call graph. See
23+
:doc:`viewers`.
2524

2625
* **My Windows profile is malformed?**: Please ensure that you open the file in binary mode. Otherwise Windows
2726
will transform ``\n`` to ``\r\n``.
2827

2928
* **Do I need to install libunwind?**: Usually not. We ship python wheels that bundle libunwind shared objects. If you install vmprof from source, then you need to install the development headers of your distribution. OSX ships libunwind per default. If your pip version is really old it does not pull wheels and it will end up compiling from source.
3029

30+
* **Why are the call counts in kcachegrind all 1?**: Because vmprof samples
31+
the stack rather than instrumenting calls, so it never sees an individual
32+
call and cannot count them. See :doc:`viewers`.

‎docs/index.rst‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,30 +7,33 @@
77
|
88
|
99
10-
VMProf Platform
11-
===============
10+
vmprof
11+
======
1212

13-
`vmprof`_ is a platform to understand and resolve performance bottlenecks in your code.
14-
It includes a *lightweight profiler* for `CPython`_ 2.7, `CPython`_ 3 and `PyPy`_
15-
and an assembler log visualizer for `PyPy`_. Currently we support Linux, Mac OS X and Windows.
13+
`vmprof`_ is a lightweight `statistical profiler`_ for `CPython`_ 3.10+ and
14+
`PyPy`_, along with an assembler log reader for `PyPy`_. It runs on Linux,
15+
Mac OS X and Windows.
1616

17-
The following provides more information about CPU profiles and JIT Compiler Logs:
17+
Profiling writes a profile file, which you open in the viewer of your choice:
18+
the bundled ``vmprofshow``, the Firefox Profiler, or kcachegrind::
19+
20+
pip install vmprof
21+
python -m vmprof -o profile.prof <program.py> <program arguments>
22+
vmprofshow profile.prof tree
1823

1924
.. toctree::
2025
:maxdepth: 2
2126

2227
vmprof
28+
viewers
2329
faq
24-
development
2530
native
2631
format
2732
jitlog
2833
query
29-
data
34+
development
3035

3136
.. _`CPython`: http://python.org
3237
.. _`PyPy`: http://pypy.org
3338
.. _`vmprof`: https://github.com/vmprof/vmprof-python
3439
.. _`statistical profiler`: https://en.wikipedia.org/wiki/Profiling_(computer_programming)#Statistical_profilers
35-
.. _`gperftools`: https://code.google.com/p/gperftools/
36-
.. _`vtune`: https://software.intel.com/en-us/intel-vtune-amplifier-xe

0 commit comments

Comments
 (0)