Need some suggestions on how to go about setting up stdout and stderr so they work properly.

[kdschlosser]

I have an application I am developing and there are quit a few C extension modules in it and those extension modules do attach to the input and output streams. I am running into a bump in the road where errors that occur are not being forwarded because those streams don't exist. Even if I create new streams and set them to stdout and stderr in the sys module it doesn't function properly.

My issue is my application is getting stuck and I am not able to tell where it is getting stuck. It only gets stuck when it is compiled as an executable. When I have not compiled it the script runs without any issues. I don't see anywhere in pyinstaller where I am able to have it build the debugging files when it builds my executable. Perhaps I missed it when I was looking over the documentation...

This is what I am doing currently is I am wrapping sys.stdout and sys.stderr in classes that use io.TextIOWrapper as a parent class and I am simply dropping the instances into sys. While this works properly without compiling it doesn't once compiled and I am not getting all of the information I should be getting.

[rokm]

If this is for debugging purposes, why don't you just build application with console enabled, until you've sorted out the issues? And only then switch to noconsole/windowed.

If you launch the windowed/noconsole executable from command-prompt and redirect output to a file (in the shell, e.g., program.exe > output.txt 2>&1), does the output get captured as expected?

[rokm]

I would have thought that this would be handled by pyinstaller by either providing a module that gets packaged into the application that a user is able to import or by some other mechanism.

Why? All PyInstaller does is package your application in self-contained application bundle, while trying to mimic the behavior of the original python interpreter as much as possible.

So in windowed/no-console mode, you get no output streams - same as if you run your code with pythonw.exe.

Which means that if your code is not compatible with pythonw.exe and its implications, it will not be compatible with windowed/noconsole PyInstaller build, either.

It's an issue with the streams I know it is. I just don't have any idea how to solve it and how to solve it while keeping things working for macOS and also Linux.

Windows is the only platform where there is a distinction between console and windowed/noconsole mode - because it involves different (bootloader) executables, built with either /SUBSYSTEM:CONSOLE or /SUBSYSTEM:WINDOWS.

So if you want to apply a work-around for re-instating the streams, you can do it in a Windows-specific codepath, without affecting other platforms.

Similarly, I don't expect you to see the problems caused by lack of streams on macOS and Linux, because the stream handles are valid even if application does not appear to have a console, for example due to being launched via a .desktop file (Linux) or bundled as an .app bundle (macOS) and launched through Finder.

---

As for the streams problem, you can start by reading the link that bwiecek1 provided below. There are couple of more things to be aware of on Windows, however:

• on Windows, the C and C++ level streams (stdout / stderr and std::cout / std::cerr) seem to be initialized from win32 file handles rather than file descriptors. See GetStdHandle() and SetStdHandle(). So in addition to (or in-lieu of) setting file descriptors through os.dup2(), you might also need to fetch the file handle of the opened file (msvcrt.get_ofshandle()) and set is as a corresponding standard output or standard error handle for the process.
• it is important how the extensions are built - i.e., whether they are dynamically or statically linked against the CRT (/MT vs /MD flag passed to MSVC).

If they are linked statically (/MT), then each extension has their own instance of C/C++ level streams, and they are initialized from process' file descriptors (and/or handles) for standard streams when extension is loaded. Which means that you need to ensure that these file descriptors (and/or handles) are properly set before extensions are loaded; in other words, you could open your output files at the very start of your python program, substitute sys.stdout and sys.stderr (for output within python itself), replace the file descriptors 1 and 2 with those of the opened files (the os.dup2() part), obtain file handles (msvcrt.get_ofshandle()) and set them via SetStdHandle() (bound through ctypes). And then import your extensions.

On the other hand, if they are dynamically linked against CRT (/MD), then the C/C++ level streams are shared with the python process; and are invalid, due to having being initialized from unavailable file descriptors / handles at the time when python shared library was loaded. So even if you replace file descriptors 1 and 2, and/or try to set file handles through SetStdHandle(), that will probably have no effect on C/C++ level streams. I suspect in this case, you would need to force their re-initialization, and there is no standard/portable way to do so. It seems that freopen() does that for both C and C++ streams. So one way that might work is to create another extension with a function that accepts two filenames, and calls freopen() to redirect stdout and stderr to the given files:

// cl.exe redirect.c /MD /link /DLL /OUT:redirect.dll
#include <stdio.h>

void __declspec(dllexport) redirect_streams (const char *stdout_filename, const char *stderr_filename)
{
    freopen(stdout_filename, "w", stdout);
    freopen(stderr_filename, "w", stderr);
}

And then have your python program load this extension and call its function before it loads other extensions. (Note that this extension also needs to be dynamically linked against CRT, so /MD flag is passed to compiler).

[kdschlosser]

You answered your own question on this one

Why? All PyInstaller does is package your application in self-contained application bundle, while trying to mimic the behavior of the original python interpreter as much as possible.

The answer is trying to mimic the behavior of the original python interpreter as much as possible. That would be why.

Providing the solution when dealing with C extension modules which is not just a simple replace the stdout and stderr in sys. Providing the code that correctly creates the stdout and stderr so C extension modules will have access to them if the user so desires them to or in my case where it is a necessity for them to exist is part of pyinstaller packaging a users code into an executable successfully. There are 2 pieces to this, the first one is creating the executable and the second part is it running properly. The issues with stdout and stderr and C extension modules is a real issue and that issue stems from how pyinstaller handles building an executable that will run on a Windows machine without having a console. This is an option that is provided by pyinstaller and that option should work correctly all of he time. To me it seems like the streams would need to exist exist when the application is initially prior to the python interpreter being run. We know that adding them from inside of Python doesn't work as I have already done this.

I did do a test and I pulled out the code that handles the streams and I use pyinstaller to create the executable with no console and as it turns out. My code works without any issues. I am almost 100% sure the issue has got to be stemming from something happening inside of a C extension module but which module is causing the issue I couldn't tell you and where in that module is the problem happening is yet another mystery. I am using a decent number of C extension modules.,...

• OCP
• CV2
• Matplotlib
• VTK
• wxPython
• OpenGL
• PyMuPDF
• MySQL connector
• Numpy
• Pillow (PIL)
• mpmath
• lib3mf
• scipy
• sympy
• pyfqmr
• meshio
• ifcopenshell
• keyring
• PyOpenGL-accelerate
• assimp

just to name some of them. There are more than that because those have requirements that have to be there as well and quite a few of those requirements are C extension modules.

[rokm]

You answered your own question on this one

Why? All PyInstaller does is package your application in self-contained application bundle, while trying to mimic the behavior of the original python interpreter as much as possible.

The answer is trying to mimic the behavior of the original python interpreter as much as possible. That would be why.

Providing the solution when dealing with C extension modules which is not just a simple replace the stdout and stderr in sys. Providing the code that correctly creates the stdout and stderr so C extension modules will have access to them if the user so desires them to or in my case where it is a necessity for them to exist is part of pyinstaller packaging a users code into an executable successfully. There are 2 pieces to this, the first one is creating the executable and the second part is it running properly. The issues with stdout and stderr and C extension modules is a real issue and that issue stems from how pyinstaller handles building an executable that will run on a Windows machine without having a console. This is an option that is provided by pyinstaller and that option should work correctly all of he time. To me it seems like the streams would need to exist exist when the application is initially prior to the python interpreter being run. We know that adding them from inside of Python doesn't work as I have already done this.

The part of my reply that you did not quote:

So in windowed/no-console mode, you get no output streams - same as if you run your code with pythonw.exe.

Which means that if your code is not compatible with pythonw.exe and its implications, it will not be compatible with windowed/noconsole PyInstaller build, either.

And this is, as far as I understand, the case here.

[kdschlosser]

Streams are able to be put into place before the interpreter gets run. This would cause Python to work exactly as it would if ran from the command line. Python doesn't really care if you compile using a console or windowed mode it's the windows kernel that normally handles providing the console streams or not. I say normally because they are able to be provided in code. The trick to making this work properly is the streams would have to be put into place before the python interpreter is executed. The pyinstaller boot loader sits in between python and the kernel in how it would norm normally run. This is done so pyinstaller has the ability to extract and expand whatever is needed from inside of the exe file in order to have Python run properly. How hard would it be to also create 2 file descriptors for buffers for stdout and stderr??. The user could then very easily capture the descriptors if they are in place. Setting the descriptors once python is running I do not believe is going to possible 100% of the time and that is because of how C extension modules are able to be compiled. But if pyinstaller sets the file descriptors then they are very easily collected and they can be changed as needed for whatever purpose is needed and this WILL work 100% of the time...

[rokm]

Well, my position on this remains unchanged - our noconsole/windowed mode is supposed to be, behavior-wise, aligned with that of the noconsole/windowed python interpreter (pythonw.exe).

But if you are convinced that PyInstaller's bootloader setting the stream handles for you is the (only) way to go, you are free to modify the bootloader code and tailor it to your specific needs.

[bwiecek1]

Here's a link discussing stdout redirection including C-level streams https://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/