Sometimes I read the following programmer’s question:
When I launch my program from the command line, I get a new prompt immediately. What API should I call to let my program display a new prompt only when it is done?
This is often related to scripting: calling this kind of program from a BAT file is a problem, because the BAT file will continue executing while the program is still running.
In fact, this behavior is not defined through coding in the program itself, but by a setting in the header of the program. For WIN32 applications, the value of the SUBSYSTEM parameter is 2 (IMAGE_SUBSYSTEM_WINDOWS_GUI) for a GUI application and 3 (IMAGE_SUBSYSTEM_WINDOWS_CUI) for a console application. A programmer can change this setting by selecting the appropriate option for his compiler. And if one can’t recompile the program, just use a binary editor or a PE file editor. It’s important to understand that console applications are not limited to console interaction, and GUI applications are not limited to GUI interaction. A console application can create windows and a GUI application can create consoles.
It is the shell (cmd.exe) that decides if it waits for the end of the launched program or not, based on the value of the SUBSYSTEM parameter. Take a look at the source code of cmd.exe for ReactOS (this is Open Source):
After your program is successfully started (CreateProcess), the shell will check if the new process is a console or GUI application (IsConsoleProcess). If it’s a console application, the shell will wait for the program to terminate (WaitForSingleObject), and then it will set the ErrorLevel to the return code. But if it’s a GUI application, it will not wait and it will set the ErrorLevel immediately to 0. That is why you immediately get a new command prompt when you launch a GUI application from the shell: it’s the shell itself that decides not to wait.
So if a programmer wants cmd.exe to wait for the end of its program, he has to declare it as a console application, even if it uses a GUI. There is a drawback however, because when this program is launched from explorer.exe (and not cmd.exe), a console will be automatically created by Windows. The only trick I found to avoid this is to hide the console programmaticaly (but it will still appear briefly when your program is stared).
You can try the following experiment to better understand the SUBSYSTEM parameter without resorting to programming:
- take a copy of notepad.exe
- change the SUBSYSTEM parameter of this copy from 2 to 3
- launch the copy from Windows Explorer
- launch the copy from cmd.exe
- launch the copy from a BAT file
Didier Stevens 