Didier Stevens

Monday 14 January 2008

Quickpost: GUI vs. CUI

Filed under: Quickpost — Didier Stevens @ 9:47

Sometimes I read the following programmer’s question:

When I launch my program from the command line, I get a new prompt immediately. What API should I call to let my program display a new prompt only when it is done?

This is often related to scripting: calling this kind of program from a BAT file is a problem, because the BAT file will continue executing while the program is still running.

In fact, this behavior is not defined through coding in the program itself, but by a setting in the header of the program. For WIN32 applications, the value of the SUBSYSTEM parameter is 2 (IMAGE_SUBSYSTEM_WINDOWS_GUI) for a GUI application and 3 (IMAGE_SUBSYSTEM_WINDOWS_CUI) for a console application. A programmer can change this setting by selecting the appropriate option for his compiler. And if one can’t recompile the program, just use a binary editor or a PE file editor. It’s important to understand that console applications are not limited to console interaction, and GUI applications are not limited to GUI interaction. A console application can create windows and a GUI application can create consoles.

It is the shell (cmd.exe) that decides if it waits for the end of the launched program or not, based on the value of the SUBSYSTEM parameter. Take a look at the source code of cmd.exe for ReactOS (this is Open Source):

reactos-cmd.png

After your program is successfully started (CreateProcess), the shell will check if the new process is a console or GUI application (IsConsoleProcess). If it’s a console application, the shell will wait for the program to terminate (WaitForSingleObject), and then it will set the ErrorLevel to the return code. But if it’s a GUI application, it will not wait and it will set the ErrorLevel immediately to 0. That is why you immediately get a new command prompt when you launch a GUI application from the shell: it’s the shell itself that decides not to wait.

So if a programmer wants cmd.exe to wait for the end of its program, he has to declare it as a console application, even if it uses a GUI. There is a drawback however, because when this program is launched from explorer.exe (and not cmd.exe), a console will be automatically created by Windows. The only trick I found to avoid this is to hide the console programmaticaly (but it will still appear briefly when your program is stared).

You can try the following experiment to better understand the SUBSYSTEM parameter without resorting to programming:

  • take a copy of notepad.exe
  • change the SUBSYSTEM parameter of this copy from 2 to 3
  • launch the copy from Windows Explorer
  • launch the copy from cmd.exe
  • launch the copy from a BAT file

Quickpost info


6 Comments »

  1. This can indeed be verry usefull.

    The following site has another nice example :

    http://www.nirsoft.net/vb/console_application_visual_basic.html

    It also has this little tool to easely make the switch to CUI :

    http://www.nirsoft.net/vb/appmodechange.html

    Nice blog!

    Comment by Anz — Monday 14 January 2008 @ 18:24

  2. Use vbscript to start console programs with the console window hidden, not just minimized.

    CreateObject(“Wscript.Shell”).Run “””” & WScript.Arguments(0) & “”””, 0, True

    The 0 sets the windowstyle to hidden, and the True tells it to wait for the window to return.
    Put the code into a single line and save it as invis.vbs or something. Then run the program

    %windir%\wscript.exe invis.vbs hiddenprogram.exe

    hiddenprogram.exe better not wait for user input, or else it’s a leaked process and your script hangs, since GUI windows and not just console windows are hidden with this script.

    Comment by Gregg — Saturday 26 January 2008 @ 8:17

  3. CreateProcess has the same option: flag CREATE_NO_WINDOW

    Comment by Didier Stevens — Tuesday 29 January 2008 @ 20:17

  4. Hi, It seems to be obvious to change the SUBSYSTEM parameter from 2 to 3. But when I use XVI32, I don’t know where is this value.
    Would you help me ? Thanks.

    Comment by jolly jumper — Saturday 27 August 2011 @ 13:38

  5. @jolly jumper Try with a PE editor instead of a hex editor, or use a hex editor that understands the PE file format, like 010 Editor with a PE template.

    Comment by Didier Stevens — Saturday 27 August 2011 @ 18:46

  6. @Didier Stevens : thank you for your quick answer ! It’s very nice.

    Comment by jolly jumper — Sunday 28 August 2011 @ 8:03


RSS feed for comments on this post. TrackBack URI

Leave a Reply (comments are moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 234 other followers

%d bloggers like this: